So, where is your data? (Part 2)

In Part 1 of this series, I discussed some of the difficulties AGHOST has faced with archiving its massive collection of research data (photographs, audio and video) and how we ultimately utilized an open-source software tool to provide free access to that data to all members. But what about reports? All AGHOST investigators are required to submit a report after a walkthrough or an investigation. How do we assure consistent and accurate reports, and provide immediate access to that reporting data online? How can we be sure that the investigator has turned in their reports on time? How can we immediately correlate and search data from, say, 100 reports from 15 different investigations? For this we needed to build a custom solution.

In the past, reporting has been done using a document template that would typically be completed using a text editor of some sort, such as Microsoft Word or OpenOffice Writer. The reports were then either emailed or printed, and submitted to the team lead for review. I believe many other groups utilize a similar process as well. There are a number of issues with this procedure and workflow, however, some of which I will discuss below.

Investigation reports need to be easily published, and made available to everyone in the group so that other investigators can review and learn from each other's experiences. They also need to be easily searched and correlated with other reports. It would also be nice if we could automatically attach third party data to every report, such as the current space weather, for use later when we start analyzing and correlating the data from multiple investigations. Unfortunately many of these goals proved very difficult with a paper or document-based reporting system.

Another problem with paper reports is assuring that the reports are submitted in a timely manner. We have a rule here at AGHOST which states that if an investigator fails to turn in his or her reports, then they lose the privilege of participating on investigations or walkthroughs. With a group as active as ours, where we have multiple investigation coordinators organizing different investigations and teams, it can often be difficult to immediately determine who turned in their reports (and their data) for any given investigation. Furthermore, when reports are not turned in, who really wants to be the guy to have to hound one of our volunteer investigators for a paper report? As it turns out, none of our investigation coordinators enjoy that sort of thing!

So, it became clear that AGHOST, and likely many other paranormal research groups out there, desperately needed some kind of software reporting system - preferably web-based with some kind of database back-end. So, around July of 2006 I began the process of building the AGHOST Reporting Database. Codenamed Project October, the system was due to be finished by the end of October 2006, but actually required a couple more months to test and tweak. The software was finally rolled out around mid January, 2007 to AGHOST investigators.


About the AGHOST Reporting Database

The AGHOST Reporting Database consists of a web-based interface for case management, client relations and investigative reporting. The software allows AGHOST investigators to complete their reports online and AGHOST team leads to manage cases, manage investigators and present the client with a well-organized and consistent final report. All data is stored in a back-end database for easy querying and searching.

The software is case oriented. That is, case managers can create cases, to which all investigators and their reports are linked. Each case, then, represents a single investigation or walkthrough. Case managers can check up on the progress of individual investigators' reports, and send email reminders when needed. While designing and building the software, a fair amount of thought was put into creating a simple, yet functional user interface. Simple tasks such as publishing a report, or slightly more complex tasks such as listing all the cases and reports for a particular user, can be accomplished with just a couple clicks.

By integrating the AGHOST Reporting Database with our
Space Weather Archive developed earlier this year by AGHOST, we are also able to automatically pull space weather and moon phase data into the report on the fly. This data can now be automatically linked to the report, so that investigators are no longer required to manually obtain this information before or after the investigation.

Ultimately, in the short term this software will make reporting and case management a lot easier for our researchers, and in the long term will provide an invaluable resource of investigation data that can be queried and sorted for useful statistical information and trend analysis. So far it has been a great success, and we're all very excited about the possibilities of this new tool.


Is the Software Available for Download?

Although it's still a very new tool, a few folks have expressed an interest in obtaining this software. The AGHOST Reporting Database was built using entirely free software tools - including PHP, PERL, Apache and MySQL. It is, of course, feasible to distribute the code as-is, but at the moment more development will be required for it to be really useful to everyone else. For example, the templating system is a bit ugly, and the authentication code was built specifically to plug into our existing Joomla!-based back-end. So I will need to rewrite some of that so folks may do their own user management. Eventually, probably toward the middle or end of this year, I will release the tool under an open-source license. If you have in-depth knowledge of PHP, however, and are interested in deploying this tool for your own research group, please feel free to email me.