In a recent blog post I presented a gloss of the components that go in to making a dynamic data analysis system. Although the high level picture I presented there is fairly straightforward, in practice the design and functional requirements of each of the parts require a fair amount of attention.
Here I’ll provide a few quick follow up notes on some of the system design considerations that need to come into play.
- Data Collection: The data collection components of the system must be designed to collect the right kinds of data, in the right format, at the right level of detail, in a way that ensures high quality data that can be analyzed in useful ways. Also very importantly, the data collection user interface, if there is one, must be carefully designed to allow users to easily provide high quality data.
- Data Storage: The database must be designed with a solid underlying data model that understands and properly formalizes the structure, relationships and properties of the objects for which data is being collected, in such a way that the desired analysis can be performed. The database must also be designed with sufficient functionality and efficiency to support the analysis operations being carried out on the dataset.
- Data Restructuring and Analysis: The data analysis component of the system must be designed to take into consideration the accuracy of the data, the way the data is representing the objects behind the data and what analysis results will be useful and informative to the end-consumers of the analysis.
- Data Presentation and Visualization: The data and analysis presentation interface must be designed to clearly, accurately and effectively display the results of the analysis. From a functional requirements point of view, it must be able to deliver and display up-to-date results of the analysis in a timely fashion, based on the requirements of the end-user.
From these considerations alone, it should be fairly apparent that designing and implementing a successful dynamic data analysis system will almost always be a group effort, requiring experts and experienced practitioners from several different domains. This can add to the scope of the project, but from my perspective it’s also what makes this work fun and compelling – working together to effectively build something cool and useful.