One part of our Open Economics project (http://www.openeconomics.net/) involves maintaining a data store:
http://www.openeconomics.net/store/ Corresponding data is stored in subversion: http://knowledgeforge.net/econ/svn/trunk/data/ Actually having to store some data (in a simple way) has been a great exercise in terms of thinking about how one does this kind of thing. In keeping with the KISS approach basic structure of the data sets (or 'bundles' as I have termed them) is: * metadata.txt: follows ini style conventions * data itself stored either: * plain csv: data.csv * and/or script file of some kind: data.py (which helps generate data.csv in some way via downloading, parsing etc etc). To illustrate one dataset we've recently been working to include is that from Millenium Development Goals Project <http://mdgs.un.org/unsd/mdg/> The corresponding (still not complete) Open Economics dataset is at: <http://knowledgeforge.net/econ/svn/trunk/data/mdg/> and can be browsed from the store browser at (note data and plot links won't work because they would 'barf' if trying to display a 9mb csv): <http://www.openeconomics.net/store/d0b2917e-bfb8-40be-8819-5867f155c1a3> Below I detail some thoughts and questions from dealing with this particular item. I know this is a bit thinking out loud but I'd appreciate any comments/ideas -- after all people have been thinking about data storage for as long as they have using computers ... Regards, Rufus Issues raised by the MDG data ============================= 1. MDG Data has at least 3 basic dimensions: * Country * Series * Time Can deal with this by taking one dimension as primary (e.g. series) though this has the cost that it is harder to divide up in other ways (e.g. show me these 3 series for this specific country). 2. Values are slightly complex in that they have a type (related to how reliable the estimates are how they were obtained) and (potentially) an associated footnote. * since fn and type are not directly relevant most of the time perhaps we can just ignore (made easier once we have normalized). 3. Number of countries (~241) make displaying a bit of an issue. * in web interface this should be dealt with simply by restricting number of rows we display to a sample (say 50). 3. Also lots of blank values for many series for many countries. * again not so much of a problem once we normalize ### How to Design (Sub-)Data Bundles ... One way to deal with the massive amount of data in the single csv file would be to create 'sub-bundles' corresponding to particular slices through the data. E.g. could create bundles for each country or each time series (or even each country by each time series). But is this a good idea? Perhaps one should just have some script that can generate such things 'on-demand' though one would still need to register somewhere what particular 'bundles' could be built on demand ... Another alternative would be to load said data into a db (after all that is where it came from and that is the easiest way to deal with getting multiple views on same dataset). Question then is does one do this in a way that preserves the ability for data to be provided in a simple (easily usable) form to other users (and how do we load the data? On request or permanently into some db or ...). Related to this is what one would want to if one wanted to provide more functionality than pure dataset browsing of visualization plugin which would manipulate a given dataset for the web user -- though still not sure how manipulating a 9mb csv file. ### What Was the Aim? But all of this adds significant complexity. What is the aim of openeconomics.net. At present think its core aim is to provide a simple repository for datasets. In this regard the web interface is there simply to provide: * A dataset browser. * Persistent urls. * [?] Simple way to upload and download data. Providing complex data analysis tools etc is not central to this but is definitely a possibly extension. ### Left-over questions * How can we integrate with OpeNDAP (and pydap). Should seem an obvious option. * PlotKit javascript graphing library no longer seems actively developed. what are the other good js graphing libraries ... _______________________________________________ okfn-discuss mailing list [email protected] http://lists.okfn.org/cgi-bin/mailman/listinfo/okfn-discuss
