We have mountains of data here too. The best why to store it is in a "real" database of some kind. There are several that are free, open source and multi-platform. The best for this use is "Postgres". As this is free and open source there is no reason not to use it.
In the past I've kept snapshots for simulations that have run for hours/days/weeks and we got many hundreds of millions of data points. Then we are able to query for almost any conditions and expression, for example "Give me a A, B where A-B less than 4 from July 5th 1998" I can tell you first hand that having a billion lines of tab separated data is worse than useless. You need itcataloged such that you can very quickly (seconds) find useful subsets of the data and you can NEVER know in advance what subset you might need. On Mon, Jan 10, 2011 at 12:22 PM, Peter Vince <pvi...@theiet.org> wrote: > Would a TSB (Tab Separated Value) format be preferable? Full-stops > and commas are used in numbers as decimal and thousands separators (or > vice versa), so using tab character would avoid any problems with > commas in the actual data (and make it is a bit easier to quickly > eyeball when viewed in a text editor). > > Peter (G8ZZR, London, England) > > > On 9 January 2011 17:15, Bob Camp <li...@rtty.us> wrote: > ... >> I doubt very much I'm the only one taking a mountain of timing data and not >> properly cataloging it. My guess is that maybe > 90% of the list members are >> in the same boat. How about: >> >> 1) A set of not to restrictive data format standards (CSV with a few >> restrictions ...) > ... > > _______________________________________________ > time-nuts mailing list -- time-nuts@febo.com > To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts > and follow the instructions there. > -- ===== Chris Albertson Redondo Beach, California _______________________________________________ time-nuts mailing list -- time-nuts@febo.com To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts and follow the instructions there.
