On 8/7/2014 4:35 AM, Mark Morgan Lloyd wrote:
waldo kitty wrote:
On 8/6/2014 4:08 AM, Mark Morgan Lloyd wrote:
I'd be inclined to start off using your method 1, i.e. text manipulation until
the format is consistent.
i don't understand "until the format is consistent"... the format has been in
use since the 60s at least (AFAIK) ;)
What I mean is, while you're doing the initial processing to e.g. add century
digits to the date and possibly to check number of decimal places etc.
oh, ok... yeah, that's all done before we reach this stage... as i posted
previously, the format is already laid out so the string spacing is already a
known factor... it is only a couple of places where one might run into
historical TLE records that are space filled instead of zero filled in the
mathematical number areas... the NORAD, COSPAR and Epoch areas are the only ones
that one might find
FWIW: i have taken some time and reworked things to be math based while still
taking the required text format into account... i've seen a very nice increase
in processing speed and now need to just make sure that i don't run into any
of the basic and well known flaws that math processing of date strings seem to
have ;)
Flatten the original record and save it in a database, create a new flat text
this appears that you are speaking of a sql database or similar? that may be a
later feature but for now, everything is/has to be done with the raw TLE
files...
Databases, even for plain-text records, can be incredibly useful.
agreed very much... however, in this case, with the old convoluted string
stuffings that were being done, processing 80000+ records takes only a minute or
two whereas the previous tool was limited by the old 64k memory limit and
processing the same files took over 45 minutes and resulted in only 10% of the
total output of the new tool i've written... it is very late and my math may be
off but output file sizes of 274K compared to 2.5Meg is 10%, right?
but as previously noted, a real database is a possibility for the future...
currently, my personal processing uses a text file containing all the latest
TLEs that i have been able to accumulate... granted, the historical ones are not
available in this format but then again, i don't have room for 9million records
like other systems ;)
i'm not sure what you mean by "flatten", either... currently i break down the
TLEs into their major records for storage in the in-memory ""database""... the
processing i posted is done before that storage takes place...
I was thinking that the first thing you could do was convert the two lines into
a single one for processing, but on reflection it would be better to save the
original with as little modification as possible- possibly with any accession
info you had (i.e. what body had provided that particular TLE).
when i move to incorporate ""real"" database capabilities, something like this
will be being done... that's an understood :) the reason i hesitate to do this
now is due to the processing speed which has been achieved compared to the old
tool written by someone else... adding database support will really slow all the
processing down compared to what has been achieved at this point... granted,
flat file text processing is ""slow"" and ""ugly"" but in this case, it is a
major GoodThing<tm> :)
the goal of the program is to build the in-memory database from all specified
TLE files and then to write out new TLE files which may be filtered on a
selection property so that only certain matching TLE records are saved...
The problem there being that once the program stops you've then got to rebuild
the next time.
that's not really a problem in this case... when i allow others to use the tool
(which is still considered by myself as alpha but may be beta or gamma level by
others), they may choose to use it the same way that i do... or not... as noted
above, i maintain a master ""database"" file which is loaded first and then the
latest live data and finally all the "just in" update files... from these are
built the desired output files... every once in a while i output an updated
""master database""... out of over 40000 possible records, i think i'm missing
only a thousand or so and those are unlikely to ever be filled due to their
military aspect and the lack of publicly available data on their orbital
activities... so back to your statement above, i guess i'm already maintaining a
fairly up-to-date ""master database" which is quickly loaded and then updated by
the other files being processed...
What I normally do when handling large bodies of tabular info is
to either have a series of database tables or a series of text files, where
ideally the text files are absolutely predictable (all fields a known length and
appropriately padded).
i guess that's what i'm doing as described above ;)
What I'm normally looking for is rate-of-change over multiple records with
irregular timestamps, which is an awkward job however it's done.
i do have extensive logs which i regularly review... one night's processing
generates some 45Meg or so for roughly 60 output files ;)
--
NOTE: No off-list assistance is given without prior approval.
Please *keep mailing list traffic on the list* unless
private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
