I have ~ 100MB of daily logs that I need to process. I need to provide
historical information in the same file as I provide recent daily
information, but I want to throw away the fine-grained detail as it
becomes older.
Ideally, I'd have 15 days worth of fine-grained log analysis, 30 days of
of medium-grained analysis (for days -16 -> -45), followed by all
historical data at a low-grained level.
I *think* the following will work, but I wanted to run it by this group
to get some feedback before I start writing scripts.
1. process nightly logs into distinct nightly cache files with
fine-grained info.
2. keep nightly cache files for 45 days
3. every night, create a new history cache file out of previous
history cache + oldest nightly (low-grained info).
4. delete oldest nightly.
5. every night, create a brand new cache file of medium-grained
info out of the oldest 30 nightly cache files
6. every night, create a brand new cache file of fine-grained info
out of the newest 15 nightly cache files
7. every night, create a single fine-grained run of the history
cache file, the medium-grained cache file, and the fine-grained
cache file.
This seems like it should work, and it seems like it should provide the
information we need while providing savings of processing time and
memory usage.
Is there anything in here that doesn't make sense? Anything that I need
to consider? Will the final nightly run suffer in any way from
including sparse historical data?
Thanks in advance for any responses,
Andrew
p.s. I'm moving to this (sort of complicated) scheme after suddenly
finding that nightly analog runs were running out of memory; I hadn't
considered the size that the cache files grow to.
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
| Digest version: http://lists.isite.net/listgate/analog-help-digest/
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------
