I just wanted to follow-up on this thread for anyone else looking at processing
large logfiles. In testing Analog on the stats files, *LOWMEM 3 was able to run
within the memory space of the server. We subsequently ran each of the LOWMEM
options at 2, one at a time. Then used a script to collect the reports from each
piece. Since the data is all output in CRO for Report Magic this was a fairly
simple Perl / grep script.
The trick, however, was to take the relevant reports from each specific data
sections. So, for example, the run with HOSTLOWMEM 2 (and the rest of the
LOWMEMs at 3) gave us the host, domain and organization reports. Similarly for
the FILELOWMEM 2 and request, directory, file size and file type. The balance of
the reports (time reports, status code) came from a run with all LOWMEMs at 3.
This piecewise approach seems to work, in fact it used less memory on a monthly
run (2GB per run, 6 distinct runs) than it did trying one run with *LOWMEM 2
(ran out on 14GB). Running a conversion test on previously produced weekly
reports showed the numbers to match.
Stangely enough (and maybe Stephen can shed light on this) is that the domain
report, which was included in all runs, had different number with all LOWMEMs at
3 than with HOSTLOWMEM 2. Shouldn't the count be the same if the report is going
to be displayed.
Hope this is uselful,
Jeremy Wadsack
Wadsack-Allen Digital Group
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/analog-help@lists.isite.net/
------------------------------------------------------------------------
