Hi Stephen
I think that is it. This site (nypost.com) typically has 5-10 million
hits a day-on over 2.5 million distinct files. I can't imagine how many
distinct files in a month but it must be a huge number. So Analog would
need gigabytes and gigabytes of RAM to keep track of all of those files,
even through I only wanted the top 200.
I would be running the request report on its own with all six lowmem
values set to 3 and I just could not understand what the %$#*@ it needed
all of that memory for.
Anyway, looks like I have displayed a touch of foot-in-mouth disease; it
IS in fact because I disregard some of the data. My apologies.
Thanks,
Tristan
Stephen Turner wrote:
>
> On Tue, 4 Apr 2000, Tristan Lawrence wrote:
> >
> > I think Analog is a wonderful program, but for whatever reason it uses
> > an absolutely ridiculous amount of memory doing large reports like this.
> > My script unquestionably takes a long time to run (hours) but is
> > dramatically more memory-efficient than running Analog from logs or
> > cache files. This isn't due to the fact that I throw away some of the
> > data but to some inefficiency in the way Analog keeps track of things on
> > this scale.
>
> I don't think that it does use much more memory than necessary. By far the
> largest amount of memory is in recording the names of every host and every
> file in the logfile, even though most of them never get listed in the end.
> This is maybe half the memory typically, but you can't get round it without
> losing data. Then you need to record the number of times you've seen each
> item, the date you last saw it, etc...
>
> There are minor improvements you could make, but I have thought pretty hard
> about this issue already, and I don't think it can be improved much.
>
> --
> Stephen Turner http://www.statslab.cam.ac.uk/~sret1/
> Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
> "8th March 2000. National No Smoking Day. Ash Wednesday." (On a calendar)
>
> ------------------------------------------------------------------------
> This is the analog-help mailing list. To unsubscribe from this
> mailing list, send mail to [EMAIL PROTECTED]
> with "unsubscribe" in the main BODY OF THE MESSAGE.
> List archived at http://www.mail-archive.com/analog-help@lists.isite.net/
> ------------------------------------------------------------------------
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/analog-help@lists.isite.net/
------------------------------------------------------------------------
