Re: [analog-help] analog can not count users... but other programs can...

[analog-help]

On 10/25/02 5:47 PM Jeremy Wadsack ([EMAIL PROTECTED]) wrote:

>Estimating visits, sessions or unique users is prone to error. It may
>be "easy" but that doesn't mean it's right. It's possible to get a
>very close estimate of sessions with some fancy heuristics, but even
>that has at least a 10% margin of error.

Let me put it a slightly different way. Any system we might come up with 
to count visits, or users, would really be counting something else. HTTP 
doesn't track visit or user information and there is no way to recreate 
that information exactly.

Other programs "solve" this by inventing a rule for counting and calling 
those things visits or users. For example a common rule is that a visit 
is a sequence of requests from a single host with no gap between requests 
of more than half an hour. That, however, has little if anything to do 
with what an (impossible) all knowing "true" analysis would call a visit.

There are many many ways in which rules like that one miscount things. 
The most common problem is that AOL uses proxy clusters. Each "true 
visit" results in hits on your server from perhaps ten different hosts 
(all part of a single cluster). If most of your visitors are AOL users, 
your so called visit counts would be six to ten times what the "real" 
number would be. Similar problems come up with user counting.

You can come up with more complex rules, and get closer to what the 
"real" numbers are, but even then you will never be able to get an exact 
count. Even the best rules can sometimes be way off, and there isn't any 
obvious way to know when that is happening.

Despite the non-reality of numbers produced by rules like the one I gave 
above, many people find them useful. The absolute number might not mean 
anything, but it can be meaningful to do things like comparing this weeks 
count to last weeks count.

This is where the political issues come in. We can never know users or 
visits, but we can know the value of various synthetic measures, for 
example the ones that other programs misleadingly call users or visits. 
Is the value of having the synthetic numbers greater than the confusion 
they cause? While they are useful in carefully chosen contexts, they can 
be incredibly misleading if presented to less knowledgeable people as if 
they were the real thing.

I'll end with a little story. I had a client who really really wanted 
visit counts. So I got them a program which produced visit counts using 
the rule I gave above. The client went on and on about wanting the most 
accurate possible information, so when I found another program that 
produced more accurate numbers I switched them to that. Unfortunately 
they were very upset with the more accurate numbers. Because the new 
program corrected for AOL proxy clusters their visit counts were one 
fifth of what they used to be. Since they appeared to management to have 
lost four fifths of their audience, their project was canceled. I can't 
say that the moral of this story is completely clear, but it does show 
one of the possible risks of using synthetic numbers and representing 
them as if they were real.

Jason

-
[EMAIL PROTECTED]
-
Dr. Seuss books . . . can be read and enjoyed on several levels. For
example, 'One Fish Two Fish, Red Fish Blue Fish' can be deconstructed
as a searing indictment of the narrow-minded binary counting system.
  -- Peter van der Linden, Expert C Programming, Deep C Secrets


+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] analog can not count users... but other programs can...

[analog-help]

cesar martin.iworkwithyou ([EMAIL PROTECTED]; Friday, October 25, 2002 2:07 PM):

> sometimes the ability to count the "number of users" looks more like a
> political decision than a "technical decision".

> I mean, is easy to count users and looks like analog can not do it just because
> "we" don't want to be able to do it...

> but why?
> and don't tell me is the way internet works...

It's some of both. Tehcnical and political.

The only mechanism that HTTP provides for this is authentication and
most web sites don't require any log in. In addition, HTTP and Web
browsers support mechanisms that make is complicated to get full
details on distinct users (caching and proxying). This is covered in
http://analog.cx/docs/webworks.html, which it sounds like you have
read.

If you do have a system in your web server that manages state (for
example an e-commerce system or a client-side solution) then you could
use that information to track users. You can re-use one of the "extra"
variables, like %u or %v, in a custom log format and get a report on
users from that. But unfortunately, there are no standards for this
and in reality, the majority of Analog users don't have any
state-generating systems on their web sites (and yes, I realize this
is self-fulfilling).

Estimating visits, sessions or unique users is prone to error. It may
be "easy" but that doesn't mean it's right. It's possible to get a
very close estimate of sessions with some fancy heuristics, but even
that has at least a 10% margin of error.

Even provided that you could generate a state variable that was
countable, the best Analog would do is provide a total of the number
of "visits" or "users" and list each (with some meaningless key, no
doubt) and the number of total requests, pages, and bytes each visit
or user requested. See http://analog.cx/docs/faq.html#faq128. (Yes,
this is also a technical/political decision.)

That's the technical end of the argument. For the political one, I
suppose will have to speak up. :-)


-- 

Jeremy Wadsack
Wadsack-Allen Digital Group

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

[analog-help] analog can not count users... but other programs can...

[analog-help]

sometimes the ability to count the "number of users" looks more like a
political decision than a "technical decision".

I mean, is easy to count users and looks like analog can not do it just because
"we" don't want to be able to do it...

but why?
and don't tell me is the way internet works...

thanks.
cesar.

=
Usabilidad / http://home.earthlink.net/~cesarmartin/think/
IMF and Russia / http://home.earthlink.net/~cesarmartin/russia.html

__
Do you Yahoo!?
Y! Web Hosting - Let the expert host your web site
http://webhosting.yahoo.com/
+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] Newbie q: trouble with to/from & file size

[analog-help]

On 10/25/02 2:44 PM, "Mark Farmer" <[EMAIL PROTECTED]> wrote:

> Hi, my name's Mark and I'm new to the list.
> 
> I'm trying to specify a date range for a log file I want analzyed. As far as
> I can tell, this is done with the FROM and TO command-line instructions.
> 
> When I double-click on analog.exe, a command window comes up, but won't
> accept commands and then closes itself. The help file told me this is
> normal, but if so, how do I enter a FROM and TO command in it?
> 
> On another question, the log file I'm operating on is huge: 350 Megs (the
> client hasn't it touched it in years, so it's bloated to its current size).


That is small-ish.  My Bb551 server log file is 3GB per school session and,
while analog takes a long time, it handles them with little effort on a
Debian Sparc64 box with 640MB RAM...

Cheers!
-Bill-  :]
_Sx
  ('>iudicium ferat
  //\   Have Computer -
  v_/_Will Hack...


+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] Newbie q: trouble with to/from & file size

[analog-help]

Mark Farmer ([EMAIL PROTECTED]; Friday, October 25, 2002 11:44 AM):

> Hi, my name's Mark and I'm new to the list.

> I'm trying to specify a date range for a log file I want analzyed. As far as 
> I can tell, this is done with the FROM and TO command-line instructions.

FROM and TO are configuration file commands. Enter them into your
analog.cfg file. Use -F and -T from the command line.


> When I double-click on analog.exe, a command window comes up, but won't 
> accept commands and then closes itself. The help file told me this is 
> normal, but if so, how do I enter a FROM and TO command in it?

If you want to enter command-line options (rather than making changes
to the configuration file) you need to run analog from a DOS window
(Command Prompt).


> On another question, the log file I'm operating on is huge: 350 Megs (the 
> client hasn't it touched it in years, so it's bloated to its current size). 
> I'm assuming analog will choke on something this size,

Not at all. Analog can handle GB log files. Of course, this does
depend on the power of the system you are using. If you limit with
FROM and TO, Analog will just dump all the lines it doesn't want, so
it should work just fine.


> so what is the best 
> way to chop it into bite-sized pieces (I'm assuming this is in fact 
> possible)?

Use a log rotation program like logrotate or similar to archive and
compress logs on a regular basis (e.g. monthly) and start new ones.
See http://www.rt.com/man/logrotate.8.html for documentation. This
command ships on most linux and many Unix systems now.

For an existing file, you can use Perl, sed or other text manipulation
tool to parse it into monthly chunks or whatever, if you are
comfortable with them.

Otherwise, you could open it in a text editor and chop it up yourself.
You will need to find an editor that can read 350MB files and will not
wrap long lines. Notepad and Wordpad will not do.

-- 

Jeremy Wadsack
Wadsack-Allen Digital Group

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

[analog-help] Newbie q: trouble with to/from & file size

[analog-help]

Hi, my name's Mark and I'm new to the list.

I'm trying to specify a date range for a log file I want analzyed. As far as 
I can tell, this is done with the FROM and TO command-line instructions.

When I double-click on analog.exe, a command window comes up, but won't 
accept commands and then closes itself. The help file told me this is 
normal, but if so, how do I enter a FROM and TO command in it?

On another question, the log file I'm operating on is huge: 350 Megs (the 
client hasn't it touched it in years, so it's bloated to its current size). 
I'm assuming analog will choke on something this size, so what is the best 
way to chop it into bite-sized pieces (I'm assuming this is in fact 
possible)?

Thanks for the help.

-- Mark Farmer
-- [EMAIL PROTECTED]

_
Unlimited Internet access for only $21.95/month.  Try MSN! 
http://resourcecenter.msn.com/access/plans/2monthsfree.asp

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] Can configuration parts be included from other file s?

[analog-help]

Elfring, Markus ([EMAIL PROTECTED]; Friday, October 25, 2002 2:07 AM):

> Inclusion of log entries is documented.
> (http://www.gekko.de/analog/docs/include.html)
> But can settings be included in several configuration files? I would like
> for example that a lot of aliases
> (http://www.gekko.de/analog/docs/alias.html) are shared and not duplicated
> in the configurations.

Use the CONFIGFILE command to include a configuration file within
another one.

-- 

Jeremy Wadsack
Wadsack-Allen Digital Group

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

RE: [analog-help] Excluding url in Request Report!

[analog-help]

On Fri, 25 Oct 2002, Arthur McBryan wrote:

> So how do I exclude a url for the Request Report?
> 

REQEXCLUDE file
was correct.

-- 
Stephen Turner, Cambridge, UKhttp://homepage.ntlworld.com/adelie/stephen/
 "Reserve your 2 hour delivery time, which means you'll see the same special
  offers as you would instore" - Sainsbury's internet shopping instructions

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

RE: [analog-help] Excluding url in Request Report!

[analog-help]

So how do I exclude a url for the Request Report?

Arthur McBryan

Webmaster
http://www.hcmr-photos.org.uk


-Original Message-
From: [EMAIL PROTECTED]
[mailto:owner-analog-help@;lists.isite.net]On Behalf Of Stephen Turner
Sent: 25 October 2002 12:46
To: Analog-Help@Lists. Isite. Net
Subject: Re: [analog-help] Excluding url in Request Report!


On Fri, 25 Oct 2002, Arthur McBryan wrote:

> If I need to exclude all requests made by a certain url from being
reported
> in the REQUEST REPORT do I use REFSITEEXCLUDE url and REQEXCLUDE url.
>

REQEXCLUDE. REFSITEEXCLUDE would exclude something from the Referring Site
Report.

--
Stephen Turner, Cambridge, UK
http://homepage.ntlworld.com/adelie/stephen/
 "Reserve your 2 hour delivery time, which means you'll see the same special
  offers as you would instore" - Sainsbury's internet shopping instructions

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+


+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] Excluding url in Request Report!

[analog-help]

On Fri, 25 Oct 2002, Arthur McBryan wrote:

> If I need to exclude all requests made by a certain url from being reported
> in the REQUEST REPORT do I use REFSITEEXCLUDE url and REQEXCLUDE url.
> 

REQEXCLUDE. REFSITEEXCLUDE would exclude something from the Referring Site
Report.

-- 
Stephen Turner, Cambridge, UKhttp://homepage.ntlworld.com/adelie/stephen/
 "Reserve your 2 hour delivery time, which means you'll see the same special
  offers as you would instore" - Sainsbury's internet shopping instructions

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

Re: [analog-help] How to turn on only needed reports

[analog-help]

Elfring, Markus <[EMAIL PROTECTED]> wrote:

> I'm working with the program "analog version 5.1/Unix".
> 
> I've used the setting "ALL OFF"
> (http://www.gekko.de/analog/docs/basiccmd.html) and I've set "HOST
> ON" with "ORGANISATION ON" after it. But I get more reports than I
> want in this case. How can the standard report selection be avoided
> completely? 

ALL OFF
ORGANISATION ON
HOST ON

should give you just the General Summary and 2 reports.

if you're getting any more than that, run Analog with 

SETTINGS ON

and see what other reports are being turned on.

Aengus


+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

[analog-help] How to turn on only needed reports

[analog-help]

Title: [analog-help] How to turn on only needed reports





I'm working with the program "analog version 5.1/Unix".


I've used the setting "ALL OFF" (http://www.gekko.de/analog/docs/basiccmd.html) and I've set "HOST ON" with "ORGANISATION ON" after it. But I get more reports than I want in this case.

How can the standard report selection be avoided completely?

RE: [analog-help] Can configuration parts be included from other file s?

[analog-help]

Title: [analog-help] Can configuration parts be included from other files?



Hi,
 
Yes, I 
do this with our configs. I have a general configuration file (analog.cfg) and a 
config file for each website. Without any include/exclude lines in either 
config, I run analog with the +g switch. This 
uses the analog.cfg file initially and then the company cfg file. Any settings 
duplicated, the company one overides i.e. the analog.cfg is used as the base and 
can contain your general configuration settings, the company one then holds the 
specific settings for this company only.
 
Does this help?
 
Matt

  -Original Message-From: Elfring, Markus 
  [mailto:[EMAIL PROTECTED]]Sent: 25 October 2002 10:08To: 
  [EMAIL PROTECTED]Subject: [analog-help] Can configuration 
  parts be included from other file s?
  Inclusion of log entries is documented. 
  (http://www.gekko.de/analog/docs/include.html) But can settings be 
  included in several configuration files? I would like for example that a lot 
  of aliases (http://www.gekko.de/analog/docs/alias.html) are shared and not duplicated in the 
  configurations.

Re: [analog-help] Can configuration parts be included from other file s?

[analog-help]

At 11:07 25-10-2002, you wrote:


Inclusion of log entries is documented. 
(http://www.gekko.de/analog/docs/include.html) 

But can settings be included in several configuration files? I would like 
for example that a lot of aliases 
(http://www.gekko.de/analog/docs/alias.html) 
are shared and not duplicated in the configurations.

Of course analog does allow a lot.
You can include, all at the same time:

configfile myspecial.cfg
configfile  myrobots.cfg
configfile mysearchengines.cfg
configfile mytypealias.cfg

etc.

+
|  This is the analog-help mailing list. To unsubscribe from this
|  mailing list, go to
|http://lists.isite.net/listgate/analog-help/unsubscribe.html
|
|  List archives are available at
|http://www.mail-archive.com/analog-help@;lists.isite.net/
|http://lists.isite.net/listgate/analog-help/archives/
|http://www.tallylist.com/archives/index.cfm/mlist.7
+

[analog-help] Can configuration parts be included from other files?

[analog-help]

Title: [analog-help] Can configuration parts be included from other files?





Inclusion of log entries is documented. (http://www.gekko.de/analog/docs/include.html)
But can settings be included in several configuration files? I would like for example that a lot of aliases (http://www.gekko.de/analog/docs/alias.html) are shared and not duplicated in the configurations.