Re: Simple web log analysis for multiple sites?
On Thu, 15 Nov 2001 14:19, John Ackermann N8UR wrote: > I'm looking for a program that will analyze the logs across the multiple > virtual sites that I run and provide summary-level info (e.g., number of > hits/bytes per site per day, with monthly summaries, etc). > > I'm currently using a slightly hacked version of webstat with some shell > scripts that cat the various logfiles together, add an identifying tag, > sort the result, and feed it into the analyzer, but that really generates > more info than I need for top-level summary purposes and doesn't provide > easy per-site statistics. > > Thanks for any suggestions. Check out my logtools package. It has everything I used to solve this issue at one site I used to run and a site I still run. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page
Re: Simple web log analysis for multiple sites?
> I'm not familiar with webalizer, but what I want is a single report that > lists summaries for all the virtual sites, not a separate report for each > site. Can webalizer do that? > > John Im sorry John, List, it seems I need to read your mails with more care afaik, webalizer has no out-of-box facility for doing a summarized report. I thought you needed per site stats. Now, i also wouldnt be surprised if you COULD do it with weblizerfor example, if you report all vhosts to a single log file, webalizer would give you a global hit count (wouldnt it?)Just check it out, im shure it cant hurt Alex > > > > John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com > President, TAPR[EMAIL PROTECTED]http://www.tapr.org >
Re: woody and ip masq options - resolved
Thanks for the pointer. Once I got your stuff, I ran 'man interfaces' and found out what it was doing. The other issue is there is a file 'options' under the network directory. In it is an option to turn on ip forwarding. Without that, the ipchains command won't work. Greg *** REPLY SEPARATOR *** On 11/14/2001 at 10:53 AM Radek Hnilica wrote: On Wed, Nov 14, 2001 at 12:59:52AM +0100, Florian Friesdorf wrote: > On Tue, Nov 13, 2001 at 05:51:09PM -0600, Gregory Wood wrote: > > Hello all, > > > > I've been working through 'woody', /etc/init.d/networking and > > /etc/network/options but I'm missing how to set my options so that I > > activate ip masq. > > > > I could type in the command manually or just add it to the script but that > > just gets around the issue. > > > > Someone point me to the right HOW-TO file. > > Have a look at the ipmasq package. I think it will do exactly what you > want. If it helps you, I give you a snippet from my /etc/network/interfaces file # Connected to internal web network iface eth1 inet static address 192.168.1.10 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 # Zakaz forwardovani mimo explicitne uvedene forwardovaci pravidla nize pre-up ipchains -P forward DENY # Maskarada pro webovaci site v Breclavi a Praze pre-up ipchains -A forward -s 192.168.1.0/24 -d 0/0 -j MASQ post-down ipchains -D forward -s 192.168.1.0/24 -d 0/0 -j MASQ ... -- Radek Hnilica === No matter how far down the wrong road you've gone, turn back. Turkish proverb -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] Gregory Wood 1219 West University Blvd. Odessa TX 79764-7119 915-335-0879
Re: Simple web log analysis for multiple sites?
--On Thursday, November 15, 2001 10:39 AM -0600 Alejandro Borges <[EMAIL PROTECTED]> wrote: En jue, 2001-11-15 a 07:19, John Ackermann N8UR escribió: Hi -- I'm looking for a program that will analyze the logs across the multiple virtual sites that I run and provide summary-level info (e.g., number of hits/bytes per site per day, with monthly summaries, etc). I'm currently using a slightly hacked version of webstat with some shell scripts that cat the various logfiles together, add an identifying tag, sort the result, and feed it into the analyzer, but that really generates more info than I need for top-level summary purposes and doesn't provide easy per-site statistics. Log for each virtual host, then instance webalizer for each different log its that easy. Alex Step One Group I'm not familiar with webalizer, but what I want is a single report that lists summaries for all the virtual sites, not a separate report for each site. Can webalizer do that? John John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org
Re: Simple web log analysis for multiple sites?
On Thu, 15 Nov 2001 14:19, John Ackermann N8UR wrote: > I'm looking for a program that will analyze the logs across the multiple > virtual sites that I run and provide summary-level info (e.g., number of > hits/bytes per site per day, with monthly summaries, etc). > > I'm currently using a slightly hacked version of webstat with some shell > scripts that cat the various logfiles together, add an identifying tag, > sort the result, and feed it into the analyzer, but that really generates > more info than I need for top-level summary purposes and doesn't provide > easy per-site statistics. > > Thanks for any suggestions. Check out my logtools package. It has everything I used to solve this issue at one site I used to run and a site I still run. -- http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Simple web log analysis for multiple sites?
En jue, 2001-11-15 a 07:19, John Ackermann N8UR escribió: > Hi -- > > I'm looking for a program that will analyze the logs across the multiple > virtual sites that I run and provide summary-level info (e.g., number of > hits/bytes per site per day, with monthly summaries, etc). > > I'm currently using a slightly hacked version of webstat with some shell > scripts that cat the various logfiles together, add an identifying tag, > sort the result, and feed it into the analyzer, but that really generates > more info than I need for top-level summary purposes and doesn't provide > easy per-site statistics. > > Thanks for any suggestions. > > John > [EMAIL PROTECTED] > > > > John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com > President, TAPR[EMAIL PROTECTED]http://www.tapr.org > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > Log for each virtual host, then instance webalizer for each different log its that easy. Alex Step One Group
Re: Simple web log analysis for multiple sites?
> I'm not familiar with webalizer, but what I want is a single report that > lists summaries for all the virtual sites, not a separate report for each > site. Can webalizer do that? > > John Im sorry John, List, it seems I need to read your mails with more care afaik, webalizer has no out-of-box facility for doing a summarized report. I thought you needed per site stats. Now, i also wouldnt be surprised if you COULD do it with weblizerfor example, if you report all vhosts to a single log file, webalizer would give you a global hit count (wouldnt it?)Just check it out, im shure it cant hurt Alex > > > > John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com > President, TAPR[EMAIL PROTECTED]http://www.tapr.org > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: woody and ip masq options - resolved
Thanks for the pointer. Once I got your stuff, I ran 'man interfaces' and found out what it was doing. The other issue is there is a file 'options' under the network directory. In it is an option to turn on ip forwarding. Without that, the ipchains command won't work. Greg *** REPLY SEPARATOR *** On 11/14/2001 at 10:53 AM Radek Hnilica wrote: On Wed, Nov 14, 2001 at 12:59:52AM +0100, Florian Friesdorf wrote: > On Tue, Nov 13, 2001 at 05:51:09PM -0600, Gregory Wood wrote: > > Hello all, > > > > I've been working through 'woody', /etc/init.d/networking and /etc/network/options >but I'm missing how to set my options so that I activate ip masq. > > > > I could type in the command manually or just add it to the script but that just >gets around the issue. > > > > Someone point me to the right HOW-TO file. > > Have a look at the ipmasq package. I think it will do exactly what you > want. If it helps you, I give you a snippet from my /etc/network/interfaces file # Connected to internal web network iface eth1 inet static address 192.168.1.10 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 # Zakaz forwardovani mimo explicitne uvedene forwardovaci pravidla nize pre-up ipchains -P forward DENY # Maskarada pro webovaci site v Breclavi a Praze pre-up ipchains -A forward -s 192.168.1.0/24 -d 0/0 -j MASQ post-down ipchains -D forward -s 192.168.1.0/24 -d 0/0 -j MASQ ... -- Radek Hnilica === No matter how far down the wrong road you've gone, turn back. Turkish proverb -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] Gregory Wood 1219 West University Blvd. Odessa TX 79764-7119 915-335-0879 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Simple web log analysis for multiple sites?
--On Thursday, November 15, 2001 10:39 AM -0600 Alejandro Borges <[EMAIL PROTECTED]> wrote: > En jue, 2001-11-15 a 07:19, John Ackermann N8UR escribió: >> Hi -- >> >> I'm looking for a program that will analyze the logs across the multiple >> virtual sites that I run and provide summary-level info (e.g., number of >> hits/bytes per site per day, with monthly summaries, etc). >> >> I'm currently using a slightly hacked version of webstat with some shell >> scripts that cat the various logfiles together, add an identifying tag, >> sort the result, and feed it into the analyzer, but that really >> generates more info than I need for top-level summary purposes and >> doesn't provide easy per-site statistics. > Log for each virtual host, then instance webalizer for each different > log > > its that easy. > Alex > Step One Group I'm not familiar with webalizer, but what I want is a single report that lists summaries for all the virtual sites, not a separate report for each site. Can webalizer do that? John John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Simple web log analysis for multiple sites?
En jue, 2001-11-15 a 07:19, John Ackermann N8UR escribió: > Hi -- > > I'm looking for a program that will analyze the logs across the multiple > virtual sites that I run and provide summary-level info (e.g., number of > hits/bytes per site per day, with monthly summaries, etc). > > I'm currently using a slightly hacked version of webstat with some shell > scripts that cat the various logfiles together, add an identifying tag, > sort the result, and feed it into the analyzer, but that really generates > more info than I need for top-level summary purposes and doesn't provide > easy per-site statistics. > > Thanks for any suggestions. > > John > [EMAIL PROTECTED] > > > > John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com > President, TAPR[EMAIL PROTECTED]http://www.tapr.org > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] > Log for each virtual host, then instance webalizer for each different log its that easy. Alex Step One Group -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
RE: Simple web log analysis for multiple sites?
I'd also be interested to know what other folks are doing for this. We use webalizer, but we keep seperate stats & reports per each web site. I then have a program that reads the webalizer.hist file for each site and updates an SQL table with information for each site. If someone needed more data they could probably extract it from webalizer's HTML files, but using webalizer.current is a bad idea since it destroys it at the end of each month and starts over fresh. I wish it preserved that data because there is a lot of good stuff in there I might like to use. Perhaps I'll contribute a patch someday; but more likely we will just reinvent the wheel so we can get over some other shortcomings of webalizer. If this program would be useful to anyone else I could share it. Below are some records for one site we host. The "ws" column is the web server on which the site resides. "ds" is the datestamp column, basically in mysql the easiest way to do this was to use the first day of each month to represent data for the whole month. "complete" indicates if the data for that month is either partial data, or if it has data for that month, as well as for the following month. While that doesn't mean there are no holes in the data, it does at least give you an indication of if you should use it for billing/etc yet :-) ts is just a timestamp column. - jsw mysql> SELECT * FROM WlfMSum WHERE ws="fire" AND sn="memepool.com" AND ds="2001-10-01"\G *** 1. row *** ws: fire sn: memepool.com ds: 2001-10-01 hits: 1133903 files: 1012502 sites: 163933 kbytes: 30512375 pages: 988517 visits: 577638 complete: 1 ts: 20011101113012 1 row in set (0.04 sec) mysql> SELECT * FROM WlfMSum WHERE ws="fire" AND sn="memepool.com"; +--+--++-+-++--+ ++--++ | ws | sn | ds | hits| files | sites | kbytes | pages | visits | complete | ts | +--+--++-+-++--+ ++--++ | fire | memepool.com | 2001-08-01 | 233195 | 210099 | 47498 | 7255226 | 214404 | 122863 |1 | 20011101113012 | | fire | memepool.com | 2001-09-01 | 931873 | 823837 | 147449 | 29061296 | 817547 | 485257 |1 | 20011101113012 | | fire | memepool.com | 2001-10-01 | 1133903 | 1012502 | 163933 | 30512375 | 988517 | 577638 |1 | 20011101113012 | +--+--++-+-++--+ ++--++ 4 rows in set (0.07 sec) -Original Message- From: John Ackermann N8UR [mailto:[EMAIL PROTECTED] Sent: Thursday, November 15, 2001 8:19 AM To: debian-isp@lists.debian.org Subject: Simple web log analysis for multiple sites? Hi -- I'm looking for a program that will analyze the logs across the multiple virtual sites that I run and provide summary-level info (e.g., number of hits/bytes per site per day, with monthly summaries, etc). I'm currently using a slightly hacked version of webstat with some shell scripts that cat the various logfiles together, add an identifying tag, sort the result, and feed it into the analyzer, but that really generates more info than I need for top-level summary purposes and doesn't provide easy per-site statistics. Thanks for any suggestions. John [EMAIL PROTECTED] John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Simple web log analysis for multiple sites?
Hi -- I'm looking for a program that will analyze the logs across the multiple virtual sites that I run and provide summary-level info (e.g., number of hits/bytes per site per day, with monthly summaries, etc). I'm currently using a slightly hacked version of webstat with some shell scripts that cat the various logfiles together, add an identifying tag, sort the result, and feed it into the analyzer, but that really generates more info than I need for top-level summary purposes and doesn't provide easy per-site statistics. Thanks for any suggestions. John [EMAIL PROTECTED] John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org
Re: SNMP Monitoring
Use ucd-snmp: get it on site: - http://ucd-snmp.ucdavis.edu
RE: Simple web log analysis for multiple sites?
I'd also be interested to know what other folks are doing for this. We use webalizer, but we keep seperate stats & reports per each web site. I then have a program that reads the webalizer.hist file for each site and updates an SQL table with information for each site. If someone needed more data they could probably extract it from webalizer's HTML files, but using webalizer.current is a bad idea since it destroys it at the end of each month and starts over fresh. I wish it preserved that data because there is a lot of good stuff in there I might like to use. Perhaps I'll contribute a patch someday; but more likely we will just reinvent the wheel so we can get over some other shortcomings of webalizer. If this program would be useful to anyone else I could share it. Below are some records for one site we host. The "ws" column is the web server on which the site resides. "ds" is the datestamp column, basically in mysql the easiest way to do this was to use the first day of each month to represent data for the whole month. "complete" indicates if the data for that month is either partial data, or if it has data for that month, as well as for the following month. While that doesn't mean there are no holes in the data, it does at least give you an indication of if you should use it for billing/etc yet :-) ts is just a timestamp column. - jsw mysql> SELECT * FROM WlfMSum WHERE ws="fire" AND sn="memepool.com" AND ds="2001-10-01"\G *** 1. row *** ws: fire sn: memepool.com ds: 2001-10-01 hits: 1133903 files: 1012502 sites: 163933 kbytes: 30512375 pages: 988517 visits: 577638 complete: 1 ts: 20011101113012 1 row in set (0.04 sec) mysql> SELECT * FROM WlfMSum WHERE ws="fire" AND sn="memepool.com"; +--+--++-+-++--+ ++--++ | ws | sn | ds | hits| files | sites | kbytes | pages | visits | complete | ts | +--+--++-+-++--+ ++--++ | fire | memepool.com | 2001-08-01 | 233195 | 210099 | 47498 | 7255226 | 214404 | 122863 |1 | 20011101113012 | | fire | memepool.com | 2001-09-01 | 931873 | 823837 | 147449 | 29061296 | 817547 | 485257 |1 | 20011101113012 | | fire | memepool.com | 2001-10-01 | 1133903 | 1012502 | 163933 | 30512375 | 988517 | 577638 |1 | 20011101113012 | +--+--++-+-++--+ ++--++ 4 rows in set (0.07 sec) -Original Message- From: John Ackermann N8UR [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 15, 2001 8:19 AM To: [EMAIL PROTECTED] Subject: Simple web log analysis for multiple sites? Hi -- I'm looking for a program that will analyze the logs across the multiple virtual sites that I run and provide summary-level info (e.g., number of hits/bytes per site per day, with monthly summaries, etc). I'm currently using a slightly hacked version of webstat with some shell scripts that cat the various logfiles together, add an identifying tag, sort the result, and feed it into the analyzer, but that really generates more info than I need for top-level summary purposes and doesn't provide easy per-site statistics. Thanks for any suggestions. John [EMAIL PROTECTED] John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
secure remote logging
Hello *, anyone using a method to do secure remote logging? I need a advice how this could be realized. I need data encryption and want to avoid misuse of the loghost. (for examble DoS) Some idea's of me: a) ipsec or cipe to tunnel the complete traffic between loghost and the other servers. Is multipoint to central point possible? b) a better syslog than the default + a packetfilter (iptables/ipchains) Didn't find one yet. syslog-ng seems not to have encryption support out of the box. c) normal syslog + secure tunneling + packetfliter stunnel, ssl, ssh, netcat or zebedee. There are other possibilities, as to send logging information over serial connections or to rsnyc (over ssh) regularly the logs. But serial lines are not possible in my environment and rsync gives me not the ability to establish a IDS-system on base of logging. I also can't make an extra IP network for only sending logging information. What's best practice? How do you establish a loghost in an unsecure environment? bye Waldemar
Re: SNMP Monitoring
Hello Jason, >From the keyboard of Jason, > Hello, > > Please forgive my unsolicited intrusion, but I am looking for an SNMP > Management solution. > > To give you some detail, we are developing in-house SNMP agents to monitor > our applications/services. > I require a robust and flexible Management tool or suite for these agents as > well as proprietary agents. > > Any help in this direction will be most appreciated. Take a look at this site: http://www.opennms.org/ bye Waldemar
Simple web log analysis for multiple sites?
Hi -- I'm looking for a program that will analyze the logs across the multiple virtual sites that I run and provide summary-level info (e.g., number of hits/bytes per site per day, with monthly summaries, etc). I'm currently using a slightly hacked version of webstat with some shell scripts that cat the various logfiles together, add an identifying tag, sort the result, and feed it into the analyzer, but that really generates more info than I need for top-level summary purposes and doesn't provide easy per-site statistics. Thanks for any suggestions. John [EMAIL PROTECTED] John AckermannN8UR [EMAIL PROTECTED] http://www.febo.com President, TAPR[EMAIL PROTECTED]http://www.tapr.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: SNMP Monitoring
Use ucd-snmp: get it on site: - http://ucd-snmp.ucdavis.edu -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
secure remote logging
Hello *, anyone using a method to do secure remote logging? I need a advice how this could be realized. I need data encryption and want to avoid misuse of the loghost. (for examble DoS) Some idea's of me: a) ipsec or cipe to tunnel the complete traffic between loghost and the other servers. Is multipoint to central point possible? b) a better syslog than the default + a packetfilter (iptables/ipchains) Didn't find one yet. syslog-ng seems not to have encryption support out of the box. c) normal syslog + secure tunneling + packetfliter stunnel, ssl, ssh, netcat or zebedee. There are other possibilities, as to send logging information over serial connections or to rsnyc (over ssh) regularly the logs. But serial lines are not possible in my environment and rsync gives me not the ability to establish a IDS-system on base of logging. I also can't make an extra IP network for only sending logging information. What's best practice? How do you establish a loghost in an unsecure environment? bye Waldemar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: SNMP Monitoring
Hello Jason, >From the keyboard of Jason, > Hello, > > Please forgive my unsolicited intrusion, but I am looking for an SNMP > Management solution. > > To give you some detail, we are developing in-house SNMP agents to monitor > our applications/services. > I require a robust and flexible Management tool or suite for these agents as > well as proprietary agents. > > Any help in this direction will be most appreciated. Take a look at this site: http://www.opennms.org/ bye Waldemar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
