ISOQLOG looks terrific, but:
1) It solves a different problem than the one I am trying to solve. It
appears to provide traffic summaries.
2) The problem I am trying to solve is to create a detailed trace from start
to finish about the entire life cycle of a single email message. I dislike
s**dmail just as much as everyone else here, but being a monolithic program,
it did have a detailed trace for a given message. I'm fine with qmail being
broken up into components, but I need an integrated log for debugging.
3) There are several posts in the archives about isoqlog not running if the
multilog is under 100k and not rotating. I am only testing now, so my
multilogs are quite small. The solutions are not clear to me. One is to find
a patch for multilog which rotates the logfile, whenever it receives a HUP.
A) where is the patch? B) exactly how am I supposed to supply the HUP in my
scripts?
>From: "hari_bhr" <[EMAIL PROTECTED]>
>Reply-To: "hari_bhr" <[EMAIL PROTECTED]>
>To: "pop corn" <[EMAIL PROTECTED]>, <[EMAIL PROTECTED]>
>Subject: Re: Integrating the logs
>Date: Thu, 28 Jun 2001 10:54:03 +0530
>
>look for ISOQLOG nice
>
>----- Original Message -----
>From: pop corn <[EMAIL PROTECTED]>
>To: <[EMAIL PROTECTED]>
>Sent: Thursday, June 28, 2001 10:00 AM
>Subject: Integrating the logs
>
>
> > I have my different daemons logging into the various log subdirectories
>via
> > multilog.
> >
> > My problem now is integrating them so that I have a continuous line of
> > activity from the beginning to end for a given email.
> >
> > For example, I can do a "tail -f current" log for qmail-pop3 while
>running
> > tests. However, I would like to know what related activities are
>occurring
> > in other logs for this same email test.
> >
> > I have pulled the following info about qmail-analog from the following
> > length thread in the archives. It includes an example script. I
>cut/paste
> > quickly, so not everyone gets the credit they deserve for their posts in
> > this thread. I have at minimum two questions after reading all of the
>info
> > below:
> >
> > 1) what are all the z... files in the example script?
> > for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
> >
> > 2) where is a real working example of qmail-mrtg?
> >
> > ==========
> >
> > I want to know how many messages were sent/failed etc. for a given
>period
>of
> > time (say the last three days).
> > I have done the following in both /var/log/qmail/qmail-send and
> > /var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't
> > know the difference between the two. Is qmail-send local deliveries and
> > qmail-smtpd remote deliveries?):
> > 1) Ran "matchup" on /var/log/qmail/qmail-send(smtpd)/current
> > 2) Converted the "matchedup" version of "current" into human readable
> > format using tai64nlocal
> > 3) Pulled out dates for which I want to see log results from the file
> > created above
> > 4) Convert the data above to tai64 format using tai64n
> > 5) Ran this data through zoverall to see qmailanalog results
> > Regardless of whether I run it against /var/log/qmail/qmail-send or
> > /var/log/qmail/qmail-smtpd I get the following:
> > ++++++++++++
> > Completed messages: 0
> > Total delivery attempts: 0
> > ++++++++++++
> > Am I anywhere near doing this right?
> > ++++++++++++
> > Here are my actual commands
> > 1) cat /var/log/qmail/qmail-smtpd/current |
> > /usr/local/qmailanalog/bin/matchup >
>/var/log/qmail/qmail-smtpd/matchedup
> > 2) cat /var/log/qmail/qmail-smtpd/matchedup |
>/usr/local/bin/tai64nlocal
> >
> > human_readable_current
> > 3) vi human_readable_current (remove all unneeded data)
> > 4) cat /var/log/qmail/qmail-send/human_readable_current |
> > /usr/local/bin/tai64n > tai64_current
> > 5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall >
>overall_log
> > No. qmail-smtpd is incoming mail via SMTP. qmail-send is all
>deliveries,
> > local and remote.
> > No. Instead of converting the tai64n timestamps to human-readable, you
>need
> > to convert them to the fractional seconds (tai) that qmail-analog
>expects.
> > You can do this with tai64n2tai, included in Bruce Guenter's qlogtools
> > package if I remember correctly. His software is at untroubled.org.
>Thanks
> > for the info Charles, but I'm confused. How do most of you folks pull
>out
> > information from your logs? Log files generated by qmail are
> > unreadable/unusable in the current (multilog) format. In order for them
>to
> > make sense to me, and in order to sift them for specific dates I have to
> > convert them to human readable format. I can do this with tai64nlocal.
>Once
> > I have removed data that is not pertinent I then have to change them
>back
> > into multilog format using tai64n, and then convert them into the older
> > TAI64 format that qmailanalog understands, then run them through the
> > qmailanalog scripts.
> > Wow, that's a convoluted process using tools that until now had worked
> > together to provide a graceful solution to my email needs.
> > >Thanks for the info Charles, but I'm confused. How do most of you
>folks
> > >pull out information from your logs?
> > With qmail-analog, tai64nlocal, and "less", in my case. Most people
>here
> > probably use something similar.
> > >Log files generated by qmail are unreadable/unusable in the current
> > >(multilog) format.
> > tai64n timestamps aren't supposed to be human readable. They're
>supposed
>to
> > be easily parsable by programs. That's the whole point of tai64nlocal
>--
> > you log with tai64n timestamps, and if you want to read the log with
> > human-readable timestamps, you do:
> > tai64nlocal < log | pager_of_choice
> > Don't run the logs through tai64nlocal before they hit the disk.
> > >In order for them to make sense to me, and in order to sift them for
> > >specific dates I have to convert them to human readable format.
> > No, it's much simpler than that. A program to filter a log with
>tai64nlocal
> > timestamps for particular dates is trivial; Bruce's qlogtools probably
> > includes one (though I haven't checked). After you've filtered them,
>you
> > run it through tai64nlocal before reading it.
> > >Once I have removed data that is not pertinent I then have to change
>them
> > >back into multilog format using tai64n, and then convert them into the
> > >older
> > >TAI64 format that qmailanalog understands, then run them through the
> > >qmailanalog scripts.
> > Don't remove any data. What isn't pertinent? qmail-analog needs all of
>the
> > various data that qmail-send logs to be able to accurately summarize it.
>I
> > have a script that runs every night to give me a summary of the day's
> > activity on each mail server. There's a slightly different version that
> > does it at the end of the month for a month's logs. This script may
>have
> > bash-specific constructions, it's not optimized, and it uses tools from
> > Bruce Guenter's qlogtools package in addition to daemontools and
> > qmail-analog. Pick up Bruce's software at untroubled.org.
> > #!/bin/bash
> > set -e
> > HOST=`hostname -f`
> > MAILTO=admin-mailstats@your_domain
> > export PATH="$PATH:/root/bin:/usr/bin/qmailanalog"
> > tmpdir=/tmp/qmail-cron.$$.$RANDOM
> > mkdir $tmpdir
> > pushd $tmpdir >/dev/null
> > s_year=`date -d '1 day ago' +%Y`
> > s_month=`date -d '1 day ago' +%m`
> > s_day=`date -d '1 day ago' +%d`
> > e_year=`date +%Y`
> > e_month=`date +%m`
> > e_day=`date +%d`
> > start="$s_year-$s_month-$s_day"
> > end="$e_year-$e_month-$e_day"
> > LOGDIR=/var/log/qmail
> > cat "$LOGDIR"/{"@",cur}* \
> > | tai64n2tai \
> > | qlogselect start $start end $end \
> > | matchup >logfile 5>/dev/null
> > for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
>do
> > $ana 2>/dev/null <logfile >$ana
> > attach="$attach -a $ana"
> > done
> > rm -f logfile
> > mutt -s "$HOST: mailstats for $s_year-$s_month-$s_day" \
> > -x $attach $MAILTO </dev/null
> > popd >/dev/null
> > rm -rf $tmpdir
> > Change the list list of reports you want as appropriate. You'll also
>likely
> > have to change the PATH setting, etc in the script.
> > >For Charles sake, I don't want to simply look at the log files. I want
>a
> > >qmailanalog-style report on a subset of the information contained
>within
>my
> > >"current" file.
> > Just like the above? :)
> > Charles
> > P.S. Russell, if there's any interest in putting this up on your site,
>feel
> > free to make a local copy. I don't have it on any ftp or http servers
>at
> > the moment.
> > >This script may have
> > >bash-specific constructions,
> > >s_year=`date -d '1 day ago' +%Y`
> > >s_month=`date -d '1 day ago' +%m`
> > >s_day=`date -d '1 day ago' +%d`
> > This works only with GNU date - the original date doesn't have -d. Your
> > script is nice, though.
> > >for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids
>zrxdelay;
>do
> > Well, for mailservers beeing somehow busy I'd _really_ avoid at least
> > zrhosts and zrxdelay - these lists become ___very___ long. If you aren't
> > running a virtual user setup I'd also avoid zsuids.
> > I found qmail-mrtg (the version that doesn't use qmail-analog) very
>helpful.
> >
> > ==========
> >
> >
> >
> > _________________________________________________________________
> > Get your FREE download of MSN Explorer at http://explorer.msn.com
> >
>
>
>_________________________________________________________
>Do You Yahoo!?
>Get your free @yahoo.com address at http://mail.yahoo.com
>
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com
