I have my different daemons logging into the various log subdirectories via
multilog.
My problem now is integrating them so that I have a continuous line of
activity from the beginning to end for a given email.
For example, I can do a "tail -f current" log for qmail-pop3 while running
tests. However, I would like to know what related activities are occurring
in other logs for this same email test.
I have pulled the following info about qmail-analog from the following
length thread in the archives. It includes an example script. I cut/paste
quickly, so not everyone gets the credit they deserve for their posts in
this thread. I have at minimum two questions after reading all of the info
below:
1) what are all the z... files in the example script?
for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay;
2) where is a real working example of qmail-mrtg?
==========
I want to know how many messages were sent/failed etc. for a given period of
time (say the last three days).
I have done the following in both /var/log/qmail/qmail-send and
/var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't
know the difference between the two. Is qmail-send local deliveries and
qmail-smtpd remote deliveries?):
1) Ran "matchup" on /var/log/qmail/qmail-send(smtpd)/current
2) Converted the "matchedup" version of "current" into human readable
format using tai64nlocal
3) Pulled out dates for which I want to see log results from the file
created above
4) Convert the data above to tai64 format using tai64n
5) Ran this data through zoverall to see qmailanalog results
Regardless of whether I run it against /var/log/qmail/qmail-send or
/var/log/qmail/qmail-smtpd I get the following:
++++++++++++
Completed messages: 0
Total delivery attempts: 0
++++++++++++
Am I anywhere near doing this right?
++++++++++++
Here are my actual commands
1) cat /var/log/qmail/qmail-smtpd/current |
/usr/local/qmailanalog/bin/matchup > /var/log/qmail/qmail-smtpd/matchedup
2) cat /var/log/qmail/qmail-smtpd/matchedup | /usr/local/bin/tai64nlocal >
human_readable_current
3) vi human_readable_current (remove all unneeded data)
4) cat /var/log/qmail/qmail-send/human_readable_current |
/usr/local/bin/tai64n > tai64_current
5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall > overall_log
No. qmail-smtpd is incoming mail via SMTP. qmail-send is all deliveries,
local and remote.
No. Instead of converting the tai64n timestamps to human-readable, you need
to convert them to the fractional seconds (tai) that qmail-analog expects.
You can do this with tai64n2tai, included in Bruce Guenter's qlogtools
package if I remember correctly. His software is at untroubled.org. Thanks
for the info Charles, but I'm confused. How do most of you folks pull out
information from your logs? Log files generated by qmail are
unreadable/unusable in the current (multilog) format. In order for them to
make sense to me, and in order to sift them for specific dates I have to
convert them to human readable format. I can do this with tai64nlocal. Once
I have removed data that is not pertinent I then have to change them back
into multilog format using tai64n, and then convert them into the older
TAI64 format that qmailanalog understands, then run them through the
qmailanalog scripts.
Wow, that's a convoluted process using tools that until now had worked
together to provide a graceful solution to my email needs.
>Thanks for the info Charles, but I'm confused. How do most of you folks
>pull out information from your logs?
With qmail-analog, tai64nlocal, and "less", in my case. Most people here
probably use something similar.
>Log files generated by qmail are unreadable/unusable in the current
>(multilog) format.
tai64n timestamps aren't supposed to be human readable. They're supposed to
be easily parsable by programs. That's the whole point of tai64nlocal --
you log with tai64n timestamps, and if you want to read the log with
human-readable timestamps, you do:
tai64nlocal < log | pager_of_choice
Don't run the logs through tai64nlocal before they hit the disk.
>In order for them to make sense to me, and in order to sift them for
>specific dates I have to convert them to human readable format.
No, it's much simpler than that. A program to filter a log with tai64nlocal
timestamps for particular dates is trivial; Bruce's qlogtools probably
includes one (though I haven't checked). After you've filtered them, you
run it through tai64nlocal before reading it.
>Once I have removed data that is not pertinent I then have to change them
>back into multilog format using tai64n, and then convert them into the
>older
>TAI64 format that qmailanalog understands, then run them through the
>qmailanalog scripts.
Don't remove any data. What isn't pertinent? qmail-analog needs all of the
various data that qmail-send logs to be able to accurately summarize it. I
have a script that runs every night to give me a summary of the day's
activity on each mail server. There's a slightly different version that
does it at the end of the month for a month's logs. This script may have
bash-specific constructions, it's not optimized, and it uses tools from
Bruce Guenter's qlogtools package in addition to daemontools and
qmail-analog. Pick up Bruce's software at untroubled.org.
#!/bin/bash
set -e
HOST=`hostname -f`
MAILTO=admin-mailstats@your_domain
export PATH="$PATH:/root/bin:/usr/bin/qmailanalog"
tmpdir=/tmp/qmail-cron.$$.$RANDOM
mkdir $tmpdir
pushd $tmpdir >/dev/null
s_year=`date -d '1 day ago' +%Y`
s_month=`date -d '1 day ago' +%m`
s_day=`date -d '1 day ago' +%d`
e_year=`date +%Y`
e_month=`date +%m`
e_day=`date +%d`
start="$s_year-$s_month-$s_day"
end="$e_year-$e_month-$e_day"
LOGDIR=/var/log/qmail
cat "$LOGDIR"/{"@",cur}* \
| tai64n2tai \
| qlogselect start $start end $end \
| matchup >logfile 5>/dev/null
for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; do
$ana 2>/dev/null <logfile >$ana
attach="$attach -a $ana"
done
rm -f logfile
mutt -s "$HOST: mailstats for $s_year-$s_month-$s_day" \
-x $attach $MAILTO </dev/null
popd >/dev/null
rm -rf $tmpdir
Change the list list of reports you want as appropriate. You'll also likely
have to change the PATH setting, etc in the script.
>For Charles sake, I don't want to simply look at the log files. I want a
>qmailanalog-style report on a subset of the information contained within my
>"current" file.
Just like the above? :)
Charles
P.S. Russell, if there's any interest in putting this up on your site, feel
free to make a local copy. I don't have it on any ftp or http servers at
the moment.
>This script may have
>bash-specific constructions,
>s_year=`date -d '1 day ago' +%Y`
>s_month=`date -d '1 day ago' +%m`
>s_day=`date -d '1 day ago' +%d`
This works only with GNU date - the original date doesn't have -d. Your
script is nice, though.
>for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; do
Well, for mailservers beeing somehow busy I'd _really_ avoid at least
zrhosts and zrxdelay - these lists become ___very___ long. If you aren't
running a virtual user setup I'd also avoid zsuids.
I found qmail-mrtg (the version that doesn't use qmail-analog) very helpful.
==========
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com
