Okay... due to the number of requests, I've attached the sed and awk
scripts, as well as the short command-line scripts I use to initiate the
log processing.
THEORY OF OPERATION:
Since diald can dial out multiple times per hour, I only wanted the summary
of total time connected, with one number per hour. I chose to use the
first dial-out per hour to give the start time for the total use that hour.
Because of the fact that a call initiated at xx:59:59 may in fact last
several hours, you will occasionally see data points that list total usage
for any given hour as more than one hour. But if any given datum is more
than two hours, you shouldn't see a separate datum for the subsequent hour
(because diald was still connected from the previous hour...).
Using arrays in awk, I compare the start time of each record with the
previous record to determine if the date or hour has changed. If so, then
I sum the use from all records matching that hour until I find a start time
from a new hour. Then I print the output of the previous hour. This data
is then post-processed from one record per hour per line (see output
examples below) into tab-separated fields, with all hours for any given day
listed. Each day has one line, with 24 columns of data.
I also have provided a raw output of data from the diald accounting log.
This is for debugging. Run netlog.raw to see all data points, with no
processing other than to summarize the 4 lines of data from each diald
accounting log entry into a single line.
MISCELLANEOUS NOTES:
1. In diald.conf, enable logging with the accounting log option. In my
case, as you'll see in the sh script, I save it to /var/log/diald.log for
collocation with other system logs. My diald.conf line option reads as
follows:
"accounting-log /var/log/diald.log"
2. The following files are tarred and gzipped:
netlog (executable -- normally this is the one you'll run)
netlog.raw (executable -- for debugging. See below)
netlog.sed (first and only sed script)
netlog.awk (the first awk script)
netlog2.awk (second awk script)
netlog.awk.raw (awk script used for netlog.raw)
3. You'll have to change the path/file locations in netlog and netlog.raw
to accommodate where you choose to put your log, as well as your scripts.
This part should be obvious when you look at the files.
4. I currently make no attempt to "prune" the diald log. It just grows and
grows... You may want to remove it weekly after processing, via a cron
command (which is how I run netlog itself).
5. The only bug that I'm aware of is that I haven't yet bothered trying to
figure out how to print out the last line of information. Because of the
method I use to tally up multiple uses per hour (and print it as a single
instance of total use started during that hour), the script doesn't ever
print the final record read into awk. An END statement may do the trick...
6. In netlog.raw, I provide the bytes transmitted and received. The same
summing logic I use for the call duration will work for the data, in case
you need the total data transmitted or received per hour. I do not provide
that data in netlog, as my spreadsheet cannot graph 4 variables or 5
variables simultaneously for comparison. (Date, hour, and duration are
tabulated in netlog. To add xmit and rcv would require too many dimensions
(variables in the table), which I cannot figure out how to enter, much less
plot. ;-)
7. I process the summed call duration from seconds into hh:mm:ss format,
since that's easier for us humans to understand (and for the bosses to
grasp). It does make plotting a little more complex, so if it causes too
many headaches, then just print the seconds instead of the hhmmss variable
in the netlog.awk script.
SAMPLE OUTPUTS OF DIFFERENT SCRIPTS:
Following are samples of the various data outputs from the different parts
of the program:
1. diald: accounting-log produces the following:
Sat Oct 10 23:15:00 1998 CDT: Calling site 209.215.198.69.
Sat Oct 10 23:15:25 1998 CDT: Connected to site 209.215.198.69.
Sat Oct 10 23:22:38 1998 CDT: Disconnected. Call duration 433 seconds.
IP transmitted 1054 bytes and received 3104 bytes.
Sun Oct 11 4:22:13 1998 CDT: Calling site 209.215.198.193.
Sun Oct 11 4:22:40 1998 CDT: Connected to site 209.215.198.193.
Sun Oct 11 4:29:56 1998 CDT: Disconnected. Call duration 436 seconds.
IP transmitted 210 bytes and received 456 bytes.
2. netlog.sed produces the following from above:
Sat Oct 10 23:15:00 1998 CDT: Calling site 209.215.198.69. Sat Oct 10
23:22:38 1998 CDT: Disconnected. Call duration 433 seconds. IP transmitted
1054 bytes and received 3104 bytes.
Sun Oct 11 4:22:13 1998 CDT: Calling site 209.215.198.193. Sun Oct 11
4:29:56 1998 CDT: Disconnected. Call duration 436 seconds. IP transmitted
210 bytes and received 456 bytes.
3. netlog.awk.raw produces following from above: (date, time started,
hh:mm:ss of duration, bytes xmit, bytes rcvd)
10/10/98 23:15:00 0:07:13 1054 3104
10/11/98 4:22:13 0:07:15 210 456
4. netlog.awk produces the following from the output of step 2: (date,
hour only of the first time a connection was established that hour, and
hh:mm:ss of total duration on-line for that hour)
10/10/98 23 0:07:13
10/11/98 4 0:07:15
5. netlog2.awk post-processes the output from step 4, and gives:
0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20
21 22 23
10/10/98 0:17:10 0:18:52 0:17:07 0:07:30
0:12:37
0:07:13
10/11/98 0:07:15
This last step doesn't look right (in text readers) until you can see the
tab stops. There's one per hour, giving 24 tabs across for each day. (The
first line is a header.) The position of each data point (call duration)
is under the column for that hour. Each day's history is separated by a
newline at the end of each record, which consists of tab-separated fields.
This final output is directed to a file, which then can be opened as a
tab-delimited text file into any spreadsheet. Then I do the following to
see it in graphical layout:
1. Format the hh:mm:ss duration fields as time (otherwise my spreadsheet
treats them as a fraction of 24 hours, showing decimal numbers.)
2. Highlight the table, and create a 3-D columnar chart. The date in my
first column gives the names for the series of data, the hour header line
at top gives the names for the rows of data, and the series of data for
each date gets printed in a row. Following days are printed in front of
the previous day, with all hours lining up. The height of the columns is
given as the length of total time connected that hour.
If you have questions, feel free to write me. As I mentioned in the first
message, I'm not a programmer, so I had to muddle my way through some
things. I'm sure there are faster, more efficient ways of doing some
things I do manually, but it works. I'm open to suggestions. Just please
document any code so I know where you changed things...
Happy logging and plotting, tabulating, or whatever it is you want to do!
Sherrod
netlog.tar.gz
