Re: tai64n -- why?
On Sat, Aug 05, 2000 at 08:07:38PM -0700, Eric Cox wrote: But I'm getting back into it. When I get it into a usable state I'll post it on my website at www.ericcox.com. Keep checking back... Your mailstats program looks pretty cool. Does it work with multilog logs? With tai64n and all that? Ben -- Ben Beuchler [EMAIL PROTECTED] MAILER-DAEMON (612) 321-9290 x101 Bitstream Underground www.bitstream.net
Re: tai64n -- why?
On Fri, Aug 04, 2000 at 07:11:52PM -0700, Russ Allbery wrote: syslog timestamps are amazingly annoying to try to parse. TAI64 is trivial to parse. This is a significant improvement. ISO date/time format would also have been easy to parse, and I would have been slightly happier with that, but TAI64 is definitely a *huge* improvement over syslog if you want to do anything at all automated with the logs. OK. I can see where that is a strong argument for TAI64n. I have tried to develop a way of converting from TAI64n to a human readable format but I'm afraid I'm having difficulty comprehending the format. Has anyone written any perl/python code that translates to a human readable format? Or can offer a brief "pseudo-code" algorithm? Or even a more clear description of the format would be helpful. Dan's writeup left me a bit lost... Gracias, Ben -- Ben Beuchler [EMAIL PROTECTED] MAILER-DAEMON (612) 321-9290 x101 Bitstream Underground www.bitstream.net
Re: tai64n -- why?
Ben Beuchler wrote: On Fri, Aug 04, 2000 at 07:11:52PM -0700, Russ Allbery wrote: syslog timestamps are amazingly annoying to try to parse. TAI64 is trivial to parse. This is a significant improvement. ISO date/time format would also have been easy to parse, and I would have been slightly happier with that, but TAI64 is definitely a *huge* improvement over syslog if you want to do anything at all automated with the logs. OK. I can see where that is a strong argument for TAI64n. I have tried to develop a way of converting from TAI64n to a human readable format but I'm afraid I'm having difficulty comprehending the format. Has anyone written any perl/python code that translates to a human readable format? Or can offer a brief "pseudo-code" algorithm? Or even a more clear description of the format would be helpful. Dan's writeup left me a bit lost... Have you looked into using libtai in your app? It's Dan's library for using tai, and could be linked with your application. I currently trying to write a Python wrapper for it so I can use it from my Python scripts... Eric
Re: tai64n -- why?
Ben Beuchler [EMAIL PROTECTED] writes:
OK. I can see where that is a strong argument for TAI64n. I have tried
to develop a way of converting from TAI64n to a human readable format
but I'm afraid I'm having difficulty comprehending the format. Has
anyone written any perl/python code that translates to a human readable
format? Or can offer a brief "pseudo-code" algorithm?
Here is some stuff that I use in perl scripts. There is plenty to
criticise here. It assumes a 32-bit platform and fails after 2038.
It is much less efficient than tai64nlocal. Handy if you just need
to output some lines that the script has selected, though.
# Simplistic tai64n converter. Returns float time_t value or undef.
sub tai64n {
my(@x) = map { unpack "N", pack "H8", $_ }
$_[0] =~ /^\@?([0-9a-f]{8})([0-9a-f]{8})([0-9a-f]{8})/;
return unless @x == 3 and $x[0] == 0x4000;
$x[1]+$x[2]/1_000_000_000;
}
# replace tai64n timestamp at beginning of line and print result
sub putline {
my($stamp,$msg) = $_[0] =~ /^(\S+) (.*)/;
$stamp = tai64n($stamp);
my $int = int($stamp);
my $frac = int(($stamp-$int)*1_000_000_000);
my @t = localtime $int;
printf("%d-%02d-%02d %02d:%02d:%02d.%09d %s\n",
$t[5]+1900,$t[4]+1,@t[3,2,1,0],$frac,$msg);
}
--
Frank Cringle, [EMAIL PROTECTED]
voice: (+49 2304) 467101; fax: 943357
Re: tai64n -- why?
On Sat, Aug 05, 2000 at 11:54:41AM -0700, Eric Cox wrote: Have you looked into using libtai in your app? It's Dan's library for using tai, and could be linked with your application. I currently trying to write a Python wrapper for it so I can use it from my Python scripts... You're my hero! The reason I don't use libtai is that I don't program in C. I program in Python. How much headway have you made? Is it usable? Gracias, Ben -- Ben Beuchler [EMAIL PROTECTED] MAILER-DAEMON (612) 321-9290 x101 Bitstream Underground www.bitstream.net
Re: tai64n -- why?
Ben Beuchler wrote: On Sat, Aug 05, 2000 at 11:54:41AM -0700, Eric Cox wrote: Have you looked into using libtai in your app? It's Dan's library for using tai, and could be linked with your application. I currently trying to write a Python wrapper for it so I can use it from my Python scripts... You're my hero! The reason I don't use libtai is that I don't program in C. I program in Python. How much headway have you made? Is it usable? Heh, no way... :( But I'm getting back into it. When I get it into a usable state I'll post it on my website at www.ericcox.com. Keep checking back... Eric
Re: tai64n -- why?
David Dyer-Bennet [EMAIL PROTECTED] writes: Yes, when I first looked at it. As is often the case with Dan, I just disagree. It's not straight text in the sense I mean; it's not human readable. Of all the strange choices Dan's made that I've encountered in working with qmail, this is the first one that I fail completely to understand. All the others, I see the tradeoffs and I see why he chose as he did, even if I might have chosen otherwise. This one makes zero sense. It's non-functional. It doesn't connect to the way I work. syslog timestamps are amazingly annoying to try to parse. TAI64 is trivial to parse. This is a significant improvement. ISO date/time format would also have been easy to parse, and I would have been slightly happier with that, but TAI64 is definitely a *huge* improvement over syslog if you want to do anything at all automated with the logs. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/
Re: tai64n -- why?
Are you asking more for something like: 2000/07/31 06:02:10.42 (GMT+05) This has always been the date format I've prefered ... its sortable (as the year comes first -- although its quite narrow-minded of me to not allow for 5 digit dates), its human-readable, and quite parsable. The decimal portion after the time allows for sub-second time measurement. I'm pretty sure adding a function to your copy of multilog/whatever that converted from the tai64n format to something more like the above wouldn't be that hard. David Dyer-Bennet wrote: Would you prefer the splogger format (to wit, Unix timestamp with fractional part) instead? I'd do anything to use a logging format that avoids timezone dependency, and multilog/tai64n seems to do that job well. You mean this: Jul 31 06:02:10 gw qmail: 965023330.820010 status: local 1/10 remote 0/50 ? It's better than tai64n, because syslog puts a real timestamp on, but that big chunk of meaningless numbers in the middle wastes a lot of the line and adds no useful information. It's what I'm using now on my main server, but it's quite wasteful and annoying. (But qmailanalog expects it)
Re: tai64n -- why?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1 Aug 00, at 9:58, Michael T. Babcock wrote: Are you asking more for something like: 2000/07/31 06:02:10.42 (GMT+05) This has always been the date format I've prefered ... its sortable (as the year comes first -- although its quite narrow-minded of me to not allow for 5 digit dates), It's not sortable: 1. '+' precedes '-' in ASCII. Yuck. 2. How do you compare (cutting unimportant stuff) 17:30 (GMT+1) and 18:00 (GMT+0)? Go figure. 3. Fixing seconds precision to two decimals is wrong, too. There goes. You mean this: Jul 31 06:02:10 gw qmail: 965023330.820010 status: local 1/10 remote 0/50 ? What? You're timestamping your logs both by syslog and some timestamper from old daemontools, whatever-was-its-name, accustamp? Why on Earth? Either you want to log through syslogd or something which adds the stamps (you call them human readable), or through accustamp or multilog which adds tai or tai64 timestamps. And, for what it's worth, _my_ logs should be machine readable, not human readable. I inspect qmail logs only seldom, and I can surely pipe them through tai(64n)local; however, my log analyzing tools run much more often, and I expect lots of reliability from them. And I don't want to spend several hours explaining the log parser that on my system, the timestamp looks like "%D %d %M %h:%m:%.4s %TZ" or whatever. -BEGIN PGP SIGNATURE- Version: PGP 6.0.2 -- QDPGP 2.60 Comment: http://community.wow.net/grt/qdpgp.html iQA/AwUBOYbMfVMwP8g7qbw/EQICmACeLP7qwvTtHsOz9Exks6t+8YaF2u0AoOzr YJ5FgM9B3kdr8pUKnRZk3s0O =1OBP -END PGP SIGNATURE- -- Petr Novotny, ANTEK CS [EMAIL PROTECTED] http://www.antek.cz PGP key ID: 0x3BA9BC3F -- Don't you know there ain't no devil there's just God when he's drunk. [Tom Waits]
Re: tai64n -- why?
[EMAIL PROTECTED] [EMAIL PROTECTED] writes on 30 July 2000 at 21:21:19 -0700 On Sun, Jul 30, 2000 at 11:06:38PM -0500, Ben Beuchler wrote: I understand from DJB's website that TAI is a better way to deal with time functions than the typical unix localtime(). However, it seems to make a lot of things really awkward when it is used as the time stamp in a log file. Really? If I want to tail a log file, eg, I go like this: tail ../someservice/current | tai64nlocal and it all looks fine for humans. Yeah, it works fine for people who check log files by tailing them. I check them by bringing them into an emacs buffer, so the funny timestamps make them darned near useless. I actually find second resolution entirely adequate for log entries; the great detail TAI provides is just wasted space to me, and displaces other, more useful, information from the screen. -- Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]
Re: tai64n -- why?
Charles Cazabon [EMAIL PROTECTED] writes on 31 July 2000 at 11:20:48 -0600 David Dyer-Bennet [EMAIL PROTECTED] wrote: Really? If I want to tail a log file, eg, I go like this: tail ../someservice/current | tai64nlocal and it all looks fine for humans. Yeah, it works fine for people who check log files by tailing them. I check them by bringing them into an emacs buffer, so the funny timestamps make them darned near useless. So why not tail them to a temp file and use emacs to view the temp file? Or write an emacs-lisp function to convert the timestamps. If I'm going to go to effort to make it work the way I want, I think I'll just change multilog to use a sensible format. It's silly having archival log files sitting there that don't mean anything without a conversion program; straight text is the appropriate format for log files. I had thought about writing a mode for TAI stamped log files that converts the timestamps, but that will be pretty slow since it'll have to change every line of the file. -- Photos: http://dd-b.lighthunters.net/ Minicon: http://www.mnstf.org/minicon Bookworms: http://ouroboros.demesne.com/ SF: http://www.dd-b.net/dd-b David Dyer-Bennet / Welcome to the future! / [EMAIL PROTECTED]
Re: tai64n -- why?
On Mon, Jul 31, 2000 at 12:23:38PM -0500, David Dyer-Bennet wrote: Charles Cazabon [EMAIL PROTECTED] writes on 31 July 2000 at 11:20:48 -0600 David Dyer-Bennet [EMAIL PROTECTED] wrote: Really? If I want to tail a log file, eg, I go like this: tail ../someservice/current | tai64nlocal and it all looks fine for humans. Yeah, it works fine for people who check log files by tailing them. I check them by bringing them into an emacs buffer, so the funny timestamps make them darned near useless. So why not tail them to a temp file and use emacs to view the temp file? Or write an emacs-lisp function to convert the timestamps. If I'm going to go to effort to make it work the way I want, I think I'll just change multilog to use a sensible format. It's silly having archival log files sitting there that don't mean anything without a conversion program; straight text is the appropriate format for log files. But it *is* straight text. The point about tai is that it's entirely appropriate for log files that may live for a long time. Have you read the rationale for tai at all? Regards.
Re: tai64n -- why?
On Mon, Jul 31, 2000 at 12:54:23PM -0500, David Dyer-Bennet wrote: Yes, when I first looked at it. As is often the case with Dan, I just disagree. It's not straight text in the sense I mean; it's not human readable. Of all the strange choices Dan's made that I've encountered in working with qmail, this is the first one that I fail completely to understand. All the others, I see the tradeoffs and I see why he chose as he did, even if I might have chosen otherwise. This one makes zero sense. It's non-functional. It doesn't connect to the way I work. You have expressed my sentiments precisely. It has left me sad and confused. Ben -- Ben Beuchler [EMAIL PROTECTED] MAILER-DAEMON (612) 321-9290 x101 Bitstream Underground www.bitstream.net
Re: tai64n -- why?
On Mon, Jul 31, 2000 at 12:54:23PM -0500, David Dyer-Bennet wrote: ! This one ! makes zero sense. It's non-functional. It doesn't connect to the way ! I work. Would you prefer the splogger format (to wit, Unix timestamp with fractional part) instead? I'd do anything to use a logging format that avoids timezone dependency, and multilog/tai64n seems to do that job well. ---Chris K. -- Chris, the Young One |_ Never brag about how your machines haven't been Auckland, New Zealand |_ hacked, or your code hasn't been broken. It's http://cloud9.hedgee.com/ |_ guaranteed to bring the wrong kind of PGP: 0xCCC6114E/0x706A6AAD |_ attention. ---Neil Schneider
tai64n -- why?
I understand from DJB's website that TAI is a better way to deal with time functions than the typical unix localtime(). However, it seems to make a lot of things really awkward when it is used as the time stamp in a log file. Any particular idea why DJB chose to use it for output in multilog? It seems to me that all it accomplishes is adding extra steps to gleaning any useful info from the logs... Ben -- Ben Beuchler [EMAIL PROTECTED] MAILER-DAEMON (612) 321-9290 x101 Bitstream Underground www.bitstream.net
Re: tai64n -- why?
On Sun, Jul 30, 2000 at 11:06:38PM -0500, Ben Beuchler wrote: I understand from DJB's website that TAI is a better way to deal with time functions than the typical unix localtime(). However, it seems to make a lot of things really awkward when it is used as the time stamp in a log file. Really? If I want to tail a log file, eg, I go like this: tail ../someservice/current | tai64nlocal and it all looks fine for humans. Any particular idea why DJB chose to use it for output in multilog? It seems to me that all it accomplishes is adding extra steps to gleaning any useful info from the logs... Well, there are two issues here. One is the use of that particular form of timestamp and the granularity of the timestamp. The granularity of syslog is a second which is no where near good enough for timing events that occur at sub-second rates. Knowing the author of syslog, I can guess why he thought that a second was plenty accurate enough for his MTA. But in the real world of modern computing, events happen more briefly. Once you accept that the current mainstream logging system isn't giving sufficiently useful timestamps, then any change is going to carry some level of awkwardness. What DjB has done is pick a timestamp that has some pretty useful characteristics. Importantly, he has chosen one which is as close to absolute as is possible and one which is easy to use with date arithmetic. Regards.
