Re: tail does not exit
On Dec 20, 2007, at 1:58 AM, Mikhail Teterin wrote: On середа 19 грудень 2007, Chuck Swiger wrote: = A quick test suggests that "tail -f" will close when it gets a SIGPIPE. SIGPIPE? How is that relevant? Does tail get a SIGPIPE, when awk disappears in my example? If it does not, why do you bring it up? tail should get a SIGPIPE when it tries to write to a pipeline where the other end has closed. And if it does get SIGPIPE, then you are wrong, because the posted "quick test" shows the exact opposite behavior -- tail does NOT go away. Please, clarify... Thanks. Worked for me. I opened two SSH sessions to a FreeBSD 5.5 system, and did this in one: % touch /tmp/logfile % echo "line 1" >> /tmp/logfile ...and this in the other: % tail -f /tmp/logfile | awk '{print "Line: " $1 ; exit(0)}END{print "Bye"}' ...when I then did a: % echo "line 2" >> /tmp/logfile ...in the first, the tail -f process terminated in the second. -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
четвер 20 грудень 2007 11:58 до, Erik Osterholm Ви написали: > Ah, I see. With very, very long lines, tail doesn't send the output > all at once. The cutoff seems to be 65536 bytes on my system. They don't even have to be very very long -- unless in an artificial example, such as the one I posted. Normal-width text files can also trigger inconsistent behavior in some real-life scenario, where awk actually does some real processing of its input for a while. The awk script may decide to quit after processing the first 1000 (normal-length) lines, for example... The behavior of the program will then be different depending on whether the average line-length is above, at, or below 65.536 characters. Maybe, it is awk's fault -- it should not be read-ing more than one line at a time, because the script may cause it to ignore some of the read data. Using line-buffering or some such? -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On Thu, Dec 20, 2007 at 11:02:59AM -0500, Mikhail Teterin wrote: > On ?? 20 ??? 2007, Erik Osterholm wrote: > = The same behavior happens if I use a larger file. I see no > = inconsistent behavior, nor any bugs. > > The inconsistency is in the fact, that the behavior depends on the size of > the > buffer and length of the lines (not the size of the file). > > If the 10 lines, which tail tries to output initially, exceed the size of the > buffer, tail learns about awk going away immediately. If the lines are not > long enough, it does not. > > Also, I would expect a program to be notified (by SIGPIPE?) /immediately/, > when any of its output pipes are closed -- instead of waiting for it to try > to write into the pipe. But this issue is not, it seems, FreeBSD-specific... > > -mi Ah, I see. With very, very long lines, tail doesn't send the output all at once. The cutoff seems to be 65536 bytes on my system. If tail has to write more than this amount, it breaks it up into mutliple writes of a maximum of 65536 characters each. The problem is that after the first 65536 characters, awk has exited, causing the next 65536 characters which tail attempts to write to cause a SIGPIPE. It seems to be working as intended, though. When piping, you have to be aware of these issues, but I do not think that it is a bug. There must be some boundary where tail splits the output into multiple writes. If, after the first write, a \n hasn't been encountered yet, awk will consider at least some portion of the next write (up until the first \n) to be the same line, at least until it hits its own limit. I have not checked to see what this limit might be. As for SIGPIPE, that's just how the POSIX standard works. The signal is sent to the writing process when it attempts to write to a broken or closed pipe, not when the pipe has closed. If you think that this behavior is bad, you might want to contact IEEE. Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On Thu, Dec 20, 2007 at 05:40:11AM -0500, Mikhail Teterin wrote: > On ?? 20 ??? 2007, Max N. Boyarov wrote: = | MT> Is not that > a bug in itself? = = | Tail write buffer at all, i.e. all 10 lines > writes to pipe. > > So, the behavior depends on the size of the buffer -- and thus the > size of the input lines. > > A bug indeed... I don't understand. aleph:~$ cat test blah1 blah2 blah3 aleph:~$ tail -f test | awk '{print $1; exit 0}' blah1 (hangs) This is expected. Awk printed one time and exited, per the given script. The output from tail/input from awk went all at once, awk printed the first line, exited, and the rest of the input disappeared. 'tail' sent "blah1\nblah2\nblah3\n" to awk, awk printed until the first newline and exited. If I now write to test from another terminal: aleph:~$ echo "blah4" >> test Tail tries to write to the pipe, which it finds closed. It receives a SIGPIPE (tried to write to a pope with no reader--see man signal), and it terminates. The same behavior happens if I use a larger file. I see no inconsistent behavior, nor any bugs. Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On четвер 20 грудень 2007, Erik Osterholm wrote: = The same behavior happens if I use a larger file. I see no = inconsistent behavior, nor any bugs. The inconsistency is in the fact, that the behavior depends on the size of the buffer and length of the lines (not the size of the file). If the 10 lines, which tail tries to output initially, exceed the size of the buffer, tail learns about awk going away immediately. If the lines are not long enough, it does not. Also, I would expect a program to be notified (by SIGPIPE?) /immediately/, when any of its output pipes are closed -- instead of waiting for it to try to write into the pipe. But this issue is not, it seems, FreeBSD-specific... -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On четвер 20 грудень 2007, Max N. Boyarov wrote: = MT> Is not that a bug in itself? = = Tail write buffer at all, i.e. all 10 lines writes to pipe. So, the behavior depends on the size of the buffer -- and thus the size of the input lines. A bug indeed... -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
> "MT" == Mikhail Teterin writes: MT> On четвер 20 грудень 2007, Max N. Boyarov wrote: MT> = after something writeln to /var/log/messages tail get SIGPIPE MT> But why is that needed for tail to notice? It is trying to output 10 lines. MT> After it outputs the very first one of them, awk exits, and the 9 subsequent MT> lines go into thin air /without tail noticing/. MT> Is not that a bug in itself? Tail write buffer at all, i.e. all 10 lines writes to pipe. $ cat test line1 line2 line3 line4 line5 line6 line7 line8 line9 line10 line11 tail -f test | awk '{print "Exiting" $1; exit 0}' open("test", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=68, ...}) = 0 syscall_477(0, 0x44, 0x1, 0x1, 0x3, 0, 0) = 0x28175000 write(1, "line2\nline3\nline4\nline5\nline6\nli"..., 62Exitingline2 ) = 62 fstat(3, {st_mode=S_ISUID|S_ISVTX|070, st_size=0, ...}) = 0 syscall_478(0x3, 0, 0, 0x1) = 0 syscall_478(0x3, 0, 0, 0) = 0 read(3, "line1\nline2\nline3\nline4\nline5\nli"..., 4096) = 68 ^^ write buff munmap(0x28175000, 68) = 0 read(3, "", 4096) = 0 kqueue(0x28172d40) = 4 syscall_397(0x3, 0xbfbfe44c)= 0 kevent(0x4, 0x28205040, 0x1, 0, 0, 0xbfbfe624) = 0 kevent(0x4, 0, 0, 0x28205040, 0x1, 0) = 1 read(3, "line++\n", 4096) = 7 ^^ new line added ^ fstat(1, {st_mode=031545, st_size=7596457873570623081, ...}) = 0 read(3, "", 4096) = 0 write(1, "line++\n", 7) = -1 EPIPE (Broken pipe) ^ try write ^^ --- SIGPIPE (Broken pipe: 13) --- --- SIGPIPE (Broken pipe: 13) --- exit ^ -- Max N. Boyarov pgpQlaBQK0jet.pgp Description: PGP signature
Re: tail does not exit
On четвер 20 грудень 2007, Max N. Boyarov wrote: = after something writeln to /var/log/messages tail get SIGPIPE But why is that needed for tail to notice? It is trying to output 10 lines. After it outputs the very first one of them, awk exits, and the 9 subsequent lines go into thin air /without tail noticing/. Is not that a bug in itself? -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
> "MT" == Mikhail Teterin writes: [...] MT> I'm sorry, this does not make sense to me. Starting with an empty MT> file, as you do in 1), /may/ make tail not notice, that awk went MT> away, because tail has nothing to write to stdout. MT> But /var/log/messages is not empty, and awk -- in my example -- would MT> exit upon seeing the very first line of its input (tail's output). MT> Yet tail fails to notice, that its subsequent output (starting with the MT> second line) is written to nowhere... MT> Why? Because nothing writeln to /var/log/messages $ sh -x /tmp/x.sh + + tail -f /var/log/messages awk {print "Exiting"; exit 0} Exiting + echo Exited Exited + exit 0 after something writeln to /var/log/messages tail get SIGPIPE open("/var/log/messages",O_RDONLY,0666) = 3 (0x3) fstat(3,{mode=-rw-r--r-- ,inode=141327,size=4997,blksize=4096}) = 0 (0x0) mmap(0x0,4997,PROT_READ,MAP_SHARED,3,0x0)= 672616448 (0x28175000) Exiting write(1,"Dec 20 11:59:04 solar kernel: da"...,1090) = 1090 (0x442) fstat(3,{mode=-rw-r--r-- ,inode=141327,size=4997,blksize=4096}) = 0 (0x0) lseek(3,0x0,SEEK_CUR)= 0 (0x0) lseek(3,0x1000,SEEK_SET) = 4096 (0x1000) read(3,"kernel: da1: 1.000MB/s transfers"...,4096) = 901 (0x385) munmap(0x28175000,4997) = 0 (0x0) read(3,0x28204000,4096) = 0 (0x0) kqueue(0x28172d40,0x4,0xa,0x0,0x28201088,0x1)= 4 (0x4) fstatfs(0x3,0xbfbfe44c,0x2815fe98,0x28172d40,0x0,0x2806dee4) = 0 (0x0) kevent(4,{0x3,EVFILT_READ,EV_ADD|EV_ENABLE|EV_CLEAR,0,0x0,0x0},1,0x0,0,{0.0}) = 0 (0x0) kevent(4,0x0,0,{0x3,EVFILT_READ,EV_CLEAR,0,0x3e,0x0},1,0x0) = 1 (0x1) read(3,"Dec 20 12:06:27 solar su: BAD SU"...,4096) = 62 (0x3e) fstat(1,{mode=p- ,inode=0,size=0,blksize=4096}) = 0 (0x0) read(3,0x28204000,4096) = 0 (0x0) write(1,"Dec 20 12:06:27 solar su: BAD SU"...,62) ERR#32 'Broken pipe' SIGNAL 13 (SIGPIPE) -- Max N. Boyarov pgp6Vf07VlDFE.pgp Description: PGP signature
Re: tail does not exit
On середа 19 грудень 2007, Chuck Swiger wrote: = A quick test suggests that "tail -f" will close when it gets a SIGPIPE. SIGPIPE? How is that relevant? Does tail get a SIGPIPE, when awk disappears in my example? If it does not, why do you bring it up? And if it does get SIGPIPE, then you are wrong, because the posted "quick test" shows the exact opposite behavior -- tail does NOT go away. Please, clarify... Thanks. On середа 19 грудень 2007, Max N. Boyarov wrote: = try to test your script with anoter file and add somthing to it = = 1) cons1$ touch /tmp/test = 2) cons1$ tail -f /tmp/test | awk '{print "Line: " $1 ; exit(0)}END{print "Bye"}' = 2a) Line: Line1 = 2b) Bye I'm sorry, this does not make sense to me. Starting with an empty file, as you do in 1), /may/ make tail not notice, that awk went away, because tail has nothing to write to stdout. But /var/log/messages is not empty, and awk -- in my example -- would exit upon seeing the very first line of its input (tail's output). Yet tail fails to notice, that its subsequent output (starting with the second line) is written to nowhere... Why? -mi P.S. Here is the example again: #!/bin/sh if tail -f /var/log/messages | awk '{print "Exiting"; exit 0}' then echo Exited else echo Failed fi exit 0 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On Dec 19, 2007, at 4:06 PM, Mikhail Teterin wrote: Josh Tolbert: Cause the -f option to tail doesn't work that way. -f always waits for more input. I know very well about -f waiting for more *input*. What puzzles me, is that tail does not quit, when its *output* is closed. James Harrison: Is there a reason you want the -f flag? Yes, I want awk to be processing the lines, which are appended to the file, until it finds, what it is looking for, and exits. I expect tail to go away, when its stdout is closed, but it does not. tail -f holds on for dear life until a ctrl-c happens. IT HAS A DEATH GRIP! This seems like a bug to me... It should hold on to its input file(s), but exit peacefully, when its stdout closes. No? A quick test suggests that "tail -f" will close when it gets a SIGPIPE. -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
> "MT" == Mikhail Teterin writes: MT> Max N. Boyarov: >> -f The -f option causes tail to not stop when end of file is >> reached, but rather to wait for additional data to be appended >> to the input. The -f option is ignored if the standard input is a pipe, >> but not if it is a FIFO. MT> Josh Tolbert: >> Cause the -f option to tail doesn't work that way. -f always waits for more >> input. MT> I know very well about -f waiting for more *input*. What puzzles me, is that MT> tail does not quit, when its *output* is closed. MT> James Harrison: >> Is there a reason you want the -f flag? MT> Yes, I want awk to be processing the lines, which are appended to the file, MT> until it finds, what it is looking for, and exits. I expect tail to go away, MT> when its stdout is closed, but it does not. >> tail -f holds on for dear life until a ctrl-c happens. IT HAS A DEATH >> GRIP! MT> This seems like a bug to me... It should hold on to its input file(s), but MT> exit peacefully, when its stdout closes. No? tail -f /tmp/foo | awk '{print "$1"} tail write to pipe (reader wait data) (2) reader get data (2a) and exit (2b). action (3) from another console tail write data to pipe again. action (4) from another console tail get SIGPIPE tail exit try to test your script with anoter file and add somthing to it 1) cons1$ touch /tmp/test 2) cons1$ tail -f /tmp/test | awk '{print "Line: " $1 ; exit(0)}END{print "Bye"}' 2a) Line: Line1 2b) Bye 3) cons2$ echo Line1 >> /tmp/test 4) cons2$ echo Line2 >> /tmp/test -- Max N. Boyarov pgpgAFND1AZND.pgp Description: PGP signature
Re: tail does not exit
Max N. Boyarov: > -f The -f option causes tail to not stop when end of file is > reached, but rather to wait for additional data to be appended > to the input. The -f option is ignored if the standard input is a pipe, > but not if it is a FIFO. Josh Tolbert: > Cause the -f option to tail doesn't work that way. -f always waits for more > input. I know very well about -f waiting for more *input*. What puzzles me, is that tail does not quit, when its *output* is closed. James Harrison: > Is there a reason you want the -f flag? Yes, I want awk to be processing the lines, which are appended to the file, until it finds, what it is looking for, and exits. I expect tail to go away, when its stdout is closed, but it does not. > tail -f holds on for dear life until a ctrl-c happens. IT HAS A DEATH > GRIP! This seems like a bug to me... It should hold on to its input file(s), but exit peacefully, when its stdout closes. No? -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
> "MT" == Mikhail T. writes: MT> Why does not the script below actually ever exit? MT> #!/bin/sh MT> if tail -f /var/log/messages | awk '{print "Exiting"; exit 0}' MT> then MT> echo Exited MT> else MT> echo Failed MT> fi MT> exit 0 MT> Awk exits as advertised, but tail stays around -- even though its MT> stdout is closed... Why? ---start quote from man 1 tail--- -f The -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the input. The -f option is ignored if the standard input is a pipe, but not if it is a FIFO. ---end quote--- try without "-f" option -- Max N. Boyarov pgp85bt53VpkK.pgp Description: PGP signature
Re: tail does not exit
James Harrison writes: > tail -f holds on for dear life until a ctrl-c happens. IT HAS A > DEATH GRIP! Agreed. Robert Huff ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: tail does not exit
On Wed, 2007-12-19 at 18:22 -0500, Mikhail T. wrote: > #!/bin/sh > > if tail -f /var/log/messages | awk '{print "Exiting"; exit 0}' > then > echo Exited > else > echo Failed > fi > > exit 0 I assume it has something to do with tail -f being used, rather than tail. Is there a reason you want the -f flag? tail -f holds on for dear life until a ctrl-c happens. IT HAS A DEATH GRIP! James ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
tail does not exit
Why does not the script below actually ever exit? #!/bin/sh if tail -f /var/log/messages | awk '{print "Exiting"; exit 0}' then echo Exited else echo Failed fi exit 0 Awk exits as advertised, but tail stays around -- even though its stdout is closed... Why? Thanks! -mi ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"