Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 13, 2007, at 11:07 AM, Jean-Paul Calderone wrote: > Likely differing buffering behavior. Prior to Linux 2.6, the pipe > implementation allowed only a single buffer (that is, the bytes from > a single write call) in a pipe at a time, and blocked subsequent > writes until that buffer was read. Recently this has changed to allow > multiple buffers up to 4k total length, so multiple short writes won't > block anymore. OS X may have some other buffering behavior which is > causing writes to block where they don't on Linux. All these details > are left to the platform, and there are a variety of behaviors which > can be considered valid. > > Of course, I don't actually /know/ the cause of the problem here, but > this explanation seems plausible to me, and I'd investigate it before > looking for platform-specific pty bugs (although OS X is a good > platform > on which to go looking for those ;). Seems plausible to me too. If I change the child writes to > 4000 bytes, Linux w/2.6 kernel will block too. I'm going to change the test_pty.py code to read from the master_fd. You have to wrap the read in a try/except to catch the OSError input/ output error you'll get on Linux when you read past the end of the buffer and the child process has already exited. Thanks, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRh+gO3EjvBPtnXfVAQJkQAP9E2Jp/EeF3ChIeNZEPVMDeG4Kd+PVslSs blNFO2oO6eO8Yn9X3hlnLwCe3W6+89bS9u7MyN1C2CCZTBz6N1QAkfDaAnxuhbNC 3hfvytMguTY0v3RiZhEVY+y/h2PfbWe6fJGXeBIfcth1HWhP0KWIgAoDA+odDm1B vZSvcxBRN1Y= =fDrh -END PGP SIGNATURE- ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
On Fri, 13 Apr 2007 11:02:01 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote: >-BEGIN PGP SIGNED MESSAGE- >Hash: SHA1 > >On Apr 13, 2007, at 10:57 AM, Jean-Paul Calderone wrote: >>>I don't know if this is caused by a bug in the Mac's pty >>>implementation or something we're doing wrong on that platform. I >>>played around with several modifications to pty.fork() on the Mac, >>>including letting it drop down to the openpty()/os.fork() code, even >>>adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens >>>chapter 19 recommends for 4.3+BSD. I can't get it to not block. >> >>What about reading from the child in the parent before calling waitpid? > >Yep, this is what I suggested below. Porting the same change over to Linux >produced an OSError, but that's probably just because I wasn't as careful >as I should have been late last night. >>>Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, >>>then I would like to at least make test_pty.py not block when run in >>>verbose mode. A very simple hack would add something like this to >>>the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly >>>protected by a "if verbose:". A less icky hack would be to read the >>>output from the master_fd in the parent, though you have to be >>>careful with that on Linux else the read can throw an input/output >>>error. >>> >>>Disabling debug output is band-aid yes, and any application on the >>>Mac like the above snippet will still fail. If anybody has any >>>suggestions, I'm all ears, but I've reached the limit of my pty-fu. >> >>I don't think this is an OS X PTY bug. Writing to a blocking file >>descriptor can block. Programs that do this need to account for the >>possibility. > >Why doesn't it block on Linux then? > Likely differing buffering behavior. Prior to Linux 2.6, the pipe implementation allowed only a single buffer (that is, the bytes from a single write call) in a pipe at a time, and blocked subsequent writes until that buffer was read. Recently this has changed to allow multiple buffers up to 4k total length, so multiple short writes won't block anymore. OS X may have some other buffering behavior which is causing writes to block where they don't on Linux. All these details are left to the platform, and there are a variety of behaviors which can be considered valid. Of course, I don't actually /know/ the cause of the problem here, but this explanation seems plausible to me, and I'd investigate it before looking for platform-specific pty bugs (although OS X is a good platform on which to go looking for those ;). Jean-Paul ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 13, 2007, at 10:57 AM, Jean-Paul Calderone wrote: >> I don't know if this is caused by a bug in the Mac's pty >> implementation or something we're doing wrong on that platform. I >> played around with several modifications to pty.fork() on the Mac, >> including letting it drop down to the openpty()/os.fork() code, even >> adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens >> chapter 19 recommends for 4.3+BSD. I can't get it to not block. > > What about reading from the child in the parent before calling > waitpid? Yep, this is what I suggested below. Porting the same change over to Linux produced an OSError, but that's probably just because I wasn't as careful as I should have been late last night. >> Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, >> then I would like to at least make test_pty.py not block when run in >> verbose mode. A very simple hack would add something like this to >> the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly >> protected by a "if verbose:". A less icky hack would be to read the >> output from the master_fd in the parent, though you have to be >> careful with that on Linux else the read can throw an input/output >> error. >> >> Disabling debug output is band-aid yes, and any application on the >> Mac like the above snippet will still fail. If anybody has any >> suggestions, I'm all ears, but I've reached the limit of my pty-fu. >> > > I don't think this is an OS X PTY bug. Writing to a blocking file > descriptor can block. Programs that do this need to account for the > possibility. Why doesn't it block on Linux then? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRh+banEjvBPtnXfVAQJxFQP+LRAdGVHuqquvGNT9fncGeJFuCjGm+dsS VnE8cirvfoSEdJ0nZ11wHMoMH2vtbXyfjLJJ4G/sROMmHKWJ2t5ieNIxmVutiiLa OPyls2bzuXL8IoyYh+c8tKRyBd76O5GN2EZABxGm2Tie5nt72ezVSEfiDovc6qEu 4lYERTD1YKA= =fZaZ -END PGP SIGNATURE- ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
On Fri, 13 Apr 2007 10:32:28 -0400, Barry Warsaw <[EMAIL PROTECTED]> wrote: >-BEGIN PGP SIGNED MESSAGE- >Hash: SHA1 > >I've been getting some test failures in Python 2.5 svn head on Mac OS >X 10.4.9 which I'm not getting on Linux (Ubuntu feisty beta). >test_sqlite and test_zipimport both fail, however, when run in >verbose mode (e.g. ./python.exe Lib/test/test_sqlite.py) both pass. > >But that's not exactly why I'm writing this email . In the >course of trying to debug this, I ran the following on my Mac: > >make TESTOPTS=-v test > >This runs the entire test suite in verbose mode, and you do get a lot >of output. However the test suite hangs on test_pty.py. In fact, if >you run that test alone: > >./python.exe Lib/test/test_pty.py > >it too hangs for me. The reason is that in verbose mode, debug() >actually prints stuff to stdout and on the Mac, when the child of the >pty.fork() writes to its stdout, it blocks and so the parent's waitpid >() never returns. This doesn't happen on Linux though; the child's >stdout prints don't block, it exits, and the parent continues after >the waitpid(). > >Here's a very simple program that reproduces the problem: > >- -snip snip- >import os, pty, sys > >pid, fd = pty.fork() >print >> sys.stderr, pid, fd >if pid: > os.waitpid(pid, 0) >else: > os._exit(0) >- -snip snip- > >stderr, stdout, doesn't matter. This hangs on the Mac but completes >successfully on Linux. Of course, in neither case do you see the >child's output. > >I don't know if this is caused by a bug in the Mac's pty >implementation or something we're doing wrong on that platform. I >played around with several modifications to pty.fork() on the Mac, >including letting it drop down to the openpty()/os.fork() code, even >adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens >chapter 19 recommends for 4.3+BSD. I can't get it to not block. What about reading from the child in the parent before calling waitpid? > >Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, >then I would like to at least make test_pty.py not block when run in >verbose mode. A very simple hack would add something like this to >the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly >protected by a "if verbose:". A less icky hack would be to read the >output from the master_fd in the parent, though you have to be >careful with that on Linux else the read can throw an input/output >error. > >Disabling debug output is band-aid yes, and any application on the >Mac like the above snippet will still fail. If anybody has any >suggestions, I'm all ears, but I've reached the limit of my pty-fu. > I don't think this is an OS X PTY bug. Writing to a blocking file descriptor can block. Programs that do this need to account for the possibility. Jean-Paul ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've been getting some test failures in Python 2.5 svn head on Mac OS X 10.4.9 which I'm not getting on Linux (Ubuntu feisty beta). test_sqlite and test_zipimport both fail, however, when run in verbose mode (e.g. ./python.exe Lib/test/test_sqlite.py) both pass. But that's not exactly why I'm writing this email . In the course of trying to debug this, I ran the following on my Mac: make TESTOPTS=-v test This runs the entire test suite in verbose mode, and you do get a lot of output. However the test suite hangs on test_pty.py. In fact, if you run that test alone: ./python.exe Lib/test/test_pty.py it too hangs for me. The reason is that in verbose mode, debug() actually prints stuff to stdout and on the Mac, when the child of the pty.fork() writes to its stdout, it blocks and so the parent's waitpid () never returns. This doesn't happen on Linux though; the child's stdout prints don't block, it exits, and the parent continues after the waitpid(). Here's a very simple program that reproduces the problem: - -snip snip- import os, pty, sys pid, fd = pty.fork() print >> sys.stderr, pid, fd if pid: os.waitpid(pid, 0) else: os._exit(0) - -snip snip- stderr, stdout, doesn't matter. This hangs on the Mac but completes successfully on Linux. Of course, in neither case do you see the child's output. I don't know if this is caused by a bug in the Mac's pty implementation or something we're doing wrong on that platform. I played around with several modifications to pty.fork() on the Mac, including letting it drop down to the openpty()/os.fork() code, even adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens chapter 19 recommends for 4.3+BSD. I can't get it to not block. Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, then I would like to at least make test_pty.py not block when run in verbose mode. A very simple hack would add something like this to the "if pid == pty.CHILD" stanza: "def debug(msg): pass", possibly protected by a "if verbose:". A less icky hack would be to read the output from the master_fd in the parent, though you have to be careful with that on Linux else the read can throw an input/output error. Disabling debug output is band-aid yes, and any application on the Mac like the above snippet will still fail. If anybody has any suggestions, I'm all ears, but I've reached the limit of my pty-fu. mr-t-says-i-pty-the-fu-ly y'rs, - -Barry P.S. chime on on the test_sqlite and test_zipimport oddities if you want. :) -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRh+Ug3EjvBPtnXfVAQJ4HgQAkm3XUq27GPBDKJj1NaIHuYijUerSE2oW 9YWiCYNQ+hBaYJJ16gdqZ2f7t/ENYVBzj3r9fNj+uyI1a6OFFMlE1tOdDFscQWUE e04XGwxo1fEPelJveIw8/dLHCPCpmCqvzLrT8fAtr7HL3thW3rk8s69i9i+CtJ1L LP7AY/SYXxg= =I+pF -END PGP SIGNATURE- ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
