Re: [Python-Dev] Accepted: PEP 446 -- Make newly created file descriptors non-inheritable
On 27 August 2013 23:17, Guido van Rossum wrote: > Thanks for your tiresome work I'm guessing you meant "tireless" here :-) Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepted: PEP 446 -- Make newly created file descriptors non-inheritable
2013/8/28 Guido van Rossum : > Congratulations Victor, PEP 446 is accepted! Thanks. I just commited the implementation into default (future Python 3.4): http://hg.python.org/cpython/rev/ef889c3d5dc6 http://bugs.python.org/issue18571 I tested it on Linux, FreeBSD 9, OpenIndiana and Windows 7. Let see if the buildbots appreciate non-inheritable file descriptors. Please test python default on your favorite application to see if the PEP 446 broke or not! If the PEP 446 breaks your application, don't worry, it's for the "good of mankind" :-) Adding some calls to os.set_inheritable(fd, True) and using subprocess with pass_fds should fix your issues. I changed the status of the PEP to Final. I also closed the issues related to the PEP 446: #10115: Support accept4() for atomic setting of flags at socket creation #12107: TCP listening sockets created without FD_CLOEXEC flag #16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined #17070: PEP 433: Use the new cloexec to improve security and avoid bugs #18571: Implementation of the PEP 446: non-inheritable file descriptors Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Accepted: PEP 446 -- Make newly created file descriptors non-inheritable
Congratulations Victor, PEP 446 is accepted! Thanks for your tiresome work and the last-minute changes. I will update the PEP's status to "Accepted" right away. You can change it to "Final" after all the changes have been committed to the default branch for inclusion into the next 3.4 alpha. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 446 (make FD non inheritable) ready for a final review
2013/8/27 Antoine Pitrou :
>> On UNIX, the subprocess module closes almost all file descriptors in
>> the child process. This operation requires MAXFD system calls, where
>> MAXFD is the maximum number of file descriptors, even if there are
>> only few open file descriptors. This maximum can be read using:
>> os.sysconf("SC_OPEN_MAX").
>
> If your intent is to remove the closerange() call from subprocess, be
> aware that it may let through some file descriptors opened by
> third-party code (such as C extensions). This may or may not be
> something we want to worry about, but there's still a small potential
> for security regressions.
The PEP doesn't change the default value of the close_fds parameter of
subprocess: file descriptors and handles are still closed in the child
process.
I modified the PEP to explain the link between non-inheritable FDs and
performances:
http://hg.python.org/peps/rev/d88fbf9941fa
If you don't use third party code, or if you control third party code
and you know that these modules only create non-inheritable FDs, it is
now safe (thanks to the PEP 446) to use close_fds=False... which
avoids the cost of closing MAXFD file descriptors explicitly in the
child process.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 446 (make FD non inheritable) ready for a final review
Hi,
I have a small comment to make:
> On UNIX, the subprocess module closes almost all file descriptors in
> the child process. This operation requires MAXFD system calls, where
> MAXFD is the maximum number of file descriptors, even if there are
> only few open file descriptors. This maximum can be read using:
> os.sysconf("SC_OPEN_MAX").
If your intent is to remove the closerange() call from subprocess, be
aware that it may let through some file descriptors opened by
third-party code (such as C extensions). This may or may not be
something we want to worry about, but there's still a small potential
for security regressions.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
On Mon, 26 Aug 2013 23:45:55 -0400 Scott Dial wrote: > On 8/26/2013 8:51 AM, Antoine Pitrou wrote: > > Le Mon, 26 Aug 2013 08:24:58 -0400, > > Tres Seaver a écrit : > >> On 08/26/2013 04:36 AM, Antoine Pitrou wrote: > >>> event-driven processing using network libraries > >> > >> Maybe I missed something: why should considerations from that topic > >> influence the design of an API for XML processing? > > > > Because this API is mostly useful when the data is received (*) at a > > slow enough speed - which usually means from the network, not from a > > hard drive. > ... > > The whole *point* of adding IncrementalParser was to parse incoming > > XML data in a way that is friendly with event-driven network > > programming, other use cases being *already* covered by existing > > APIs. This is why it's far from nonsensical to re-use an existing > > terminology from that world. > > Since when is Tulip the OOWTDI? If this was Twisted, it would be "write" > and "finish"[1]. Tulip's Protocol ABC isn't even a good match for the > application. There is reason that Twisted has a separate > Consumer/Producer interface from the network I/O interface. > I'm sure > there is other existing practice in this specific area too (e.g., > XMLParser). I'm really not convinced further bikeshedding on this issue has any point. If you have any concrete concerns, you can voice them on the issue tracker. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg.python.org is slow
Le Tue, 27 Aug 2013 10:37:00 +0200, Charles-François Natali a écrit : > 2013/8/27 Antoine Pitrou : > > Sounds a lot like a network problem, then? > > If I'm the only one, it's likely, although these pathological timeouts > are transient, and I don't have any problem with other servers (my > line sustains 8Mb/s without problem). Well, "network problem" doesn't mean the problem is on your side :-) We've had network problems with the hosting in the past, although they were noticed by many people usually (mostly non-North American people). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg.python.org is slow
2013/8/27 Antoine Pitrou : > Sounds a lot like a network problem, then? If I'm the only one, it's likely, although these pathological timeouts are transient, and I don't have any problem with other servers (my line sustains 8Mb/s without problem). > Have you tried a traceroute? I'll try tonight if this persists, and keep you posted. 2013/8/27 Ned Deily : > BTW, do you have ssh compression enabled for that host? Yep. cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg.python.org is slow
Le Tue, 27 Aug 2013 08:00:09 +0200, Charles-François Natali a écrit : > Hi, > > I'm trying to checkout a pristine clone from > ssh://h...@hg.python.org/cpython, and it's taking forever: > """ > 07:45:35.605941 IP 192.168.0.23.43098 > > virt-7yvsjn.psf.osuosl.org.ssh: Flags [.], ack 22081460, win 14225, > options [nop,nop,TS val 368519 ecr 2401783356], length 0 > 07:45:38.558348 IP virt-7yvsjn.psf.osuosl.org.ssh > > 192.168.0.23.43098: Flags [.], seq 22081460:22082908, ack 53985, win > 501, options [nop,nop,TS val 2401784064 ecr 368519], length 1448 > 07:45:38.558404 IP 192.168.0.23.43098 > > virt-7yvsjn.psf.osuosl.org.ssh: Flags [.], ack 22082908, win 14225, > options [nop,nop,TS val 369257 ecr 2401784064], length 0 > 07:45:39.649995 IP virt-7yvsjn.psf.osuosl.org.ssh > > 192.168.0.23.43098: Flags [.], seq 22082908:22084356, ack 53985, win > 501, options [nop,nop,TS val 2401784367 ecr 369257], length 1448 > """ > > See the time to just get an ACK? Sounds a lot like a network problem, then? Have you tried a traceroute? (HTTP cloning works fine here, from a free.fr connection; I'll try a ssh clone tonight if you're still having problems.) cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
Simon Cross, 27.08.2013 09:58: > On Mon, Aug 26, 2013 at 5:57 PM, Antoine Pitrou wrote: >> What do you mean, "all events have to be ready"? >> If you look at the unit tests, the events are generated on-the-fly, >> not at the end of the document. >> (exactly the same as iterparse(), except that iterparse() is blocking) > > So you have to poll .events()? That also seems unhelpful from an event > driven programming perspective. > > What I'm driving at is that I'd expect to have access to some sort of > deferred that fires when the next event is ready to be processed and I > don't see that here. The idea is that you pass data into the parser and then ask read_events() for an event iterator. If/When that's empty, you're done. No repeated polling or anything, just all in one shot whenever data is available. It's a really nice interface by design, just badly integrated into the existing API. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
Le Tue, 27 Aug 2013 09:58:57 +0200, Simon Cross a écrit : > On Mon, Aug 26, 2013 at 5:57 PM, Antoine Pitrou > wrote: > > What do you mean, "all events have to be ready"? > > If you look at the unit tests, the events are generated on-the-fly, > > not at the end of the document. > > (exactly the same as iterparse(), except that iterparse() is > > blocking) > > So you have to poll .events()? That also seems unhelpful from an event > driven programming perspective. At most, you're gonna poll events() after calling data_received() (there's no other way some events can be generated, after all). You can also poll it less often, depending on how often you're interested in new events. That is, it is amenable to both push and pull modes. > What I'm driving at is that I'd expect to have access to some sort of > deferred that fires when the next event is ready to be processed and I > don't see that here. That would be sensible to do if Deferred was a construct shared amongst major async frameworks, but it isn't ;-) (and it looks like Guido won't include a Deferred-alike in PEP 3156) Letting people "poll" events() lets them plug any async-callback-firing primitive they like, almost trivially. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
On Mon, Aug 26, 2013 at 5:57 PM, Antoine Pitrou wrote: > What do you mean, "all events have to be ready"? > If you look at the unit tests, the events are generated on-the-fly, > not at the end of the document. > (exactly the same as iterparse(), except that iterparse() is blocking) So you have to poll .events()? That also seems unhelpful from an event driven programming perspective. What I'm driving at is that I'd expect to have access to some sort of deferred that fires when the next event is ready to be processed and I don't see that here. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg.python.org is slow
In article , Charles-Francois Natali wrote: > I'm trying to checkout a pristine clone from > ssh://h...@hg.python.org/cpython, and it's taking forever: > """ > 07:45:35.605941 IP 192.168.0.23.43098 > > virt-7yvsjn.psf.osuosl.org.ssh: Flags [.], ack 22081460, win 14225, > options [nop,nop,TS val 368519 ecr 2401783356], length 0 > 07:45:38.558348 IP virt-7yvsjn.psf.osuosl.org.ssh > > 192.168.0.23.43098: Flags [.], seq 22081460:22082908, ack 53985, win > 501, options [nop,nop,TS val 2401784064 ecr 368519], length 1448 > 07:45:38.558404 IP 192.168.0.23.43098 > > virt-7yvsjn.psf.osuosl.org.ssh: Flags [.], ack 22082908, win 14225, > options [nop,nop,TS val 369257 ecr 2401784064], length 0 > 07:45:39.649995 IP virt-7yvsjn.psf.osuosl.org.ssh > > 192.168.0.23.43098: Flags [.], seq 22082908:22084356, ack 53985, win > 501, options [nop,nop,TS val 2401784367 ecr 369257], length 1448 > """ > > See the time to just get an ACK? > > Am I the only one experiencing this? At the moment (about 90 minutes after you posted this), I was just did a reasonable-sized pull via ssh: with no apparent delays. But I'm a *lot* closer to the server than you are. BTW, do you have ssh compression enabled for that host? -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
