Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Jan 8, 2013, at 9:14 PM, Guido van Rossum wrote: > But which half? A socket is two independent streams, one in each > direction. Twisted uses half_close() for this concept but unless you > already know what this is for you are left wondering which half. Which > is why I like using 'write' in the name. I should add, if you don't already know what this means you really shouldn't be trying to do it ;-). -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
Hi Yuriy, For the record, it isn't necessary to cross-post. python-ideas is the place for discussing this, and most interested people will be subscribed to both python-ideas and python-dev, and therefore they get duplicate messages. Regards Antoine. Le Wed, 9 Jan 2013 05:14:02 +0400, Yuriy Taraday a écrit : > Hello. > > I've read the PEP and some things raise questions in my > consciousness. Here they are. [snip] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 8:50 AM, Guido van Rossum wrote: > On Tue, Jan 8, 2013 at 8:31 PM, Guido van Rossum wrote: > > On Tue, Jan 8, 2013 at 5:14 PM, Yuriy Taraday > wrote: > >> - pause() and resume() work with reading only, so they should be > suffixed > >> (prefixed) with read(ing), like pause_reading(), resume_reading(). > > > > Agreed. > > I think I want to take that back. I think it is more common for a > protocol to want to pause the transport (i.e. hold back > data_received() calls) than it is for a transport to want to pause the > protocol (i.e. hold back write() calls). So the more common method can > have a shorter name. Also, pause_reading() is almost confusing, since > the protocol's method is named data_received(), not read_data(). Also, > there's no reason for the protocol to want to pause the *write* (send) > actions of the transport -- if wanted to write less it should not have > called write(). The reason to distinguish between the two modes of > pausing is because it is sometimes useful to "stack" multiple > protocols, and then a protocol in the middle of the stack acts as a > transport to the protocol next to it (and vice versa). See the > discussion on this list previously, e.g. > http://mail.python.org/pipermail/python-ideas/2013-January/018522.html > (search for the keyword "stack" in this long message to find the > relevant section). I totally agree with protocol/transport stacking, anyone should be able to do some ugly thing like FTP over SSL over SOCKS over SSL over HTTP (j/k). Just take a look at what you can do with netgraph in *BSD (anything over anything with any number of layers). But still we shouldn't sacrifice ease of understanding (both docs and code) for couple extra chars (10 actually). Yes, 'reading' is misleading, pause_receiving and resume_receiving are better. -- Kind regards, Yuriy. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 10:02 AM, Guido van Rossum wrote: > Changing event loops in the middle of event processing is not a common > (or even useful) pattern. You start the event loop and then leave it > alone. > Yes. It was not-so-great morning idea. > Yes, 'write' part is good, I should mention it. I meant to say that I > won't > > need to explain that there were days when we had to handle a special > marker > > at the end of file. > > But even today you have to mark the end somehow, to distinguish it > from "not done yet, more could be coming". The equivalent is typing ^D > into a UNIX terminal (or ^Z on Windows). My interns told me that they remember EOF as special object only from high school when they had to study Pascal. I guess, in 5 years students won't understand how one can write an EOF. (and schools will finally replace Pascal with Python) -- Kind regards, Yuriy. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 1:39 PM, Antoine Pitrou wrote: > > Hi Yuriy, > > For the record, it isn't necessary to cross-post. python-ideas is > the place for discussing this, and most interested people will be > subscribed to both python-ideas and python-dev, and therefore they get > duplicate messages. > Oh, sorry. I just found this thread in both MLs, so decided to send it to both. This will be my last email (for now) on this topic at python-dev. -- Kind regards, Yuriy. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 8:55 PM, Yuriy Taraday wrote: > My interns told me that they remember EOF as special object only from high > school when they had to study Pascal. I guess, in 5 years students won't > understand how one can write an EOF. (and schools will finally replace > Pascal with Python) Python really doesn't try to avoid the concept of an End-of-file marker. $ python3 Python 3.2.3 (default, Jun 8 2012, 05:36:09) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> quit Use quit() or Ctrl-D (i.e. EOF) to exit >>> import io >>> print(io.FileIO.read.__doc__) read(size: int) -> bytes. read at most size bytes, returned as bytes. Only makes one system call, so less data may be returned than requested In non-blocking mode, returns None if no data is available. On end-of-file, returns ''. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Set close-on-exec flag by default in SocketServer
Hi, The SocketServer class creates a socket to listen on clients, and a new socket per client (only for stream server like TCPServer, not for UDPServer). Until recently (2011-05-24, issue #5715), the listening socket was not closed after fork for the ForkingMixIn flavor. This caused two issues: it's a security leak, and it causes "address already in use" error if the server is restarted (see the first message of #12107 for an example with Django). Our implementation of the XMLRPC server uses fcntl() + FD_CLOEXEC to fix this issue since long time ago (2005-12-04, #1222790). IMO it would be safer to always enable close-on-exec flag on all sockets (listening+clients) *by default*. According to Charles-François Natali, it will break applications. Ruby made a similar brave choice one year ago (2011-10-22), not only on sockets but on all file descriptors: http://bugs.ruby-lang.org/issues/5041 Developers of Ruby servers relying on inheriting sockets in child processes have to explicitly disable close-on-exec flag in their server. Unicorn has been fixed for example. My question is: would you accept to break backward compatibility (in Python 3.4) to fix a potential security vulnerability? If not, an alternative is to add an option, disabled by default, to enable (or disable) explicitly close-on-exec in Python 3.4, and wait for 3.5 to enable the option by default. So applications might disable the flag explicitly in Python 3.4. I wrote a patch attached to the issue #12107 which adds a flag to enable or disable close-on-exec, I chose to enable the flag by default: http://bugs.python.org/issue12107 -- I'm not sure that close-on-exec flag must be set on the listening socket *and* on the client sockets. What do you think? -- It would be nice to be able to set close-on-exec flag directly using socket.socket() constructor. For example, an optional cloexec keyword-only argument can be added to socket.socket(family, type, *, cloexec=False). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Set close-on-exec flag by default in SocketServer
On Wed, Jan 9, 2013 at 4:48 AM, Victor Stinner wrote: > My question is: would you accept to break backward compatibility (in > Python 3.4) to fix a potential security vulnerability? > > If not, an alternative is to add an option, disabled by default, to > enable (or disable) explicitly close-on-exec in Python 3.4, and wait > for 3.5 to enable the option by default. So applications might disable > the flag explicitly in Python 3.4. If the end goal is indeed going to close-on-exec ON by default, then I think having it 3.4 itself is a good idea. OFF for one release just gives the framework developers who use SocketServer some additional time. Usually, I have realized that framework devs try our release candidates and see if they see any potential changes to be done. If they realize this change in their testing, it would be good for both parties. So, my vote. +1 for making that in 3.4 Thank you, Senthil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Arena terminology (PyArena vs obmalloc.c:"arena")
There's no correlation between PyArenas and the extensive use of the term "arena" in obmalloc.c, right? I initially assumed there was, based solely on the common use of the term arena. However, after more investigation, it *appears* as though there's absolutely no correlation. Just wanted to make 100% sure. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Arena terminology (PyArena vs obmalloc.c:"arena")
2013/1/9 Trent Nelson : > There's no correlation between PyArenas and the extensive use of the > term "arena" in obmalloc.c, right? Correct. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Set close-on-exec flag by default in SocketServer
> My question is: would you accept to break backward compatibility (in
> Python 3.4) to fix a potential security vulnerability?
Although obvious, the security implications are not restricted to
sockets (yes, it's a contrived example):
"""
# cat test_inherit.py
import fcntl
import os
import pwd
import sys
f = open("/tmp/passwd", 'w+')
#fcntl.fcntl(f.fileno(), fcntl.F_SETFD, fcntl.FD_CLOEXEC)
if os.fork() == 0:
os.setuid(pwd.getpwnam('nobody').pw_uid)
os.execv(sys.executable, ['python', '-c', 'import os; os.write(3,
"owned")'])
else:
os.waitpid(-1, 0)
f.seek(0)
print(f.read())
f.close()
# python test_inherit.py
owned
"""
> I'm not sure that close-on-exec flag must be set on the listening
> socket *and* on the client sockets. What do you think?
In the listening socket is inherited, it can lead to EADDRINUSE, or
the child process "hijacking" new connections (by accept()ing on the
same socket).
As for the client sockets, there's at least one reason to set them
close-on-exec: if a second forked process inherits the first process'
client socket, even when the first client closes its file descriptor
(and exits), the socket won't be closed until the the second process
exits too: so one long-running child process can delay other child
processes connection shutdown for arbitrarily long.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Set close-on-exec flag by default in SocketServer
2013/1/9 Charles-François Natali :
>> My question is: would you accept to break backward compatibility (in
>> Python 3.4) to fix a potential security vulnerability?
>
> Although obvious, the security implications are not restricted to
> sockets (yes, it's a contrived example):
> ...
> f = open("/tmp/passwd", 'w+')
> if os.fork() == 0:
> ...
Do you mean that we should provide a way to enable close-on-exec flag
by default on all file descriptors? Adding an "e" flag to open() mode
is not compatible with this requirement: we would need the opposite
flag to disable explicitly close-on-exec.
A better API is maybe to add a "cloexec" argument to open(),
socket.socket(), pipe(), etc. It's easier to choose the default value
of this argument:
* False: backward compatible
* True: backward incompatible
* value retrieved from a global option, ex: sys.getdefaultcloexec()
So a simple call to "sys.setdefaultcloexec(True)" would make your
program make secure and less error-prone, globally, without having to
break backward compatibility.
Example without setting cloexec globally:
f = open("/tmp/passwd", "w+", cloexec=True)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, cloexec=True)
rfd, wfd = os.pipe(cloexec=True)
--
"os.pipe(cloexec=True)" doesn't respect 1-to-1 "rule" of OS functions
exposed in the Python module "os", because it it may use pipe2() with
O_CLOEXEC internally, whereas os.pipe2() does exist...
... But other Python functions of the os module use a different C
function depending on the option. Examples:
- os.open() uses open(), or openat() if dir_fd is set.
- os.utime() uses utimensat(), utime(), utimes(), futimens(),
futimes(), futimesat(), ...
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Set close-on-exec flag by default in SocketServer
2013/1/10 Victor Stinner : > A better API is maybe to add a "cloexec" argument to open(), ... I realized that setting close-on-exec flag is a non trivial problem. There are many ways to set it depending on the function, on the OS, and on the OS version. There is also the difficult question of the "default value". I started to work on a PEP: https://bitbucket.org/haypo/misc/src/tip/python/pep_cloexec.rst Contact me if you would like to contribute. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
