Re: [Python-Dev] Seeming unintended difference between list comprehensions and generator expressions...
Nick Coghlan wrote: Josiah Carlson wrote: Similarly, a 3.x list comprehension [i*i for i in x] is very roughly translated as: def _lc(arg): result = [] for i in arg: result.append(i*i) return result expr_value = _lc(x) I was under the impression that in 3.x, it was equivalent to list(genexp) . Which would explain the difference between 2.6 and 3.0. Semantically, the two translations end up being the same. Actually, they're not quite the same. list(genexp) will swallow any StopIteration exceptions that end up getting thrown inside the generator, but even in Python 3, list comprehensions will not catch StopIteration. So for explanatory purposes, thinking of a list comprehension as a function that returns a list is more useful than thinking of it as list(genexp). -- Carl ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Attention Bazaar mirror users
Steve Holden wrote: Steven Bethard wrote: On Sat, Feb 21, 2009 at 1:11 PM, Paul Moore p.f.mo...@gmail.com wrote: PS Just for my own information, am I correct in thinking that it is *only* Bazaar in the (D)VCS world that has this problem, to any real extent? I know old Mercurial clients can interact with newer servers (ie, the wire protocol hasn't changed), I'm fairly sure that older Subversion clients can talk to newer servers (at least, I've never cared what client version I'm running). I've definitely had an SVN server tell me that I needed to upgrade my client to 1.5. Interestingly, I've had 1.4 servers tell me I shouldn't have upgraded my client to 1.5, which is a little tedious = particularly as the information comes in the form of a hard-to-decipher error message and a refusal to work. So much for backward compatibility. Hmm... I've been using 1.5 clients with 1.4 and earlier servers for quite a while now. My guess is that you deciphered the error message wrong! A working copy created by a 1.5 client can only be manipulated by a 1.5 client (I sometimes have several clients on windows boxen) but work fine with earlier servers in my experience. Michael Fortunately I still had the SVN 1.4 client on the Ubuntu machine that was hosting the Samba shares, so I could use ssh and the command line to maintain things. regards Steve -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Attention Bazaar mirror users
Michael Foord wrote: Steve Holden wrote: Steven Bethard wrote: On Sat, Feb 21, 2009 at 1:11 PM, Paul Moore p.f.mo...@gmail.com wrote: PS Just for my own information, am I correct in thinking that it is *only* Bazaar in the (D)VCS world that has this problem, to any real extent? I know old Mercurial clients can interact with newer servers (ie, the wire protocol hasn't changed), I'm fairly sure that older Subversion clients can talk to newer servers (at least, I've never cared what client version I'm running). I've definitely had an SVN server tell me that I needed to upgrade my client to 1.5. Interestingly, I've had 1.4 servers tell me I shouldn't have upgraded my client to 1.5, which is a little tedious = particularly as the information comes in the form of a hard-to-decipher error message and a refusal to work. So much for backward compatibility. Hmm... I've been using 1.5 clients with 1.4 and earlier servers for quite a while now. My guess is that you deciphered the error message wrong! A working copy created by a 1.5 client can only be manipulated by a 1.5 client (I sometimes have several clients on windows boxen) but work fine with earlier servers in my experience. You are correct. In fact I got over the issue by using the 1.4 client from Cygwin. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Reviving restricted mode?
I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Sun, Feb 22, 2009 at 10:45 AM, Guido van Rossum gu...@python.org wrote: I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) We have to remember that it's quite trivial to segfault the interpreter with pure Python. (See Lib/test/crashers) Even if this patch manages to plug all the holes in the current Python, do we really want to commit our selves to maintaining it through language evolution which will surely introduce new subtle ways to circumvent the guard? IMO, the only fairly close to fool proof method of running restricted python is through something like the PyPy sandbox where all os level calls have to be checked by the master process. I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? It was broken by the introduction of new-style classes: http://mail.python.org/pipermail/python-dev/2002-December/031160.html -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
On Feb 21, 2009, at 2:17 PM, Brett Cannon wrote: The other approach is having pickle contain code known not to be overridden by anyone, import _pypickle for stuff that may be overridden, and then import _pickle for whatever is available. This approach has the perk of using a standard practice for how to pull in different implementation. But the drawback, thanks to how globals are bound, is that any code pulled in from _pickle/_pypickle will not be able to call into other optimized code; it's a take or leave it once the call chain enters one of those modules as they will always call the implementations in the module they originate from. fwiw, I believe this is the approach that I've been using when I'm faced with the need to optimize code, but still want to retain a pure-Python version. Thankfully, I haven't had a need for implementation intersections(well, it almost works. I think. ;) for access to common module globals as the optimizations tend to be simple transformations or isolated classes. However, if I were faced with the problem of needing some common global data/functionality, I'd probably put it in yet-another-module and just import it explicitly in each implementation. Sure, it seems like it might be annoying, but so is maintaining multiple implementations. ;) Specifically: pbuffer.py - The python implementation buffer.c - cbuffer.so - The c implementation buffer.py - The abstraction module, trying to import the contents of the fast one first. And in my unittest: if c_buffer_module is not None: class c_buffer(buffer_test, unittest.TestCase): bufferclass = c_buffer_module.pq_message_stream class p_buffer(buffer_test, unittest.TestCase): bufferclass = p_buffer_module.pq_message_stream Of course, buffer_test is not invoked because it's not a TestCase. However, Aahz is probably right about this thread belonging elsewhere.? Hrm, day old, maybe it's been moved already.. sigh. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
Guido van Rossum wrote: I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? I don't have much time these days, for sure not until pycon us, to look at the proposed code. A remark though, in many cases people want to use restricted execution to allow trusted and untrusted code to live side by side. In such a scenario the issue is not only how to prevent for example the untrusted code to get hold of open() but also how to deal with the security issues at the boundary when untrusted code calls into trusted code. AFAIK E provides and incorporate a lot of thinking around the concept of guards, a sort of generalized runtime type checking with very strong guarantees, in a language like Java instead type checking and final classes can be used to provide the required safety. Python instead over the years has grown more hooks for object to cheat about their types. In general safely programming such trusted code in python will be delicate and slightly cumbersome. Think for example of thinhs that can be done by objects redefining equality wrt data structures that old sensitive information unless such arguments are blocked. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
On Fri, Feb 20, 2009 at 1:45 PM, Brett Cannon br...@python.org wrote: But there is another issue with this: the pure Python code will never call the extension code because the globals will be bound to _pypickle and not _pickle. So if you have something like:: # _pypickle def A(): return _B() def _B(): return -13 # _pickle def _B(): return 42 # pickle from _pypickle import * try: from _pickle import * except ImportError: pass If you import pickle and call pickle.A() you will get -13 which is not what you are after. Maybe I've missed this and someone else already suggested it, but couldn't we provide a (probably C-coded) function ``replace_globals(module, globals)`` that would, say, replace the globals in _pypickle with the globals in pickle? Then you could write something like:: from _pypickle import * try: from _pickle import * module = __import__('_pickle') except ImportError: module = __import__('_pypickle') replace_globals(module, globals()) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
Steven Bethard wrote: On Fri, Feb 20, 2009 at 1:45 PM, Brett Cannon br...@python.org wrote: But there is another issue with this: the pure Python code will never call the extension code because the globals will be bound to _pypickle and not _pickle. So if you have something like:: # _pypickle def A(): return _B() def _B(): return -13 # _pickle def _B(): return 42 # pickle from _pypickle import * try: from _pickle import * except ImportError: pass If you import pickle and call pickle.A() you will get -13 which is not what you are after. Maybe I've missed this and someone else already suggested it, but couldn't we provide a (probably C-coded) function ``replace_globals(module, globals)`` that would, say, replace the globals in _pypickle with the globals in pickle? Then you could write something like:: from _pypickle import * try: from _pickle import * module = __import__('_pickle') except ImportError: module = __import__('_pypickle') replace_globals(module, globals()) Steve Swapping out module globals seems to be a backwards way to do things to me. Why not have one set of tests that test the low level APIs (identical tests whether they are written in C or Python) - and as they live in their own module are easy to test in isolation. And then a separate set of tests for the higher level APIs, which can even mock out the low level APIs they use. There shouldn't be any need for switching out objects in the scope of the lower level APIs - that seems like a design smell to me. Michael -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Silencing IO errors on del/dealloc?
Hello, The Python version of IO lib has taken the stance that all errors happening in the destructor of an IO object are silenced. Here is the relevant code in IOBase: def __del__(self) - None: Destructor. Calls close(). # The try/except block is in case this is called at program # exit time, when it's possible that globals have already been # deleted, and then the close() call might fail. Since # there's nothing we can do about such failures and they annoy # the end users, we suppress the traceback. try: self.close() except: pass However, this seems very unreliable and dangerous to me. Users will not be warned if their IO objects are not closed properly (which might be due to programming errors), and data possibly gets corrupted. I would like to change this behaviour so that the try/except enclosure above is removed, letting __del__ at least print those (unraisable) errors on the console. Of course I also advocate doing so in the C version of the IO lib. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Sun, Feb 22, 2009 at 9:54 AM, Benjamin Peterson benja...@python.org wrote: On Sun, Feb 22, 2009 at 10:45 AM, Guido van Rossum gu...@python.org wrote: I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) We have to remember that it's quite trivial to segfault the interpreter with pure Python. (See Lib/test/crashers) I know it well, but for this particular use case that doesn't matter. Tav is interested in using this on app engine, which doesn't care about segfaults -- the process is simply restarted, nobody gains access to information they shouldn't have. App engine does care about overwriting memory, but none of the crashers get that far in the configuration on app engine. Even if this patch manages to plug all the holes in the current Python, do we really want to commit our selves to maintaining it through language evolution which will surely introduce new subtle ways to circumvent the guard? You'd have to talk to Tav about this (I've CC'ed him on this message). He seems quite convinced that his current approach is secure, all he wants is a way to do the equivalent in app engine (where ctypes is not supported and never will be). IMO, the only fairly close to fool proof method of running restricted python is through something like the PyPy sandbox where all os level calls have to be checked by the master process. Trust me, app engine has something similar -- Tav's desire is just to provide an added barrier between his app and code that his app will let users run within a context it provides. He's relying (forced to rely :-) on app engine's sandbox for protection against illegal syscalls. I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? It was broken by the introduction of new-style classes: http://mail.python.org/pipermail/python-dev/2002-December/031160.html I think there were other threads where Samule showed some quite devious ways to access the os module. I don't see any posts by Samuele in that thread (and he just replied that he's too busy). The attack demonstrated at the top of that thread isn't particularly interesting -- it just shows that class-based security doesn't work so well, but that doesn't mean some other approach based more on functions couldn't work. For Tav's benefit, I think it would be good to at least add IsRestricted checks to __subclasses__(), go_code and gi_frame -- that's a trivial patch and if he believes it's enough he can create a sandbox on app engine and invite people to try to break out of it... If someone succeeds, the damage is limited by app engine's own perimeter defenses (which haven't been broken through since it was released 9 months ago). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
On Sun, Feb 22, 2009 at 11:02 AM, Antoine Pitrou solip...@pitrou.net wrote: The Python version of IO lib has taken the stance that all errors happening in the destructor of an IO object are silenced. Here is the relevant code in IOBase: def __del__(self) - None: Destructor. Calls close(). # The try/except block is in case this is called at program # exit time, when it's possible that globals have already been # deleted, and then the close() call might fail. Since # there's nothing we can do about such failures and they annoy # the end users, we suppress the traceback. try: self.close() except: pass However, this seems very unreliable and dangerous to me. Users will not be warned if their IO objects are not closed properly (which might be due to programming errors), and data possibly gets corrupted. OTOH the is a much larger category of false positives, where a close() call raise an exception for some spurious reason (see the quoted comment) but no harm is done; in the past the tracebacks printed for __del__ by default have caused nothing but confuse users (who think something is seriously wrong but have no way to debug it). The svn history of those lines may have more pointers. I would like to change this behaviour so that the try/except enclosure above is removed, letting __del__ at least print those (unraisable) errors on the console. Of course I also advocate doing so in the C version of the IO lib. I'd be very wary of this change. That code has enough comments to make me believe that I put it in for a very good reason... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
Antoine Pitrou wrote: I would like to change this behaviour so that the try/except enclosure above is removed, letting __del__ at least print those (unraisable) errors on the console. Of course I also advocate doing so in the C version of the IO lib. I would hope that the C version could at least ensure that buffers are flushed properly without having to rely on any global variables, so that this would not be so much of an issue. When we were using stdio, we could rely on the C runtime to flush buffers unless something very drastic has gone wrong. Bypassing stdio means we are taking that responsibility on ourselves. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
Guido van Rossum wrote: Tav is interested in using this on app engine, which doesn't care about segfaults -- the process is simply restarted, nobody gains access to information they shouldn't have. App engine does care about overwriting memory, That doesn't make sense -- how can something not care about segfaults, but care about memory overwriting? If something is capable of causing a segfault, you can't be sure it won't just corrupt memory in some way that doesn't segfault but causes some other problem. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
Greg Ewing greg.ewing at canterbury.ac.nz writes: I would hope that the C version could at least ensure that buffers are flushed properly without having to rely on any global variables, so that this would not be so much of an issue. The C version cleans up after itself just fine :) The issue is when someone defines a Python class derived from IOBase or one of its descendants (RawIOBase, BufferedIOBase, TextIOBase), and overrides the close() method. (see e.g. SocketIO in socket.py, which incidentally defined its own __del__ while it could perhaps rely on RawIOBase to do the right thing) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Sun, Feb 22, 2009 at 12:10 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Tav is interested in using this on app engine, which doesn't care about segfaults -- the process is simply restarted, nobody gains access to information they shouldn't have. App engine does care about overwriting memory, That doesn't make sense -- how can something not care about segfaults, but care about memory overwriting? If something is capable of causing a segfault, you can't be sure it won't just corrupt memory in some way that doesn't segfault but causes some other problem. To be more precise, we don't care about crashes caused by NULL pointer dereferencing. Most of the demonstrated crashers work by causing a NULL pointer dereference. Since that crashes immediately, there is no possibility for a further exploit. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
Hey guys, benjamin Even if this patch manages to plug all the holes in the benjamin current Python, do we really want to commit our benjamin selves to maintaining it through language evolution benjamin which will surely introduce new subtle ways to benjamin circumvent the guard? If it would be helpful, I am happy to maintain this as Python evolves. I've already been maintaining the PJE-inspired ctypes-based approach and monkeypatches for various Python versions for a while now. See secure.py, secure25.py, secure26.py and secure30.py in: http://github.com/tav/plexnet/tree/9dabc570a2499689e773d1af3599a29102071f80/source/plexnet/util Also, my plans for world domination depend on a secure Python, so I have the necessary incentives ;p sameule I don't have much time these days, for sure not samuele until pycon us, to look at the proposed code. Thanks in advance if/when you get the time for this Samuele! samuele E provides and incorporate a lot of thinking samuele around [snip] The functions based approach I am taking is very much taken from E and inspired by an insight that Ka-Ping Yee had on Python-Dev years ago. See http://www.erights.org/elib/capability/ode/index.html for a direct parallel to the approach I've taken... guido For Tav's benefit, I think it would be good to at guido least add IsRestricted checks to guido __subclasses__(), gi_code and gi_frame -- guido that's a trivial patch and if he believes it's guido enough he can create a sandbox on app engine guido and invite people to try to break out of it... If guido someone succeeds If someone succeeds... ...My missus might end up leaving me on account of so much crying ;p Seriously though, it's a relatively risk-free approach. The only person who stands to lose out is me if I'm wrong =) In the worst case scenario, this approach would help identify other leak attributes/methods -- which I'm hoping won't be found. And, in an ideal scenario, we'd have the basis for secure Python interpreter/programming... which, together with PyPy's sandboxed interpreter, would seriously rock! -- enthusiastically, tav plex:espians/tav | t...@espians.com | +44 (0) 7809 569 369 http://tav.espians.com | http://twitter.com/tav | skype:tavespian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
On Sun, Feb 22, 2009 at 11:32 AM, Antoine Pitrou solip...@pitrou.net wrote: Guido van Rossum guido at python.org writes: OTOH the is a much larger category of false positives, where a close() call raise an exception for some spurious reason (see the quoted comment) but no harm is done; in the past the tracebacks printed for __del__ by default have caused nothing but confuse users (who think something is seriously wrong but have no way to debug it). The svn history of those lines may have more pointers. Well this code dates back to the first checkin in the py3k branch. Apparently the old p3yk branch is not there anymore... OK, then we'll just have to understand the big comment in the code. I understand the missing globals on shutdown problem, but the error may also be a legitimate programming error, where a close() implementation fails for whatever reason. At least displaying the error may encourage the programmer to strengthen his implementation. How about the following compromise: try: closed = self.closed except: # Object is in an unusable state, don't attempt anything pass else: if not closed: self.close() No. Trust me. It is not always possible to strengthen the implementation. (At least not until we get rid of the replace all globals with None upon module deletion rule.) Your legitimate programming error is purely hypothetical, while the unactionable, confusing traceback caused by missing globals is real. If people want an exception from close() they can call it explicitly or use a context manager. When a file is closed implicitly by __del__, there is a chance of an impossible-to-avoid spurious traceback. I have dealt with such tracebacks in real life and there is nothing that can be done about them except ignore them. If you don't ignore them, users living 1000s of miles away and years later than the original programmer will be upset by spurious tracebacks that worry them to death. If you want to ensure buffers are flushed, why not call self.flush() for writable files outside the try/except? If flush() fails it *is* a real problem. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracker archeology
[Sorry for stepping in so late] On Thu, Feb 12 2009 at 02:05:23PM BRST, Daniel (ajax) Diniz aja...@gmail.com wrote: Victor Stinner wrote: Oh, I realized that there is a component called Unicode. So it should be possible to write a request to list all issues related to unicode. Nice, I'll add set this component for issues that don't have it. I can still add people to these issues, if they want. I think this one might be considered a Unicode issue: http://bugs.python.org/issue1293741 doctest runner cannot handle non-ascii characters rbp ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
On Sat, Feb 21, 2009 at 20:12, Aahz a...@pythoncraft.com wrote: On Sat, Feb 21, 2009, Brett Cannon wrote: On Sat, Feb 21, 2009 at 15:46, Aahz a...@pythoncraft.com wrote: On Sat, Feb 21, 2009, Brett Cannon wrote: I am seeing two approaches emerging. One is where pickle contains all Python code and then uses something like use_extension to make sure the original Python objects are still reachable at some point. This has the drawback that you have to use some function to make the extensions happen and there is some extra object storage. The other approach is having pickle contain code known not to be overridden by anyone, import _pypickle for stuff that may be overridden, and then import _pickle for whatever is available. This approach has the perk of using a standard practice for how to pull in different implementation. But the drawback, thanks to how globals are bound, is that any code pulled in from _pickle/_pypickle will not be able to call into other optimized code; it's a take or leave it once the call chain enters one of those modules as they will always call the implementations in the module they originate from. To what extent do we care about being able to select Python-only on a per-module basis, particularly in the face of threaded imports? That is, we could have a sys.python_only attribute that gets checked on import. That's simple and direct, and even allows per-module switching if the application really cares and import doesn't need to worry about threads. Alternatively, sys.python_only could be a set, but that gets ugly about setting from the application. (The module checks to see whether it's listed in sys.python_only.) Maybe we should move this discussion to python-ideas for now to kick around really oddball suggestions? This is all about testing. If a change is made to some extension code it should be mirrored in the Python code and vice-versa. Okay, I don't see how that is a response to my suggestion -- I can imagine that someone might want to test a combination of pure-Python and binary libraries. I don't want to move it because this isn't some idea for a new feature that may or may not be useful; this isn't an idea, it's needed. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
I've already been maintaining the PJE-inspired ctypes-based approach and monkeypatches for various Python versions for a while now. See secure.py, secure25.py, secure26.py and secure30.py in: http://github.com/tav/plexnet/tree/9dabc570a2499689e773d1af3599a29102071f80/source/plexnet/util What is the objective of this code? Is it a complete sandbox? If not, is a complete sandbox based on it available somehow for review? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
On Sun, Feb 22, 2009 at 10:29, Michael Foord fuzzy...@voidspace.org.ukwrote: Steven Bethard wrote: On Fri, Feb 20, 2009 at 1:45 PM, Brett Cannon br...@python.org wrote: But there is another issue with this: the pure Python code will never call the extension code because the globals will be bound to _pypickle and not _pickle. So if you have something like:: # _pypickle def A(): return _B() def _B(): return -13 # _pickle def _B(): return 42 # pickle from _pypickle import * try: from _pickle import * except ImportError: pass If you import pickle and call pickle.A() you will get -13 which is not what you are after. Maybe I've missed this and someone else already suggested it, but couldn't we provide a (probably C-coded) function ``replace_globals(module, globals)`` that would, say, replace the globals in _pypickle with the globals in pickle? Then you could write something like:: from _pypickle import * try: from _pickle import * module = __import__('_pickle') except ImportError: module = __import__('_pypickle') replace_globals(module, globals()) Steve Swapping out module globals seems to be a backwards way to do things to me. I agree; I would rather muck with sys.modules at that point. Why not have one set of tests that test the low level APIs (identical tests whether they are written in C or Python) - and as they live in their own module are easy to test in isolation. And then a separate set of tests for the higher level APIs, which can even mock out the low level APIs they use. There shouldn't be any need for switching out objects in the scope of the lower level APIs - that seems like a design smell to me. That's possible. As I have said, my only worry with the separate py/extension module approach is that any time someone wants to do an extension version of something the Python code will need to be moved. But at this point I am honestly burning out on this topic (got a lot on my plate right now) so I am going to let somebody else lead this to the finish line. =) -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
Guido van Rossum guido at python.org writes: If you want to ensure buffers are flushed, why not call self.flush() for writable files outside the try/except? If flush() fails it *is* a real problem. Well, my concern is less about flushing writable files (they will be implemented in C and the flushing will probably be ok) than Python-written file-like types, like SocketIO or any third-party class. Anyway, let's keep silencing exceptions and we'll see if anyone complains about it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Silencing IO errors on del/dealloc?
Antoine Pitrou wrote: Guido van Rossum guido at python.org writes: The svn history of those lines may have more pointers. Well this code dates back to the first checkin in the py3k branch. Apparently the old p3yk branch is not there anymore... The history is available (see below), but tells nothing that would be useful/relevant. Daniel History of io.py is available on ViewVC: http://svn.python.org/view/python/branches/p3yk/Lib/io.py?view=logpathrev=56853 It's possible to checkout that as a peg revision: svn co -r54910 http://svn.python.org/projects/python/branches/p3yk/Lib/@r54910 Then, svn blame tells where it comes from: 54728 guido.van.rossum def __del__(self) - None: 54728 guido.van.rossum Destructor. Calls close(). 54728 guido.van.rossum # The try/except block is in case this is called at program 54728 guido.van.rossum # exit time, when it's possible that globals have already been 54728 guido.van.rossum # deleted, and then the close() call might fail. Since 54728 guido.van.rossum # there's nothing we can do about such failures and they annoy 54728 guido.van.rossum # the end users, we suppress the traceback. 54728 guido.van.rossum try: 54728 guido.van.rossum self.close() 54728 guido.van.rossum except: 54728 guido.van.rossum pass And here's the log for that rev: http://svn.python.org/view?view=revrevision=54728 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Feb 22, 2009, at 5:15 PM, Martin v. Löwis wrote: What is the objective of this code? Is it a complete sandbox? If not, is a complete sandbox based on it available somehow for review? From a cursory look at Tav's CPython patch, I'd describe it as follows: Requires: an existing Python code execution environment which does not expose unsafe interpreter functionality to the (untrusted) code it runs, Provides: a way for the trusted code outside of that restricted environment to wrap functions (which may reference trusted objects) in closures which can be passed into the restricted environment as completely opaque objects whose internal state (including any referenced trusted objects) cannot be accessed from the untrusted code. When I say 'requires', I mean 'requires to be useful'; the patch can be applied to vanilla CPython, but isn't useful on its own. Building blocks for a restricted execution environment as required by the patch are commonly provided in templating libraries; Tav mentions Genshi in particular. By default, Genshi templates don't come with security built in -- you can do what you please in a template. However, since Genshi manually constructs a Python AST from Python code in the template, one can restrict the AST and modify the builtins exposed to the template environment, making things like filesystem access, import and other sensitive system facilities unavailable. Even with those unavailable, untrusted code can break out of containment by accessing object.__subclasses__ or gaining access to gi_frame and gi_code. This means you can't, for instance, pass into the untrusted code a closure which operates on trusted objects. Tav's patch addresses this particular problem. To give you a complete sandbox to play with, I just violently ripped out the relevant code from Genshi, added some glue, and slapped it all together in a single file along with Tav's pure-Python code which is functionally equivalent to the CPython patch he submitted. The result, tested on 2.5.1 and nothing else: http://radian.org/~krstic/sandbox.py As is, sandbox allows you to execute untrusted Python code which won't have access to import, __import__, eval, exec, execfile, open, and to which you can pass closures which reference trusted objects which the untrusted code can't get at. Example: import sandbox import sys import md5 def trusted_pwd_closure(password): ...def get_pwd(): ...return md5.md5(password).hexdigest() ...return get_pwd ... context = dict(pwd=trusted_pwd_closure('secret')) sandbox.run_string(print pwd(), context) sandbox.run_string(print pwd.func_closure[0].cell_contents, context) [snip] AttributeError: 'function' object has no attribute 'func_closure' Without Tav's patch, the untrusted code can extract our password ('secret') from the closure with that last statement. To run whole files in the sandbox: sandbox.run_file('/some/path/file.py') Finally, disclaimer: I put this together in *15 minutes*. I'm sure I missed something, this isn't secure, etc. It's just a proof of concept. Void where prohibited, etc. Cheers, -- Ivan Krstić krs...@solarsail.hcs.harvard.edu | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] .pythonrc.py in man page
The python man page, dated 2005 even in 3.1, has this curious entry: ~/.pythonrc.py User-specific initialization file loaded by the user module; not used by default or by most applications. 1. I couldn't figure out what the user module is. 2. I couldn't figure out what not used by default or by most applications meant: what would cause it to get loaded when python starts up? how would an application load it even if the user's environment didn't cause it to get loaded? 3. Why would this file exist if the environment variable PYTHONSTARTUP can specify a file to be loaded at startup? Perhaps this entry in the man page is obsolete and should be removed? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .pythonrc.py in man page
On Sun, Feb 22, 2009 at 5:58 PM, Mitchell L Model mlmli...@comcast.net wrote: The python man page, dated 2005 even in 3.1, has this curious entry: ~/.pythonrc.py User-specific initialization file loaded by the user module; not used by default or by most applications. 1. I couldn't figure out what the user module is. user.py. 2. I couldn't figure out what not used by default or by most applications meant: what would cause it to get loaded when python starts up? how would an application load it even if the user's environment didn't cause it to get loaded? The docstring in user.py explains this. 3. Why would this file exist if the environment variable PYTHONSTARTUP can specify a file to be loaded at startup? PYTHONSTARTUP is only used in interactive sessions. Perhaps this entry in the man page is obsolete and should be removed? Not at all. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .pythonrc.py in man page
On Sun, Feb 22, 2009 at 8:25 PM, Guido van Rossum gu...@python.org wrote: Perhaps this entry in the man page is obsolete and should be removed? Not at all. For what it's worth, the 2.6.1 documentation states: Deprecated since version 2.6: The user module has been removed in Python 3.0. If user.py has indeed been removed, then Mitchell is correct. http://docs.python.org/library/user.html -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
I'm not familiar with Genshi -- what is the purpose of the AST transformation here? Traditionally, sandboxing environments in Python usually just exec the untrusted code in an environment with a __builtins__ dict that limits the built-in functions and overrides __import__ so as to provide an import implementation that allows import of pure-Python modules but not extensions (or only allows certain extensions that have been previously audited, or provides wrappers or subsets of selected others). --Guido On Sun, Feb 22, 2009 at 6:09 PM, Ivan Krstić krs...@solarsail.hcs.harvard.edu wrote: On Feb 22, 2009, at 5:15 PM, Martin v. Löwis wrote: What is the objective of this code? Is it a complete sandbox? If not, is a complete sandbox based on it available somehow for review? From a cursory look at Tav's CPython patch, I'd describe it as follows: Requires: an existing Python code execution environment which does not expose unsafe interpreter functionality to the (untrusted) code it runs, Provides: a way for the trusted code outside of that restricted environment to wrap functions (which may reference trusted objects) in closures which can be passed into the restricted environment as completely opaque objects whose internal state (including any referenced trusted objects) cannot be accessed from the untrusted code. When I say 'requires', I mean 'requires to be useful'; the patch can be applied to vanilla CPython, but isn't useful on its own. Building blocks for a restricted execution environment as required by the patch are commonly provided in templating libraries; Tav mentions Genshi in particular. By default, Genshi templates don't come with security built in -- you can do what you please in a template. However, since Genshi manually constructs a Python AST from Python code in the template, one can restrict the AST and modify the builtins exposed to the template environment, making things like filesystem access, import and other sensitive system facilities unavailable. Even with those unavailable, untrusted code can break out of containment by accessing object.__subclasses__ or gaining access to gi_frame and gi_code. This means you can't, for instance, pass into the untrusted code a closure which operates on trusted objects. Tav's patch addresses this particular problem. To give you a complete sandbox to play with, I just violently ripped out the relevant code from Genshi, added some glue, and slapped it all together in a single file along with Tav's pure-Python code which is functionally equivalent to the CPython patch he submitted. The result, tested on 2.5.1 and nothing else: http://radian.org/~krstic/sandbox.py As is, sandbox allows you to execute untrusted Python code which won't have access to import, __import__, eval, exec, execfile, open, and to which you can pass closures which reference trusted objects which the untrusted code can't get at. Example: import sandbox import sys import md5 def trusted_pwd_closure(password): ...def get_pwd(): ...return md5.md5(password).hexdigest() ...return get_pwd ... context = dict(pwd=trusted_pwd_closure('secret')) sandbox.run_string(print pwd(), context) sandbox.run_string(print pwd.func_closure[0].cell_contents, context) [snip] AttributeError: 'function' object has no attribute 'func_closure' Without Tav's patch, the untrusted code can extract our password ('secret') from the closure with that last statement. To run whole files in the sandbox: sandbox.run_file('/some/path/file.py') Finally, disclaimer: I put this together in *15 minutes*. I'm sure I missed something, this isn't secure, etc. It's just a proof of concept. Void where prohibited, etc. Cheers, -- Ivan Krstić krs...@solarsail.hcs.harvard.edu | http://radian.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .pythonrc.py in man page
On Sun, Feb 22, 2009 at 6:30 PM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Sun, Feb 22, 2009 at 8:25 PM, Guido van Rossum gu...@python.org wrote: Perhaps this entry in the man page is obsolete and should be removed? Not at all. For what it's worth, the 2.6.1 documentation states: Deprecated since version 2.6: The user module has been removed in Python 3.0. If user.py has indeed been removed, then Mitchell is correct. http://docs.python.org/library/user.html Fair enough. I missed the 3.x context. Go ahead and fix the man page! -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Feb 22, 2009, at 9:43 PM, Guido van Rossum wrote: I'm not familiar with Genshi -- what is the purpose of the AST transformation here? Sorry, I should have been clearer. If the only goal is to provide a restricted bare interpreter, you can certainly just exec with a restricted set of builtins and no __import__. Since Tav mentioned wanting restricted execution of Genshi templates in particular (which have a rather complicated mechanism for executing inline Python code), I threw together a realistic, self-contained 'restricting Genshi' demo which doesn't rely on outside restrictions, such as those provided by GAE. You can ignore the AST stuff; Genshi does it for its own (non- security) purposes. -- Ivan Krstić krs...@solarsail.hcs.harvard.edu | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
At 08:45 AM 2/22/2009 -0800, Guido van Rossum wrote: I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? Just a question, but, if you just need a pure-python restricted environment for App Engine, why not just use the RestrictedPython package (i.e., http://pypi.python.org/pypi/RestrictedPython )? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Sun, Feb 22, 2009 at 7:39 PM, P.J. Eby p...@telecommunity.com wrote: At 08:45 AM 2/22/2009 -0800, Guido van Rossum wrote: I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add restricted guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? Just a question, but, if you just need a pure-python restricted environment for App Engine, why not just use the RestrictedPython package (i.e., http://pypi.python.org/pypi/RestrictedPython )? How does that work? Remember, app engine doesn't support certain things, and bytecode manipulations (if that's what RestrictedPython does) are one of the unsupported things. The other reason I can think of is that Tav is a capabilities purist. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
At 07:56 PM 2/22/2009 -0800, Guido van Rossum wrote: On Sun, Feb 22, 2009 at 7:39 PM, P.J. Eby p...@telecommunity.com wrote: Just a question, but, if you just need a pure-python restricted environment for App Engine, why not just use the RestrictedPython package (i.e., http://pypi.python.org/pypi/RestrictedPython )? How does that work? Remember, app engine doesn't support certain things, and bytecode manipulations (if that's what RestrictedPython does) are one of the unsupported things. It doesn't modify bytecode, it modifies an AST. It basically replaces prints, and attribute/item read/writes with function calls. Unfortunately, it does this AST modification by running as a traversal against the stdlib compiler package's AST, not a modern AST. So, I suppose it might not be usable as-is on app engine. It does, however, have the advantage of having been used in Zope for oh, six or seven years now? ISTM that it first came out around the same time as Python 2.3, and the latest version just dropped support for Python 2.1 and 2.2. So, if you want something that wasn't thrown together in an afternoon, it might be a good thing to take a look at. ;-) The other reason I can think of is that Tav is a capabilities purist. :-) You can implement capabilities on top of RestrictedPython; it's simply a policy-neutral enforcement framework. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Choosing a best practice solution for Python/extension modules
On Sun, Feb 22, 2009, Brett Cannon wrote: On Sat, Feb 21, 2009 at 20:12, Aahz a...@pythoncraft.com wrote: On Sat, Feb 21, 2009, Brett Cannon wrote: On Sat, Feb 21, 2009 at 15:46, Aahz a...@pythoncraft.com wrote: On Sat, Feb 21, 2009, Brett Cannon wrote: I am seeing two approaches emerging. One is where pickle contains all Python code and then uses something like use_extension to make sure the original Python objects are still reachable at some point. This has the drawback that you have to use some function to make the extensions happen and there is some extra object storage. The other approach is having pickle contain code known not to be overridden by anyone, import _pypickle for stuff that may be overridden, and then import _pickle for whatever is available. This approach has the perk of using a standard practice for how to pull in different implementation. But the drawback, thanks to how globals are bound, is that any code pulled in from _pickle/_pypickle will not be able to call into other optimized code; it's a take or leave it once the call chain enters one of those modules as they will always call the implementations in the module they originate from. To what extent do we care about being able to select Python-only on a per-module basis, particularly in the face of threaded imports? That is, we could have a sys.python_only attribute that gets checked on import. That's simple and direct, and even allows per-module switching if the application really cares and import doesn't need to worry about threads. Alternatively, sys.python_only could be a set, but that gets ugly about setting from the application. (The module checks to see whether it's listed in sys.python_only.) Maybe we should move this discussion to python-ideas for now to kick around really oddball suggestions? This is all about testing. If a change is made to some extension code it should be mirrored in the Python code and vice-versa. Okay, I don't see how that is a response to my suggestion -- I can imagine that someone might want to test a combination of pure-Python and binary libraries. I don't want to move it because this isn't some idea for a new feature that may or may not be useful; this isn't an idea, it's needed. That's fine, but what about my idea of using sys.python_only? -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com