[Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Wed, Nov 3, 2010 at 9:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But what is alternative in stdlib? Don't you think that Python doesn't provide any? -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85902 - in python/branches/py3k/Lib: os.py test/test_os.py
On Wednesday 03 November 2010 23:12:01 Nick Coghlan wrote: On Thu, Nov 4, 2010 at 1:01 AM, Benjamin Peterson benja...@python.org wrote: 2010/11/3 Nick Coghlan ncogh...@gmail.com: On Wed, Nov 3, 2010 at 10:19 PM, Benjamin Peterson benja...@python.org wrote: Warnings is loaded every time anyway. I would have agreed with you, but the contents of sys.modules in a just-started interactive interpreter suggests that isn't true any more (which surprised me). Is that perhaps because of _warnings? I suspect it's a combination of that and the patch to allow non-blocking module imports (which turns some things that would previously have been deadlocks into runtime exceptions). So do you still think that I should patch the os module to use a global import or not? Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On 06:28 am, techto...@gmail.com wrote: On Wed, Nov 3, 2010 at 9:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But what is alternative in stdlib? Don't you think that Python doesn't provide any? Persistence is a very hard problem. Lots and lots of trade-offs need to be made, and you generally want to tailor those trade-offs to the particular application at hand. This probably means that the stdlib isn't a suitable place to try to solve the problem. Look outside the stdlib and you'll find an extremely vibrant and diverse collection of software which is aimed at solving this problem, though. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, Nov 4, 2010 at 2:44 PM, Allan McRae al...@archlinux.org wrote: The second case was particularly interesting. These software would change some of their #! to point at the python2 symlink and leave the rest pointing at python. Note that python-2.7 itself falls into this category as many files in /usr/lib/python2.7 still have #!/usr/bin/env python even when installed with make altinstall. I can not remember the exact details, but I recall that some of these files were installed with executable permissions which would be bad, but I need to look into this again now things have calmed down... The packages that did not auto-detect and work with /usr/bin/python2 or /usr/bin/python2.7 mostly required a sed of their shebangs or a patch to any hardcoded /usr/bin/python paths so were easily fixed. A very interesting exercise, indeed - especially the observation regarding software (including python itself) that supports installation under alternate names, but doesn't subsequently ensure use of that name in its shebang lines. I just did a quick grep of Lib in my py3k directory, and it looks like cgi.py is incorrectly set to use /usr/local/bin/python, while the other files with shebang lines are set to /usr/bin/env python3 as expected. Tools also had a few discrepancies: scripts/2to3.py: /usr/bin/env python (necessary, I think - I believe 2to3 is a 2.x only program) scripts/gprof2html.py: /usr/bin/env python32.3 (Huh? Automated correction gone wrong, perhaps?) scripts/reindent-rst.py: /usr/bin/env python (probably incorrect) pybench: /usr/bin/env python (not sure - has pybench been forward ported on the 3.x branch?) world: /usr/bin/env python (I have no idea what this script is even for) (Note that these examples are a matter of simply respecting the *default* install location for python3, without even getting into questions of altinstall or configured installation locations) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Thu, Nov 4, 2010 at 4:28 PM, anatoly techtonik techto...@gmail.com wrote: On Wed, Nov 3, 2010 at 9:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But what is alternative in stdlib? Don't you think that Python doesn't provide any? Python 3.2a3+ (py3k:85817, Oct 24 2010, 19:25:28) [GCC 4.4.3] on linux2 Type help, copyright, credits or license for more information. import json dir(json) ['JSONDecoder', 'JSONEncoder', '__all__', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_default_decoder', '_default_encoder', 'decoder', 'dump', 'dumps', 'encoder', 'load', 'loads', 'scanner'] pickle gets overspecific in many ways, and hence (despite our best efforts, and those of third parties) may break when changing Python versions. Serialising to something more language natural (be it JSON, YAML, XML or one of the multitude of other state encoding formats out there) is far more likely to be future proof. As a tool for communicating between different instances of the *same* version of Python though, pickle is fine. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Code coverage doesn't show .py stats
On 03.11.10 19:21, anatoly techtonik wrote: Hi, Python code coverage doesn't include any .py files. What happened? http://coverage.livinglogic.de/ Did it work before? It did, however currently the logfile http://coverage.livinglogic.de/testlog.txt shows the following exception: Traceback (most recent call last): File Lib/test/regrtest.py, line 1500, in module main() File Lib/test/regrtest.py, line 696, in main r.write_results(show_missing=True, summary=True, coverdir=coverdir) File /home/coverage/python/Lib/trace.py, line 319, in write_results lnotab, count) File /home/coverage/python/Lib/trace.py, line 369, in write_results_file outfile.write(line.expandtabs(8)) UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 30: ordinal not in range(128) BTW, this is the py3k branch (i.e. http://svn.python.org/snapshots/python3k.tar.bz2) It seems the trace module has a problem with unicode. Servus, Walter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, 4 Nov 2010 23:33:38 +1000 Nick Coghlan ncogh...@gmail.com wrote: Tools also had a few discrepancies: scripts/2to3.py: /usr/bin/env python (necessary, I think - I believe 2to3 is a 2.x only program) scripts/gprof2html.py: /usr/bin/env python32.3 (Huh? Automated correction gone wrong, perhaps?) Or time machine gone wild? I think it is the version which automatically renames your classes and methods based on good taste, but still has the old assertLessEqual method at the bottom of the now 5-level deep unittest package hierarchy (while Michael enjoys his 3251st PSF community award after it was decided to make it a daily ceremony). pyclbr has been patched to handle it fine, though. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
2010/11/4 Nick Coghlan ncogh...@gmail.com: On Thu, Nov 4, 2010 at 2:44 PM, Allan McRae al...@archlinux.org wrote: The second case was particularly interesting. These software would change some of their #! to point at the python2 symlink and leave the rest pointing at python. Note that python-2.7 itself falls into this category as many files in /usr/lib/python2.7 still have #!/usr/bin/env python even when installed with make altinstall. I can not remember the exact details, but I recall that some of these files were installed with executable permissions which would be bad, but I need to look into this again now things have calmed down... The packages that did not auto-detect and work with /usr/bin/python2 or /usr/bin/python2.7 mostly required a sed of their shebangs or a patch to any hardcoded /usr/bin/python paths so were easily fixed. A very interesting exercise, indeed - especially the observation regarding software (including python itself) that supports installation under alternate names, but doesn't subsequently ensure use of that name in its shebang lines. I just did a quick grep of Lib in my py3k directory, and it looks like cgi.py is incorrectly set to use /usr/local/bin/python, while the other files with shebang lines are set to /usr/bin/env python3 as expected. Tools also had a few discrepancies: scripts/2to3.py: /usr/bin/env python (necessary, I think - I believe 2to3 is a 2.x only program) No, I believe distutils is supposed to patch that up, though. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85987 - python/branches/py3k/Lib/test/test_os.py
On 2010/11/02 1:30, Nick Coghlan wrote: On Tue, Nov 2, 2010 at 2:10 AM, Hirokazu Yamamoto ocean-c...@m2.ccsnet.ne.jp wrote: Does this really cause resource warning? I think os.popen instance won't be into traceback because it's not declared as variable. So I suppose it will be deleted by reference count == 0 even when exception occurs. Any time __del__ has to close the resource triggers ResourceWarning, regardless of whether that is due to the cyclic garbage collector or the refcount naturally falling to zero. In the past dealing with this was clumsy, so it made sense to rely on CPython's refcounting to do the work. However, we have better tools for deterministic resource management now (in the form of context managers), so these updates help make the standard library and its test suite more suitable for use with non-refcounting Python implementations (such as PyPy, Jython and IronPython). Cheers, Nick. Thank you for reply. Probably this is difficult problem. I often use with statement, but it's also true sometimes I feel this warning is a bit noisy. Is there a way to turn this off? C:\Documents and Settings\Oceanpy3k Python 3.2a3+ (py3k, Nov 3 2010, 00:27:28) [MSC v.1200 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. open(a.py).read() __main__:1: ResourceWarning: unclosed file _io.TextIOWrapper name='a.py' encodi ng='cp932' '\nimport timeit\n\nt = timeit.Timer(\nos.stat(e:/voltest/lnk)\n, \ni mport os\n)\n\nprint(t.timeit(1000))\n\n' [49593 refs] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85987 - python/branches/py3k/Lib/test/test_os.py
On Thu, 04 Nov 2010 23:09:39 +0900 Hirokazu Yamamoto ocean-c...@m2.ccsnet.ne.jp wrote: On 2010/11/02 1:30, Nick Coghlan wrote: On Tue, Nov 2, 2010 at 2:10 AM, Hirokazu Yamamoto ocean-c...@m2.ccsnet.ne.jp wrote: Does this really cause resource warning? I think os.popen instance won't be into traceback because it's not declared as variable. So I suppose it will be deleted by reference count == 0 even when exception occurs. Any time __del__ has to close the resource triggers ResourceWarning, regardless of whether that is due to the cyclic garbage collector or the refcount naturally falling to zero. In the past dealing with this was clumsy, so it made sense to rely on CPython's refcounting to do the work. However, we have better tools for deterministic resource management now (in the form of context managers), so these updates help make the standard library and its test suite more suitable for use with non-refcounting Python implementations (such as PyPy, Jython and IronPython). Cheers, Nick. Thank you for reply. Probably this is difficult problem. I often use with statement, but it's also true sometimes I feel this warning is a bit noisy. Is there a way to turn this off? You can use all the usual means of controlling emission of warnings, so for example python -Wi would work to silence them all. Also, ResourceWarning is silenced by default in release builds. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85987 - python/branches/py3k/Lib/test/test_os.py
On 2010/11/04 23:23, Antoine Pitrou wrote: You can use all the usual means of controlling emission of warnings, so for example python -Wi would work to silence them all. Also, ResourceWarning is silenced by default in release builds. Regards Antoine. Thank you, this works. (I couldn't find the way from python --help) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Wed, Nov 3, 2010 at 9:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But *that* set of types can safely be marshalled using the marshal module... -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Thu, Nov 4, 2010 at 10:51 AM, Guido van Rossum gu...@python.org wrote: .. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But *that* set of types can safely be marshalled using the marshal module... Not if the instances contain reference cycles. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85902 - in python/branches/py3k/Lib: os.py test/test_os.py
On Thu, Nov 4, 2010 at 9:16 PM, Victor Stinner victor.stin...@haypocalc.com wrote: So do you still think that I should patch the os module to use a global import or not? I'm actually more inclined to suggest we avoid triggering the warning under -bb in the first place by iterating over the environment in that case instead of using the mapping interface. (I was going to suggest a smarter version that used a SafeKey class instead, but it turns out os.environ only works with real string objects). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Nov 04, 2010, at 02:44 PM, Allan McRae wrote: While this is not strictly related to python development, I thought that developers of python might be interested in some of the lessons provided by this. So forgive me if this is really wrong for this list... Recently Arch Linux did a big transition with respect to python. Now we support two python packages: python and python2. Very cool to hear about this first hand, thanks for posting it Allan. I was recently at the Ubuntu Developers Summit and Arch Linux's transition was a source of several hallway discussions. It's good to read about your work and successes in blazing that trail. I really do not want to debate the sanity of pointing /usr/bin/python at python-3.x here, but it suffices to say that I am of the opinion that if python-3.x is really the future of python, then /usr/bin/python must eventually point to a 3.x version. Also, Arch Linux is very bleeding edge and we expect our users to be competent enough to deal with thing like this. According to #python, we are all idiots And I have been (figuratively) yelled at by a couple of Debian developers (which is incidentally the only major distro I found without a /usr/bin/python2 symlink). Ah too bad, no one needs to yell :). It's an interesting discussion topic though and it's something I think other distros should start considering. In Ubuntu 11.04, we'll have Python 3.1 and 3.2, 2.6 and 2.7, with the default (i.e. /usr/bin/python) either at 2.6 (probable) or 2.7 (possible). `python3` currently points to 3.1.2, but we've talked about getting that to 3.2 for this cycle. Ubuntu's next Long Term Support release is scheduled for April 2012. It's an ambitious but worthy goal to see if we can transition to Python 3 as the default Python by then. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Nov 04, 2010, at 11:33 PM, Nick Coghlan wrote: world: /usr/bin/env python (I have no idea what this script is even for) It's basically a front-end to ISO 3166 country codes. IOW, it prints the expansion of top-level domain names and can do some reverse lookups too. E.g. % Tools/world/world us us originated from United States I once started to rip it out into a separate package but haven't gotten too far with that. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Thu, Nov 4, 2010 at 3:38 PM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Nov 4, 2010 at 4:28 PM, anatoly techtonik techto...@gmail.com wrote: On Wed, Nov 3, 2010 at 9:08 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But what is alternative in stdlib? Don't you think that Python doesn't provide any? Python 3.2a3+ (py3k:85817, Oct 24 2010, 19:25:28) [GCC 4.4.3] on linux2 Type help, copyright, credits or license for more information. import json dir(json) ['JSONDecoder', 'JSONEncoder', '__all__', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_default_decoder', '_default_encoder', 'decoder', 'dump', 'dumps', 'encoder', 'load', 'loads', 'scanner'] pickle gets overspecific in many ways, and hence (despite our best efforts, and those of third parties) may break when changing Python versions. Serialising to something more language natural (be it JSON, YAML, XML or one of the multitude of other state encoding formats out there) is far more likely to be future proof. As a tool for communicating between different instances of the *same* version of Python though, pickle is fine. pickle is insecure, marshal too. What about JSON? IIUC you need a definition of a class to be able to unserialize it in all cases. I wonder how is this definition validated, i.e. what to watch for when modifying classes that can be serialized. -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Thu, Nov 4, 2010 at 9:15 AM, anatoly techtonik techto...@gmail.com wrote: pickle is insecure, marshal too. What's the attack you're thinking of on marshal? It never executes any code while unmarshalling (although it can unmarshal code objects -- but the receiving program has to do something additionally to execute those). What about JSON? IIUC you need a definition of a class to be able to unserialize it in all cases. I wonder how is this definition validated, i.e. what to watch for when modifying classes that can be serialized. Security is all in the code used to deserialize. I haven't analyzed the json library that comes in the stdlib these days, but couldn't it in theory be as safe as XML? (Not that there haven't been any attacks on XML -- but they depended on bugs in the unmarshalling code, the format itself is not insecure.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, Nov 4, 2010 at 05:44, Allan McRae al...@archlinux.org wrote: According to #python, we are all idiots To clarify (but I dont speak for the rest of #python, just myself), I think the move was premature, but I don't use Arch and I don't know what typical Arch users expect. The reason I think it's premature is that 'python2' just doesn't work everywhere, and I would have gone for a transitionary period where '/usr/bin/python' is something that screams loudly that it shouldn't be used before it executes 'python2'. That would've allowed for more time to fix things that use the wrong shebang line, or tools that use 'python' instead of letting distutils set it for them. I hope that's something other distributions will consider before changing the meaning of /usr/bin/python. As for #python, well, we got this storm of people utterly confused about how their stuff doesn't work anymore, and putting the blame in the wrong place. I don't think a distribution should ever cause that (even though many do in lesser ways) -- but as I said, I don't use Arch so maybe I don't understand the purpose of it. The complaints seem to have died down now (though possibly because of the 'no arch' topic :) -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
To clarify (but I dont speak for the rest of #python, just myself), I think the move was premature, but I don't use Arch and I don't know what typical Arch users expect. The reason I think it's premature is that 'python2' just doesn't work everywhere, and I would have gone for a transitionary period where '/usr/bin/python' is something that screams loudly that it shouldn't be used before it executes 'python2'. I really do think the key point here is don't know what typical Arch users expect. I don't know either, but my personal feeling is that Arch isn't that widely used, but ISTM that Arch users are expected to be technically advanced, compared to the wider community of Linux users. So if these user find a problem, they might know how to fix it, and they might know how to make bug reports. In essence, you are asking that there should be a smoother path to making /usr/bin/python Python 3 - and I observe that Arch's switching actually is a very useful step on that smoother path. If they figure out what changes to make, many of the changes may have been done when other Linux distributions just start to consider the change. As for #python, well, we got this storm of people utterly confused about how their stuff doesn't work anymore, and putting the blame in the wrong place. I don't think a distribution should ever cause that (even though many do in lesser ways) -- but as I said, I don't use Arch so maybe I don't understand the purpose of it. The complaints seem to have died down now (though possibly because of the 'no arch' topic :) So apparently, there is quite a number of Arch users, and they do make bug reports. Good :-) If this gets attributed correctly (i.e. as a deliberate decision by Arch, revealing bugs in many packages that have long existed), and if Google picks the canonical resolution quickly, I don't think any harm is done - and in the long run, it will smooth the migration for everybody else. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Nov 4, 2010, at 12:49 PM, Guido van Rossum wrote: What's the attack you're thinking of on marshal? It never executes any code while unmarshalling (although it can unmarshal code objects -- but the receiving program has to do something additionally to execute those). These issues may have been fixed now, but a long time ago I recall seeing some nasty segfaults which looked exploitable when feeding marshal malformed data. If they still exist, running a fuzzer on some pyc files should reveal them pretty quickly. When I ran across them I didn't think much of them, and probably did not even report the bug, since marshal is mostly used to load code anyway, which is implicitly trusted. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On 04.11.2010 21:12, Martin v. Löwis wrote: To clarify (but I dont speak for the rest of #python, just myself), I think the move was premature, but I don't use Arch and I don't know what typical Arch users expect. The reason I think it's premature is that 'python2' just doesn't work everywhere, and I would have gone for a transitionary period where '/usr/bin/python' is something that screams loudly that it shouldn't be used before it executes 'python2'. Iirc, it was an explicit decision made at the 2009 language summit not to introduce a python2 symlink, but using python3 for python3.x instead. Debian/Ubuntu don't ship a python2 symlink by intent. Did the plans change, i.e. are there plans to provide a python symlink for python 3.x altinstall in a future release, e.g in 3.4 or 3.5? Matthias ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, Nov 4, 2010 at 21:12, Martin v. Löwis mar...@v.loewis.de wrote: As for #python, well, we got this storm of people utterly confused about how their stuff doesn't work anymore, and putting the blame in the wrong place. I don't think a distribution should ever cause that (even though many do in lesser ways) -- but as I said, I don't use Arch so maybe I don't understand the purpose of it. The complaints seem to have died down now (though possibly because of the 'no arch' topic :) So apparently, there is quite a number of Arch users, and they do make bug reports. Good :-) I don't know that they do. I just know that people came to #python and complained, which is unfortunately something completely different. (We did ask every single one to take it up with the right forum, and I know at least one person did file a bug, but that's about it.) -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Thu, Nov 4, 2010 at 1:25 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Nov 4, 2010, at 12:49 PM, Guido van Rossum wrote: What's the attack you're thinking of on marshal? It never executes any code while unmarshalling (although it can unmarshal code objects -- but the receiving program has to do something additionally to execute those). These issues may have been fixed now, but a long time ago I recall seeing some nasty segfaults which looked exploitable when feeding marshal malformed data. If they still exist, running a fuzzer on some pyc files should reveal them pretty quickly. When I ran across them I didn't think much of them, and probably did not even report the bug, since marshal is mostly used to load code anyway, which is implicitly trusted. I'm not sure that all these were fixed but it would be a finite (and probably small) amount of work to get it fixed -- unlike fixing pickling, which is impossible (unless you implemented some kind of sandboxing solution :-). A good use for pickling is when it's optional. Example: putting pickles in memcache. The source of the pickles is (presumably) trusted, so the only remaining problem is occasional version skew. If the unpickling fails it can just be treated as a cache miss. (Tricky: when unpickling succeeds but returns a broken object. Nobody's perfect. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, Nov 4, 2010 at 5:44 AM, Allan McRae al...@archlinux.org wrote: According to #python, we are all idiots I realize this is not really what your message was about and for sake of brevity you used a bit of a hyperbole, but like Thomas I would still like to nip in right there. #python is a pretty big channel. I think everyone understands that reducing it in its entirety to a single opinion as inflammatory as you're all idiots is at best oversimplifying and at worst offensive. (FWIW, Thomas has already said a bunch of stuff I completely agree with, so +1 everything he said.) What is true is that there's a new and temporary NO ARCH rule in the topic, and it's the for the same reason there's a NO LOL in the topic: to keep the signal to noise ratio high. Apparently there is a large number of packages (or perhaps just commonly used ones) either in Arch itself or AUR that didn't work anymore. This caused a lot of people to complain about problems that are actually Arch-specific problems: not really something #python is there for nor something it is good at helping with. That wouldn't be helping people with Python, that would be helping people with Arch. It is not intended as, and should not be interpreted as, some kind of public declaration of war against Arch. It simply means that #python isn't going to do Arch-specific support for packages that no longer work after an update, since that's not our job nor expertise. I don't think grudges or misunderstandings help anyone, and Python in particular, further along. I think I've demonstrated that I'm eager to get rid of them before. If you (or anyone else for that matter) are worried about behavior or policy in #python in the future (I assure you there's really not as much as people generally seem to think there is) and would like clarification, there's an easy way to access a list of the ops: /msg chanserv access #python list Or just shout are there any ops on in #python whenever you like. These people should be able to tell you what you want to know or at least point you to the right person to ask. But basically, to reiterate a point I've made a bunch of times and have already made (not to you in particular, just in general): #python is a bunch of people, please don't extrapolate the opinions of a few to the opinions of many. It's easy and tempting, but it often leads to demonizing a bunch of people and putting words in people's mouths which they didn't say or even agree with. cheers and good luck lvh ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On 05/11/10 08:40, Laurens Van Houtven wrote: On Thu, Nov 4, 2010 at 5:44 AM, Allan McRaeal...@archlinux.org wrote: According to #python, we are all idiots I realize this is not really what your message was about and for sake of brevity you used a bit of a hyperbole, but like Thomas I would still like to nip in right there. #python is a pretty big channel. I think everyone understands that reducing it in its entirety to a single opinion as inflammatory as you're all idiots is at best oversimplifying and at worst offensive. snip Of course, and I was not intending to offend here. It was more of a running commentary on the unintended influx of Arch Linux users to the channel and some of the responses posted to them (some of which I found rather amusing when forwarded to me - especially the early response as people were figuring out what was going on). I also agree with the NO ARCH topic at the moment. I was fairly surprised so many people went to #python for help given we had made news posts and had a topic in our IRC channel pointing to how to start fixing issues. Allan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help with warnings not being raised
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all. I just committed r86180, but there is something I don't like. If you read the tests I did (by hand)at http://bugs.python.org/issue9675#msg120462 , python should show the unraisable and THEN the C API unavailable warning, but it is not showing the warning. I don't know why. I have committed the patch because it solves the original bug, but I am pretty uncomfy not knowing what Python is not doing exactly what I want... Any idea?. Sorry for wasting your time with probably trivial stuff, but I need to know... :-? PS: I am using PyErr_Warn(), that is deprecated, because this code should work in Python 2.3 too. I tried PyErr_WarnEx() too, it didn't work either. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTNNPN5lgi5GaxT1NAQKt9gP/VbS1Fycg7SBBy9nTFhvw7NOAOSnxM8Dt B0Wq/I9Rnr+YOYGpCIvgVut8CuqT3oVRtRPeBnajjinEo7rcZSi79rQlcMcNq1VS JwQELp9bd3Az5Xmbpf+FeKNBE8K+1bpezAcGHv/QTsPXSIsU+fTH1sKwXoj9S4Vg CMUXTP9InkE= =6LVa -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help with warnings not being raised
2010/11/4 Jesus Cea j...@jcea.es: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all. I just committed r86180, but there is something I don't like. If you read the tests I did (by hand)at http://bugs.python.org/issue9675#msg120462 , python should show the unraisable and THEN the C API unavailable warning, but it is not showing the warning. I don't know why. Are you passing -3 -Wall? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
Am 04.11.2010 17:15, schrieb anatoly techtonik:
pickle is insecure, marshal too.
If the transport or storage layer is not save, you should
cryptographically sign the data anyway::
def pickle_encode(data, key):
msg = base64.b64encode(pickle.dumps(data, -1))
sig = base64.b64encode(hmac.new(key, msg).digest())
return sig + ':' + msg
def pickle_decode(data, key):
if data and ':' in data:
sig, msg = data.split(':', 1)
if sig == base64.b64encode(hmac.new(key, msg).digest()):
return pickle.loads(base64.b64decode(msg))
raise pickle.UnpicklingError(Wrong or missing signature.)
Bottle (a web framework) uses a similar approach to store non-string
data in client-side cookies. I don't see a (security) problem here.
--
Mit freundlichen Grüßen
Marcel Hellkamp
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
Thomas Wouters writes: To clarify (but I dont speak for the rest of #python, just myself), I think the move was premature, but I don't use Arch and I don't know what typical Arch users expect. All of the Arch users I know expect Arch to occasionally do radical things because they're the right things to do in the long run. But every avant garde distribution picks up its share of wannabes who don't understand how the process works. The reason I think it's premature is that 'python2' just doesn't work everywhere, and I would have gone for a transitionary period where '/usr/bin/python' is something that screams loudly that it shouldn't be used before it executes 'python2'. This is unrealistic. It would seriously annoy Arch's intended audience. (Eg, recently I've become a lot more favorable to using Word instead of OOo because Word doesn't pop up a useless warning every time I save a .doc file.) Practically speaking, it would have to be off by default, like Python pending deprecation warnings. Anyway, I bet that anybody capable of upgrading their *Arch* packages and complaining to *#python* about resulting breakage would be capable of complaining to #python about the weird warning about python2. And you can't have a NO /USR/BIN/PYTHON topic, can you?wink As for #python, well, we got this storm of people utterly confused about how their stuff doesn't work anymore, and putting the blame in the wrong place. How so? Ultimately, Guido is responsible for this. Sure, the immediate symptom was caused by Arch's action, but Python 3 *is* rather incompatible with Python 2. You're going to get a storm every time a distro changes, and in a year or two, it's no longer going to be something you can dispose of by setting a hotkey to Google for 'BOGUS Linux python' -- it's going to be stuff that requires a real understanding of how Python 3 differs from Python 2, and often will be pretty subtle. I don't think a distribution should ever cause that (even though many do in lesser ways) Sure, and Guido should have exercised the Time Machine a little harder so that Python 3 never needed to happen. IOW, this is the price of success and wide distribution. BTW, I hope the next distribution make the jump does try your suggestion to make /usr/bin/python scream. It might work, even work well. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Thu, Nov 4, 2010 at 7:19 PM, Allan McRae al...@archlinux.org wrote: I also agree with the NO ARCH topic at the moment. I was fairly surprised so many people went to #python for help given we had made news posts and had a topic in our IRC channel pointing to how to start fixing issues. Allan I don't remember seeing any warning about it during the upgrade. That may have helped people (ones that read the warnings, at least) figure out what was going on. I think a warning from /usr/bin/python may have helped as well, but I do suppose might be a bit extreme. FWIW, I found those news posts and the Python wiki page pretty quickly after I realized my scripts weren't working anymore. -Devin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
Nick Coghlan wrote: As a tool for communicating between different instances of the *same* version of Python though, pickle is fine. I'm using pickle to pass a list and dict of floats and strings from Python 2.6 to 3.1. I've never had any problems with it. Am I living in a state of sin or is that okay? -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On Nov 4, 2010, at 8:43 PM, Stephen J. Turnbull wrote: All of the Arch users I know expect Arch to occasionally do radical things because they're the right things to do in the long run. But the previous consensus (at least, as I, and presumably many other people understood it) was that python2 would remain the owner of the name /usr/bin/python for the indefinite future, and python3 would be invoked with /usr/bin/python3. Given that, it's not at all clear that Arch's actions are the right thing to do. IMO, moving away from that consensus should've been brought up on python-dev rather than just one distro just doing it all alone, causing incompatibilities and annoyance. If python-dev wants python3 to inherit the name /usr/bin/python, then python2 should've been installing a binary called /usr/bin/python2 for a couple years ahead of time, and recommending that everyone use that in their #! lines, so that the switch could've been done without breaking everything... Sure, and Guido should have exercised the Time Machine a little harder so that Python 3 never needed to happen. IOW, this is the price of success and wide distribution. Well, other programming languages seem to have avoided making sweeping bidirectionally-incompatible changes, despite being successful and widely distributed. But that's a whole other discussion. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help with warnings not being raised
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/11/10 01:36, Benjamin Peterson wrote: I don't know why. Are you passing -3 -Wall? I am passing -3 -Werror, to induce the error control I have committed. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTNNaF5lgi5GaxT1NAQJt0wQAk29q0qXXDzBINyamo7I3ehD75U185165 UXYvCw987N74ye+CyUfpkTENjFkxY+cUMEmTmB0N/mhGblveHFLbaC0Kz831SWGM OdNDi6tBQB1CpyxeOyhQN4e5NzabljoFc7XuLh32rbYY15dqYnZXYgUaXZ+8W84t DXiK08P0TrU= =g732 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/11/10 05:44, Allan McRae wrote: The second case was particularly interesting. These software would change some of their #! to point at the python2 symlink and leave the rest pointing at python. Note that python-2.7 itself falls into this category as many files in /usr/lib/python2.7 still have #!/usr/bin/env python even when installed with make altinstall. I can not remember the exact details, but I recall that some of these files were installed with executable permissions which would be bad, but I need to look into this again now things have calmed down... PLEASE, open a bug with this. It is a serious bug. make altinstall *SHOULD* be respected. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTNNb+Jlgi5GaxT1NAQIScgP+NFE98E98rFOMh85RJBdFT3U3nwWNktz8 2uYBPQq5p28eQ6LY6TNkv4/iSIoF++o40xuveuuL+1ys7I/QRne0P/Wipspr2eLZ oOMkwNVfrYnaX0MV/pu750uqsh62dQZIqxe9oWtD4FS00gHgqtfIvlI/EZYkVy0m WwzT9zBo5Lw= =fQx7 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/11/10 15:57, Alexander Belopolsky wrote: On Thu, Nov 4, 2010 at 10:51 AM, Guido van Rossum gu...@python.org wrote: .. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. But *that* set of types can safely be marshalled using the marshal module... Not if the instances contain reference cycles. Moreover, in the docs the marshall module EXPLICITLY says that the format is undocumented on purpose, and subject to change. Seems a pretty bad option for persistence, if you expect to read your data back in the future. http://docs.python.org/library/marshal.html This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine architecture issues (e.g., you can write a Python value to a file on a PC, transport the file to a Sun, and read it back there). Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does). - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTNNeWZlgi5GaxT1NAQLU/wP/aZOJkgYWHyYMUT0diDsh+pFg6TxDPWMu /cNx5l6wNaW8DQ5cuMSHkelfVYpx6EQwTCZPu9jiAAOJmFfNURt1Q+P4ikf5eobA 7mRlFrr+C3Lmi9CA3thuwBh4IkLHUl3mk6eQ0mqJPzpdbJLWhPmkOEN7L31nk9// YQHdU4e795U= =+cB0 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
James Y Knight wrote: But the previous consensus (at least, as I, and presumably many other people understood it) was that python2 would remain the owner of the name /usr/bin/python for the indefinite future, and python3 would be invoked with /usr/bin/python3. Given that, it's not at all clear that Arch's actions are the right thing to do. The time will come when even Python 2.7 is long obsolete. I think it is silly to insist that people invoke python3 to run their Python 3.7 scripts. Arch might be jumping the gun a little, or even a lot, but sooner or later it should be done. Besides, this is another sign that the Python 3 haters are wrong. We now have a distro that has made Python 3 the standard system python. It might be a bleeding-edge distro not recommended for non-experts, but it's still pretty cool that *somebody* has done it. IMO, moving away from that consensus should've been brought up on python-dev rather than just one distro just doing it all alone, causing incompatibilities and annoyance. We're all adults here. If Arch wants to live on the bleeding edge, more power to them. That's why my server runs Centos :) -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On 12:21 am, m...@gsites.de wrote:
Am 04.11.2010 17:15, schrieb anatoly techtonik:
pickle is insecure, marshal too.
If the transport or storage layer is not save, you should
cryptographically sign the data anyway::
def pickle_encode(data, key):
msg = base64.b64encode(pickle.dumps(data, -1))
sig = base64.b64encode(hmac.new(key, msg).digest())
return sig + ':' + msg
def pickle_decode(data, key):
if data and ':' in data:
sig, msg = data.split(':', 1)
if sig == base64.b64encode(hmac.new(key, msg).digest()):
return pickle.loads(base64.b64decode(msg))
raise pickle.UnpicklingError(Wrong or missing signature.)
Bottle (a web framework) uses a similar approach to store non-string
data in client-side cookies. I don't see a (security) problem here.
Your pickle_decode leaks information about the key. An attacker will
eventually (a few seconds to a few minutes, depending on how they have
access to this system) be able to determine your key and send you
arbitrary pickles (ie, execute arbitrary code on your system).
Oops.
This stuff is hard. If you're going to mess around with it, make sure
you're *serious* (better approach: don't mess around with it).
Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)
On Friday, November 5, 2010, exar...@twistedmatrix.com wrote:
On 12:21 am, m...@gsites.de wrote:
Am 04.11.2010 17:15, schrieb anatoly techtonik:
pickle is insecure, marshal too.
If the transport or storage layer is not save, you should cryptographically
sign the data anyway::
def pickle_encode(data, key):
msg = base64.b64encode(pickle.dumps(data, -1))
sig = base64.b64encode(hmac.new(key, msg).digest())
return sig + ':' + msg
def pickle_decode(data, key):
if data and ':' in data:
sig, msg = data.split(':', 1)
if sig == base64.b64encode(hmac.new(key, msg).digest()):
return pickle.loads(base64.b64decode(msg))
raise pickle.UnpicklingError(Wrong or missing signature.)
Bottle (a web framework) uses a similar approach to store non-string data in
client-side cookies. I don't see a (security) problem here.
Your pickle_decode leaks information about the key. An attacker will
eventually (a few seconds to a few minutes, depending on how they have access
to this system) be able to determine your key and send you arbitrary pickles
(ie, execute arbitrary code on your system).
Oops.
This stuff is hard. If you're going to mess around with it, make sure you're
*serious* (better approach: don't mess around with it).
Specifically you need to use a constant time signature verification or
else there are possible timing attacks. Sounds like something a hmac
module should provide in the first place.
But yeah, this stuff is hard, better to just not have a code execution
hole in the first place.
-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-3 transition in Arch Linux
On 05/11/10 11:20, Jesus Cea wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/11/10 05:44, Allan McRae wrote: The second case was particularly interesting. These software would change some of their #! to point at the python2 symlink and leave the rest pointing at python. Note that python-2.7 itself falls into this category as many files in /usr/lib/python2.7 still have #!/usr/bin/env python even when installed with make altinstall. I can not remember the exact details, but I recall that some of these files were installed with executable permissions which would be bad, but I need to look into this again now things have calmed down... PLEASE, open a bug with this. It is a serious bug. make altinstall *SHOULD* be respected. Done: http://bugs.python.org/issue10318 Allan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
