[issue6631] Disallow relative files paths in urllib*.open()
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: Sorry, why was this change backported? Does this fix a specific issue in 2.7 or 3.2? On the contrary, it seems to me that code which (incorrectly) used urllib.urlopen() to allow both urls and local files will suddenly break. -- nosy: +amaury.forgeotdarc stage: committed/rejected - commit review status: closed - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6631 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6631] Disallow relative files paths in urllib*.open()
Senthil Kumaran sent...@uthcode.com added the comment: Actually, I saw this as a bug with urllib.urlopen and urllib2 had exhibited proper behaviour previously. Now, both behaviour will be consistent now. But, you are right that an *incorrect* usage of urllib.urlopen would break in 2.7.2. If we need to be lenient on that incorrect usage, then this change can be there in 3.x series, because of urllib.request.urlopen would be interface which users will be using and it can be reverted from 2.7. Personally, I am +/- 0 on reverting this in 2.7. Initially, I saw this as a bug, but later when I added tests for ValueError and checkedin, I realized that it can break some incorrect usages, as you say. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6631 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13790] In str.format an incorrect error message for list, tuple, dict, set
Eric V. Smith e...@trueblade.com added the comment: While looking at object.__format__, I recall that we've already addressed this, sort of. For a different reason, this is already deprecated in 3.3 and will become an error in 3.4. See issues 9856 and 7994. $ ./python -Wd Python 3.3.0a0 (default:40e1be1e0707, Jan 15 2012, 00:58:51) [GCC 4.6.1] on linux Type help, copyright, credits or license for more information. format([], 'd') __main__:1: DeprecationWarning: object.__format__ with a non-empty format string is deprecated Traceback (most recent call last): File stdin, line 1, in module ValueError: Unknown format code 'd' for object of type 'str' [67288 refs] We could still have object.__format__ catch and re-throw the ValueError with a better message. I'd have to think it through if we could catch all ValueErrors, or if it's possible for another ValueError to be thrown and we'd only catch and rethrow this specific ValueError. But since this is deprecated, I'm not sure it's worth the hassle. I'd advocate closing this issue as won't fix. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13790 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13609] Add os.get_terminal_size() function
STINNER Victor victor.stin...@haypocalc.com added the comment: Does this need need more discussion, code review, testing, or just more time? As I already wrote, I would prefer a very simple os.get_terminal_size() function: don't read environment varaiables, use a simple tuple instead of a new type, and raise an error if the size cannot be read (so no need of default values). The os module is written as a thin wrapper between Python and the OS. A more high level function (read environment variables, handle the error, use a namedtuple) can be written in your application, or maybe in another module. This is just my opinion, other core developers may prefer your version :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13609 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13790] In str.format an incorrect error message for list, tuple, dict, set
R. David Murray rdmur...@bitdance.com added the comment: So the error is going to be something about the source type not supporting '__format__'? That change will also address the OP's concern about truncated reprs when a fixed string length is specified, so I agree that the title issue can be closed. Terry's patch with the ({}) removed should be committed, though. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13790 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: Thoughts? (apart from ugh! it's ugly! yes I know - it's late here) Is it guaranteed that no usage pattern can render this protection inefficient? What if a dict is constructed by intermingling lookups and inserts? Similarly, what happens with e.g. the common use case of dictdefault(list), where you append() after the lookup/insert? Does some key distribution allow the attack while circumventing the protection? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13706] non-ascii fill characters no longer work in formatting
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 231c6042c40c by Victor Stinner in branch 'default': Issue #13706: Support non-ASCII fill characters http://hg.python.org/cpython/rev/231c6042c40c -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13706] non-ascii fill characters no longer work in formatting
STINNER Victor victor.stin...@haypocalc.com added the comment: I fixed the original report, but there is still an issue with non-ASCII thousands separator. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Zbyszek Szmek zbys...@in.waw.pl added the comment: The hashing with random seed is only marginally slower or more complicated than current version. The patch is big because it moves random number generator initialization code around. There's no per object tax, and the cost of the random number generator initialization is only significant on windows. Basically, there's no tax. Zbyszek -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: On Sat, 2012-01-21 at 14:27 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: Thoughts? (apart from ugh! it's ugly! yes I know - it's late here) Is it guaranteed that no usage pattern can render this protection inefficient? What if a dict is constructed by intermingling lookups and inserts? Similarly, what happens with e.g. the common use case of dictdefault(list), where you append() after the lookup/insert? Does some key distribution allow the attack while circumventing the protection? Yes, I agree that I was making an unrealistic assumption about usage patterns. There was also some global state (the is_inserting variable). I've tweaked the approach somewhat, moved the global to be per-dict, and am attaching a revised version of the patch: amortized-probe-counting-dmalcolm-2012-01-21-003.patch In this patch, rather than reset the count each time, I keep track of the total number of calls to insertdict() that have happened for each large dict (i.e. for which ma_table != ma_smalltable), and the total number of probe iterations that have been needed to service those insertions/overwrites. It raises the exception when the *number of probe iterations per insertion* exceeds a threshold factor (rather than merely comparing the number of iterations against the current ma_used of the dict). I believe this means that it's tracking and checking every time the dict is modified, and (I hope) protects us against any data that drives the dict implementation away from linear behavior (because that's essentially what it's testing for). [the per-dict stats are reset each time that it shrinks down to using ma_smalltable again, but I think at-risk usage patterns in which that occurs are uncommon relative to those in which it doesn't]. When attacked, this leads to exceptions like this: AlgorithmicComplexityError: dict construction used 1697 probes whilst performing 53 insertions (len() == 58) at key 58 with hash 42 i.e we have a dictionary containing 58 keys, which has seen 53 insert/overwrite operations since transitioning to the non-ma_smalltable representation (at size 6); presumably it has 128 PyDictEntries. Servicing those 53 operations has required a total 1697 iterations through the probing loop, or a little over 32 probes per insert. I just did a full run of the test suite (using run_tests.py), and it mostly passed the new tests I've added (included the test for scenario 2 from Frank's email). There were two failures: == FAIL: test_inheritance (test.test_pep352.ExceptionClassTests) -- AssertionError: 1 != 0 : {'AlgorithmicComplexityError'} not accounted for -- which is obviously fixable (given a decision on where the exception lives in the hierarchy) and this one: test test_mutants crashed -- Traceback (most recent call last): File /home/david/coding/python-hg/cpython-count-collisions/Lib/test/regrtest.py, line 1214, in runtest_inner the_package = __import__(abstest, globals(), locals(), []) File /home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py, line 159, in module test(100) File /home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py, line 156, in test test_one(random.randrange(1, 100)) File /home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py, line 132, in test_one dict2keys = fill_dict(dict2, range(n), n) File /home/david/coding/python-hg/cpython-count-collisions/Lib/test/test_mutants.py, line 118, in fill_dict Horrid(random.choice(candidates)) AlgorithmicComplexityError: dict construction used 2753 probes whilst performing 86 insertions (len() == 64) at key Horrid(86) with hash 42 though that seems to be deliberately degenerate code. Caveats: * no overflow handling (what happens after 2**32 modifications to a long-lived dict on a 32-bit build?) - though that's fixable. * no idea what the scaling factor for the threshold should be (there may also be a deep mathematical objection here, based on how big-O notation is defined in terms of an arbitrary scaling factor and limit) * not optimized; I haven't looked at performance yet * doesn't cover set(), though that also has spare space (I hope) via its own smalltable array. BTW, note that although I've been working on this variant of the collision counting approach, I'm not opposed to the hash randomization approach, or to adding extra checks in strategic places within the stdlib: I'm keen to get some kind of appropriate fix approved by the upstream Python development community so I can backport it to the various recent-to-ancient versions of CPython I support in RHEL (and Fedora), before we start seeing real-world attacks. Hope this is helpful
[issue13829] exception error
Brett Cannon br...@python.org added the comment: Then I'm going to assume the bug lies with Moviegrabber doing something wrong and it isn't Python's direct fault. -- resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13829 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: (or combination of fixes, of course) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13790] In str.format an incorrect error message for list, tuple, dict, set
Eric V. Smith e...@trueblade.com added the comment: The error message will be: non-empty format string passed to object.__format__. I agree with your comment about Terry's patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13790 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: In this patch, rather than reset the count each time, I keep track of the total number of calls to insertdict() that have happened for each large dict (i.e. for which ma_table != ma_smalltable), and the total number of probe iterations that have been needed to service those insertions/overwrites. It raises the exception when the *number of probe iterations per insertion* exceeds a threshold factor (rather than merely comparing the number of iterations against the current ma_used of the dict). This sounds much more robust than the previous attempt. When attacked, this leads to exceptions like this: AlgorithmicComplexityError: dict construction used 1697 probes whilst performing 53 insertions (len() == 58) at key 58 with hash 42 We'll have to discuss the name of the exception and the error message :) Caveats: * no overflow handling (what happens after 2**32 modifications to a long-lived dict on a 32-bit build?) - though that's fixable. How do you suggest to fix it? * no idea what the scaling factor for the threshold should be (there may also be a deep mathematical objection here, based on how big-O notation is defined in terms of an arbitrary scaling factor and limit) I'd make the threshold factor a constant, e.g. 64 or 128 (it should not be too small, to avoid false positives). We're interested in the actual slowdown factor, which a constant factor models adequately. It's the slowdown factor which makes a DOS attack using this technique efficient. Whether or not dict construction truely degenerates into a O(n**2) operation is less relevant. There needs to be a way to disable it: an environment variable would be the minimum IMO. Also, in 3.3 there should probably be a sys function to enable or disable it at runtime. Not sure it should be backported since it's a new API. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13609] Add os.get_terminal_size() function
Antoine Pitrou pit...@free.fr added the comment: Does this need need more discussion, code review, testing, or just more time? As I already wrote, I would prefer a very simple os.get_terminal_size() function: don't read environment varaiables, use a simple tuple instead of a new type, and raise an error if the size cannot be read (so no need of default values). The os module is written as a thin wrapper between Python and the OS. A more high level function (read environment variables, handle the error, use a namedtuple) can be written in your application, or maybe in another module. I think we have reached the point where we won't be in total agreement over the API, so let's choose whatever is submitted as a patch. I only have two remaining issues with the patch: - the tests needn't be in a separate file, they can go in test_os - there should be a test for get_terminal_size_raw as well (and of course it should be skipped if the function doesn't exist) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13609 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12922] StringIO and seek()
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 03e61104f7a2 by Antoine Pitrou in branch '3.2': Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods. http://hg.python.org/cpython/rev/03e61104f7a2 New changeset f7e5abfb31ea by Antoine Pitrou in branch 'default': Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods. http://hg.python.org/cpython/rev/f7e5abfb31ea New changeset fcf4d547bed8 by Antoine Pitrou in branch '2.7': Issue #12922: fix the TextIOBase documentation to include a description of seek() and tell() methods. http://hg.python.org/cpython/rev/fcf4d547bed8 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12922] StringIO and seek()
Changes by Antoine Pitrou pit...@free.fr: -- resolution: - fixed stage: needs patch - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13609] Add os.get_terminal_size() function
Giampaolo Rodola' g.rod...@gmail.com added the comment: read environment varaiables [...] and raise an error if the size cannot be read (so no need of default values). The os module is written as a thin wrapper between Python and the OS. A more high level function (read environment variables, handle the error, use a namedtuple) can be written in your application, or maybe in another module. +1. I also find weird that a function, especially one living in the os module, has such a high level of abstraction (basically this is why I was originally proposing shutil module for this to go in). Given the different opinions about the API, I think it's best to expose the lowest level functionality as-is, and let the user decide what to do (read env vars first, suppress the exception, use a fallback, etc.). I think we have reached the point where we won't be in total agreement over the API, so let's choose whatever is submitted as a patch. I'd be more careful. Once this gets in it will be too late for a change. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13609 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13816] Two typos in the docs
Stefan Krah stefan-use...@bytereef.org added the comment: ... with *n*th (italic n) as alternate form Knuth uses that in TAOCP, too. I think with or without italics it's the most frequently used form overall. Also the Lisp function is called nth and not n-th, even though in Lisp it is possible to use hyphens in function names. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13816 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: Well, the old attempt was hardly robust :) Can anyone see any vulnerabilities in this approach? Yeah; I was mostly trying to add raw data (to help me debug the implementation). I wonder if the dict statistics should be exposed with extra attributes or a method on the dict; e.g. a __stats__ attribute, something like this: LargeDictStats(keys=58, mask=127, insertions=53, iterations=1697) SmallDictStats(keys=3, mask=7) or somesuch. Though that's a detail, I think. Caveats: * no overflow handling (what happens after 2**32 modifications to a long-lived dict on a 32-bit build?) - though that's fixable. How do you suggest to fix it? If the dict is heading towards overflow of these counters, it's either long-lived, or *huge*. Possible approaches: (a) use 64-bit counters rather than 32-bit, though that's simply delaying the inevitable (b) when one of the counters gets large, divide both of them by a constant (e.g. 2). We're interested in their ratio, so dividing both by a constant preserves this. By a constant do you mean from the perspective of big-O notation, or do you mean that it should be hardcoded (I was wondering if it should be a sys variable/environment variable etc?). We're interested in the actual slowdown factor, which a constant factor models adequately. It's the slowdown factor which makes a DOS attack using this technique efficient. Whether or not dict construction truely degenerates into a O(n**2) operation is less relevant. OK. There needs to be a way to disable it: an environment variable would be the minimum IMO. e.g. set it to 0 to enable it, set it to nonzero to set the scale factor. Any idea what to call it? PYTHONALGORITHMICCOMPLEXITYTHRESHOLD=0 would be quite a mouthful. OK BTW, presumably if we do it, we should do it for sets as well? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: I wonder if the dict statistics should be exposed with extra attributes or a method on the dict; e.g. a __stats__ attribute, something like this: LargeDictStats(keys=58, mask=127, insertions=53, iterations=1697) SmallDictStats(keys=3, mask=7) Sounds a bit overkill, and it shouldn't be a public API (which __methods__ are). Even a private API on dicts would quickly become visible, since dicts are so pervasive. Caveats: * no overflow handling (what happens after 2**32 modifications to a long-lived dict on a 32-bit build?) - though that's fixable. How do you suggest to fix it? If the dict is heading towards overflow of these counters, it's either long-lived, or *huge*. Possible approaches: (a) use 64-bit counters rather than 32-bit, though that's simply delaying the inevitable Well, even assuming one billion lookup probes per second on a single dictionary, the inevitable will happen in 584 years with a 64-bit counter (but only 4 seconds with a 32-bit counter). A real issue, though, may be the cost of 64-bit arithmetic on 32-bit CPUs. (b) when one of the counters gets large, divide both of them by a constant (e.g. 2). We're interested in their ratio, so dividing both by a constant preserves this. Sounds good, although we may want to pull this outside of the critical loop. By a constant do you mean from the perspective of big-O notation, or do you mean that it should be hardcoded (I was wondering if it should be a sys variable/environment variable etc?). Hardcoded, as in your patch. There needs to be a way to disable it: an environment variable would be the minimum IMO. e.g. set it to 0 to enable it, set it to nonzero to set the scale factor. 0 to enable it sounds misleading. I'd say: - 0 to disable it - 1 to enable it and use the default scaling factor - = 2 to enable it and set the scaling factor Any idea what to call it? PYTHONDICTPROTECTION ? Most people should either enable or disable it, not change the scaling factor. BTW, presumably if we do it, we should do it for sets as well? Yeah, and use the same env var / sys function. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8052] subprocess close_fds behavior should only close open fds
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 61aa484a3e54 by Gregory P. Smith in branch '3.2': Fixes issue #8052: The posix subprocess module's close_fds behavior was http://hg.python.org/cpython/rev/61aa484a3e54 New changeset 8879874d66a2 by Gregory P. Smith in branch 'default': Fixes issue #8052: The posix subprocess module's close_fds behavior was http://hg.python.org/cpython/rev/8879874d66a2 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8052 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: On Sat, 2012-01-21 at 22:20 +, Antoine Pitrou wrote: Sounds a bit overkill, and it shouldn't be a public API (which __methods__ are). Even a private API on dicts would quickly become visible, since dicts are so pervasive. Fair enough. Caveats: * no overflow handling (what happens after 2**32 modifications to a long-lived dict on a 32-bit build?) - though that's fixable. How do you suggest to fix it? If the dict is heading towards overflow of these counters, it's either long-lived, or *huge*. Possible approaches: (a) use 64-bit counters rather than 32-bit, though that's simply delaying the inevitable Well, even assuming one billion lookup probes per second on a single dictionary, the inevitable will happen in 584 years with a 64-bit counter (but only 4 seconds with a 32-bit counter). A real issue, though, may be the cost of 64-bit arithmetic on 32-bit CPUs. (b) when one of the counters gets large, divide both of them by a constant (e.g. 2). We're interested in their ratio, so dividing both by a constant preserves this. Sounds good, although we may want to pull this outside of the critical loop. OK; I'll look at implementing (b). Oops, yeah, that was a typo; I meant 0 to disable. - 0 to disable it - 1 to enable it and use the default scaling factor - = 2 to enable it and set the scaling factor You said above that it should be hardcoded; if so, how can it be changed at run-time from an environment variable? Or am I misunderstanding. Works for me. BTW, presumably if we do it, we should do it for sets as well? Yeah, and use the same env var / sys function. Despite the DICT in the title? OK. Thanks for the feedback. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: You said above that it should be hardcoded; if so, how can it be changed at run-time from an environment variable? Or am I misunderstanding. You're right, I used the wrong word. I meant it should be a constant independently of the dict size. But, indeed, not hard-coded in the source. BTW, presumably if we do it, we should do it for sets as well? Yeah, and use the same env var / sys function. Despite the DICT in the title? OK. Well, dict is the most likely target for these attacks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13609] Add os.get_terminal_size() function
Antoine Pitrou pit...@free.fr added the comment: +1. I also find weird that a function, especially one living in the os module, has such a high level of abstraction (basically this is why I was originally proposing shutil module for this to go in). Given the different opinions about the API, I think it's best to expose the lowest level functionality as-is, and let the user decide what to do (read env vars first, suppress the exception, use a fallback, etc.). Fair enough, but other people expressed sympathy for the two-function approach :) I'm personally indifferent, although I find get_terminal_size_raw a bit ugly and liked query_terminal_size better. (and looking up ROWS and COLUMNS make sense, since they are de facto standards) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13609 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8052] subprocess close_fds behavior should only close open fds
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 780992c9afea by Gregory P. Smith in branch '3.2': Add a Misc/NEWS entry for issue 8052. http://hg.python.org/cpython/rev/780992c9afea New changeset 1f0a01dc723c by Gregory P. Smith in branch 'default': A Misc/NEWS entry for issue 8052. http://hg.python.org/cpython/rev/1f0a01dc723c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8052 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13405] Add DTrace probes
Antoine Pitrou pit...@free.fr added the comment: So, yes. The code is intrusive. The code deals with a lot of internal machinery (PEP393 support in the ustack helper was quite difficult). It is going to break from time to time, sure. At the same time, I am committed to support it. And even if it is dropped in 3.4, no Python program will be affected. To ease the concerns, I think you should make it so that dtrace-specific code gets out of the way as much as possible. I suggest you create a Python/ceval-dtrace.h header and put most dtrace-specific code from ceval.c there. Its inclusion should be conditional on WITH_DTRACE so that other core devs can ignore its presence. A couple other comments: - in the makefile, DTRACEOBJS is inconsistent with DTRACE_STATIC and LIBRARY_OBJS. You should make it DTRACE_OBJS. - please add comments at the top of whatever header files you add, to make it clear that they are dtrace-specific. Mentions of ustack helper are a bit too specific to be helpful. - some code lacks error checking, e.g. when calling PyUnicode_AsUTF8. - is co_linenos ever freed or is it a memory leak? - your indices and offsets should be Py_ssize_t, not int - is an empty dtrace module really needed? a flag variable in the sys module should be enough - as you can see the Makefile uses -rm -f, you should probably do the same instead of rm -f - you have a rather strange if true in your configure.in additions Thanks in advance. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13405 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8052] subprocess close_fds behavior should only close open fds
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset d0acd8169c2a by Gregory P. Smith in branch '3.2': Bugfix for issue #8052 fix on *BSD variants. http://hg.python.org/cpython/rev/d0acd8169c2a New changeset 5be3dadd2eef by Gregory P. Smith in branch '3.2': Another issue #8052 bugfix (related to previous commit). http://hg.python.org/cpython/rev/5be3dadd2eef New changeset e52d81e0c750 by Gregory P. Smith in branch 'default': bugfix for issue 8052 fixes on *BSD platforms. http://hg.python.org/cpython/rev/e52d81e0c750 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8052 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Gregory P. Smith g...@krypto.org added the comment: On Sat, Jan 21, 2012 at 2:45 PM, Antoine Pitrou rep...@bugs.python.org wrote: Antoine Pitrou pit...@free.fr added the comment: You said above that it should be hardcoded; if so, how can it be changed at run-time from an environment variable? Or am I misunderstanding. You're right, I used the wrong word. I meant it should be a constant independently of the dict size. But, indeed, not hard-coded in the source. BTW, presumably if we do it, we should do it for sets as well? Yeah, and use the same env var / sys function. Despite the DICT in the title? OK. Well, dict is the most likely target for these attacks. While true I wouldn't make that claim as there will be applications using a set in a vulnerable manner. I'd prefer to see any such environment variable name used to configure this behavior not mention DICT or SET but just say HASHTABLE. That is a much better bikeshed color. ;) I'm still in the hash seed randomization camp but I'm finding it interesting all of the creative ways others are trying to solve this problem in a way that could be enabled by default in stable versions regardless. :) -gps -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Alex Gaynor alex.gay...@gmail.com added the comment: On Sat, Jan 21, 2012 at 5:42 PM, Gregory P. Smith rep...@bugs.python.orgwrote: Gregory P. Smith g...@krypto.org added the comment: On Sat, Jan 21, 2012 at 2:45 PM, Antoine Pitrou rep...@bugs.python.org wrote: Antoine Pitrou pit...@free.fr added the comment: You said above that it should be hardcoded; if so, how can it be changed at run-time from an environment variable? Or am I misunderstanding. You're right, I used the wrong word. I meant it should be a constant independently of the dict size. But, indeed, not hard-coded in the source. BTW, presumably if we do it, we should do it for sets as well? Yeah, and use the same env var / sys function. Despite the DICT in the title? OK. Well, dict is the most likely target for these attacks. While true I wouldn't make that claim as there will be applications using a set in a vulnerable manner. I'd prefer to see any such environment variable name used to configure this behavior not mention DICT or SET but just say HASHTABLE. That is a much better bikeshed color. ;) I'm still in the hash seed randomization camp but I'm finding it interesting all of the creative ways others are trying to solve this problem in a way that could be enabled by default in stable versions regardless. :) -gps -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ I'm a little slow, so bear with me, but David, does this counting scheme in any way address the issue of: I'm able to put N pieces of data into the database on successive requests, but then *rendering* that data puts it in a dictionary, which renders that page unviewable by anyone. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13609] Add os.get_terminal_size() function
Denilson Figueiredo de Sá denilso...@gmail.com added the comment: On Sat, Jan 21, 2012 at 17:40, Giampaolo Rodola' rep...@bugs.python.org wrote: Given the different opinions about the API, I think it's best to expose the lowest level functionality as-is, and let the user decide what to do (read env vars first, suppress the exception, use a fallback, etc.). As a Python user (and not a committer), I disagree. As an user, I don't care too much where the function should be placed (although I believe os or sys are sensible choices). What I do care is that I want a extremely simple function that will just work. Don't make me add code for handling all the extra cases, such code should be inside the function. All this discussion about the API made me remember this presentation: http://python-for-humans.heroku.com/ Also, I see no downside of using a Named Tuple. Issue 4285 actually added a named tuple to the sys.version_info. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13609 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: 5 more characters: PYTHONHASHTABLEPROTECTION or PYHASHTABLEPROTECTION maybe? I'm in *both* camps: I like hash seed randomization fwiw. I'm nervous about enabling either of the approaches by default, but I can see myself backporting both approaches into RHEL's ancient Python versions, compiled in, disabled by default, but available at runtime via env vars (assuming that no major flaws are discovered in my patch e.g. performance). I'm sorry if I'm muddying the waters by working on this approach. Is the hash randomization approach ready to go, or is more work needed? If the latter, is there a clear TODO list? (for backporting to 2.*, presumably we'd want PyStringObject to be randomized; I think this means that PyBytesObject needs to be randomized also in 3.*; don't we need hash(b'foo') == hash('foo') ?). Does the patch needs to also randomize the hashes of the numeric types? (I think not; that may break too much 3rd-party code (NumPy?)). [If we're bikeshedding, I prefer the term salt to seed in the hash randomization approach: there's a per-process hash salt, which is either randomly generated, or comes from the environment, set to 0 to disable] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13790] In str.format an incorrect error message for list, tuple, dict, set
Terry J. Reedy tjre...@udel.edu added the comment: Looking further, I noticed that 'string' needed to be changed to 'specification' in the following sentence also. Then I decided that the preceding sentence Most built-in types implement the following options for format specifications, although some of the formatting options are only supported by the numeric types. should really follow the one about non-empty format specs. This positioning should make it more obvious that most of the options affect the string representation of the object after, not before, the string is produced, and are therefore applicable to all objects and not just string and number objects. I also propose to modify it so it is shorter and no longer contradictory, to read Most built-in types implement various options for such modifications, although some are only supported by the numeric types. Further on, under The available string presentation types are: I think ``'s'`` String format. This is the default type for strings and may be omitted. should have 'and other non-numeric types ' inserted after strings. New patch i13790b.diff attached The point of these additional changes is to make it clearer that the default formatting of non-number, non-string objects is to call str() and then apply the options to the resulting string. That makes something like format(range(5), '-^20s') # same with object.__format__(), 3.3.0a0 'range(0, 5)-' predictable and comprehensible. I agree with not making a temporary change (but see below ;-). But it seems that the 3.4 message should at least be numeric format string passed to object.__format__ or format string with number-only options passed to object.__format__ or object.__format__ cannot handle number-only options as string formats work fine and, I presume, are not deprecated (?). However, if the new ValueError message did not specify object.__format__ (which could still be confusing, even if more accurate), the change could be make now. For instance 'Numeric option 'd' for non-number object'. It would not really matter if it is later raised in object.__format__ instead of str.__format__. I believe *all* of the format codes 'unknown' to str (and by extension, by default, to all other non-number types) *are* number codes. -- assignee: docs@python - terry.reedy Added file: http://bugs.python.org/file24290/i13790b.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13790 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Paul McMillan p...@mcmillan.ws added the comment: On Sat, Jan 21, 2012 at 3:47 PM, Alex Gaynor rep...@bugs.python.org wrote: I'm able to put N pieces of data into the database on successive requests, but then *rendering* that data puts it in a dictionary, which renders that page unviewable by anyone. This and the problems Frank mentions are my primary concerns about the counting approach. Without the original suggestion of modifying the hash and continuing without an exception (which has its own set of problems), the valid data python can't process problem is a pretty big one. Allowing attackers to poison interactions for other users is unacceptable. The other thing I haven't seen mentioned yet is that while it is true that most web applications do have robust error handling to produce proper 500s, an unexpected error will usually result in restarting the server process - something that can carry significant weight by itself. I would consider it a serious problem if every attack request required a complete application restart, a la original cgi. I'm strongly in favor of randomization. While there are many broken applications in the wild that depend on dictionary ordering, if we ship with this feature disabled by default for security and bugfix branches, and enable it for 3.3, users can opt-in to protection as they need it and as they fix their applications. Users who have broken applications can still safely apply the security fix (without even reading the release notes) because it won't change the default behavior. Distro managers can make an appropriate choice for their user base. Most importantly, it negates the entire compute once, attack everywhere class of collision problems, even if we haven't explicitly discovered them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13783] Clean up PEP 380 C API additions
Meador Inge mead...@gmail.com added the comment: 'PyStopIteration_Create' is just a trivial wrapper: PyObject * PyStopIteration_Create(PyObject *value) { return PyObject_CallFunctionObjArgs(PyExc_StopIteration, value, NULL); } It is not needed. As for 'PyGen_FetchStopIterationValue', does it really need to be public? It is trivial to make it private because all calls to it are in 'genobject.c'. However, I am not sure if there is a strong use case for having it public. -- nosy: +meador.inge stage: - needs patch type: - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11551] test_dummy_thread.py test coverage improvement
Denver Coneybeare denver.coneybe...@gmail.com added the comment: I've looked at the review (thanks for the review) and can submit an updated patch. I don't have the Python source code pulled down to my PC anymore so it might take a week or two before I'm able to update the patch and test it out. I imagine that's not too much of a problem though :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11551 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8052] subprocess close_fds behavior should only close open fds
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 754c2eb0a92c by Gregory P. Smith in branch '3.2': Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix. http://hg.python.org/cpython/rev/754c2eb0a92c New changeset 7d4658a8de96 by Gregory P. Smith in branch 'default': Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix. http://hg.python.org/cpython/rev/7d4658a8de96 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8052 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8052] subprocess close_fds behavior should only close open fds
Gregory P. Smith g...@krypto.org added the comment: For FreeBSD, Python 3.2 and 3.3 now check to see if /dev/fd is valid. Be sure and mount -t fdescfs none /dev/fd on FreeBSD if you want faster subprocess launching. Run a FreeBSD buildbot? Please do it! For Python 3.1 the fix for #13788 would fix this, but I believe 3.1 is in security fix only mode at this point so we're not going to backport that os.closerange change that far. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8052 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13814] Document why generators don't support the context management protocol
Nick Coghlan ncogh...@gmail.com added the comment: Generators deliberately don't support the context management protocol. This is so that they raise an explicit TypeError or AttributeError (pointing out that __exit__ is missing) if you leave out the @contextmanager decorator when you're using a generator to write an actual context manager. Generators supporting the context management protocol natively would turn that into a far more subtle (and confusing) error: your code would silently fail to invoke the generator body. Ensuring this common error remains easy to detect is far more important than making it easier to invoke close() on a generator object (particularly when contextlib.closing() already makes that very easy). -- assignee: - docs@python components: +Documentation nosy: +docs@python stage: test needed - needs patch status: pending - open title: Generators as context managers. - Document why generators don't support the context management protocol ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13814 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com