[issue46794] Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs)
Michał Górny added the comment: Could you make a PR to fix the test failures? I suppose that could speed things up and if not, I'd at least have something to pull into Gentoo. -- ___ Python tracker <https://bugs.python.org/issue46794> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46794] Please update bundled libexpat to 2.4.5 with security fixes (5 CVEs)
Michał Górny added the comment:
BTW there are test regressions with expat 2.4.5, apparently due to some test
snippets now being rejected as invalid XML:
==
ERROR: test_issue3151 (test.test_xml_etree.BugsTest)
--
Traceback (most recent call last):
File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1718, in
feed
self.parser.Parse(data, False)
^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mgorny/git/cpython/Lib/test/test_xml_etree.py", line 2196, in
test_issue3151
e = ET.XML('')
^
File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1347, in
XML
parser.feed(text)
^
File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1720, in
feed
self._raiseerror(v)
^^^
File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1627, in
_raiseerror
raise err
^
xml.etree.ElementTree.ParseError: syntax error: line 1, column 0
==
ERROR: testEncodings (test.test_minidom.MinidomTest)
--
Traceback (most recent call last):
File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1150, in
testEncodings
self.assertRaises(UnicodeDecodeError, parseString,
^^
File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 734, in
assertRaises
return context.handle('assertRaises', args, kwargs)
File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 218, in handle
callable_obj(*args, **kwargs)
^
File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in
parseString
return expatbuilder.parseString(string)
File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in
parseString
return builder.parseString(string)
^^^
File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in
parseString
parser.Parse(string, True)
^^
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 5
==
ERROR: testExceptionOnSpacesInXMLNSValue (test.test_minidom.MinidomTest)
--
Traceback (most recent call last):
File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1613, in
testExceptionOnSpacesInXMLNSValue
parseString('')
File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in
parseString
return expatbuilder.parseString(string)
File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in
parseString
return builder.parseString(string)
^^^
File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in
parseString
parser.Parse(string, True)
^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0
--
nosy: +mgorny
___
Python tracker
<https://bugs.python.org/issue46794>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40379] multiprocessing's default start method of fork()-without-exec() is broken
Michał Górny added the comment: After updating PyPy3 to use Python 3.9's stdlib, we hit very bad hangs because of this — literally compiling a single file with "parallel" compileall could hang. In the end, we had to revert the change in how Python 3.9 starts workers because otherwise multiprocessing would be impossible to use: https://foss.heptapod.net/pypy/pypy/-/commit/c594b6c48a48386e8ac1f3f52d4b82f9c3e34784 This is a very bad default and what's even worse is that it often causes deadlocks that are hard to reproduce or debug. Furthermore, since "fork" is the default, people are unintentionally relying on its support for passing non-pickleable projects and are creating non-portable code. The code often becomes complex and hard to change before they discover the problem. Before we managed to figure out how to workaround the deadlocks in PyPy3, we were experimenting with switching the default to "spawn". Unfortunately, we've hit multiple projects that didn't work with this method, precisely because of pickling problems. Furthermore, they were surprised to learn that their code wouldn't work on macOS (in the end, many people perceive Python as a language for writing portable software). Finally, back in 2018 I've made one of my projects do parallel work using multiprocessing. It gave its users great speedup but for some it caused deadlocks that I couldn't reproduce nor debug. In the end, I had to revert it. Now that I've learned about this problem, I'm wondering if this wasn't precisely because of "fork" method. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue40379> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Michał Górny added the comment: I would still find it helpful to have a "proper" "blake3-c" package on normal pypi, for those of us who can't rely on Rust being present immediately. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40059] Provide a toml module in the standard library
Michał Górny added the comment: I've already asked that, and the author said it's fine but with the deprecated text file support removed: https://github.com/hukkin/tomli/issues/141#issuecomment-968056905 That said, for consistency with json and so on we'd probably want to combine tomli and tomli_w to get dumping support as well. -- ___ Python tracker <https://bugs.python.org/issue40059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40059] Provide a toml module in the standard library
Michał Górny added the comment: Is there any progress happening? FWIU the common "toml" module on pypi has been discontinued now, projects are switching to yet another one and this is exactly the kind of problem a built-in module would have avoided. -- ___ Python tracker <https://bugs.python.org/issue40059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Michał Górny added the comment: Jack, are you still working on this? I was considering allocating the time to write the bindings for the C library myself but I've stumbled upon this bug and I suppose there's no point in duplicating work. I'd love to see it on pypi, so we could play with it a bit. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Michał Górny added the comment:
The test added for this bug is insufficient to verify the fix. If I revert the
Lib/http/client.py change, the test still passes. This is because a subclass
of client.HTTPException is still raised.
If I add an explicit begin() call to trigger the exception, then without the
fix I get:
File "/tmp/cpython/Lib/test/test_httplib.py", line 1189, in
test_overflowing_header_limit_after_100
resp.begin()
File "/tmp/cpython/Lib/http/client.py", line 308, in begin
version, status, reason = self._read_status()
File "/tmp/cpython/Lib/http/client.py", line 277, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
With the fix, I get (correctly):
test test_httplib failed -- Traceback (most recent call last):
File "/tmp/cpython/Lib/test/test_httplib.py", line 1189, in
test_overflowing_header_limit_after_100
resp.begin()
File "/tmp/cpython/Lib/http/client.py", line 321, in begin
skipped_headers = _read_headers(self.fp)
File "/tmp/cpython/Lib/http/client.py", line 218, in _read_headers
raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers
However, the test considers both exceptions to match.
--
nosy: +mgorny
___
Python tracker
<https://bugs.python.org/issue44022>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Michał Górny added the comment: In my opinion, raising an exception would have been safer. Botocore and django do precisely what you say — provide a validator. To make this validator easier, they do the validation on splitted up URL parts. I disagree with the premise that they were stupid to rely on invalid data being passed through. I could understand if the function started rejecting invalid data. But until now, you could reasonably assume that urlsplit()'s output would correspond to its input. Making the output 'sanitized' means that invalid input is converted into valid output. This goes against the principle of least surprise. In the end, this opens us potential vulnerabilities in other packages. Imagine that something uses urlsplit() to perform the URL validation but uses the original URL elsewhere. By making the validation happen on a sanitized URL, you're effectively disarming the validator and letting bad URL through. Security is not only about fixing potential problems with your package. It's also about considering the consequences to your users. In this case, the chosen solution may actually open more vulnerabilities that it fixes. What's even worse, you're actively harming security in projects that actually attempted to solve the same problem earlier. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Michał Górny added the comment: I hate to be the bearer of bad news but I've already found this change to be breaking tests of botocore and django. In both cases, the test failure is apparently because upstream used to reject URLs after finding newlines in the split components, and now they're silently stripped away. Filed bugs: https://github.com/boto/botocore/issues/2377 https://code.djangoproject.com/ticket/32713 Note that I'm not saying the change should be reverted. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36384] [security] CVE-2021-29921: ipaddress Should not reject IPv4 addresses with leading zeroes as ambiguously octal
Michał Górny added the comment: > If it takes years for users to get to 3.10, we should reevaluate our > release cycle, not whether we aggressively break maintenance releases. I don't really understand how that would help. The problem is that users have major inertia for switching to newer Python versions. A part of it is that a lot of people just don't care about deprecation warnings, and don't fix stuff until it's actually broken. In the end, your projects are blocked from using new major Python version by broken dependencies with long release cycles. I can't imagine deliberately leaving 3.8 and 3.9 vulnerable when 3.10 isn't going to reach final release in the next half year. Gentoo stable is only switching to 3.9 next month. I'm pretty sure some of our (few) corporate users are still on 3.7 or earlier. Then, there are projects that literally include a vulnerable copy of Python 2.7 to get around distributions removing it. I dare say this has less breakage potential than the &/; change. It should be fixed on all affected versions. If you don't do that, distributions will have to patch it anyway, and this will only lead to incompatibility between different Python package vendors. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue36384> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18232] running a suite with no tests is not an error
Change by Michał Górny : -- versions: +Python 3.10 -Python 3.4 ___ Python tracker <https://bugs.python.org/issue18232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18232] running a suite with no tests is not an error
Change by Michał Górny : -- keywords: +patch pull_requests: +23655 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24893 ___ Python tracker <https://bugs.python.org/issue18232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18232] running a suite with no tests is not an error
Michał Górny added the comment: I'm not convinced we need something that complex here but I think it would make sense to make 'unittest discover' fail when it doesn't discover a single test. As packagers, we've been bitten more than once by packages whose tests suddenly stopped being discovered, and it would be really helpful if we were able to catch this automatically without having to resort to hacks. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue18232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43459] Race conditions when the same source file used to build mutliple extensions
New submission from Michał Górny : There is a race condition in distutils' build_ext implementation. When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel. This means that the link editor can grab the file while another compiler instance is overwriting it. The results vary from compile errors to cryptic dyld failures when attempting to load the module. I've created a trivial reproducer that I've attached in a patch form. For convenience, it's also available on my GitHub: https://github.com/mgorny/distutils-build_ext-race The reproducer consists of two extension modules sharing the same file. The race.sh script attempts to build the extension and then import it. The process is repeated until something fails, e.g.: + python3.10 setup.py build_ext -i -j4 running build_ext building 'bar' extension creating build building 'foo' extension creating build/temp.linux-x86_64-3.10 creating build/temp.linux-x86_64-3.10 x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/bar.cpython-310-x86_64-linux-gnu.so + python3.10 -c 'import foo; import bar' Traceback (most recent call last): File "", line 1, in ImportError: /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so: undefined symbol: call_shared + echo 'Reproduced at iteration 256' Reproduced at iteration 256 + break -- components: Distutils files: 0001-A-reproducer-for-distutils-build_ext-race-condition.patch keywords: patch messages: 388410 nosy: dstufft, eric.araujo, mgorny priority: normal severity: normal status: open title: Race conditions when the same source file used to build mutliple extensions type: compile error versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file49863/0001-A-reproducer-for-distutils-build_ext-race-condition.patch ___ Python tracker <https://bugs.python.org/issue43459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43386] test_ctypes hangs inside Portage build env since 'subprocess: Use vfork() instead of fork() [...]'
Michał Górny added the comment: Nevermind, I've been testing wrong and this is most likely our fault. I'm sorry about the noise. I am going to investigate further and reopen if it turns out the problem's on CPython end. -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43386> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43386] test_ctypes hangs inside Portage build env since 'subprocess: Use vfork() instead of fork() [...]'
New submission from Michał Górny : So I've finally found time to bisect this. Long story short, test_ctypes started hanging on Gentoo package builds since 3.10.0a2. Previously, the test took less than a second. Now, it just keeps running for minutes until I kill it. The weird thing is that I can't reproduce it when running it manually. I've tried hard to rebuild Portage-like environment to make it hang, to no avail. I've finally gotten around to bisecting it, and established that the problem is caused by the following change: ``` commit 976da903a746a5455998e9ca45fbc4d3ad3479d8 Author: Alexey Izbyshev Date: 2020-10-24 02:47:01 +0200 bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671) [...] ``` After running the test with a timeout, I get the following backtrace: ``` test_issue_8959_a (ctypes.test.test_callbacks.SampleCallbacksTestCase) ... Timeout (0:00:30)! Thread 0x7f72f2507740 (most recent call first): File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/subprocess.py", line 1773 in _execute_child File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/subprocess.py", line 962 in __init__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/ctypes/util.py", line 289 in _findSoname_ldconfig File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/ctypes/util.py", line 329 in find_library File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/ctypes/test/test_callbacks.py", line 183 in test_issue_8959_a File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/case.py", line 549 in _callTestMethod File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/case.py", line 592 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/case.py", line 652 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 122 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 84 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 122 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 84 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 122 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 84 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 122 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 84 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 122 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/suite.py", line 84 in __call__ File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/unittest/runner.py", line 176 in run File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/support/__init__.py", line 959 in _run_suite File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/support/__init__.py", line 1082 in run_unittest File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/runtest.py", line 211 in _test_module File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/runtest.py", line 236 in _runtest_inner2 File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/runtest.py", line 272 in _runtest_inner File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/runtest.py", line 155 in _runtest File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/runtest.py", line 195 in runtest File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/main.py", line 319 in rerun_failed_tests File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/main.py", line 696 in _main File "/var/tmp/portage/dev-lang/python-3.10.0_alpha6/work/Python-3.10.0a6/Lib/test/libregrtest/main.py", line 639 in main File "/var/tmp/portage/
[issue40059] Provide a toml module in the standard library
Michał Górny added the comment: It seems that the spec has reached 1.0.0: > 1.0.0 / 2021-01-11 I guess that there are still ~2 months left to do this before 3.10 reaches beta. -- ___ Python tracker <https://bugs.python.org/issue40059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42697] 3.8.7rc1 regression: 'free(): invalid pointer' after running backports-zoneinfo test suite
Michał Górny added the comment: Thank you for debugging this. I can confirm that this patch resolves my issue. -- ___ Python tracker <https://bugs.python.org/issue42697> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42697] 3.8.7rc1 regression: 'free(): invalid pointer' after running backports-zoneinfo test suite
Michał Górny added the comment: A more complete backtrace: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49 #1 0x7fa633b20536 in __GI_abort () at abort.c:79 #2 0x7fa633b79bf7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fa633c8c3b5 "%s\n") at ../sysdeps/posix/libc_fatal.c:155 #3 0x7fa633b81a7a in malloc_printerr (str=str@entry=0x7fa633c8a593 "free(): invalid pointer") at malloc.c:5389 #4 0x7fa633b82e5c in _int_free (av=, p=, have_lock=0) at malloc.c:4201 #5 0x55ea845c6170 in _PyMem_RawFree (ctx=0x0, ptr=0x55ea848c1a60 ) at Objects/obmalloc.c:127 #6 0x55ea845c6f46 in PyMem_RawFree (ptr=0x55ea848c1a60 ) at Objects/obmalloc.c:595 #7 0x55ea845c7fa0 in _PyObject_Free (ctx=0x0, p=0x55ea848c1a60 ) at Objects/obmalloc.c:1898 #8 0x55ea845c726e in PyObject_Free (ptr=0x55ea848c1a60 ) at Objects/obmalloc.c:709 #9 0x55ea845dda33 in object_dealloc (self=0x55ea848c1a60 ) at Objects/typeobject.c:3784 #10 0x55ea845c60a5 in _Py_Dealloc (op=0x55ea848c1a60 ) at Objects/object.c:2215 #11 0x55ea845d3b55 in _Py_DECREF (filename=0x55ea847a8de0 "./Include/object.h", lineno=541, op=0x55ea848c1a60 ) at ./Include/object.h:478 #12 0x55ea845d3b83 in _Py_XDECREF (op=0x55ea848c1a60 ) at ./Include/object.h:541 #13 0x55ea845d4627 in tupledealloc (op=0x7fa63195d6c0) at Objects/tupleobject.c:247 #14 0x55ea845c60a5 in _Py_Dealloc (op=0x7fa63195d6c0) at Objects/object.c:2215 #15 0x55ea84743afb in _Py_DECREF (filename=0x55ea8482b9a0 "./Include/object.h", lineno=541, op=0x7fa63195d6c0) at ./Include/object.h:478 #16 0x55ea84743b29 in _Py_XDECREF (op=0x7fa63195d6c0) at ./Include/object.h:541 #17 0x55ea84745b5a in code_dealloc (co=0x7fa631b68450) at Objects/codeobject.c:552 #18 0x55ea845c60a5 in _Py_Dealloc (op=0x7fa631b68450) at Objects/object.c:2215 #19 0x55ea847525d2 in _Py_DECREF (filename=0x55ea8482e47c "Objects/funcobject.c", lineno=576, op=0x7fa631b68450) at ./Include/object.h:478 #20 0x55ea84753b6a in func_clear (op=0x7fa631b768b0) at Objects/funcobject.c:576 #21 0x55ea84753de0 in func_dealloc (op=0x7fa631b768b0) at Objects/funcobject.c:597 #22 0x55ea845c60a5 in _Py_Dealloc (op=0x7fa631b768b0) at Objects/object.c:2215 #23 0x55ea845addff in _Py_DECREF (filename=0x55ea847a3320 "./Include/object.h", lineno=541, op=0x7fa631b768b0) at ./Include/object.h:478 #24 0x55ea845ade4f in _Py_XDECREF (op=0x7fa631b768b0) at ./Include/object.h:541 #25 0x55ea845aeed8 in free_keys_object (keys=0x55ea84e72580) at Objects/dictobject.c:584 #26 0x55ea845ae524 in dictkeys_decref (dk=0x55ea84e72580) at Objects/dictobject.c:324 #27 0x55ea845b1c9c in PyDict_Clear (op=0x7fa63198b740) at Objects/dictobject.c:1729 #28 0x55ea845dd683 in type_clear (type=0x55ea84e12ba0) at Objects/typeobject.c:3594 #29 0x55ea845d8720 in subtype_clear (self=0x55ea84e12ba0) at Objects/typeobject.c:1147 #30 0x55ea846ac4aa in delete_garbage (state=0x55ea848dd6f8 <_PyRuntime+344>, collectable=0x7fff13eb2e10, old=0x55ea848dd740 <_PyRuntime+416>) at Modules/gcmodule.c:948 #31 0x55ea846aca23 in collect (state=0x55ea848dd6f8 <_PyRuntime+344>, generation=2, n_collected=0x0, n_uncollectable=0x0, nofail=1) at Modules/gcmodule.c:1123 #32 0x55ea846ae0da in _PyGC_CollectNoFail () at Modules/gcmodule.c:1866 #33 0x55ea846616da in PyImport_Cleanup () at Python/import.c:599 #34 0x55ea8467df93 in Py_FinalizeEx () at Python/pylifecycle.c:1233 #35 0x55ea84577e8a in Py_RunMain () at Modules/main.c:691 #36 0x55ea84577f22 in pymain_main (args=0x7fff13eb2ff0) at Modules/main.c:719 #37 0x55ea84577fe2 in Py_BytesMain (argc=6, argv=0x7fff13eb3138) at Modules/main.c:743 #38 0x55ea84576785 in main (argc=6, argv=0x7fff13eb3138) at ./Programs/python.c:16 -- ___ Python tracker <https://bugs.python.org/issue42697> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42697] 3.8.7rc1 regression: 'free(): invalid pointer' after running backports-zoneinfo test suite
Michał Górny added the comment: aeb66c1abbf4ec214e2e80eb972546996d1a1571 is the first bad commit commit aeb66c1abbf4ec214e2e80eb972546996d1a1571 Author: Miss Skeleton (bot) <31488909+miss-isling...@users.noreply.github.com> Date: Thu Oct 15 08:51:48 2020 -0700 bpo-41984: GC track all user classes (GH-22701/GH-22707) (cherry picked from commit c13b847a6f913b72eeb71651ff626390b738d973) Lib/test/test_finalization.py | 23 -- Lib/test/test_gc.py| 6 +++--- .../2020-10-14-16-19-43.bpo-41984.SEtKMr.rst | 2 ++ Modules/_testcapimodule.c | 20 +++ Objects/typeobject.c | 22 ++--- 5 files changed, 52 insertions(+), 21 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2020-10-14-16-19-43.bpo-41984.SEtKMr.rst -- ___ Python tracker <https://bugs.python.org/issue42697> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42697] 3.8.7rc1 regression: 'free(): invalid pointer' after running backports-zoneinfo test suite
New submission from Michał Górny :
I'm still investigating the problem and I will include more information
shortly. However, I'm filing the bug early, as I'd like to prevent this
regression from hitting 3.8.7 release.
When running backports-zoneinfo-0.2.1 test suite using cpython 3.8.7rc1, all
tests pass, then python segfaults:
```
--
Ran 233 tests in 2.200s
OK (skipped=27)
free(): invalid pointer
/var/tmp/portage/dev-python/backports-zoneinfo-0.2.1/temp/environment: line
3054: 167 Aborted (core dumped) "${EPYTHON}" -m unittest
discover -v
```
The backtrace I got doesn't seem very useful:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1 0x7fd4b6c79536 in __GI_abort () at abort.c:79
#2 0x7fd4b6cd2bf7 in __libc_message (action=action@entry=do_abort,
fmt=fmt@entry=0x7fd4b6de53b5 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x7fd4b6cdaa7a in malloc_printerr (str=str@entry=0x7fd4b6de3593
"free(): invalid pointer") at malloc.c:5389
#4 0x7fd4b6cdbe5c in _int_free (av=, p=,
have_lock=0) at malloc.c:4201
#5 0x7fd4b6f00aaa in ?? () from /usr/lib64/libpython3.8.so.1.0
#6 0x7fd4b6eb8745 in ?? () from /usr/lib64/libpython3.8.so.1.0
#7 0x7fd4b6ece115 in ?? () from /usr/lib64/libpython3.8.so.1.0
#8 0x7fd4b6ece2f2 in ?? () from /usr/lib64/libpython3.8.so.1.0
#9 0x562239cd1a60 in ?? ()
#10 0x7fd4b7086967 in ?? () from /usr/lib64/libpython3.8.so.1.0
#11 0x7fd4b7167e20 in ?? () from /usr/lib64/libpython3.8.so.1.0
#12 0x562239cd1a60 in ?? ()
#13 0x7fd4b6f05d26 in ?? () from /usr/lib64/libpython3.8.so.1.0
#14 0x7fd4b6fccf0d in ?? () from /usr/lib64/libpython3.8.so.1.0
#15 0x7fd4b6fcdc1d in PyGC_Collect () from /usr/lib64/libpython3.8.so.1.0
#16 0x56223996c670 in ?? ()
#17 0x7fd4b6f93e8a in PyImport_Cleanup () from
/usr/lib64/libpython3.8.so.1.0
#18 0x7fd4b6faa55c in Py_NewInterpreter () from
/usr/lib64/libpython3.8.so.1.0
#19 0x in ?? ()
I'm going to start by trying to bisect this, and let you know the results when
I manage them.
--
components: Interpreter Core
messages: 383464
nosy: mgorny
priority: normal
severity: normal
status: open
title: 3.8.7rc1 regression: 'free(): invalid pointer' after running
backports-zoneinfo test suite
type: crash
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue42697>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40791] hmac.compare_digest could try harder to be constant-time.
Michał Górny added the comment: Any reason this wasn't backported to 3.6? FWICS it's supposed to be security supported still. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue40791> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41725] bz2 would use a dedicated exception for data error
Michał Górny added the comment:
> An OSError for a file the OS can read seems a bit off. What is the
> accompanying message?
For example:
OSError: Invalid data stream
(I've just put random string into a file, and opened it as bzip2)
> Would something like ValueError("bz2 compressor failed with BZ_DATA_ERROR")
> be better?
It would be a partial improvement. However, it would also break backwards
compatibility with code already catching OSError. It might be better to go
with custom exception type derived from OSError to keep existing code working.
> Michał, are there precedents or models for this in the other file compression
> modules?
Python 3.8 has introduced BadGzipFile in gzip module, to replace their OSError.
lzma has used LZMAError from day one.
--
___
Python tracker
<https://bugs.python.org/issue41725>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41725] bz2 would use a dedicated exception for data error
New submission from Michał Górny : Currently, the bz2 extension functions raise a generic OSError() on BZ_DATA_ERROR and BZ_DATA_ERROR_MAGIC. When working on files, this makes it unnecessarily hard to catch compressor errors (i.e. invalid bz2 input file) without catching system I/O errors. Could you please consider adding a new exception type for bz2 errors, akin to gzip.BadGzipFile? -- components: Extension Modules messages: 376427 nosy: mgorny priority: normal severity: normal status: open title: bz2 would use a dedicated exception for data error type: enhancement ___ Python tracker <https://bugs.python.org/issue41725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39017] Infinite loop in the tarfile module
Michał Górny added the comment: Given that a CVE was assigned for this, I think it'd be better if the news were in the 'Security' category and not 'Library'. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue39017> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35247] test.test_socket.RDSTest.testPeek hangs indefinitely
Michał Górny added the comment: I can reproduce test_socket hanging in 2 out of 3 runs. However, in my case testPeek doesn't seem to be the (only) culprit. Disabling whole RDSTest helps. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue35247> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27657] urlparse fails if the path is numeric
Michał Górny added the comment: I wonder if it would be feasible to support new behavior in earlier versions of Python via __future__. I suppose that could help software avoid having Python version-dependent behavior in the long run. -- ___ Python tracker <https://bugs.python.org/issue27657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: Thank you! -- ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27657] urlparse fails if the path is numeric
Michał Górny added the comment: I'm sorry but does this change mean that it's not final or...? My main concern is whether we should be adjusting our packages to the new behavior in py3.9, or wait for further changes. -- ___ Python tracker <https://bugs.python.org/issue27657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27657] urlparse fails if the path is numeric
Michał Górny added the comment: Do I understand correctly that the new behavior is intentional in 3.9, or is that still being discussed? -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue27657> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: Ok, it seems that I misunderstood it at first. Judging by the code, it also affects extensions installed into site-packages, so I've tried to make that clear and add one more example to the bullet list. -- ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE
Change by Michał Górny : -- pull_requests: +19600 pull_request: https://github.com/python/cpython/pull/20332 ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: > Can you please propose a different wording? English is not my first language. Mine neither but I'll try. How about: 'Name of the platform-specific library directory. It is used to build the path of the standard library and C extension modules of the standard library.' -- ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs? -- ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: > Not at all. This means that it is possible to do so. It remains optional. ...but then sys.platlibdir is going to incorrectly list 'lib', isn't it? According to https://docs.python.org/3.9/library/sys.html#sys.platlibdir it's used 'to build the path of platform-specific dynamic libraries and the path of the standard library'. This means that people relying on it for the former will get 32-bit libraries instead. -- ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1294959] Add sys.platlibdir to use /usr/lib64 on Fedora and SuSE
Michał Górny added the comment: Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*? -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue1294959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40059] Provide a toml module in the standard library
New submission from Michał Górny : PEP 518 uses the TOML format to specify build system requirements. AFAIU this means that all new build systems will require a TOML parser. Could you consider adding one to the standard library to reduce the number of chicken-egg problems? The referenced PEP states that 'pytoml TOML parser is ~300 lines of pure Python code', so I don't think integrating it would be a large maintenance cost. [1] https://www.python.org/dev/peps/pep-0518/ -- components: Library (Lib) messages: 364979 nosy: brett.cannon, dstufft, mgorny, njs priority: normal severity: normal status: open title: Provide a toml module in the standard library type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler
Change by Michał Górny : -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue39503> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38826] Regular Expression Denial of Service in urllib.request.AbstractBasicAuthHandler
Change by Michał Górny : -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue38826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36699] building for riscv multilib (patch attached)
Michał Górny added the comment: This is not really an upstream issue, it is due to Gentoo-specific patching. -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue36699> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35227] [RFE] tarfile: support adding file objects without prior known size
Michał Górny added the comment: Thanks a lot! I've left a few comments based on eyeball review. I'm going to test it later today. -- ___ Python tracker <https://bugs.python.org/issue35227> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35227] [RFE] tarfile: support adding file objects without prior known size
Michał Górny added the comment: > Adding this API would require to provide a way to set file status like mode, > uid, gid, mtime, type, linkname, uname and gname. That's why I mentioned addfile() -- it takes TarInfo object for that purpose. I suppose the new function should have the same parameters, except it would set size in the TarInfo instance instead of getting it. -- ___ Python tracker <https://bugs.python.org/issue35227> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35227] [RFE] tarfile: support adding file objects without prior known size
New submission from Michał Górny : Currently, the tarfile module only supports adding files if their size is known prior to adding. However, I think it'd be helpful to be able to store large dynamically generated streams straight into the (uncompressed) .tar file without being able to precalculate the final size and without having to use a temporary file. I'm not really sure how the API should look like (i.e. whether it should be a new method or extension of addfile()) but the mechanism would be rather simple -- write TarInfo with size of 0, write data until end of stream, write padding appropriately to written data, seek back and update TarInfo. Of course, the use of this API would have to be restricted to cases when underlying file supports seeking back and random writes, i.e. not stream, not compressed. -- components: Library (Lib) messages: 329818 nosy: mgorny priority: normal severity: normal status: open title: [RFE] tarfile: support adding file objects without prior known size type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35227> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34883] test_lzma: Multiple test failures when liblzma is built without complete codec set
New submission from Michał Górny : xz-utils has four options to configure codecs supported by liblzma: --enable-encoders --enable-decoders --enable-match-finders --enable-checks In Gentoo, we're using those options to optionally provide smaller footprint liblzma builds that include only the standard set of codecs used by .xz archives (i.e. that work with any archive created by xz/lzma without expert options used). However, it seems that the CPython test suite wrongly presumes that all codecs are always available and it fails on LZMAErrors when those are not present. I'm attaching the verbose output of test_lzma. I think the best solution here would be to catch LZMAError and skipTest() if it's LZMA_OPTIONS_ERROR. I think this would be best done by exposing the underlying error code in LZMAError exception (rather than string-matching, though the latter would also work). -- components: Tests files: lzma-test.txt messages: 326960 nosy: mgorny priority: normal severity: normal status: open title: test_lzma: Multiple test failures when liblzma is built without complete codec set type: enhancement versions: Python 3.8 Added file: https://bugs.python.org/file47845/lzma-test.txt ___ Python tracker <https://bugs.python.org/issue34883> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34366] _uuid module fails to compile on FreeBSD when libuuid is installed
Change by Michał Górny : -- keywords: +patch pull_requests: +8198 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34366> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34366] _uuid module fails to compile on FreeBSD when libuuid is installed
New submission from Michał Górny : The _uuid extension fails to build on my Gentoo/FreeBSD system: building '_uuid' extension x86_64-gentoo-freebsd11.1-gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -pipe -march=native -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include -I. -I/var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0/Include -I/var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0 -c /var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0/Modules/_uuidmodule.c -o build/temp.freebsd-11.1-RELEASE-amd64-3.7/var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0/Modules/_uuidmodule.o In file included from /usr/include/uuid.h:34:0, from /var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0/Modules/_uuidmodule.c:8: /usr/include/sys/uuid.h:77:21: error: conflicting types for 'uuid_t' typedef struct uuid uuid_t; ^~ In file included from /var/tmp/portage/dev-lang/python-3.7.0/work/Python-3.7.0/Modules/_uuidmodule.c:5:0: /usr/include/uuid/uuid.h:44:23: note: previous declaration of 'uuid_t' was here typedef unsigned char uuid_t[16]; ^~ [...] Apparently the cause is that it includes headers belonging to the system uuid library and libuuid (from util-linux) simultaneously. The whole module seems to be written with the assumption that the two different implementations will not be present simultaneously. However, some software supports libuuid only, so we're forced to have both. Attaching the complete build log. I will submit a PR soonish. -- components: Extension Modules files: dev-lang:python-3.7.0:20180809-062928.log messages: 323317 nosy: mgorny priority: normal severity: normal status: open title: _uuid module fails to compile on FreeBSD when libuuid is installed type: compile error versions: Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47735/dev-lang:python-3.7.0:20180809-062928.log ___ Python tracker <https://bugs.python.org/issue34366> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32576] concurrent.futures.thread deadlock due to Queue in weakref callback
Michał Górny added the comment: Well, according to the reporters disabling GC doesn't help at all. Maybe it's another issue. -- ___ Python tracker <https://bugs.python.org/issue32576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32576] concurrent.futures.thread deadlock due to Queue in weakref callback
Michał Górny added the comment: Could you get this fixed in earlier versions of CPython? Given that 3.7 is not yet released, having this broken in 3.5 and 3.6 is highly undesirable. This apparently seems to affect our tooling [1] and telling our users to use 3.7 beta is not an option. [1]:https://bugs.gentoo.org/647964 -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue32576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32500] PySequence_Length() raises TypeError on dict type
Michał Górny added the comment: Well, my two main concerns are: 1) whether it is acceptable for PyPy to not raise TypeError in this case, or if I should report it to PyPy upstream as a bug, 2) and whether programmers can appropriately rely on PySequence_Length() raising TypeError for non-sequence types or if they should use e.g. PySequence_Check() explicitly. -- ___ Python tracker <https://bugs.python.org/issue32500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32500] PySequence_Length() raises TypeError on dict type
Michał Górny added the comment: So is the documentation mistaken or just confusing? It says straight: > For objects that do not provide sequence protocol, this is equivalent to the > Python expression len(o). So it the 'do not' mistaken or does it mean objects that are sequences but do not provide the sequence protocol? -- ___ Python tracker <https://bugs.python.org/issue32500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32500] PySequence_Length() raises TypeError on dict type
New submission from Michał Górny :
While debugging PyPy test failure on backports.lzma [1], I've noticed that
PySequence_Check() on a dict type raises TypeError, e.g.:
Traceback (most recent call last):
File "test/test_lzma.py", line 273, in test_bad_args
b"", format=lzma.FORMAT_RAW, filters={})
File
"/home/mgorny/git/backports.lzma/build/lib.linux-x86_64-3.6/backports/lzma/__init__.py",
line 463, in decompress
decomp = LZMADecompressor(format, memlimit, filters)
TypeError: object of type 'dict' has no len()
The relevant C code is:
static int
parse_filter_chain_spec(lzma_filter filters[], PyObject *filterspecs)
{
Py_ssize_t i, num_filters;
num_filters = PySequence_Length(filterspecs);
...
where filterspecs is the object corresponding to the {} dict in Python snippet.
According to the documentation [2], PySequence_Length() should be 'equivalent
to the Python expression len(o).' The Python expression obviously does not
raise TypeError:
>>> len({})
0
Therefore, I think that the behavior of PySequence_Length() is a bug, and the
function should successfully return the dict length instead.
[1]:https://github.com/peterjc/backports.lzma
[2]:https://docs.python.org/3/c-api/sequence.html#c.PySequence_Length
--
components: Extension Modules
messages: 309534
nosy: mgorny
priority: normal
severity: normal
status: open
title: PySequence_Length() raises TypeError on dict type
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6
___
Python tracker
<https://bugs.python.org/issue32500>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31834] BLAKE2: the (pure) SSE2 impl forced on x86_64 is slower than reference
Change by Michał Górny : -- keywords: +patch pull_requests: +4036 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue31834> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31834] BLAKE2: the (pure) SSE2 impl forced on x86_64 is slower than reference
New submission from Michał Górny :
The setup.py file for Python states:
if (not cross_compiling and
os.uname().machine == "x86_64" and
sys.maxsize > 2**32):
# Every x86_64 machine has at least SSE2. Check for sys.maxsize
# in case that kernel is 64-bit but userspace is 32-bit.
blake2_macros.append(('BLAKE2_USE_SSE', '1'))
While the assertion about having SSE2 is true, it doesn't mean that it's
worthwhile to use. I've tested pure (i.e. without SSSE3 and so on) on three
different machines, getting the following results:
Athlon64 X2 (SSE2 is the best supported variant), 540 MiB of data:
SSE2: [5.18998800443, 5.070812243997352]
ref: [2.0161159170020255, 2.0475422790041193]
Core i3, same data file:
SSE2: [1.92442592602, 1.9246174693, 1.929803750191]
ref: [1.794020974667, 1.790085556976, 1.783553876418]
Xeon E5630 server, 230 MiB data file:
SSE2: [0.7671358410007088, 0.7797677099879365, 0.7648976119962754]
ref: [0.5784736709902063, 0.5717909929953748, 0.5717219939979259]
So in all the tested cases, pure SSE2 implementation is *slower* than the
reference implementation. SSSE3 and other variants are faster and AFAIU they
are enabled automatically based on CFLAGS, so it doesn't matter for most of the
systems.
However, for old CPUs that do not support SSSE3, the choice of SSE2 makes the
algorithm prohibitively slow -- it's 2.5 times slower than the reference
implementation!
--
components: Extension Modules
messages: 304696
nosy: mgorny
priority: normal
severity: normal
status: open
title: BLAKE2: the (pure) SSE2 impl forced on x86_64 is slower than reference
type: performance
versions: Python 3.6
___
Python tracker
<https://bugs.python.org/issue31834>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22140] "python-config --includes" returns a wrong path (double prefix)
Michał Górny added the comment: I've submitted a pull request with another fix. I've tried to keep the changes at minimal but I couldn't stand keeping meaningless 'echo $x | sed -e s/$x/$y/' ;-). I have to point out that the attached patch is wrong since it does readlink on CFLAGS &c. -- versions: +Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue22140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22140] "python-config --includes" returns a wrong path (double prefix)
Changes by Michał Górny : -- keywords: +patch pull_requests: +3754 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue22140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22140] "python-config --includes" returns a wrong path (double prefix)
Changes by Michał Górny : -- nosy: +mgorny ___ Python tracker <https://bugs.python.org/issue22140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16113] Add SHA-3 and SHAKE (Keccak) support
Michał Górny added the comment: Christian, since the code is now integrated in Python 3.6+ (with some bugfixes AFAICS), could you consider updating your bitbucket package to match it? It would be helpful as a backport package for older Python versions. -- nosy: +mgorny ___ Python tracker <http://bugs.python.org/issue16113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27226] distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously
Michał Górny added the comment: Yes, you could put it like this. -- ___ Python tracker <http://bugs.python.org/issue27226> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27226] distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously
Michał Górny added the comment: Brett, .pyc was controlled by --compile and .pyo by --optimize (either 1 or 2). Technically only two variants could be used simultaneously, and distutils accounted for that. Now you have .pyc + .opt-1.pyc + .opt-2.pyc, so three variants instead of two. -- ___ Python tracker <http://bugs.python.org/issue27226> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27226] distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously
Changes by Michał Górny : -- keywords: +patch Added file: http://bugs.python.org/file43231/0001-distutils-make-OO-enable-both-opt-1-and-opt-2-optimi.patch ___ Python tracker <http://bugs.python.org/issue27226> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27226] distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously
New submission from Michał Górny: Since 3.5, optimized Python modules for -O and -OO are installed using different names. The Python build system itself seems to been prepared for this as Python itself is built with both .opt-1.pyc and .opt-2.pyc files built. However, distutils at the moment can only install one of them. Gentoo is relying on 'setup.py install --compile -O2' doing the right thing. We can change the arguments if necessary but calling it twice would require a lot of conditional changes to how we build packages. Of course, not installing both variants is not an option since it causes Python to attempt to write them at runtime later. Could we please get some way to build both .opt-1.pyc and .opt-2.pyc in setup.py install? Either through making -O2 build also -O1 implicitly, or providing an ability to specify multiple optimization levels. Relevant Gentoo bug: https://bugs.gentoo.org/585060 -- components: Distutils messages: 267389 nosy: dstufft, eric.araujo, mgorny priority: normal severity: normal status: open title: distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously type: behavior versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue27226> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21356] Support LibreSSL (instead of OpenSSL): make RAND_egd optional
Michał Górny added the comment: > In CPython, the _ssl module is compiled in C. How can we check if libssl > provides RAND_egd() or not at compile time? How about... checking whether the function is provided? Unless I'm missing some major point, AC_CHECK_FUNC should be good enough. > Is there a way to check if libssl is OpenSSL or LibreSSL? Why would you want to do that? Do you want to make silly assumptions on API depending on provider name, and then add extra conditionals for versions? -- nosy: +mgorny ___ Python tracker <http://bugs.python.org/issue21356> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18201] distutils write into symlinks instead of replacing them
Michał Górny added the comment: Oh, sorry. I've looked throughout the code again and it seems that distutils is doing the right thing. It's setuptools/distribute that is broken. -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue18201> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18201] distutils write into symlinks instead of replacing them
Michał Górny added the comment: Well, I don't see much relevance between the two bugs, to be honest :). I think this bug is more of a symptom of a deeper issue with the way distutils is installing files. But the issue is causing repeating issues for our users, and I don't really know what is the best way of at least helping our users avoid them. If there were an agreement that this specific behavior of distutils is unintended, we can prepare a patch. I don't really want to diverge from the 'upstream' behavior in Gentoo just to fix our issue, and I don't really see another good way of fixing it. -- ___ Python tracker <http://bugs.python.org/issue18201> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18201] distutils write into symlinks instead of replacing them
Changes by Michał Górny : -- type: -> behavior ___ Python tracker <http://bugs.python.org/issue18201> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18201] distutils write into symlinks instead of replacing them
New submission from Michał Górny: We're doing heavy wrapping of Python scripts on Gentoo in order to efficiently support having multiple versions of Python installed. For that reason, every Python script installed using the package manager is renamed, and a symlink to a common wrapper binary is installed in its place. For example, setuptools install looks like: /usr/bin/easy_install -> python-exec /usr/bin/easy_install-python2.7 /usr/bin/easy_install-python3.2 Using the wrappers like this allow us to actively support user preference of Python version and map it to supported ones. Symlinking a common binary makes this more maintainable since there's just one place where wrapper needs to be updated. However, it seems that setup.py is failing hard in replacing symlinks. Whenever user upgrades any installed package, either externally or due to some tool like easy_install or pip, the relevant setup.py writes into /usr/bin/easy_install rather than properly replacing it with a new file. The effect, as you may imagine, is the wrapper being replaced with a random Python script and system-wide breakage. The proper thing to do would be to write into temporary file in the destination directory, then use os.rename() to atomically replace the destination with the temporary file. Since distutils is no longer maintained, I would be equally happy with simply os.remove() before writing the scripts. The problem may affect distutils2 as well. -- assignee: eric.araujo components: Distutils messages: 191057 nosy: eric.araujo, mgorny, tarek priority: normal severity: normal status: open title: distutils write into symlinks instead of replacing them versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue18201> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16879] distutils.command.config uses fragile constant temporary file name
Michał Górny added the comment: While using. We run parallel merges with out-of-source builds in Gentoo, and I found this to result in a failure when building the 'egenix-mx-base' package [1] which uses the 'config' command. [1]:http://pypi.python.org/pypi/egenix-mx-base -- ___ Python tracker <http://bugs.python.org/issue16879> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16879] distutils.command.config uses fragile constant temporary file name
New submission from Michał Górny: The distutils.command.config module reads: def _gen_temp_sourcefile(self, body, headers, lang): filename = "_configtest" + LANG_EXT[lang] which means that every time one of the functions is used, a temporary file with the same name will be created. This is really fragile and fails terribly whenever more than one process tries to use the relevant functions in the same working directory. I believe this code should use the tempfile module to generate unique temporary files names instead. -- assignee: eric.araujo components: Distutils messages: 179179 nosy: eric.araujo, mgorny, tarek priority: normal severity: normal status: open title: distutils.command.config uses fragile constant temporary file name versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker <http://bugs.python.org/issue16879> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9561] distutils: set encoding to utf-8 for input and output files
Michał Górny added the comment: Ping. What's the progress on this? Will this ever be fixed? -- ___ Python tracker <http://bugs.python.org/issue9561> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12087] install_egg_info fails with UnicodeEncodeError depending on locale
Changes by Michał Górny : -- nosy: +mgorny ___ Python tracker <http://bugs.python.org/issue12087> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9561] distutils: set encoding to utf-8 for input and output files
Michał Górny added the comment: Now that installing scripts with unicode characters was fixed, shall I open a separate bug for writing egg files with utf8 chars in author name? -- ___ Python tracker <http://bugs.python.org/issue9561> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10419] distutils command build_scripts fails with UnicodeDecodeError
Changes by Michał Górny : -- nosy: +mgorny ___ Python tracker <http://bugs.python.org/issue10419> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9561] distutils: set encoding to utf-8 for input and output files
Changes by Michał Górny : -- nosy: +mgorny ___ Python tracker <http://bugs.python.org/issue9561> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10051] distutils fail to install unicode-encoded files with POSIX locale
New submission from Michał Górny : I was reported an issue where the user was unable to install my package [1] using python3.1. I was able to guess the issue comes from locale being set to POSIX and reproduce it. It seems that the reasons are utf8-encoded characters in the installed script file. All the project files carry encoding declaration as per PEP-0263. They install fine with python2 but fail both with 3.1 and 3.2. I think the build system should be locale-agnostic, and work correctly even if LC_ALL=C. If it's necessary to use text mode for scripts, I think a similar encoding guessing method should be used as in python itself. I'm attaching a complete build log (for multiple versions of Python). Below I'm repasting the backtrace: Traceback (most recent call last): File "setup.py", line 35, in 'Topic :: System :: Installation/Setup' File "/usr/lib64/python3.1/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib64/python3.1/distutils/dist.py", line 919, in run_commands self.run_command(cmd) File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/usr/lib64/python3.1/distutils/command/build.py", line 128, in run self.run_command(cmd_name) File "/usr/lib64/python3.1/distutils/cmd.py", line 315, in run_command self.distribution.run_command(command) File "/usr/lib64/python3.1/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/usr/lib64/python3.1/distutils/command/build_scripts.py", line 51, in run self.copy_scripts() File "/usr/lib64/python3.1/distutils/command/build_scripts.py", line 83, in copy_scripts first_line = f.readline() File "/usr/lib64/python3.1/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 127: ordinal not in range(128) [1] http://github.com/mgorny/smart-live-rebuild -- assignee: tarek components: Distutils files: build.log messages: 118188 nosy: eric.araujo, mgorny, tarek priority: normal severity: normal status: open title: distutils fail to install unicode-encoded files with POSIX locale type: behavior versions: Python 3.1, Python 3.2 Added file: http://bugs.python.org/file19159/build.log ___ Python tracker <http://bugs.python.org/issue10051> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
