[issue43895] Unnecessary Cache of Shared Object Handles
Gregory P. Smith added the comment: New changeset fed2fc4443235fa9669b73817827fd6da88e3417 by Ian Henriksen in branch 'main': bpo-43895: Remove an unnecessary cache of shared object handles (GH-25487) https://github.com/python/cpython/commit/fed2fc4443235fa9669b73817827fd6da88e3417 -- ___ Python tracker <https://bugs.python.org/issue43895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44145] hmac.update is not releasing the GIL when openssl's hmac is used
Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue44145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43895] Unnecessary Cache of Shared Object Handles
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith nosy: +gregory.p.smith, twouters versions: +Python 3.11 ___ Python tracker <https://bugs.python.org/issue43895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41486] Add _BlocksOutputBuffer for bz2/lzma/zlib module
Gregory P. Smith added the comment: New changeset a9a69bb3ea1e6cf54513717212aaeae0d61b24ee by Ma Lin in branch 'main': bpo-41486: zlib uses an UINT32_MAX sliding window for the output buffer (GH-26143) https://github.com/python/cpython/commit/a9a69bb3ea1e6cf54513717212aaeae0d61b24ee -- ___ Python tracker <https://bugs.python.org/issue41486> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13788] os.closerange optimization
Gregory P. Smith added the comment: Code needed in a modern patch: 1) Use the system call if compile time configure detected it may be available. (if we just use syscall() rather than a libc wrapper, a configure check may not be necessary, but various #ifdefs likely are) 2) If (1) produces an error saying it isn't available at runtime (ie: running on a kernel that doesn't support it, regardless of where we were built), fall back to another approach (3) 3) if iterating over /proc/$pid/fd works at runtime, use that; else (4) 4) the existing brute force code. -- stage: -> needs patch versions: +Python 3.11 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue13788> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36991] zipfile: AttributeError on extract
Change by Gregory P. Smith : -- stage: patch review -> resolved status: pending -> closed ___ Python tracker <https://bugs.python.org/issue36991> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44458] Duplicate symbol _BUFFER_BLOCK_SIZE when statically linking multiple modules
Gregory P. Smith added the comment: Good catch. We don't have buildbots statically linking the modules in these days, that'd be useful to prevent things like this from happening. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44458> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44458] Duplicate symbol _BUFFER_BLOCK_SIZE when statically linking multiple modules
Gregory P. Smith added the comment: New changeset cf739332bd039cd2303b58663a804f784883820d by Miss Islington (bot) in branch '3.10': bpo-44458: Ensure BUFFER_BLOCK_SIZE symbol is statically allocated. (GH-26808) (GH-26844) https://github.com/python/cpython/commit/cf739332bd039cd2303b58663a804f784883820d -- ___ Python tracker <https://bugs.python.org/issue44458> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44458] Duplicate symbol _BUFFER_BLOCK_SIZE when statically linking multiple modules
Gregory P. Smith added the comment: New changeset 92c2e91580521ba5c85aa3205a0211df5b48689b by Russell Keith-Magee in branch 'main': bpo-44458: Ensure BUFFER_BLOCK_SIZE symbol is statically allocated. (GH-26808) https://github.com/python/cpython/commit/92c2e91580521ba5c85aa3205a0211df5b48689b -- message_count: 1.0 -> 2.0 nosy: +miss-islington nosy_count: 2.0 -> 3.0 pull_requests: +25425 pull_request: https://github.com/python/cpython/pull/26844 ___ Python tracker <https://bugs.python.org/issue44458> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44458] Duplicate symbol _BUFFER_BLOCK_SIZE when statically linking multiple modules
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44458> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44107] HTTPServer can't close http client completely
Change by Gregory P. Smith : -- resolution: later -> not a bug status: open -> closed ___ Python tracker <https://bugs.python.org/issue44107> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40120] Undefined C behavior going beyond end of struct via a [1] arrays.
Change by Gregory P. Smith : -- versions: +Python 3.11 -Python 3.9 ___ Python tracker <https://bugs.python.org/issue40120> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44334] Use bytearray in urllib.unquote_to_bytes
Change by Gregory P. Smith : -- versions: +Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44334] Use bytearray in urllib.unquote_to_bytes
Gregory P. Smith added the comment: fwiw this sort of thing may be reasonable to backport to 3.9 as it is more than just a performance enhancement but also a resource consumption bug and should result in no behavior change. """ In case of form contain very large data ( in my case the string to parse was about 600 byte ) Old code use list of bytes during parsing consumes a lot of memory New code will use bytearry , which use less memory """ - text from the original PR -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43776] Popen with shell=True yield mangled repr output
Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue43776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43776] Popen with shell=True yield mangled repr output
Gregory P. Smith added the comment: New changeset 5a8ddcc4524dca3880d7fc2818814ffae1cfb8a2 by Gregory P. Smith in branch '3.9': [3.9] bpo-43776: Remove list call from args in Popen repr (GH-25338) (GH-26510) https://github.com/python/cpython/commit/5a8ddcc4524dca3880d7fc2818814ffae1cfb8a2 -- ___ Python tracker <https://bugs.python.org/issue43776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43776] Popen with shell=True yield mangled repr output
Change by Gregory P. Smith : -- pull_requests: +25106 pull_request: https://github.com/python/cpython/pull/26510 ___ Python tracker <https://bugs.python.org/issue43776> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: New changeset e60ab843cbb016fb6ff8b4f418641ac05a9b2fcc by Gregory P. Smith in branch 'main': bpo-44022: Improve the regression test. (GH-26503) https://github.com/python/cpython/commit/e60ab843cbb016fb6ff8b4f418641ac05a9b2fcc -- ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: Great catch! The new PR should address that. -- ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Change by Gregory P. Smith : -- pull_requests: +25099 pull_request: https://github.com/python/cpython/pull/26503 ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42392] [document removal of] the deprecated 'loop' parameter asyncio API in 3.10
Gregory P. Smith added the comment: thanks! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue42392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42392] [document removal of] the deprecated 'loop' parameter asyncio API in 3.10
Gregory P. Smith added the comment: New changeset d8fd8c8568cbc2f53c1abeda3596a89a46f0e3d7 by Ken Jin in branch 'main': bpo-42392: [docs] Add deprecated-removed loop labels for asyncio (GH-26357) https://github.com/python/cpython/commit/d8fd8c8568cbc2f53c1abeda3596a89a46f0e3d7 -- ___ Python tracker <https://bugs.python.org/issue42392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37439] Add random.binomialvariate()
Gregory P. Smith added the comment: A presumed optimal version of this is already available in numpy. https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.binomial.html https://github.com/numpy/numpy/blob/2232a473f8713f532c8164c8cf616f7bd05f54a7/numpy/random/_generator.pyx#L2805 -- nosy: +gregory.p.smith versions: +Python 3.11 -Python 3.9 ___ Python tracker <https://bugs.python.org/issue37439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42392] [document removal of] the deprecated 'loop' parameter asyncio API in 3.10
Change by Gregory P. Smith : -- title: remove the deprecated 'loop' parameter asyncio API -> [document removal of] the deprecated 'loop' parameter asyncio API in 3.10 ___ Python tracker <https://bugs.python.org/issue42392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42392] remove the deprecated 'loop' parameter asyncio API
Change by Gregory P. Smith : -- assignee: -> docs@python components: +Documentation nosy: +docs@python ___ Python tracker <https://bugs.python.org/issue42392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42392] remove the deprecated 'loop' parameter asyncio API
Gregory P. Smith added the comment: There appear to be no versionchanged:: 3.10 in the asyncio docs on the APIs that formerly accepted a loop= parameter linking people to information on when that went away (3.10) and why. Specifically I'm talking about https://docs.python.org/3.10/library/asyncio-stream.html. The asyncio stack traces people will face when porting code to 3.10 are mystifying (they may not even show use of a loop parameter) when this comes up, so we should really leave more bread crumbs than expecting people to find the What's New doc. ``` ... Eserver = event_loop.run_until_complete(coro) E File "/opt/hostedtoolcache/Python/3.10.0-beta.1/x64/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete Ereturn future.result() E File "/opt/hostedtoolcache/Python/3.10.0-beta.1/x64/lib/python3.10/asyncio/streams.py", line 113, in start_unix_server Ereturn await loop.create_unix_server(factory, path, **kwds) E TypeError: _UnixSelectorEventLoop.create_unix_server() got an unexpected keyword argument 'loop' ``` Arguably something similar to that whatsnew text should've been added to the docs in 3.8 with the loop deprecation. Something like this? ``` .. versionchanged:: 3.7 This function now implicitly gets the current thread's running event loop. .. versionchanged:: 3.10 That `loop` parameter has been removed. ``` including a ReST link to more info in the whats new doc on the last entry would be useful. -- nosy: +gregory.p.smith resolution: fixed -> stage: resolved -> needs patch status: closed -> open ___ Python tracker <https://bugs.python.org/issue42392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: Lets get equivalent whatsnew text into the 3.8 and 3.9 and 3.10 branches before closing it. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42443] Provide Thread creation hook support
Gregory P. Smith added the comment: > > * IMO it should be called after profiling and tracing hook, so non-trivial > > hooks can be profiled and traced. > Makes sense, Done. I updated the PR do this. (if your "Done" indicated you had done this somewhere, it appears you never pushed your changes to the PR branch on github :) > > * It's important to define and document, which thread runs the hook > (calling thread or new thread). > Will update documentation when we agree upon appropriate API. I updated the docstring in the PR. > > * Since the hook is designed to monitor thread creation, would it make > > sense to pass the thread object to the hook? > As it's called within the context of the created thread I guess this is not > needed (but as you prefer). I updated the PR to do this. > > * How does the hook behave when the callback raises an exception? > I guess it makes sense to do similar thing as in case of profile/trace hooks > (Error in the profile function will cause itself unset). WDYT? I like the consistency. I've updated the PR. > > * Is a single hook good enough or should the API behave more like atexit, > > which supports an arbitrary amount of hooks? > Feels like it makes sense to keep API similar to what profile/trace provides, > otherwise it would be inconsistent. Agreed. when writing the docs we can mention this. Anyone setting these hooks would be wise to check what's already set and decide if they want to set theirs at all or save that and chain call it or not. It is expected to be rare to set this. Our own use case is effectively platform wide for use in most applications. > > * Instead of just a creation hook, how about lifetime hooks are also called > > when a thread ends? > Sounds fine, do you mean two independent hooks (_thread_creation_hook, > _thread_termination_hook) or some other form? Lets leave this as a separate feature to add if the need arises. I can imagine uses in threading tests, but that doesn't feel like a strong use case. -- versions: +Python 3.11 ___ Python tracker <https://bugs.python.org/issue42443> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44174] Unclear meaning of _Private__names in enum docs.
New submission from Gregory P. Smith : https://docs.python.org/3/library/enum.html#private-names """ _Private__names Private names will be normal attributes in Python 3.10 instead of either an error or a member (depending on if the name ends with an underscore). Using these names in 3.9 will issue a DeprecationWarning. """ It isn't clear from this documentation what is meant by "private names". Please expand on this to be explicit about what name pattern is being described. -- assignee: docs@python components: Documentation messages: 393919 nosy: docs@python, gregory.p.smith priority: normal severity: normal stage: needs patch status: open title: Unclear meaning of _Private__names in enum docs. versions: Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44174> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22891] code removal from urllib.parse.urlsplit()
Gregory P. Smith added the comment: There are still a lot of cleanliness and even logical issues lurking within urllib.parse. But this bug is dated and other changes have happened since. I'm closing it as out of date. Open new issues for specific action items. -- nosy: +gregory.p.smith resolution: -> out of date stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue22891> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37901] 21 tests fail when run on an IPv6-only host
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +24842 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/26225 ___ Python tracker <https://bugs.python.org/issue37901> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29639] test suite intentionally avoids referring to localhost, destroying abstraction away from IPv6 vs IPv4
Gregory P. Smith added the comment: I'm making progress on https://bugs.python.org/issue37901 and related IPv6-only issues, the changes of which tend to shake out improper uses of 127.0.0.1 hardcoded. I'm closing this issue as there aren't specific needs that the rest won't cover. -- resolution: -> out of date stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue29639> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40339] Instead of skipping, IPV6 test(s) fail on a non-IPV6 machine
Change by Gregory P. Smith : -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Test suite fails when Ipv6 is unavailable ___ Python tracker <https://bugs.python.org/issue40339> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33433] ipaddress is_private misleading for IPv4 mapped IPv6 addresses
Gregory P. Smith added the comment: thanks! -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue33433> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43650] MemoryError on zip.read in shutil._unpack_zipfile
Gregory P. Smith added the comment: thanks for the patch! -- resolution: remind -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue43650> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43650] MemoryError on zip.read in shutil._unpack_zipfile
Gregory P. Smith added the comment: New changeset 7a588621c2854bcef6ce9eeb54349b84ac078c45 by Miss Islington (bot) in branch '3.10': bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058) (GH-26190) https://github.com/python/cpython/commit/7a588621c2854bcef6ce9eeb54349b84ac078c45 -- ___ Python tracker <https://bugs.python.org/issue43650> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44107] HTTPServer can't close http client completely
Gregory P. Smith added the comment: Python is calling close() after the HTTP/1.0 request. This isn't a bug on the CPython side. -- nosy: +gregory.p.smith resolution: remind -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44107> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43650] MemoryError on zip.read in shutil._unpack_zipfile
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith nosy: +gregory.p.smith type: crash -> resource usage versions: +Python 3.10, Python 3.11 -Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue43650> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33433] ipaddress is_private misleading for IPv4 mapped IPv6 addresses
Gregory P. Smith added the comment: Questionable as to if this counts as a bug or not, but I agree it is counter-intuitive given the purpose of actually checking the .is_private property of an address. So updating the behavior makes sense, but I'll be conservative w.r.t. behavior changes and not apply this to a 3.9 or earlier. -- nosy: +gregory.p.smith type: -> enhancement versions: +Python 3.10, Python 3.11 -Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33433> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39950] Add pathlib.Path.hardlink_to()
Gregory P. Smith added the comment: New changeset b913f47e87f788e705716ae037ee9f68b4265e69 by Miss Islington (bot) in branch '3.10': bpo-39950: Fix deprecation warning in test for `pathlib.Path.link_to()` (GH-26155) (GH-26178) https://github.com/python/cpython/commit/b913f47e87f788e705716ae037ee9f68b4265e69 -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue39950> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44145] hmac.update is not releasing the GIL when openssl's hmac is used
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +24791 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/26157 ___ Python tracker <https://bugs.python.org/issue44145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44145] hmac.update is not releasing the GIL when openssl's hmac is used
New submission from Gregory P. Smith : This prevents parallel hmac computations. see michaelforney's comment left on https://github.com/python/cpython/pull/20129 where the problem was introduced when adding support for using OpenSSL's HMAC implementations. easy fix. PR coming. We don't really have a way to unittest for regressions on things like this. -- assignee: gregory.p.smith components: Extension Modules keywords: 3.9regression messages: 393733 nosy: christian.heimes, gregory.p.smith priority: normal severity: normal stage: needs patch status: open title: hmac.update is not releasing the GIL when openssl's hmac is used type: performance versions: Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38768] [feature request] Add lldb equivalent to Tools/gdb
Gregory P. Smith added the comment: This would be useful beyond macOS. Clang/LLVM is the compiler toolchain of choice for a lot of the world these days. Including lldb. gdb cannot be assumed to be available or even work everywhere. This would be a great thing to have. -- components: +Demos and Tools nosy: +gregory.p.smith stage: -> needs patch type: -> enhancement versions: -Python 3.10 ___ Python tracker <https://bugs.python.org/issue38768> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44002] Use functools.lru_cache in urllib.parse instead of 1996 custom caching
Change by Gregory P. Smith : -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue44002> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43743] BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS.
Gregory P. Smith added the comment: The logic for bailing out to a slow copy is currently: https://github.com/python/cpython/blob/main/Lib/shutil.py#L158 that condition appears to not be happening in Alexei's test. Suggesting that either at least one sendfile call succeeded and thus offset is non-zero or the lseek failed. run that test under pdb and walk thru the code, or under strace to look at the syscalls and find out. The question seems to be is if it should be okay to _GiveUpOnFastCopy after a partial (incomplete) copy has already occurred via sendfile. -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue43743> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44091] traceback & inspect modules should verify that the .py source file matches the one that the running process is using
New submission from Gregory P. Smith :
A long-standing wart in Python is that once a module is loaded, when rendering
a traceback and including source lines, we do not verify if the source file
we're loading is the same as the one representing the code we are running.
It could have been replaced. As is normal during software upgrades.
If our code was loaded from .py source, we should be recording the
timestamp/size||hash of the source file and referencing that from each code
object. If our code was loaded from a .pyc source, the .pyc already contains a
timestamp/size||hash for the corresponding .py source file that could be
referenced.
When traceback.StackSummary and FrameSummary use the linecache module, we
should plumb this source metainfo in from the relevant code object.
A traceback being rendered with potentially modified source code could choose
to omit the source lines, or at least annotate them with a " ## this source
{timestamp/size||hash} does not match the running code {timestamp/size||hash}."
marker so that anyone seeing the traceback knows the displayed line may not be
trustworthy. (If the pyc was written using the "unchecked-hash" mode, no
source/pyc synchronization check should be made)
The inspect module also needs the ability to do indicate this to the caller.
--
components: Library (Lib)
messages: 393328
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: traceback & inspect modules should verify that the .py source file
matches the one that the running process is using
type: enhancement
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue44091>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44023] "tarfile" library will lead to "write any content to any file on the host".
Gregory P. Smith added the comment: TL;DR - A tar file being extracted doesn't check to see if it is overwriting an existing file, which could be a symlink to elsewhere leading to elsewhere's contents being clobbered assuming the elsewhere file exists. doing an unlink before opening the destination file (ignoring either success or FileNotFound) during extract would avoid this _specific_ case. But tarfile is already documented with a warning about untrusted inputs being able to do bad things: https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall fixing this one serialized case doesn't do anything about other cases or race conditions we won't claim protection against, so I'm not sure this issue is serious from a stdlib perspective. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44023> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44058] 'master' refs in 3.10 version of .azure-pipelines
Change by Gregory P. Smith : -- assignee: -> pablogsal nosy: +pablogsal ___ Python tracker <https://bugs.python.org/issue44058> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35663] webbrowser.py firefox bug [python3, windows 10]
Gregory P. Smith added the comment: Others are reporting this works. if that is not true on the most recent python 3.9 release, please reopen with more detailed reproduction instructions. -- nosy: +gregory.p.smith resolution: -> works for me stage: -> needs patch status: open -> closed ___ Python tracker <https://bugs.python.org/issue35663> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44070] Regression with relative paths in sys.path in python 3.8.10
Gregory P. Smith added the comment: without inspecting the code: I wonder if this is related to the same change as https://bugs.python.org/issue44061 ? -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44070] Regression with relative paths in sys.path in python 3.8.10
Change by Gregory P. Smith : -- keywords: +3.10regression, 3.8regression, 3.9regression priority: normal -> high ___ Python tracker <https://bugs.python.org/issue44070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44070] Regression with relative paths in sys.path in python 3.8.10
Change by Gregory P. Smith : -- nosy: +brett.cannon ___ Python tracker <https://bugs.python.org/issue44070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44070] Regression with relative paths in sys.path in python 3.8.10
Change by Gregory P. Smith : -- nosy: +steve.dower ___ Python tracker <https://bugs.python.org/issue44070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: There is no less intrusive fix as far as I can see. I believe we're down to either stick with what we've done, or do nothing. It doesn't have to be the same choice in all release branches, being more conservative with changes the older the stable branch is okay. (ie: removing this from 3.6 and 3.7 seems fine even if more recent ones do otherwise) Based on my testing, raising an exception is more intrusive to existing tests (which we can only ever hope is representative of code) than stripping. At least as exposed by running the changes through many tens of thousands of unittest suites at work. ie: if we raise an exception, pandas.read_json() starts failing because that winds up using urlsplit in hopes of extracting the scheme and comparing that to known values as their method of deciding if something should be treated as a URL to data rather than data. Pandas would need to be fixed. That urlsplit() API use pattern is repeated in various other pieces of code: urlsplit is not expected to raise an exception. The caller then has a conditional or two testing some parts of the urlsplit result to make a guess as to if something should be considered a URL or not. Doing code inspection, pandas included, this code pretty much always then goes on to pass the original url value off to some other library, be it urllib, or requests, or ...). Consequences of that code inspection finding? With our existing character stripping change, new data is then allowed to pass through these urlsplit uses and be considered a URL. Which leads to some code sending the url with embedded \r\n\t chars on to other APIs - a concern expressed a couple of times above. Even though urlsplit isn't itself a validation API, it gets used as an early step in peoples custom identification and validation attempts. So *any* change we make to it at all in any way breaks someones expectations, even if they probably shouldn't have had those expectations and aren't doing wise validation. Treat this analysis as a sign that we should provide an explicit url validator because almost everyone is doing it some form of wrong. (issue43883) I did wonder if Mike's suggestion of removing the characters during processing, but leaving them in the final result in https://bugs.python.org/issue43882#msg393033 is feasible as remediation for this? My gut feeling is that it isn't. It doesn't solve the problem of preventing the bad data from going where it shouldn't. Even if we happen to parse that example differently, the unwanted characters are still retained in other places they don't belong. Fundamantelly: We require people to make a different series of API call and choices in the end user code to **explicitly not use unvalidated inputs**. Our stdlib API surface can't satisfy that today and use of unvalidated data in wrong places is a broad software security antipattern theme. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment:
httplib.py is a Python 2 concept. Python 2 is end of life. bugs.python.org no
longer tracks issues with its code. I don't doubt that Python 2.7 has bugs.
As a matter of policy, we don't care -
https://www.python.org/doc/sunset-python-2/. Python 3.6 as that is the oldest
branch still open for security fixes.
The PRs associated with this issue fixed a codepath in Python 3 that only
happened after a '100' response. That codepath did not accumulate headers:
```
if status != CONTINUE:
break
# skip the header from the 100 response
while True:
skip = self.fp.readline(_MAXLINE + 1)
if len(skip) > _MAXLINE:
raise LineTooLong("header line")
skip = skip.strip()
if not skip:
break
```
CONTINUE = 100; meaning that loop only runs after receiving what appears to be
a 100 continue response. And it does not accumulate data.
There is no `hlist` in the original pre-fix Python 3.6+ code. Nor any header
accumulation caused by this the client.py talking to evil_server.py as
described in this issues opening message.
--
___
Python tracker
<https://bugs.python.org/issue44022>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44059] Support SerenityOS Browser in webbrowser module
Gregory P. Smith added the comment: thanks for the PR! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: FWIW, if we were to add a parameter, I'd lean towards a name of "invalid_url_characters = None" defaulting to using what's in our private _UNSAFE_URL_BYTES_TO_REMOVE global when None but otherwise letting the user specify a sequence of characters. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment:
Of note: If we had chosen to raise a ValueError (or similar) for these
characters by default, the cloud-init code would also fail to behave as
intended today (based on what I see in
https://github.com/canonical/cloud-init/commit/c478d0bff412c67280dfe8f08568de733f9425a1)
Recommendation for cloud-init - Do your hostname transformation early using as
simple as possible logic. By virtue of accepting (and encouraging?) invalid
characters and transforming them, what you have today that you call urlsplit on
is more of a url template, not really a url. something like this:
```
if m :=
re.search(r'^(?P[^/:]+://|)(?P[^/]+)(?P/.*)',
url_template):
start, hostname, end = m.groups()
for transformation in transformations:
... fixup hostname ...
url = f'{start}{hostname}{end}'
else:
... # doesn't look like a URL template
```
yes this simplicity would allow your transformations to apply to the :port
number. you could simplify further by including the scheme_colon_slashes in
the part transformed. as your values are coming from user written config
files, do you need to care about invalid characters in those transforming into
invalid in the scheme or port number - characters in the resulting url anyways?
after that, do you even need urlsplit at all in your
`_apply_hostname_transformations_to_url()` function?
--
___
Python tracker
<https://bugs.python.org/issue43882>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: We try to not add a new parameter in a bugfix release as that can be difficult to use. That said, adding a new bool keyword only parameter to control this behavior seems feasible. Unfortunately you already have to deal with the existence of 3.9.5 having the new behavior but not having a control. (maybe you resort to the global behavior change there by monkeypatching the list? removing tab is probably enough as I doubt you rely on newlines in your scenario?) Code wanting to support versions before this patch _and pass that new parameter_ winds up needing to resort to inspect.signature(), or a call with the parameter, catch the NameError, and retry calling without it pattern. Not unheard of. And so long as most code doesn't ever need to do that trick, is fine. (it can be checked for at module import time rather than doing it on every call to save overhead if that matters) meta: Marking the issue as open while we decide if we'll be doing something here. -- status: closed -> open ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44059] Support SerenityOS Browser in webbrowser module
Gregory P. Smith added the comment: New changeset fbefdaf92e40134c43142c182f7c5bfe406e4c2b by Miss Islington (bot) in branch '3.10': bpo-44059: Register the SerenityOS Browser in the webbrowser module (GH-25947) (GH-25950) https://github.com/python/cpython/commit/fbefdaf92e40134c43142c182f7c5bfe406e4c2b -- ___ Python tracker <https://bugs.python.org/issue44059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: If anyone wants a CVE for it, that's up to them. This bug is in the CPython http.client module which is what urllib uses for http/https. I'd rate it low severity. A malicious server can hold a http connection from this library open as a network traffic sink. There are other ways to do that. ex: Just use omit a content-length header in a server response and start streaming an infinite response. The difference in this case being that since the data is thrown away, it isn't going to result in memory exhaustion and kill the unfortunate process as trying to read an infinite response would. That's the primary DoS potential from my point of view. -- ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: For completeness reference, the 'main' branch after the master->main rename also got fixed to check it early the same was as the release branches via: https://github.com/python/cpython/commit/985ac016373403e8ad41f8d563c4355ffa8d49ff our robot updating bug comments presumably didn't know about the master -> main rename yet so didn't leave a comment here. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43743] BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS.
Gregory P. Smith added the comment: I don't believe CPython should be working around a bug in specific Linux kernel versions in the standard library unless they are extremely pernicious and not considered to be a bug and thus ever be fixed in the OS kernel. As the sendfile system call appears to infinitely return one of EAGAIN, EALREADY, EWOULDBLOCK, or EINPROGRESS in this case, there isn't anything CPython could do. A retry/backoff loop won't help. This should be worked around at the application level by whatever means are appropriate. -- nosy: +gregory.p.smith resolution: -> not a bug stage: -> resolved status: open -> closed type: crash -> behavior ___ Python tracker <https://bugs.python.org/issue43743> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44059] Support SerenityOS Browser in webbrowser module
Gregory P. Smith added the comment: Quite a humorous long list of historical browsers still in the webbrowser module. This can join the list. The PR should automerge after the CI runs complete. -- assignee: -> gregory.p.smith nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue44059> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44061] Regression in pkgutil: iter_modules stopped taking Path argument in python 3.8.10 and 3.9.5
Gregory P. Smith added the comment: (obviously we're missing any tests for use of Path objects in this situation) -- stage: -> needs patch type: crash -> behavior ___ Python tracker <https://bugs.python.org/issue44061> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43105] [Windows] Can't import extension modules resolved via relative paths in sys.path
Gregory P. Smith added the comment: This caused a regression described in https://bugs.python.org/issue44061 -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue43105> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44061] Regression in pkgutil: iter_modules stopped taking Path argument in python 3.8.10 and 3.9.5
Gregory P. Smith added the comment: https://bugs.python.org/issue43105 and https://github.com/python/cpython/pull/25121 caused this regression. -- assignee: -> steve.dower keywords: +3.8regression, 3.9regression nosy: +gregory.p.smith, lukasz.langa, steve.dower priority: normal -> release blocker versions: +Python 3.10 ___ Python tracker <https://bugs.python.org/issue44061> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: New changeset 60ba0b68470a584103e28958d91e93a6db37ec92 by Miss Islington (bot) in branch '3.10': bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916) (GH-25931) https://github.com/python/cpython/commit/60ba0b68470a584103e28958d91e93a6db37ec92 -- ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: Thanks guangli dong (leveryd)! This is in and the 3.10-3.6 PRs should automerge (thru 3.9) after the CI runs, or be merged by the release managers (3.6-3.8). -- resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: Mike: There may be multiple ways to read that WHATWG recommendation? The linked to section is about implementing a state machine for parsing a URL into parts safely. But that may not imply that anything that passed through that state machine should be considered "valid". Just that this spec is able to make some sense out of otherwise messy data. Network adage: Be lenient in what you accept. I doubt anyone would want something producing URLs for consumption by something else to allow these in their output. I have yet to read the _entire_ WHATWG spec from head to toe to try and better understand the context they had in mind. I agree that it is unfortunate that the original URL may have these issues and go on to be (re)used in another context where it passes to something that might not treat it in the same manner. That's in some sense a flaw in our API design that we allow string based URLs to be used in APIs rather than require them to have gone through a parsing sanitization step into a instance of a URL object (for example). API design like that is clearly out of scope for this issue. :) Regardless my gut feeling is that we continue with the existing fix that remove the characters as we've already started releasing. If we need to change our mind on how we've done that, so be it, we can, that'll show up in later patches. -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Gregory P. Smith added the comment: The bug: Our http client can get stuck infinitely reading len(line) < 64k lines after receiving a '100 Continue' http response. So yes, this could lead to our client being a bandwidth sink for anyone in control of a server. Clear issue: That's a denial of network bandwidth and the denial of service in terms of CPU needed to process read and skip such lines. The infinite lines are size bounded and are not buffered so there is no memory based DoS. Maybe issue: If a the underlying socket has a timeout set on it, it can be used to prevent the timeout from triggering by sending a line more often than the timeout. this is a denial of service by making a http client connection that an author may have assumed would timeout based on their socket.setdefaulttimeout() settings hang forever. I expect there are plenty of other ways to accomplish the latter in our http client code though. Ex: A regular response with a huge content length where one byte is transmitted occasionally could also effectively accomplish that. The stdlib http stack doesn't have its own overall http transaction timeout as a feature. -- ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44022] urllib http client possible infinite loop on a 100 Continue response
Change by Gregory P. Smith : -- assignee: -> gregory.p.smith nosy: +gregory.p.smith title: "urllib" will result to deny of service -> urllib http client possible infinite loop on a 100 Continue response versions: +Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44050] Exceptions in a subinterpreter are changed by another subinterpreter
Change by Gregory P. Smith : -- assignee: -> eric.snow nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue44050> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44043] 3.10 b1 armhf Bus Error in hashlib test: test_gil
Change by Gregory P. Smith : -- dependencies: +unaligned memory access in the _sha3 extension ___ Python tracker <https://bugs.python.org/issue44043> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36515] unaligned memory access in the _sha3 extension
Change by Gregory P. Smith : -- pull_requests: +24594 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25927 ___ Python tracker <https://bugs.python.org/issue36515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36515] unaligned memory access in the _sha3 extension
Gregory P. Smith added the comment: that patch looks fine, i'll turn it into a PR. -- assignee: christian.heimes -> gregory.p.smith nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue36515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36515] unaligned memory access in the _sha3 extension
Change by Gregory P. Smith : -- assignee: -> christian.heimes versions: +Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue36515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment:
Thanks Florian! Indeed, I'm glad you have tests for this. (I expect anyone
writing their own validation code will have such tests)
Making urlsplit raise an exception where it never has before has other
consequences:
In CPython's own test suite test_urllib fails as many of its tests for
validation that these characters are either ignored or cause a specific
http.client.InvalidURL error on urlopen() start failing. I draw no conclusions
from that other than we'd need to rework some of those tests. It just reflects
the state of our test suite and even inconsistent behavior between excluding
the characters or erroring within the http.client code on them based on past
CVEs.
Regardless, if people would people prefer to see urlsplit `raise
SomeExistingException(f'Invalid characters in URL or scheme. url={url!r}
scheme={scheme!r}')` in 3.9.6 and the security patch releases for other 3.x
versions, evidence that it wouldn't cause alternate problems would be helpful.
I've kicked off tests at work on our huge codebase with both variants as a
datapoint to see if that is informative or not.
If we went the exception route: SomeExistingException might make sense as
`http.client.InvalidURL`, but that'd be a circular dependency (no big deal) and
heavyweight import for urllib.parse to have. `urllib.error.URLError` could
also make sense, but that's an OSError descendant and identifies itself as a
urlopen error which would be surprising. `ValueError` is a reasonable
fallback. But using that guarantees someone will wonder why it isn't one of
the other two... As this is a bugfix, defining a new exception isn't an option.
We as a community currently lack a way for security patches to CPython to be
tested against a broad swath of projects in advance of them appearing in a
release. Once upon a time there were release candidates for patches releases
that could serve this purpose...
--
___
Python tracker
<https://bugs.python.org/issue43882>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: Both Django and Botocore issues appear to be in the category of: "depending on invalid data being passed through our urlsplit API so that they could look for it later" Not much sympathy. We never guaranteed we'd pass invalid data through. They're depending on an implementation detail (Hyrum's law). Invalid data causes other people who don't check for it problems. There is no valid solution on our end within the stdlib that won't frustrate somebody. We chose to move towards safer (undoubtedly not perfect) by default. Instead of the patches as you see them, we could've raised an exception. I'm sure that would also also have tripped up existing code depending on the undesirable behavior. If one wants to reject invalid data as an application/library/framework, they need a validator. The Python stdlib does not provide a URL validation API. I'm not convinced we would even want to (though that could be something issue43883 winds up providing) given how perilous that is to get right: Who's version of right? which set of standards? when and why? Conclusion: The web... such a mess. -- versions: +Python 3.11 ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43883] Making urlparse WHATWG conformant
Gregory P. Smith added the comment: FWIW rather than implementing our own URL parsing at all... wrapping a library extracted from a compatible-license major browser (Chromium or Firefox) and keeping it updated would avoid disparities. Unfortunately, I'm not sure how feasible this really is. Do all of the API surfaces we must support in the stdlib for compatibility's sake with urllib line up with such a browser core URL parsing library? Something to ponder. Unlikely something we'll actually do. -- ___ Python tracker <https://bugs.python.org/issue43883> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41232] Python `functools.wraps` doesn't deal with defaults correctly
Gregory P. Smith added the comment: That makes sense. It seems like we're missing information signature() would need to be able to differentiate between when it should default to follow .__wrapped__ vs when it shouldn't. Neither default can satisfy everyone. -- ___ Python tracker <https://bugs.python.org/issue41232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment: rejecting. code trying to make direct use of __defaults__ is likely better off using inspect.signature(). there might be an issue with inspect in some cases (https://bugs.python.org/issue41232) but I do not believe that is true for lru_cache wrapped things. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41232] Python `functools.wraps` doesn't deal with defaults correctly
Gregory P. Smith added the comment: Even if we shouldn't blindly propagate defaults (wrong in some wrapping scenarios), there is still a problem here as noted in Thor's opening message. A bug exists between functools.wraps and inspect.signature. The signature reported is wrong. why? -- ___ Python tracker <https://bugs.python.org/issue41232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment: Oh, I didn't realize mutating those would actually change the code runtime behavior. But it makes sense, those are needed before the code object is entered. Yeah that is different, and suggests making this the default is not actually desired. (this issue and the other one) I guess our rule is that introspection code really must check for and be ready to handle .__wrapped__ if its goal is robustness? -- ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Change by Gregory P. Smith : -- type: behavior -> enhancement versions: -Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment: https://bugs.python.org/issue41232 covers the more general case of suggesting changing update_wrapper's behavior. That would alleviate the need to fix the pure python implementation within my own PR. -- ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41232] Python `functools.wraps` doesn't deal with defaults correctly
Gregory P. Smith added the comment: I'm working on a more specialized case of this for functools.lru_cache in bpo-issue44003. But I believe this issue also makes sense. It is basically suggesting that __defaults__ and __kwdefaults__ should be included in the default set of assigned= attributes that functools.update_wrapper applies. The argument in favor of this is that it already has __annotations__ in the default set. With that, the default isn't really correct for use on any decorator that modifies the meaning or overall call signature of the function if it happens to have annotations. as __annotations__ and __defaults__ and __kwdefaults__ are all abstract callable introspection interfaces. >From Python's perspective some may call each of these an Enhancement rather >than a Bugfix. I'm fine with that. It doesn't come up super often but would >make Python things more consistent if it were done. Your PR makes sense to me, though we'll need to understand and fix that test failure you identified where it appears to be potentially testing for the wrong thing (I haven't studied it enough study to say yet). -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue41232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment: An inner function can't know if somebody else might want to inspect it. This is a decorator that does not change anything about the argument signature of the wrapped function, carrying over the reference to meta-information about that by default seems to make sense. -- versions: +Python 3.11 ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment: __defaults__ and __kwdefaults__ get used for code introspection. Just as __annotations__ does. __annotations__ is already available on the lru_cache wrapped function. All of those seem to go together from a runtime inspection point of view. -- ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +24487 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/25800 ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment:
the pure python functools.lru_cache doesn't get this right either. here's a
desirable testcase for this bug:
```
def test_lru_defaults_bug44003(self):
@self.module.lru_cache(maxsize=None)
def func(arg='ARG', *, kw='KW'):
return arg, kw
self.assertEqual(func.__wrapped__.__defaults__, ('ARG',))
self.assertEqual(func.__wrapped__.__kwdefaults__, {'kw': 'KW'})
self.assertEqual(func.__defaults__, ('ARG',))
self.assertEqual(func.__kwdefaults__, {'kw': 'KW'})
```
results in
```
ERROR: test_lru_defaults_bug44003 (test.test_functools.TestLRUC)
--
Traceback (most recent call last):
File "/Users/greg/oss/python/gpshead/Lib/test/test_functools.py", line 1738,
in test_lru_defaults_bug44003
self.assertEqual(func.__defaults__, ('ARG',))
AttributeError: 'functools._lru_cache_wrapper' object has no attribute
'__defaults__'
==
FAIL: test_lru_defaults_bug44003 (test.test_functools.TestLRUPy)
--
Traceback (most recent call last):
File "/Users/greg/oss/python/gpshead/Lib/test/test_functools.py", line 1738,
in test_lru_defaults_bug44003
self.assertEqual(func.__defaults__, ('ARG',))
AssertionError: None != ('ARG',)
```
--
___
Python tracker
<https://bugs.python.org/issue44003>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
Gregory P. Smith added the comment:
That was anecdotal evidence:
```
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec 7 2020, 12:10:52)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def func(arg=1, *, kwarg=2): pass
...
>>> import functools
>>> from functools import lru_cache, update_wrapper
>>> @lru_cache
... def cached_func(arg=1, *, kwargs=2): pass
...
>>> def x(*args, **kwargs): func(*args, **kwargs)
...
>>> updated_x = update_wrapper(func, x)
>>> x
>>> updated_x
>>> updated_x.__defaults__
(1,)
>>> updated_x.__kwdefaults__
{'kwarg': 2}
>>> func.__defaults__
(1,)
>>> func.__kwdefaults__
{'kwarg': 2}
>>> cached_func.__defaults__
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'functools._lru_cache_wrapper' object has no attribute
'__defaults__'
>>> cached_func.__kwdefaults__
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'functools._lru_cache_wrapper' object has no attribute
'__kwdefaults__'
```
--
___
Python tracker
<https://bugs.python.org/issue44003>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44002] Use functools.lru_cache in urllib.parse instead of 1996 custom caching
Change by Gregory P. Smith : -- keywords: +patch pull_requests: +24486 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25798 ___ Python tracker <https://bugs.python.org/issue44002> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44003] functools.lru_cache omits __defaults__ attribute from wrapped function
New submission from Gregory P. Smith : When the C implementation of functools.lru_cache was added in bpo/issue14373, it appears to have omitted setting `.__defaults__` on its wrapped function. ``` Python 3.10.0a7+ (heads/master-dirty:823fbf4e0e, May 1 2021, 11:10:30) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import functools >>> def func(b=5): pass ... >>> @functools.lru_cache ... def cached_func(b=5): pass ... >>> func.__defaults__ (5,) >>> cached_func.__defaults__ Traceback (most recent call last): File "", line 1, in AttributeError: 'functools._lru_cache_wrapper' object has no attribute '__defaults__' ``` functools.update_wrapper() does set __defaults__ so this appears to just be an oversight in Modules/_functoolsmodule.c. -- components: Library (Lib) keywords: 3.5regression messages: 392623 nosy: gregory.p.smith, serhiy.storchaka priority: normal severity: normal stage: needs patch status: open title: functools.lru_cache omits __defaults__ attribute from wrapped function type: behavior versions: Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue44003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44002] Use functools.lru_cache in urllib.parse instead of 1996 custom caching
Gregory P. Smith added the comment: Yeah, the Quoter class seems a little odd... Past notes of some the caching performance around the character quoting for quote() can be found in https://bugs.python.org/issue1285086 circa 2005-2010. -- nosy: -rhettinger ___ Python tracker <https://bugs.python.org/issue44002> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44002] Use functools.lru_cache in urllib.parse instead of 1996 custom caching
New submission from Gregory P. Smith : `urllib.parse` has custom caching code for both `urlsplit()` and `quote()`. From 1996. https://github.com/python/cpython/commit/3fd32ecd9232fcb041b9f1f7a19a1e7e65cf11a0 https://github.com/python/cpython/commit/74495409861b357d9925937d6576229c74e2550d with a truthful comment added by Nick in 2010 that we should just use functools.lru_cache. https://github.com/python/cpython/commit/9fc443cf590c76d4b979c46dc954d3956cee0319#diff-b3712475a413ec972134c0260c8f1eb1deefb66184f740ef00c37b4487ef873e time to clean up this cruft and do that. I'm waiting for after the 3.10 cut and a still in progress urllib.parse security fix to land before rebasing my soon to be attached PR to avoid code conflicts. -- assignee: gregory.p.smith components: Library (Lib) messages: 392614 nosy: gregory.p.smith priority: normal severity: normal status: open title: Use functools.lru_cache in urllib.parse instead of 1996 custom caching type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue44002> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43882] [security] urllib.parse should sanitize urls containing ASCII newline and tabs.
Gregory P. Smith added the comment: I think there's still a flaw in the fixes implemented in 3.10 and 3.9 so far. We're closer, but probably not quite good enough yet. why? We aren't stripping the newlines+tab early enough. I think we need to do the stripping *right after* the _coerce_args(url, ...) call at the start of the function. Otherwise we (1) are storing url variants with the bad characters in _parse_cache [a mere slowdown in the worst case as it'd just overflow the cache sooner] (2) are splitting the scheme off the URL prior to stripping. in 3.9+ there is a check for valid scheme characters, which will defer to the default scheme when found. The WHATWG basic url parsing has these characters stripped before any parts are split off though, so 'ht\rtps' - for example - would wind up as 'https' rather than our behavior so far of deferring to the default scheme. I noticed this when reviewing the pending 3.8 PR as it made it more obvious due to the structure of the code and would've allowed characters through into query and fragment in some cases. https://github.com/python/cpython/pull/25726#pullrequestreview-649803605 -- ___ Python tracker <https://bugs.python.org/issue43882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41486] Add _BlocksOutputBuffer for bz2/lzma/zlib module
Gregory P. Smith added the comment: New changeset 251ffa9d2b16b091046720628deb6a7906c35d29 by Ma Lin in branch 'master': bpo-41486: Fix initial buffer size can't > UINT32_MAX in zlib module (GH-25738) https://github.com/python/cpython/commit/251ffa9d2b16b091046720628deb6a7906c35d29 -- ___ Python tracker <https://bugs.python.org/issue41486> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41486] Add _BlocksOutputBuffer for bz2/lzma/zlib module
Gregory P. Smith added the comment: Renaming to OutputBuffer sounds like a good idea. On Thu, Apr 29, 2021, 7:55 PM Ma Lin wrote: > > Ma Lin added the comment: > > Found a backward incompatible behavior. > > Before the patch, in 64-bit build, zlib module allows the initial size > > UINT32_MAX. > It creates a bytes object, and uses a sliding window to deal with the > UINT32_MAX limit: > https://github.com/python/cpython/blob/v3.9.4/Modules/zlibmodule.c#L183 > > After the patch, when init_size > UINT32_MAX, it raises a ValueError. > > PR 25738 fixes this backward incompatibility. > If the initial size > UINT32_MAX, it clamps to UINT32_MAX, rather than > raising an exception. > > Moreover, if you don't mind, I would like to take this opportunity to > rename the wrapper functions from Buffer_* to OutputBuffer_*, so that the > readers can easily distinguish between input buffer and output buffer. > If you don't think it's necessary, you may merge PR 25738 as is. > > -- > > ___ > Python tracker > <https://bugs.python.org/issue41486> > ___ > -- ___ Python tracker <https://bugs.python.org/issue41486> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
