[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows

[Eryk Sun]

Eryk Sun  added the comment:

> os.write(fd) can't ... know that the fd was opened with O_APPEND

It's possible to query the granted access of a kernel handle via NtQueryObject: 
ObjectBasicInformation -- not that I'm suggesting to use it.

> the most serious one is rather silly: we need to honor the current 
> umask to possibly create a read-only file, 

It pains me to see umask get in the way of implementing open() directly via 
CreateFileW, which we need in order to support delete-access sharing. 

Python could implement its own umask value in Windows. os.umask() would set the 
C umask value as well, but only for the sake of consistency with C extensions 
and embedding.

> os module exposes some MSVCRT-specific flags for use with os.open() 
> (like O_TEMPORARY), which a reimplementation would have to support. 

Additionally, ucrt has an undocumented O_OBTAIN_DIR flag. It opens with backup 
semantics, which would be more obvious if aliased as O_BACKUP_SEMANTICS. This 
allows an open to take advantage of SeBackupPrivilege and SeRestorePrivilege if 
they're enabled, to get read or write access regardless of the file security.

Open attribute flags could also be supported, such as O_ATTR_HIDDEN and 
O_ATTR_SYSTEM. These are needed because a hidden or system file is required to 
remain as such when it's overwritten, else CreateFileW fails.

> It seems easy in most cases, but there is O_TEXT, 

Retaining O_TEXT in 3.x was probably an accident. Text mode should use a 
TextIOWrapper instance, and I doubt that there's a serious need to support 
Ctrl+Z as EOF. 

If it's important enough, msvcrt.open() and msvcrt.O_TEXT could be provided.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42974] tokenize reports incorrect end col offset and line string when input ends without explicit newline

[Brian Romanowski]

Change by Brian Romanowski :


--
keywords: +patch
pull_requests: +23085
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24260

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42974] tokenize reports incorrect end col offset and line string when input ends without explicit newline

[Brian Romanowski]

New submission from Brian Romanowski :

The tokenize module's tokenizer functions output incorrect (or at least 
misleading) information when the content being tokenized does not end in a line 
ending character.  This is related to the fix for issue<33899>  which added the 
NEWLINE tokens for this case but did not fill out the whole token tuple 
correctly.

The bug can be seen by running a version of the test in 
Lib/test/test_tokenize.py:

import io, tokenize

newline_token_1 = 
list(tokenize.tokenize(io.BytesIO("x\n".encode('utf-8')).readline))[-2]
newline_token_2 = 
list(tokenize.tokenize(io.BytesIO("x".encode('utf-8')).readline))[-2]

print(newline_token_1)
print(newline_token_2)

# Prints:
# TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 1), end=(1, 2), line='x\n')
# TokenInfo(type=4 (NEWLINE), string='', start=(1, 1), end=(1, 2), line='')  # 
bad "end" and "line"!

Notice that "len(newline_token_2.string) == 0" but "newline_token_2.end[1] - 
newline_token_2.start[1] == 1".  Seems more consistent if the 
newline_token_2.end == (1, 1).

Also, newline_token_2.line should hold the physical line rather than the empty 
string.  This would make it consistent with newline_token_1.line.

I'll add a PR shortly with a change so the output from the two cases is:

TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 1), end=(1, 2), line='x\n')
TokenInfo(type=4 (NEWLINE), string='', start=(1, 1), end=(1, 1), line='x')

If this looks reasonable, I can backport it for the other branches.  Thanks!

--
components: Library (Lib)
messages: 385313
nosy: romanows
priority: normal
severity: normal
status: open
title: tokenize reports incorrect end col offset and line string when input 
ends without explicit newline
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

Yet another use case: the friendly-traceback project detects typos on imports 
using the list of stdlib module names:
https://aroberge.github.io/friendly-traceback-docs/docs/html/tracebacks_en_3.8.html#standard-library-module

For example, it suggest to replace "import Tkinter" with "import tkinter".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

The pants project also uses a list of stdlib module names, similar to isort, to 
infer dependencies:
https://github.com/pantsbuild/pants/tree/master/src/python/pants/backend/python/dependency_inference/python_stdlib

"Pants is a scalable build system for monorepos: codebases containing multiple 
projects, often using multiple programming languages and frameworks, in a 
single unified code repository."

See https://blog.pantsbuild.org/dependency-inference/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

sys.module_names also solves the following old StackOverflow question:
"How to check if a module/library/package is part of the python standard 
library?"
https://stackoverflow.com/questions/22195382/how-to-check-if-a-module-library-package-is-part-of-the-python-standard-library

"I have installed sooo many libraries/modules/packages with pip and now I 
cannot differentiate which is native to the python standard library and which 
is not. This causes problem when my code works on my machine but it doesn't 
work anywhere else."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35829] datetime: parse "Z" timezone suffix in fromisoformat()

[Daniel Lenski]

Daniel Lenski  added the comment:

Like many others here, I've run into this issue because I'm trying to parse 
timestamps from JSON.

(Specifically, I'm trying to parse timestamps from JSON serialization of Java 
POJOs and/or Kotlin data classes, as serialized by the Jackson serialization 
library for JVM languages, in conjunction with JavaTimeModule.
https://fasterxml.github.io/jackson-modules-java8/javadoc/datetime/2.9/com/fasterxml/jackson/datatype/jsr310/JavaTimeModule.html)

In order to "be lenient in what I accept" (adhering to the robustness 
principal), I need to add a special case for deserialization of strings ending 
with 'Z'. This gets pretty tricky and pretty subtle quickly.

Here is my Python 3.7+ code path (the strptime-based code path for earlier 
versions is much, much uglier).

from numbers import Number
from datetime import datetime, timezone
def number_or_iso8601_to_dt(ts, t=datetime):
if isinstance(ts, Number):
return datetime.utcfromtimestamp(ts).replace(tzinfo=timezone.utc)
elif ts.endswith('Z'):
# This is not strictly correct, since it would accept a string with
# two timezone specifications (e.g. ending with +01:00Z) and 
# silently pass that erroneous representation:
#
# return 
datetime.fromisoformat(ts[:-1]).replace(tzinfo=timezone.utc)
#
# This version is better:
d = datetime.fromisoformat(ts[:-1])
if d.tzinfo is not None:
raise ValueError(f"time data '{ts}' contains multiple timezone 
suffixes")
return d.replace(tzinfo=timezone.utc)
else:
return datetime.fromisoformat(ts)

I don't really understand why .fromisoformat() must be *strictly* the inverse 
of .isoformat(). As @mehaase points out, the transformation isn't strictly 
reversible as is.

There are other functions where the Python standard library has special-cased 
options for extremely common use cases. For example, `str.split(None)`, which 
is certainly not the inverse of the non-existent `None.join()`.

This feels to me like a case where the standard library should simply just 
accept an extremely-common real-world variant in the interests of 
interoperability.

I would also be in favor of @p-ganssle's proposal (3), wherein 
`datetime.isoformat` would also output the 'Z' suffix for the UTC timezone.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42957] os.readlink produces wrong result on windows

[Eryk Sun]

Eryk Sun  added the comment:

os.readlink() was generalized to support mountpoints (junctions) as well as 
symlinks, and it's more common for mountpoints to lack the print name field in 
the reparse data buffer [1]. For example, PowerShell's new-item creates 
junctions that only have a substitute path. This is allowed since the 
filesystem protocols only require that the substitute path is valid in the 
reparse data buffer, since that's all that the system actually uses when 
resolving the reparse point.

The substitute path in the reparse point is a \\?\ prefixed path (actually a 
\??\ NT path, but they're effectively the same for our purposes). This type of 
path is usually called an extended path -- or an extended device path, or 
verbatim path. It's a device path, like the \\.\ prefix, except that (1) it's 
verbatim (i.e. not normalized), (2) its length is never limited to MAX_PATH 
(260) characters, and (3) the Windows file API supports the \\?\ prefix more 
broadly than the \\.\ prefix.

You're right that some programs can't grok an extended path. Some can't even 
handle any type of UNC path. I agree that we need a simple way to remove the 
prefix. I just don't agree that removing it should be the default behavior in 
nt.readlink(), which I prefer to keep efficient and free of race conditions.

os.path.realpath() isn't necessarily what you want since it resolves the final 
path. The link may target a path that traverses any number of reparse points 
and mapped drives, so the final path may be completely different from the 
os.readlink() result. We simply need an option to remove the \\?\ or \\?\UNC 
prefix, either always or only when the path doesn't require it. It could be 
implemented by a high-level wrapper function in os.py.

---
Reasons the prefix may be required

If the length of the target path exceeds MAX_PATH, then removing the prefix may 
render the path inaccessible if the current process doesn't support long paths 
without it (e.g. Windows 10 without long paths enabled at the system level, or 
any version prior to Windows 10).

Also, reserved DOS device names are only accessible using an extended path. Say 
I have the following "spam" junction:

>>> print(os.readlink('spam'))
\\?\C:\Temp\con

The junction allows accessing the target directory normally:

>>> stat.S_ISDIR(os.stat('spam').st_mode)
True

But look what happens when I try to access the target path without the prefix:

>>> stat.S_ISDIR(os.stat(r'C:\Temp\con').st_mode)
False
>>> stat.S_ISCHR(os.stat(r'C:\Temp\con').st_mode)
True

Instead of the directory that one might expect, it's actually a character 
device!? Let's see what Windows opens:

>>> print(os.path.abspath(r'C:\Temp\con'))
\\.\con

It opens the "CON" device, for console I/O. It turns out that a bunch of names 
are reserved, including NUL, CON, CONIN$, CONOUT$, AUX, PRN, COM<1-9>, and 
LPT<1-9>. They're reserved even with an extension introduced by a colon or dot, 
preceded by zero or more spaces. For example:

>>> print(os.path.abspath(r'C:\Temp\con :whatever'))
\\.\con

Directly accessing such a name in the filesystem requires a verbatim path. For 
example:

>>> stat.S_ISDIR(os.stat(r'\\?\C:\Temp\con').st_mode)
True

Using reserved names is cautioned against, but in the real world we have to be 
defensive. We can't simply remove the prefix and hope for the best.

---

[1] 
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_reparse_data_buffer

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35829] datetime: parse "Z" timezone suffix in fromisoformat()

[Daniel Lenski]

Change by Daniel Lenski :


--
nosy: +dlenski

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42864] Improve error messages regarding unclosed parentheses

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42864] Improve error messages regarding unclosed parentheses

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset d6d6371447357c9c69b0936573977a3e60f2 by Pablo Galindo in 
branch 'master':
bpo-42864: Improve error messages regarding unclosed parentheses (GH-24161)
https://github.com/python/cpython/commit/d6d6371447357c9c69b0936573977a3e60f2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +paul.j3, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42923] Py_FatalError(): dump the list of extension modules

[STINNER Victor]

STINNER Victor  added the comment:

More "real world" example using cinder which imports 220 modules (ctypes is 
used to simulate a crash):
---
$ ./python -m venv env
$ env/bin/python -m pip install wheel
$ env/bin/python -m pip install cinder
$ env/bin/python -X dev -c 'import cinder, ctypes, sys; 
print(f"{len(sys.modules)=}"); print(); ctypes.string_at(0)'
(...)
len(sys.modules)=220

Fatal Python error: Segmentation fault

Current thread 0x7fc4a88e0740 (most recent call first):
  File "/home/vstinner/python/master/Lib/ctypes/__init__.py", line 517 in 
string_at
  File "", line 1 in 

Extension modules: greenlet._greenlet, __original_module__thread, 
__original_module_select, __original_module_time, _cffi_backend (total: 5)
Erreur de segmentation (core dumped)
---

cinder only uses 2 third party extension modules (on a total of 220 modules): 
greenlet and cffi.

Note: __original_xxx modules are aliases of stdlib modules created by eventlet 
monkey patching.

So if cinder does crash, I suggest to look at Python, but *also* look at these 
2 extensions ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)

[Eryk Sun]

Eryk Sun  added the comment:

> That's what Lib/test/win_console_handler.py:39 does. 

No, that script is currently broken.

SetConsoleCtrlHandler(None, False) clears the Ctrl+C ignore flag in the PEB 
ProcessParameters, which is normally inherited by child processes. But using 
the CREATE_NEW_PROCESS_GROUP creation flag always causes Ctrl+C to be initially 
ignored in the process. The intent is to make it a 'background' process by 
default, to support CMD's `start /b` option.

Win32KillTests in Lib/test/test_os.py incorrectly assumes it only has to enable 
Ctrl+C in the parent process. win_console_handler.py incorrectly assumes it 
does not have to call SetConsoleCtrlHandler(None, False). This is why 
test_CTRL_C_EVENT() is currently being skipped. To fix it, 
win_console_handler.py needs to call SetConsoleCtrlHandler(None, False).

> Isn't that what PyConfig.install_signal_handlers is supposed to do?

The C runtime's signal() function does not clear the Ctrl+C ignore flag when a 
SIGINT handler is set. I think Python should. It's expected behavior for POSIX 
compatibility.

>with self.assertRaises(KeyboardInterrupt):
>os.kill(os.getpid(), SIG)

Unless you know the current process was created in a new process group, it is 
not correct to call os.kill(os.getpid(), signal.CTRL_C_EVENT). You'd be better 
off using os.kill(0, signal.CTRL_C_EVENT) and ensuring that Ctrl+C is 
temporarily ignored in the parent process, if that's in your control. Also, 
Python 3.8 added raise_signal(), which calls C raise(), so a more reliable way 
to test just the signal handler is to call signal.raise_signal(signal.SIGINT). 
This also eliminates needing to synchronize with the console control thread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42958] filecmp.cmp(shallow=True) isn't actually shallow when only mtime differs

[Josh Rosenberg]

Josh Rosenberg  added the comment:

This is a problem with the docstring. The actual docs for it are a bit more 
clear, https://docs.python.org/3/library/filecmp.html#filecmp.cmp :

"If shallow is true, files with identical os.stat() signatures are taken to be 
equal. Otherwise, the contents of the files are compared."

Your patch can't be used because it changes longstanding documented behavior. 
If you'd like to submit a patch to fix the docstring, that's fine, but we're 
not going to break existing code to make the function less accurate.

The patch should probably make the documentation more clear while it's at it.

1. The original wording could be misinterpreted as having the "Otherwise" apply 
to shallow=False only, not to the case where shallow=T rue but os.stat doesn't 
match.
2. The existing wording isn't clear on what an os.stat() "signature" is, which 
can leave the impression that the entirety of os.stat is compared (which would 
only match for hardlinks of the same file), when in fact it's just the file 
type (stat.S_IFMT(st.st_mode), file vs. directory vs. symlink, etc.), size and 
mtime.

Proposed rewording of main docs would be:

"If shallow is true, files with identical os.stat() signatures (file type, 
size, and modification time) are taken to be equal. When shallow is false, or 
the file signatures are identical, the contents of the files are compared."

A similar wording (or at least, a shorter version of the above, rather than a 
strictly wrong description of the shallow parameter) could be applied to the 
docstring.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42923] Py_FatalError(): dump the list of extension modules

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42923] Py_FatalError(): dump the list of extension modules

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 66f77caca39ba39ebe1e4a95dba6d19b20d51951 by Victor Stinner in 
branch 'master':
bpo-42923: _Py_DumpExtensionModules() ignores stdlib ext (GH-24254)
https://github.com/python/cpython/commit/66f77caca39ba39ebe1e4a95dba6d19b20d51951


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
keywords: +patch
pull_requests: +23083
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
nosy: +monkeyman79
nosy_count: 9.0 -> 10.0
pull_requests: +23084
pull_request: https://github.com/python/cpython/pull/24259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42970] File path with blank causes open error in 3.8, not in 3.7

[Doug Day]

Doug Day  added the comment:

To clarify: either python version generates the same path. On 3.8.2 though an 
open error results

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

New submission from Tadek Kijkowski :

I have following use case for argparse:

./prog --tracks=80 image1 --tracks=40 image2 --tracks=80 image3 ...

I expected that the following parser would be able process that command line:

parser = argparse.ArgumentParser()
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

But it stops with 'unrecognized arguments: image2' error. It turns out that 
there is no way to prepare parser which could handle this command line.

There was a long discussion about this issue in #14191. There were two 
approaches proposed:
- Allow 'image' action to trigger multiple times with additional encountered 
parameters.
- Parse all optionals first and all positionals in second pass.

The first approach was represented by a patch by user guilherme-pg. This patch 
was slightly criticized (undeservedly IMO) and pretty much ignored.

The discussion then focused on the second approach which ended up with 
introduction of `parse_intermixed_args` into codebase.

The problem with `parse_intermixed_args` is that it defeats the purpose of 
having intermixed arguments. When the arguments are effectively reordered, 
there is no way to match "optionals" with "positionals" with they relate to.

In my option the Guilherme's approach was the correct way to go, I would say it 
was elegant. The only complaint against that patch is that it unnecessarily 
modified _StoreAction class - instead of that, the way to go is to use "extend" 
or custom action.

I allowed myself to simplify the changes a bit for readability. The patch 
doesn't change default functionality, for the cost of changing public API by 
adding new argument to ArgumentParser constructor.

With the changes applied, I can parse my command line with this code:

parser = argparse.ArgumentParser(greedy_star=True)
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

--
components: Library (Lib)
messages: 385301
nosy: monkeyman79
priority: normal
severity: normal
status: open
title: argparse: mixing optional and positional arguments... not again
type: enhancement
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42971] Some errnos for BSD/OSX are missing from errno module

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +ned.deily, ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cad8020cb83ec6d904f874c0e4f599e651022196 by Victor Stinner in 
branch 'master':
bpo-42955: Add Python/module_names.h (GH-24258)
https://github.com/python/cpython/commit/cad8020cb83ec6d904f874c0e4f599e651022196


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol

[STINNER Victor]

STINNER Victor  added the comment:

In June 2020, I create PR 20983 to attempt to automatically traverse the type:
"Provide a default tp_traverse implementation for the base object
type for heap types which have no tp_traverse function. The
traverse function visits the type if the type is a heap type."

I abandoned my PR.

I marked bpo-41036 as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41036] Visit the type of instance of heap types if tp_traverse is not implemented

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-42972 instead.

--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> [C API] Heap types (PyType_FromSpec) must fully implement the 
GC protocol

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42972] [C API] Heap types (PyType_FromSpec) must fully implement the GC protocol

[STINNER Victor]

New submission from STINNER Victor :

Copy of my email sent to python-dev:
https://mail.python.org/archives/list/python-...@python.org/thread/C4ILXGPKBJQYUN5YDMTJOEOX7RHOD4S3/

Hi,

In the Python stdlib, many heap types currently don't "properly"
(fully?) implement the GC protocol which can prevent to destroy these
types at Python exit. As a side effect, some other Python objects can
also remain alive, and so are not destroyed neither.

There is an on-going effect to destroy all Python objects at exit
(bpo-1635741). This problem is getting worse when subinterpreters are
involved: Refleaks buildbots failures which prevent to spot other
regressions, and so these "leaks" / "GC bugs" must be fixed as soon as
possible. In my experience, many leaks spotted by tests using
subinterpreters were quite old, it's just that they were ignored
previously.

It's an hard problem and I don't see any simple/obvious solution right
now, except of workarounds that I dislike. Maybe the only good
solution is to fix all heap types, one by one.

== Only the Python stdlib should be affected ==

PyType_FromSpec() was added to Python 3.2 by the PEP 384 to define
"heap types" in C, but I'm not sure if it's popular in practice (ex:
Cython doesn't use it, but defines static types). I expect that most
types to still be defined the old style (static types) in a vas
majority of third party extension modules.

To be clear, static types are not affected by this email.

Third party extension modules using the limited C API (to use the
stable ABI) and PyType_FromSpec() can be affected (if they don't fully
implement the GC protocol).

== Heap type instances now stores a strong reference to their type ==

In March 2019, the PyObject_Init() function was modified in bpo-35810
to keep a strong reference (INCREF) to the type if the type is a heap
type. The fixed problem was that heap types could be destroyed before
the last instance is destroyed.

== GC and heap types ==

The new problem is that most heap types don't collaborate well with
the garbage collector. The garbage collector doesn't know anything
about Python objects, types, reference counting or anything. It only
uses the PyGC_Head header and the traverse functions. If an object
holds a strong reference to an object but its type does not define a
traverse function, the GC cannot guess/infer this reference.

A heap type must respect the following 3 conditions to collaborate with the GC:

Have the Py_TPFLAGS_HAVE_GC flag;
Define a traverse function (tp_traverse) which visits the type: 
Py_VISIT(Py_TYPE(self));
Instances must be tracked by the GC.

If one of these conditions is not met, the GC can fail to destroy a
type during a GC collection. If an instance is kept alive late while a
Python interpreter is being deleted, it's possible that the type is
never deleted, which can keep indirectly many objects alive and so
don't delete them neither.

In practice, when a type is not deleted, a test using subinterpreter
starts to fail on Refleaks buildbot since it leaks references. Without
subinterpreters, such leak is simply ignored, whereas this is an
on-going effect to delete Python objects at exit (bpo-1635741).

== Boring traverse functions ==

Currently, there is no default traverse implementation which visits the type.

For example, I had the implement the following function for _thread.LockType:

static int
lock_traverse(lockobject self, visitproc visit, void arg)
{
Py_VISIT(Py_TYPE(self));
return 0;
}

It's a little bit annoying to have to implement the GC protocol
whereas a lock cannot contain other Python objects, it's not a
container. It's just a thin wrapper to a C lock.

There is exactly one strong reference: to the type.

== Workaround: loop on gc.collect() ==

A workaround is to run gc.collect() in a loop until it returns 0 (no
object was collected).

== Traverse automatically? Nope. ==

Pablo Galindo attempts to automatically visit the type in the traverse function:

https://bugs.python.org/issue40217
https://github.com/python/cpython/commit/0169d3003be3d072751dd14a5c84748ab63...

Moreover, What's New in Python 3.9 contains a long section suggesting
to implement a traverse function for this problem, but it doesn't
suggest to track instances:
https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-c-api

This solution causes too many troubles, and so instead, traverse
functions were defined on heap types to visit the type.

Currently in the master branch, 89 types are defined as heap types on
a total of 206 types (117 types are defined statically). I don't think
that these 89 heap types respect the 3 conditions to collaborate with
the GC.

== How should we address this issue? ==

I'm not sure what should be done. Working around the issue by
triggering multiple GC collections? Emit a warning in development mode
if a heap type doesn't collaborate well with the GC?

If core developers miss these bugs and have troubles to debug them, I
expect that extension 

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +23082
pull_request: https://github.com/python/cpython/pull/24258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)

[William Schwartz]

William Schwartz  added the comment:

>For a new process group, the cancel event is initially ignored, but the break 
>event is always handled. To enable the cancel event, the process must call 
>SetConsoleCtrlHandler(NULL, FALSE), such as via ctypes with 
>kernel32.SetConsoleCtrlHandler(None, False). I think the signal module should 
>provide a function to enable/disable Ctrl+C handling without ctypes, and 
>implicitly enable it when setting a new SIGINT handler.

That's what Lib/test/win_console_handler.py:39 does. What I don't understand is 
why that's necessary. Isn't that what PyConfig.install_signal_handlers is 
supposed to do?

Which brings me to how I ended up here in the first place: I wanted to write a 
test that PyConfig.install_signal_handlers is set in an embedded instance of 
Python I'm working with. In outline, the following test works on both Windows 
and macOS *except on Windows running under Tox*.

@unittest.removeHandler
def test_signal_handlers_installed(self):
SIG = signal.SIGINT
if sys.platform == 'win32':
SIG = signal.CTRL_C_EVENT
with self.assertRaises(KeyboardInterrupt):
os.kill(os.getpid(), SIG)
if sys.platform == 'win32':
time.sleep(.1)  # Give handler's thread time to join

Using SetConsoleCtrlHandler if I detect that I'm running on Windows under Tox 
would, if I understand correctly, hide whether PyConfig.install_signal_handlers 
was set before the Python I'm running in started, right? (I know this isn't the 
right venue for discussing my embedding/testing problem. But maybe the use case 
informs the pending discussion of what to do about os.kill's semantics.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42971] Some errnos for BSD/OSX are missing from errno module

[Enji Cooper]

Enji Cooper  added the comment:

Some items that I've noted so far that are missing:

- ENOATTR (BSD/OSX)
- ENOLINK (BSD; is reserved on OSX)
- ENOTSUP (NetBSD, OpenBSD, OSX)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36344] install_certificates.command too complicated

[Ned Deily]

Ned Deily  added the comment:

> What exactly is the backport status for this issue, and is there anything 
> with more info for this user?

Python 3.6 has been in the security-fix-only phase of its life cycle since 
2018-12 which means that python.org binary installers for Mac or Windows 
platforms have not been produced since then. If at all possible, your user 
should upgrade to a fully-supported version, currently either Python 3.9.1 or 
3.8.7 (https://www.python.org/downloads/). That said, the last python.org macOS 
installers for 3.6, 3.6.8, still work for me on the most recent macOS Sierra 
update (10.12.6) including the "Install Certificates Command".  Make sure the 
user has admin privileges or that "Install Certificates Commnad" is run under a 
user that does have admin privs. If it still fails, you could try downloading 
and re-installing Python 3.6.8 or give more information about exactly what 
error is being seen.  But this sort of problem is really off-topic for this 
issue; better to ask in a help forum.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42971] Some errnos for BSD/OSX are missing from errno module

[Enji Cooper]

New submission from Enji Cooper :

Some errnos for BSD/OSX are currently not supported by the errno module. It 
would be helpful for these to be exposed to the end-user so they could 
programmatically use them instead of reinventing the wheel in a separate 
module/C extension, or hardcoding the information in consuming pieces of code.

--
components: Extension Modules
messages: 385293
nosy: ngie
priority: normal
severity: normal
status: open
title: Some errnos for BSD/OSX are missing from errno module
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36344] install_certificates.command too complicated

[Steve Arnold]

Steve Arnold  added the comment:

I can't seem to find a usable answer for a user on Sierra with a (supposedly) 
working python 3.6 install.  From what I can tell all the related bugs are 
closed except this one, and this one doesn't include python 3.6.  Running the 
install_certificates.command does not work in his case and double-clicking 
results in a permissions error.  I suspect the target system is not in the best 
shape, but I haven't touched a Mac since the powerpc days.

What exactly is the backport status for this issue, and is there anything with 
more info for this user?

--
nosy: +sarnold

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)

[Eryk Sun]

Eryk Sun  added the comment:

> Should there be a `return NULL;` between these two lines?

In 3.x, os.kill() has always fallen back on TerminateProcess() when 
GenerateConsoleCtrlEvent() fails, and I assumed that was intentional. But I 
misunderstood that it's not actually chaining the exception via PyErr_Fetch and 
_PyErr_ChainExceptions, so it still fails with a SystemError when both 
GenerateConsoleCtrlEvent() and TerminateProcess() fail -- e.g. os.kill(4, 1). 

I checked the history for bpo-1220212. It appears that win32_kill() was 
committed to the 2.x branch first with a missing return statement, which was 
fixed a day later:

https://github.com/python/cpython/commit/ae509520de5a0321f58c79afffad10ae59dae8b9

A few days later it was ported to 3.x without that fix:

https://github.com/python/cpython/commit/eb24d7498f3e34586fee24209f5630a58bb1a04b

If Steve Dower or Victor Stinner is okay with changing the behavior to agree 
with 2.x -- after more than a decade -- then I say just add the missing return 
statement. In that case, if os.kill() is called with a sig value of 
CTRL_C_EVENT (0) or CTRL_BREAK_EVENT (1), then it will only try 
GenerateConsoleCtrlEvent() and no longer fall back on TerminateProcess(). 

That said, it's common to call TerminateProcess() with an exit status of 1. 
Maybe the enum values of signal.CTRL_C_EVENT and signal.CTRL_BREAK_EVENT can be 
checked by identity instead of checking the integer value. That way 
os.kill(pid, 1) can still be used to forcefully terminate an arbitrary process 
with an exit status of 1.

---

To reiterate previous comments, the sig values of signal.CTRL_C_EVENT and 
signal.CTRL_BREAK_EVENT are not supported for an individual, arbitrary process. 
To use them, the pid value must be a process group ID, such as the pid of a 
process created with the flag CREATE_NEW_PROCESS_GROUP, and the process must be 
in the same console session as the current process. Anything else either fails 
(assuming no fallback on TerminateProcess) or invokes buggy behavior in the 
console, which can even leave the console in a broken state.

For a new process group, the cancel event is initially ignored, but the break 
event is always handled. To enable the cancel event, the process must call 
SetConsoleCtrlHandler(NULL, FALSE), such as via ctypes with 
kernel32.SetConsoleCtrlHandler(None, False). I think the signal module should 
provide a function to enable/disable Ctrl+C handling without ctypes, and 
implicitly enable it when setting a new SIGINT handler.

---
Hypothetical la la land...

I wish os.kill() in Windows had been implemented to conform with Unix kill(), 
and that CTRL_C_EVENT and CTRL_BREAK_EVENT were not added since they are not 
signals. Route negative pid values to the process-group branch that calls 
GenerateConsoleCtrlEvent(), and special case pid -1 to send the event to all 
accessible processes in the console session. Support SIGINT and SIGBREAK sig 
values in this case, respectively mapped to the console's cancel and break 
events. Don't support pid 0 in Windows, since there's no documented function to 
get the process group ID of the current process. Route positive pid values to 
the TerminateProcess() branch. Special case the sig value 0 to test for 
existence and access via OpenProcess(). Also, the latter should only request 
PROCESS_TERMINATE access, not PROCESS_ALL_ACCESS. It's common to have terminate 
access but not all access, e.g. to an elevated process in the current desktop 
session.

--
components: +Library (Lib) -Extension Modules

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42970] File path with blank causes open error in 3.8, not in 3.7

[Doug Day]

New submission from Doug Day :

The following code generates a path that works in Pythons 3.7.6 on macOS Big 
Sur but not in Catalina  with 3.8.2.. 
mySrcFldr="~/Library/Mobile Documents/com~apple~CloudDocs/Utilities/"
srcFldr=os.path.expanduser(mySrcFldr)
f=os.path.join(srcFldr,"mortgage.py")

When python tries to run the code file(montage.py) on 3.8.2, an open error 
results.
/Library/Developer/CommandLineTools/usr/bin/python3: can't open file 
'/Users/Doug/Library/Mobile': [Errno 2] No such file or directory
 I tried putting a backslash after 'Mobile'. Does not help.
The complete path is:
'/Users/Doug/Library/Mobile Documents/com~apple~CloudDocs/Utilities/mortgage.py'

While the doc changes show some changes in the os.path module, I see no mention 
of this issue.

--
components: Interpreter Core
messages: 385290
nosy: dday52
priority: normal
severity: normal
status: open
title: File path with blank causes open error in 3.8, not in 3.7
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

[Gregory P. Smith]

Change by Gregory P. Smith :


--
title: pthread_exit & PyThread_exit_thread are harmful -> pthread_exit & 
PyThread_exit_thread from PyEval_RestoreThread etc. are harmful

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42969] pthread_exit & PyThread_exit_thread are harmful

[Gregory P. Smith]

Gregory P. Smith  added the comment:

C-APIs such as `PyEval_RestoreThreads()` are insufficient for the task they are 
asked to do.  They return void, yet have a failure mode.

They call pthread_exit() on failure today.

Instead, they need to return an error to the calling application to indicate 
that "The Python runtime is no longer available."

Callers need to act on that in whatever way is most appropriate to them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42969] pthread_exit & PyThread_exit_thread are harmful

[Gregory P. Smith]

New submission from Gregory P. Smith :

## BACKGROUND

`PyThread_exit_thread()` calls `pthread_exit(`) and is in turn called from a 
variety of APIs as documented in the C-API doc update from 
https://bugs.python.org/issue36427.

The `pthread_exit()` call was originally introduced as a way "resolve" a 
crashes from daemon threads during shutdown https://bugs.python.org/issue1856.  
It did that.  That fix to that even landed in 2.7.8 but was rolled back before 
2.7.9 due to a bug in an existing application it exposed at the time (did we 
miss the implications of that? maybe).  It remained in 3.x.

## PROBLEM

`pthread_exit()` cannot be used blindly by any application.  All code in the 
threaded application needs to be on board with it and always prepared for any 
API call they make to potentially lead to thread termination.  Quoting a 
colleague: "pthread_exit() does not result in stack unwind or local variable 
destruction".  This means that any code up the stack from the ultimate 
`pthread_exit()` call that has process-wide state implications that did not go 
out of its way to register cleanup with `pthread_cleanup_push()` could lead to 
deadlocks or lost resources. Something implausible to assume that code does.

We're seeing this happen with C/C++ code.  Our C++ builds with 
`-fno-exceptions` so uncatchable exception based stack unwinding as some 
pthread_exit implementations may trigger does not happen (and cannot be 
guaranteed anyways, see bpo-42888).  Said C/C++ code is calling back into 
Python from a thread and thus must use `PyEval_RestoreThread()` or similar APIs 
before performing Python C API calls.  If the interpreter is being finalized 
from another thread... these enter a codepath that ultimately calls 
`pthread_exit()` leaving corrupt state in the process.  In this case that 
unexpected thread simply disappearing can lead to a deadlock in our process.

Fundamentally I do not believe the CPython VM should ever call `pthread_exit()` 
when non-CPython frames are anywhere in the C stack.  This may mean we should 
never call `pthread_exit()` at all (unsure; but it'd be ideal).

The documentation suggests that all callers in user code of the four C-APIs 
with the documented `pthread_exit()` caveats need auditing and pre-call 
`_Py_IsFinalizing()` API checks.  But... I do not believe that would fix 
anything even if it were done.  `_Py_IsFinalizing()` called without the GIL 
held means that it could change by the time the `PyEval_RestoreThreads()` API 
calls it internally do determine if it should exit the thread.  Thus the race 
condition window would merely be narrowed, not eliminated.  Not good enough.

## CURRENT WORKAROUND (Big Hammer)

Change CPython to call abort() instead of pthread_exit() as that situation is 
unresolvable and the process dying is better than hanging, partially alive.  
That solution isn't friendly, but is better than being silent and allowing 
deadlock.  A failing process is always better than a hung process, especially a 
partially hung process.

## SEMI RELATED WORK

https://bugs.python.org/issue42888 - appears to be avoiding some 
`PyThread_exit_thread()` calls to stop some crashes due to `libgcc_s` being 
loaded on demand upon thread exit.

--
components: Interpreter Core
keywords: 3.2regression
messages: 385288
nosy: gregory.p.smith
priority: normal
severity: normal
status: open
title: pthread_exit & PyThread_exit_thread are harmful
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42957] os.readlink produces wrong result on windows

[simon mackenzie]

simon mackenzie  added the comment:

I note os.path.realpath("v1") does produce the right path in windows. Maybe
that is what you meant. Will that work cross-platform?

On Tue, 19 Jan 2021 at 18:48, simon mackenzie  wrote:

> For most people the expectation would be that it returns a path in the
> same format as any other path. Furthermore it seems odd to change the
> default behaviour after years when it worked as expected. I never heard of
> this substitute path before and it does not work in some circumstances e.g.
> docker does not recognise it.
>
> Note also that os.path.realpath(os.path.readlink("v1")) still returns
> ?\\d:\\v1. There needs to be some way of getting to the path that
> everyone actually uses.
>
> On Mon, 18 Jan 2021 at 21:25, Eryk Sun  wrote:
>
>>
>> Eryk Sun  added the comment:
>>
>> Symlinks and mountpoints (aka junctions) contain two forms of the target
>> path. There's a path that's intended for display in the shell, and there's
>> the actual substitute path to which the link resolves. os.readlink() was
>> changed to return the substitute path because the display form is not
>> mandated by filesystem protocols (it's sometimes missing, especially for
>> junctions) and not reliable (e.g. the display path may be a long path or
>> contain reserved names such that it's not valid without the \\?\ prefix).
>> It was decided to keep the C implementation of os.readlink() simple.
>> Whether to retain the \\?\ prefix was shifted to high-level functions that
>> consume the result of os.readlink(), such as os.path.realpath().
>>
>> There was a previous issue related to this, in that the shutil module
>> copies symlinks via os.readlink() and os.symlink(), which thus copies only
>> the substitute path now. The issue was closed as not a bug, but had it been
>> resolved with new functionality, I would have preferred to do so with a
>> low-level function to copy a reparse point, not by reverting the behavior
>> of os.readlink(). I also see no reason against adding an option to
>> readlink() to return the display path instead of the substitute path, or to
>> just remove the prefix. But I'd vote against making it the default behavior.
>>
>> --
>> components: +Library (Lib)
>> nosy: +eryksun
>> versions: +Python 3.10
>>
>> ___
>> Python tracker 
>> 
>> ___
>>
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42957] os.readlink produces wrong result on windows

[simon mackenzie]

simon mackenzie  added the comment:

For most people the expectation would be that it returns a path in the same
format as any other path. Furthermore it seems odd to change the default
behaviour after years when it worked as expected. I never heard of this
substitute path before and it does not work in some circumstances e.g.
docker does not recognise it.

Note also that os.path.realpath(os.path.readlink("v1")) still returns
?\\d:\\v1. There needs to be some way of getting to the path that
everyone actually uses.

On Mon, 18 Jan 2021 at 21:25, Eryk Sun  wrote:

>
> Eryk Sun  added the comment:
>
> Symlinks and mountpoints (aka junctions) contain two forms of the target
> path. There's a path that's intended for display in the shell, and there's
> the actual substitute path to which the link resolves. os.readlink() was
> changed to return the substitute path because the display form is not
> mandated by filesystem protocols (it's sometimes missing, especially for
> junctions) and not reliable (e.g. the display path may be a long path or
> contain reserved names such that it's not valid without the \\?\ prefix).
> It was decided to keep the C implementation of os.readlink() simple.
> Whether to retain the \\?\ prefix was shifted to high-level functions that
> consume the result of os.readlink(), such as os.path.realpath().
>
> There was a previous issue related to this, in that the shutil module
> copies symlinks via os.readlink() and os.symlink(), which thus copies only
> the substitute path now. The issue was closed as not a bug, but had it been
> resolved with new functionality, I would have preferred to do so with a
> low-level function to copy a reparse point, not by reverting the behavior
> of os.readlink(). I also see no reason against adding an option to
> readlink() to return the display path instead of the substitute path, or to
> just remove the prefix. But I'd vote against making it the default behavior.
>
> --
> components: +Library (Lib)
> nosy: +eryksun
> versions: +Python 3.10
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42968] multiprocessing handle leak on Windows when child process is killed during startup/unpickling

[Daniel Grunwald]

Daniel Grunwald  added the comment:

Fix idea: get_spawning_popen().pid could be used to directly copy the handle 
into the child process, thus avoiding the temporary copy in the main process.
This would help at least in our case (where we pass all connections during 
startup).

I don't know if the general case is solvable -- in general we don't know which 
process will unpickle the data, and "child process is killed" isn't the only 
possible reason why the call to rebuild_pipe_connection() might not happen 
(e.g. exception while unpickling an earlier part of the same message).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42968] multiprocessing handle leak on Windows when child process is killed during startup/unpickling

[Daniel Grunwald]

New submission from Daniel Grunwald :

Running the attached script deadlocks.
Uncommenting the `time.sleep(1)` in the script makes the deadlock disappear.

For context: our application uses multiple child processes 
(multiprocessing.Process) and uses pipes (multiprocessing.Pipe) to communicate 
with them.
If one process fails with an error, the main process will kill all other child 
processes running concurrently.
We noticed that sometimes (non-deterministically), when an error occurs soon 
after startup, the main process ends up hanging.

We expect that when we pass the writing half of a connection to a child process 
and close the connection in the main process, that we will receive EOFError if 
the child process terminates unexpectedly.
But sometimes the EOFError never comes and our application hangs.

I've reduced the problem to the script attached. With the reduced script, the 
deadlock happens reliably for me.

I've debugged this a bit, and I think this is happening because passing a 
connection to the process being started involves reduce_pipe_connection() which 
creates a copy of the handle within the main process.
When the pickled data is unpickled in the child process, it uses 
DUPLICATE_CLOSE_SOURCE to close the copy in the main process.
But if the pickled data is never unpickled by the child process, the handle 
ends up being leaked.
Thus the main process itself holds the writing half of the connection open, 
causing the recv() call on the reading half to block forever.

--
components: Windows
files: deadlock.py
messages: 385283
nosy: dgrunwald, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: multiprocessing handle leak on Windows when child process is killed 
during startup/unpickling
type: behavior
versions: Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49751/deadlock.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42896] Solaris 11.4 crle output not handled correctly

[David Murphy]

Change by David Murphy :


--
pull_requests: +23080
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24226

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42035] [C API] PyType_GetSlot cannot get tp_name

[STINNER Victor]

STINNER Victor  added the comment:

A type has different names:

* short name: "name"
* qualified name: "module.name"

Which one should be returned by PyType_GetName()? Is there a warranty that it's 
always short or always qualified?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42035] [C API] PyType_GetSlot cannot get tp_name

[STINNER Victor]

STINNER Victor  added the comment:

New C API functions must not return borrowed references, but strong references.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41713] _signal module leak: test_interpreters leaked [1424, 1422, 1424] references

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e8e66eab941b983b6e85cd0d57cd45838880c568 by Victor Stinner in 
branch 'master':
bpo-41713: Remove PyOS_InitInterrupts() from python3dll.c (GH-24257)
https://github.com/python/cpython/commit/e8e66eab941b983b6e85cd0d57cd45838880c568


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42887] 100000 assignments of .__sizeof__ cause a segfault on del

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[STINNER Victor]

STINNER Victor  added the comment:

Ee:
> Apologies, I gisted a version that was from attempts to debug the timeouts. 
> It's been updated.

Aaaah :-D No problem. Can you please try:

* replace "timeout client 30s" with "timeout client 1d"
* replace "timeout server 30s" with "timeout server 1d"

in buildbot-master server of the pillar config:

https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[STINNER Victor]

STINNER Victor  added the comment:

> "timeout server 1d"

Hum. I'm not sure if "1d" syntax is accepted. Maybe use "3600s".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)

[William Schwartz]

William Schwartz  added the comment:

> Fixing the SystemError should be simple. Just clear an existing error if 
> TerminateProcess() succeeds.

Should there be a `return NULL;` between these two lines? 
https://github.com/python/cpython/blob/e485be5b6bd5fde97d78f09e2e4cca7f363763c3/Modules/posixmodule.c#L7854-L7855

I'm not the best person to work on a patch for this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[Ee W. Durbin III]

Ee W. Durbin III  added the comment:

Apologies, I gisted a version that was from attempts to debug the timeouts. 
It's been updated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41713] _signal module leak: test_interpreters leaked [1424, 1422, 1424] references

[STINNER Victor]

STINNER Victor  added the comment:

Petr Viktorin:
> The PyOS_InitInterrupts function is still listed in PC/python3dll.c.

Ooops, I proposed PR 24257 to remove it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41713] _signal module leak: test_interpreters leaked [1424, 1422, 1424] references

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +23079
pull_request: https://github.com/python/cpython/pull/24257

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42035] [C API] PyType_GetSlot cannot get tp_name

[Petr Viktorin]

Petr Viktorin  added the comment:

Now that I see the implementation (and now that I'm spending a lot of time 
trying to formalize what is good stable API), I see a problem with 
PyType_GetName: it effectively returns a borrowed reference.
The proposed docs say:

   Callers can hold [the retuned] pointer until the type has been deallocated.

This is not friendly to alternate Python implementations. For example, if 
tp_name is stored as UTF-32, it would need to generate the char* data -- and 
then retain it until the class is deallocated.
I guess the "correct" way would be to return a Py_buffer, which would (in 
CPython) reference the class.

Victor, what do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

> One (poorly thought out) option is to add sys._stdlib_path with the 
> subsection of sys.path that contains the stdlib, and a function in importlib 
> that returns if a spec is for a stdlib module.

There is already sysconfig.get_paths()['stdlib']. Maybe we need to add a new 
key for extension modules.

I don't think that these two options are exclusive. For me, it can even be a 
similar but different use case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

Ronald:
> I think we agree on that point: my counter proposal won’t work in the 
> faulthandler scenario, and may be problematic in the Py_FatalError case as 
> well.

The API providing a tuple of str (sys.module_names) works with the 4 use cases 
that I listed:

* faulthandler/Py_FatalError (dump third party extensions of sys.modules)
* isort (group stdlib imports)
* trace (don't trace stdlib modules)
* pypt (ignore stdlib modules when computing dependencies)


Ronald:
> Although that might give misleading answers with tools like 
> pyinstaller/py2exe/py2app that package an application and its dependencies 
> into a single zipfile.

These tools worked for years without sys.module_names and don't need to be 
modified to use sys.module_names.

sys.module_names is well defined, extract of its doc:

"The list is the same on all platforms. Modules which are not available on some 
platforms and modules disabled at Python build are also listed."

These packaging tools may require further checks than just checking if the name 
is in sys.module_names. These tools are complex anyway ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[STINNER Victor]

STINNER Victor  added the comment:

I looked at these files.

HAProxy configuration, "defaults" section contains:

timeout connect 5000
timeout client  5
timeout server  5

These timeouts are not overriden in "frontend main" nor in "backend 
buildbot-master" sections.

pillar/base/haproxy.sls seems to try to override these variables:

https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11

I don't see these in the rendered HAProxy config, but I see "timeout tunnel 
3600s" is rendered as "timeout tunnel 1d" in "backend buildbot-master".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

Ronald:
> BTW. A list of stdlib module names is not sufficient to determine if a module 
> is in the stdlib, thanks to .pth files it is possible to have entries on 
> sys.path before the stdlib.

Yeah, I wrote it in my first message. I solved this issue with documentation:

"Note: If a third party module has the same name than a standard library module 
and it comes before the standard library in sys.path, it overrides the standard 
library module on import."

I updated sys.module_names documentation in my PR.

IMO it's an acceptable limitation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42392] remove the deprecated 'loop' parameter asyncio API

[Ken Jin]

Change by Ken Jin :


--
nosy: +kj
nosy_count: 5.0 -> 6.0
pull_requests: +23078
pull_request: https://github.com/python/cpython/pull/24256

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[Ee W. Durbin III]

Ee W. Durbin III  added the comment:

OK, I've confirmed that HAProxy seems to be the issue. WebSockets opened to the 
nginx proxy on the server or directly to the twisted server successfully remain 
indefinitely.

If anyone familiar with HAProxy would be interested in helping debug, the 
current relevant information is:

- Ubuntu 18.04.5 LTS
- HA-Proxy version 1.8.8-1ubuntu0.11 2020/06/22

Here is the current HAProxy configuration: 
https://gist.github.com/ewdurbin/d8a42c30a04d6cb5763431200acaecde which is 
generated from this salt state: 
https://github.com/python/psf-salt/tree/master/salt/haproxy and this pillar 
data: https://github.com/python/psf-salt/blob/master/pillar/base/haproxy.sls

I can provide any additional information that would be helpful to someone 
trying to sort out what's going on.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41713] _signal module leak: test_interpreters leaked [1424, 1422, 1424] references

[Petr Viktorin]

Petr Viktorin  added the comment:

The PyOS_InitInterrupts function is still listed in PC/python3dll.c.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42887] 100000 assignments of .__sizeof__ cause a segfault on del

[Mark Shannon]

Mark Shannon  added the comment:

It won't solve the problem.
Maybe make it would make it easier to avoid the segfault, but some sort of 
recursion/overflow check is needed. 

It might make the use of the trashcan cheaper, as it only need be used when 
stack space is running low.

Ultimately, the cycle GC needs to be iterative, rather than recursive. That 
will take a *lot* of work though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42967] Web cache poisoning - `;` as a query args separator

[Adam Goldschmidt]

New submission from Adam Goldschmidt :

The urlparse module treats semicolon as a separator 
(https://github.com/python/cpython/blob/master/Lib/urllib/parse.py#L739) - 
whereas most proxies today only take ampersands as separators. Link to a blog 
post explaining this vulnerability: 
https://snyk.io/blog/cache-poisoning-in-popular-open-source-packages/

When the attacker can separate query parameters using a semicolon (;), they can 
cause a difference in the interpretation of the request between the proxy 
(running with default configuration) and the server. This can result in 
malicious requests being cached as completely safe ones, as the proxy would 
usually not see the semicolon as a separator, and therefore would not include 
it in a cache key of an unkeyed parameter - such as `utm_*` parameters, which 
are usually unkeyed. Let’s take the following example of a malicious request:   
  

```
GET /?link=http://google.com_content=1;link='>alert(1) HTTP/1.1

Host: somesite.com

Upgrade-Insecure-Requests: 1

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,imag 
e/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 
Accept-Encoding: gzip, deflate 

Accept-Language: en-US,en;q=0.9 Connection: close   
```

urlparse sees 3 parameters here: `link`, `utm_content` and then `link` again. 
On the other hand, the proxy considers this full string: 
`1;link='>alert(1)` as the value of `utm_content`, which is why the 
cache key would only contain `somesite.com/?link=http://google.com`. 

A possible solution could be to allow developers to specify a separator, like 
werkzeug does:

https://github.com/pallets/werkzeug/blob/6784c44673d25c91613c6bf2e614c84465ad135b/src/werkzeug/urls.py#L833

--
components: C API
messages: 385266
nosy: AdamGold
priority: normal
severity: normal
status: open
title: Web cache poisoning - `;` as a query args separator
type: security
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42887] 100000 assignments of .__sizeof__ cause a segfault on del

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Mark, would your proposal in PEP-651 fix this case?

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42965] Optional callable raises TypeError

[Ofek Lev]

Ofek Lev  added the comment:

Ah I see, thanks!

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42966] argparse: customizable help formatter

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
title: argpare: customizable help formatter -> argparse: customizable help 
formatter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42966] argpare: customizable help formatter

[Tadek Kijkowski]

New submission from Tadek Kijkowski :

Current implementation of the argparse module doesn't make it easy for 
developers to customize the help text message format. On one hand the module 
doesn't provide any builtin ways to do it, and the only way is to provide 
developer's own formatter class, on the other hand developers are threatened, 
that all internal structures of the module, including API of the HelpFormatter 
class is subject to change. Trying to customize anything but the most basic 
things, developer risks that his or hers module will suddenly stop working with 
a new version of argparse.

For most basic customization there is a set of four alternative formatting 
classes, but their usage as well as development of additional formatting 
classes is hindered by following reasons:
- Only one formatting class can be used at a time. Although arguably devloper 
can safely derive his own formatter class from multiple argparse's formatters - 
if he or she will come up with this idea.
- At this moment there is no way to pass any configuration data to the 
formatter object, so each formatter class can only change one aspect of the 
output format.
- Any new functionality added to argparse help formatting has to take form of 
new formatter class. If it is significant change, it means duplicating parts of 
HelpFormatter code and in the future maintaining additional classes.
- Even if additional formatter classes are added to argparse, developers have 
make sure that the new class is present in the argparse used by their module - 
either add dependency to specific argparse version, or dynamically check for 
presence of the new class in argparse module.

The most important part of this enhancement request is change in argparse 
public API - addition of class CustomHelpFormat. Object of this class would 
serve as factory for CustomizableHelpFormatter objects and could be passed to 
ArgumentParser constructor in place of format class type:

custom_format = argparse.CustomHelpFormat()
custom_format.indent_increment = 4
custom_format.raw_description = True
parser = argparse.ArgumentParser(formatter_class=custom_format)

The idea of passing callable instead of class type as formatter_class argument 
is not new, but this time it would be part of official API with access to 
internal details and maintained along with the module.

In conrtast to the list of drawbacks above, the advantages of this solution are:
- It is much easier to do basic customization. Developers won't have to search 
the internet for information how to customize argparse output, when they just 
want to focus on their own application functionality.
- There is no need to check for presence of future additional help formatter in 
the argparse module. One just has to set appropriate attribute in the 
'custom_format' object. If required functionality is not present in the module 
on the system, it would be simply ignored, developer could even safely set 
attribute for future feature not present in released version.
- Detection of implemented feature would is much easier - developer would just 
call `hasattr(custom_format, "hybrid_text_formatter")` and modify displayed 
text accordingly in the absence of the requested feature.
- Development of new features should be easier - instead of creating new class 
for each possible aspect of customization, new features could be combined in 
one new class, or even added to the base HelpFormatter and just enabled by an 
attribute.

Altough this solution will not immediately help developers wanting to customize 
their argparse, the main idea is that it will unclog development of new 
formatters and incorporating existing propositions and possibly lessen or 
remove the need for client side customizations.

Initially, the new class would contain attributes corresponding to funcionality 
of existing alternative formatting classes as well as arguments to default 
HelpFormatter - indent, max help position and width.

Attached implementation of the CustomizableHelpFormatter class may seem 
rudimentary or hackish (or maybe not). Perhaps it should implement all 
appropriate methods instead of patching self.

--
components: Library (Lib)
files: CustomizableHelpFormatter.txt
messages: 385263
nosy: monkeyman79
priority: normal
severity: normal
status: open
title: argpare: customizable help formatter
type: enhancement
versions: Python 3.10
Added file: https://bugs.python.org/file49750/CustomizableHelpFormatter.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42965] Optional callable raises TypeError

[Ken Jin]

Ken Jin  added the comment:

Hello, this issue is a byproduct of issue42195. It has already been fixed on 
Python 3.10, and on Python 3.9.2 (which isn't out yet). You can see the what's 
new for it here 
https://docs.python.org/3/whatsnew/3.9.html#notable-changes-in-python-3-9-2.

The expected release date for Python 3.9.2 is Monday, 2021-02-15 according to 
PEP 596 https://www.python.org/dev/peps/pep-0596/.

For now, I guess you'll have to use the old typing.Callable, then update it in 
newer versions of Python.

On Python 3.10a4:
>>> from typing import Optional
>>> from collections.abc import Callable
>>> Optional[Callable[[bytes], bytes]]
typing.Optional[collections.abc.Callable[[bytes], bytes]]

--
nosy: +kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42951] Random and infinite loop in dealing with recursion error for "try-except "

[Xinmeng Xia]

Xinmeng Xia  added the comment:

oh,I see. By the way, I set the argument of sys.setrecursionlimit to 10 in 
this program and a segmentation fault is reported. Is that normal?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42951] Random and infinite loop in dealing with recursion error for "try-except "

[Mark Shannon]

Mark Shannon  added the comment:

Try setting the recursion limit to 10 or so and it should terminate.

The reason ctrl-C doesn't work is that you are catching the KeyboardInterrupt. 
Never use a plain `except:`, use `except Exception:`

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42961] Use-after-free (of a heap type) during finalization

[Boris Staletic]

Boris Staletic  added the comment:

Oops... I uploaded (and pasted) the wrong file. The /correct/ example can be 
found here:

https://github.com/pybind/pybind11/pull/2797/#pullrequestreview-570541151

However, I have just realized that the example doesn't really need the embedded 
module. The following also shows the use-after-free:


#include 

static void pybind11_object_dealloc(PyObject *self) {
auto type = Py_TYPE(self);
type->tp_free(self);
Py_DECREF(type);
}
static PyType_Slot base_slots[] = {{Py_tp_dealloc, 
(void*)pybind11_object_dealloc}, {0, nullptr}};
static PyType_Spec base_spec{"B", sizeof(PyObject), 0, Py_TPFLAGS_BASETYPE | 
Py_TPFLAGS_HEAPTYPE, base_slots};
int main() {
Py_InitializeEx(1);
auto base_type = PyType_FromSpec(_spec);
auto globals = PyDict_New();
PyDict_SetItemString(globals, "B", base_type);
auto derived_t = PyRun_String("def f():\n"
  "  class C:\n"
  "class D(B):pass\n"
  "b=D()\n"
  "f()", Py_file_input, globals, nullptr);
Py_DECREF(globals);
Py_DECREF(derived_t);
Py_Finalize();
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Could anybody provide their thoughts on this RFE? Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41818] Lib/pty.py major revision

[Petr Viktorin]

Petr Viktorin  added the comment:


New changeset 65cf1ad6723b6b4489fa7dda04283bb2466be531 by Petr Viktorin in 
branch 'master':
bpo-41818: Close file descriptors in test_openpty (#GH-24119)
https://github.com/python/cpython/commit/65cf1ad6723b6b4489fa7dda04283bb2466be531


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[Petr Viktorin]

Change by Petr Viktorin :


--
keywords: +patch
pull_requests: +23077
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24119

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41701] Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup

[Petr Viktorin]

Petr Viktorin  added the comment:

Is there anything I can help with to move this forward?
Investigating buildbot failures continues to be very annoying.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

BTW. A list of stdlib module names is not sufficient to determine if a module 
is in the stdlib, thanks to .pth files it is possible to have entries on 
sys.path before the stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

> On 19 Jan 2021, at 12:30, STINNER Victor  wrote:
> 
> Ronald:
>> You wouldn't necessarily have to import a module to test, this is something 
>> that could be added to importlib.  One (poorly thought out) option is to add 
>> sys._stdlib_path with the subsection of sys.path that contains the stdlib, 
>> and a function in importlib that returns if a spec is for a stdlib module.
> 
> I'm not sure how it would work. I listed different cases which have different 
> constraints:
> 
> * From a module name, check if it's part of the stdlib or not
> * From a module object, check if it's part of the stdlib or not
> 
> For the test on the module name, how would it work with sys._stdlib_path? 
> Should you import the module and then check if its path comes from 
> sys._stdlib_path?

For a module name use importlib.util.find_spec() to locate the module (or 
toplevel package if this is a module in package). The new importlib function 
could then use the spec and sys._stdlib_path to check if the spec is one for a 
stdlib module.  This is pretty handwavy, but I do something similar in py2app 
(but completely based on paths calculated outside of the import machinery).

For a module object you can extract the spec from the object and use the same 
function. 

> 
> 
> Ronald:
>> The disadvantage of this, or for the most part anything but your initial 
>> proposal, is that might not be save to use a function in importlib in 
>> Py_FatalError.
> 
> PR 24254 is a working implementation of my use case: only list third party 
> extension modules on a Python fatal error. It relies on PR 24238 
> sys.module_names list. The implementation works when called from a signal 
> handler (when faulthandler catch fatal signals like SIGSEGV), it avoids 
> memory allocations on the heap (one of the limits of a signal handler).

I think we agree on that point: my counter proposal won’t work in the 
faulthandler scenario, and may be problematic in the Py_FatalError case as well.

Ronald

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

Myself:
> Having to actually import modules to check if it's a stdlib module or not is 
> not convenient. Many stdlib modules have side effects on import.

The "trace" usecase needs an exhaustive list of all module names. It is even 
less convenient to have to list all Python modules available on the system only 
to check modules coming from the stdlib.


Ronald:
> You wouldn't necessarily have to import a module to test, this is something 
> that could be added to importlib.  One (poorly thought out) option is to add 
> sys._stdlib_path with the subsection of sys.path that contains the stdlib, 
> and a function in importlib that returns if a spec is for a stdlib module.

I'm not sure how it would work. I listed different cases which have different 
constraints:

* From a module name, check if it's part of the stdlib or not
* From a module object, check if it's part of the stdlib or not

For the test on the module name, how would it work with sys._stdlib_path? 
Should you import the module and then check if its path comes from 
sys._stdlib_path?


Ronald:
> The disadvantage of this, or for the most part anything but your initial 
> proposal, is that might not be save to use a function in importlib in 
> Py_FatalError.

PR 24254 is a working implementation of my use case: only list third party 
extension modules on a Python fatal error. It relies on PR 24238 
sys.module_names list. The implementation works when called from a signal 
handler (when faulthandler catch fatal signals like SIGSEGV), it avoids memory 
allocations on the heap (one of the limits of a signal handler).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

>> Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? 
>> Although that might give misleading answers with tools like 
>> pyinstaller/py2exe/py2app that package an application and its dependencies 
>> into a single zipfile.

> Having to actually import modules to check if it's a stdlib module or not is 
> not convenient. Many stdlib modules have side effects on import. For example, 
> "import antigravity" opens a web browser. An import can open files, spawn 
> threads, run programs, etc.

You wouldn't necessarily have to import a module to test, this is something 
that could be added to importlib.  One (poorly thought out) option is to add 
sys._stdlib_path with the subsection of sys.path that contains the stdlib, and 
a function in importlib that returns if a spec is for a stdlib module.

The disadvantage of this, or for the most part anything but your initial 
proposal, is that might not be save to use a function in importlib in 
Py_FatalError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[STINNER Victor]

STINNER Victor  added the comment:

Ronald Oussoren:
> A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. 
> if you consider that not all modules/extensions are installed on all systems 
> (both because dependencies aren't present and because packagers have decided 
> to unbundle parts of the stdlib).

My PR 24238 adds a list of module names (tuple of str): sys.module_names. It 
includes modules which are not available on the platform. For example, 
"winsound" is listed on Linux but not available, and "tkinter" is listed even 
if the extension could not be built (missing dependency).

I tried to make it clear in the sys.module_names documentation (see my PR).

Making the list conditional depending if the module is built or not is causing 
different issues. For example, for the "isort" (sort and group imports) use 
case, you want to know on Linux if "import winsound" is a stdlib import or a 
third party import (same for Linux-only modules on Windows). Moreover, there is 
a practical issue: extension modules are only built by setup.py *after* the sys 
module is built. I don't want to rebuild the sys module if building an 
extension failed.

Even if a module is available (listed in sys.module_names and the file is 
present on disk), it doesn't mean that "it works". For example, "import 
multiprocessing" fails if there is no working lock implementation. Other 
modules have similar issues.


> Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? 
> Although that might give misleading answers with tools like 
> pyinstaller/py2exe/py2app that package an application and its dependencies 
> into a single zipfile.

Having to actually import modules to check if it's a stdlib module or not is 
not convenient. Many stdlib modules have side effects on import. For example, 
"import antigravity" opens a web browser. An import can open files, spawn 
threads, run programs, etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[Ronald Oussoren]

Ronald Oussoren  added the comment:

A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. 
if you consider that not all modules/extensions are installed on all systems 
(both because dependencies aren't present and because packagers have decided to 
unbundle parts of the stdlib).

Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? 
Although that might give misleading answers with tools like 
pyinstaller/py2exe/py2app that package an application and its dependencies into 
a single zipfile.

--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39291] "pathlib.Path.link_to()" and "pathlib.Path.symlink_to()" have reversed usage

[yota moteuchi]

yota moteuchi  added the comment:

one option to could be to create a hardlink_to() method which is 
link.hardlink_to(target) and in a few release, deprecate link_to ? :)

--
nosy: +yota moteuchi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)

[Eryk Sun]

Eryk Sun  added the comment:

> I just reported it because the SystemError indicates that a C-API function
> was returning non-NULL even after PyErr_Occurred() returns true

Fixing the SystemError should be simple. Just clear an existing error if 
TerminateProcess() succeeds. 

> Windows Command shell inside *Command Prompt* (not Windows Terminal):

The cmd.exe shell (aka command prompt or command interpreter) is a console 
client application like python.exe, which attaches to a console session that's 
hosted by conhost.exe or openconsole.exe. The host also implements the console 
UI, unless it's executed in pseudoconsole mode (e.g. a tab in Windows 
Terminal). 

> C:\Users\wksch>python -c"import os, signal, time; os.kill(os.getpid(),
> signal.CTRL_C_EVENT); time.sleep(1)"

The above kill() call is implemented by calling GenerateConsoleCtrlEvent() to 
send the cancel event to clients of the console session that are members of the 
process group, os.getpid(). But there is no such group since Python isn't 
executed as a new process group. GenerateConsoleCtrlEvent() should fail with an 
invalid parameter error, and kill() should fall back on TerminateProcess(). But 
GenerateConsoleCtrlEvent() has a bug in cases like this that causes it to 
behave as if it were passed group ID 0 (i.e. all processes in the console 
session). If not for the bug in the console, this example would also raise 
SystemError.

> In the Windows Command shell inside *Windows Terminal*

The misbehavior is different in a pseudoconsole session, which is probably due 
to an unrelated bug that's affecting the expected bug. Other than Windows 
Terminal, another simple way to get a pseudoconsole session is to run cmd.exe 
from WSL. (The UI will be in the same console as WSL, but cmd.exe will actually 
be attached to a pseudoconsole session that's hosted by a headless instance of 
conhost.exe.) In pseudoconsole mode, the console will broadcast the control 
event only after a key such as enter or escape is pressed, which is obviously a 
bug in the console's input event loop. It's still a case of 
GenerateConsoleCtrlEvent() nominally succeeding with buggy behavior where it 
should fail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42965] Optional callable raises TypeError

[Ofek Lev]

Ofek Lev  added the comment:

I'm using the deprecated typing.Callable instead now and that works

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42965] Optional callable raises TypeError

[Ofek Lev]

New submission from Ofek Lev :

https://docs.python.org/3.9/library/typing.html#callable

```
Python 3.9.1 (default, Jan 12 2021, 16:45:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Optional
>>> from collections.abc import Callable
>>>
>>> Hasher = Optional[Callable[[bytes], bytes]]
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.9/typing.py", line 262, in inner
return func(*args, **kwds)
  File "/usr/local/lib/python3.9/typing.py", line 339, in __getitem__
return self._getitem(self, parameters)
  File "/usr/local/lib/python3.9/typing.py", line 463, in Optional
return Union[arg, type(None)]
  File "/usr/local/lib/python3.9/typing.py", line 262, in inner
return func(*args, **kwds)
  File "/usr/local/lib/python3.9/typing.py", line 339, in __getitem__
return self._getitem(self, parameters)
  File "/usr/local/lib/python3.9/typing.py", line 451, in Union
parameters = _remove_dups_flatten(parameters)
  File "/usr/local/lib/python3.9/typing.py", line 231, in _remove_dups_flatten
return tuple(_deduplicate(params))
  File "/usr/local/lib/python3.9/typing.py", line 205, in _deduplicate
all_params = set(params)
TypeError: unhashable type: 'list'
>>>
>>> from typing import Tuple
>>> Hasher = Optional[Callable[Tuple[bytes], bytes]]
>>>
```

Tuple type for arguments makes it work

--
components: Library (Lib)
messages: 385246
nosy: Ofekmeister
priority: normal
severity: normal
status: open
title: Optional callable raises TypeError
type: behavior
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com