[issue34435] Missing NULL check in unicode_encode_ucs1()
Alexey Izbyshev added the comment: Thank you for merging! -- resolution: -> fixed ___ Python tracker <https://bugs.python.org/issue34435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34435] Missing NULL check in unicode_encode_ucs1()
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8302 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34441] NULL dereference when issubclass() is called on a class with bogus __subclasses__
New submission from Alexey Izbyshev :
>>> from abc import ABCMeta
>>> class S(metaclass=ABCMeta):
... __subclasses__ = None
...
>>> issubclass(int, S)
Segmentation fault (core dumped)
This is the result of missing NULL check for 'subclasses' in
_abc__abc_subclasscheck_impl (Modules/_abc.c):
/* 6. Check if it's a subclass of a subclass (recursive). */
subclasses = PyObject_CallMethod(self, "__subclasses__", NULL);
if (!PyList_Check(subclasses)) {
PyErr_SetString(PyExc_TypeError, "__subclasses__() must return a list");
goto end;
}
Reported by Svace static analyzer.
--
components: Extension Modules
messages: 323789
nosy: inada.naoki, izbyshev, levkivskyi, serhiy.storchaka
priority: normal
severity: normal
status: open
title: NULL dereference when issubclass() is called on a class with bogus
__subclasses__
type: crash
versions: Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue34441>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34456] pickle: Missing NULL check in save_global()
Change by Alexey Izbyshev : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: -Python 3.5 ___ Python tracker <https://bugs.python.org/issue34456> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34456] pickle: Missing NULL check in save_global()
Alexey Izbyshev added the comment: No, I couldn't find any similar calls in save_global() in 2.7. However, there is Py_BuildValue() call in initcPickle(), and its result is passed unchecked to PyDict_SetItemString(), where it's eventually dereferenced. In fact, all PyDict_SetItemString() calls in initcPickle() are made without checking for NULL. Should I file an issue for that? -- ___ Python tracker <https://bugs.python.org/issue34456> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34457] Missing NULL check in alias_for_import_name() from Python/ast.c
Change by Alexey Izbyshev : -- pull_requests: +8329 ___ Python tracker <https://bugs.python.org/issue34457> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34471] _datetime: Missing NULL check in tzinfo_from_isoformat_results()
New submission from Alexey Izbyshev : The return value of new_delta() is not checked for NULL at https://github.com/python/cpython/blob/ccd99752675042bd5f67d332c5b0ed85ba1f2da3/Modules/_datetimemodule.c#L1292 and then dereferenced. Reported by Svace static analyzer. -- components: Extension Modules messages: 323934 nosy: belopolsky, izbyshev, p-ganssle, serhiy.storchaka priority: normal severity: normal status: open title: _datetime: Missing NULL check in tzinfo_from_isoformat_results() type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue34471> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34436] Overallocation is never disabled in _PyBytes_FormatEx()
Alexey Izbyshev added the comment: Thank you for benchmarking and merging, Victor! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue34436> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34471] _datetime: Missing NULL check in tzinfo_from_isoformat_results()
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8345 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34471> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34441] NULL dereference when issubclass() is called on a class with bogus __subclasses__
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8310 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34462] _xxsubinterpreters: Wrong NULL check in _copy_raw_string()
New submission from Alexey Izbyshev :
The code at
https://github.com/python/cpython/blob/28853a249b1d0c890b7e9ca345290bb8c1756446/Modules/_xxsubinterpretersmodule.c#L18
checks a wrong variable for NULL:
const char *str = PyUnicode_AsUTF8(strobj);
if (str == NULL) {
return NULL;
}
char *copied = PyMem_Malloc(strlen(str)+1);
if (str == NULL) {
PyErr_NoMemory();
return NULL;
}
Reported by Svace static analyzer.
--
components: Extension Modules
messages: 323897
nosy: eric.snow, izbyshev, serhiy.storchaka
priority: normal
severity: normal
status: open
title: _xxsubinterpreters: Wrong NULL check in _copy_raw_string()
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue34462>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34462] _xxsubinterpreters: Wrong NULL check in _copy_raw_string()
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8337 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34462> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34502] Docs of sys.exit() mention utf8_mode for an unclear reason
Alexey Izbyshev added the comment: Victor, but sys.flags docs *do* contain the note[1]: Changed in version 3.7: Added dev_mode attribute for the new -X dev flag and utf8_mode attribute for the new -X utf8 flag. I've tried to express that in msg324086, but obviously failed. [1] https://docs.python.org/3/library/sys.html#sys.flags -- ___ Python tracker <https://bugs.python.org/issue34502> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34512] Document platform-specific strftime() behavior for non-ASCII format strings
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8424 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34512> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34481] Different behavior of C and Python impls of datetime.strftime with non-UTF-8-encodable strings
Alexey Izbyshev added the comment: > if we can't make assertions about the behavior of strftime outputs, I think > it makes it hard to prevent regressions. Ironically, some of the changes we may have to make to fix time.strftime() inconsistencies may appear like regressions to some users. For example, regarding dropping wcsftime() on all platforms, 'man strftime' on macOS contains the following warning in BUGS section: "The strftime() function does not correctly handle multibyte characters in the format argument.". (I'm not sure what "incorrect handling" means here). That's not to discourage you, just to point out that we should be extra careful at this point when there might be users relying on every variety of the current behavior. And that's also why I don't want to mix the fix for C vs. Python issue (low risk) with changes in time.strftime() (high risk). Also, while I certainly agree with "the outsized effort" point, it seems that your PR goes far beyond supporting surrogates because falling back to escaping allows one to bypass checks in wcsftime()/strftime() and round-trip any code point regardless of the current locale. -- ___ Python tracker <https://bugs.python.org/issue34481> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34481] Different behavior of C and Python impls of datetime.strftime with non-UTF-8-encodable strings
Alexey Izbyshev added the comment: @p-ganssle > I'm finding it very difficult to reconcile these things. Paul, this issue was only about reconciling C and Python implementations of datetime module -- not fixing time.strftime(), which both of them delegate to. While the latter is certainly more important, it's also much more difficult, and IMO needs another issue report (maybe merged with #34512). As for the former, it seems that you've already fixed it in your PR. -- ___ Python tracker <https://bugs.python.org/issue34481> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32236] open() shouldn't silently ignore buffering=1 in binary mode
Alexey Izbyshev added the comment: Thank you, Gregory. I didn't intend to add the warning to stable branches -- it's just that 3.7 hasn't been released yet when this report was submitted. -- ___ Python tracker <https://bugs.python.org/issue32236> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32270] subprocess closes redirected fds even if they are in pass_fds
Alexey Izbyshev added the comment: Thank you, Gregory! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue32270> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33010] os.path.isdir() returns True for broken directory symlinks or junctions
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Thank you for the detailed response, Eryk! > A mount point is always a directory, even if the volume isn't currently > available. Do I understand correctly that you propose to additionally change os.path.exists() to return True for mount points with unavailable volumes? Сurrently, os.path.exists() (i.e, the underlying os.stat()) attempts to traverse them, and this would be consistent with os.path.isdir() if the latter were changed to traverse directory reparse points too (both would return False for such mount points). Is your idea to change the behavior to match POSIX in a similar case when, for example, the remote NFS server is down but stat() still works on the local mount point? If so, is this a new idea compared to the first paragraph of [1] where you say that non-link reparse points should always be traversed? [1] https://github.com/python/cpython/pull/5998#discussion_r172402233 -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33010] os.path.isdir() returns True for broken directory symlinks or junctions
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Hmm, actually, my NFS example is probably bad. I've run an experiment, and stat() simply hangs in the case if the NFS server is down. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- keywords: +patch pull_requests: +5775 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
New submission from Alexey Izbyshev <izbys...@ispras.ru>: The first call of GetFinalPathNameByHandleW requests the required buffer size for the NT path (VOLUME_NAME_NT), while the second call receives the DOS path (VOLUME_NAME_DOS) in the allocated buffer. Usually, NT paths are longer than DOS ones, for example: NT path: \Device\HarddiskVolume2\foo DOS path: \\?\C:\foo Or, for UNC paths: NT path: \Device\Mup\server\share\foo DOS path: \\?\UNC\server\share\foo However, it is not always the case. A volume can be mounted to an arbitrary path, and if a drive letter is not assigned to such a volume, GetFinalPathNameByHandle will use the mount point path instead of C: above. This way, a DOS path can be longer than an NT path. Since the result of the second call is not checked properly, this condition won't be detected, resulting in an out-of-bounds access and use of uninitialized memory later. Moreover, the path returned by GetFinalPathNameByHandle may change between the first and the second call, for example, because an intermediate directory was renamed. If the path becomes longer than buf_size, the same issue will occur. -- components: Extension Modules, Windows messages: 313366 nosy: izbyshev, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: nt._getfinalpathname may use uninitialized memory type: behavior versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: I do not see any point in allowing non-types in ABCMeta.__subclasscheck__. Currently, ABCs are clearly not designed to support non-types: 1. ABCMeta.register() accepts types only. 2. ABCMeta.__subclasscheck__ implicitly requires its arguments to support weak references (regardless of whether __subclasshook__ is called or not). This requirement alone doesn't make sense, so it seems to be an exposed implementation detail stemming from the fact that non-types were not intended to be supported. 3. Some ABC users already expect that the argument of __subclasshook__ is a type (see the example with collections.abc.Reversible by OP). 4. Attempting to support arbitrary arguments in ABC.__subclasscheck__ (by returning False instead of raising TypeError or worse) will not solve any 'issubclass' inconsistencies. 'issubclass' is fundamentally "fragmented": issubclass(x, y) may return True/False while issubclass(x, z) may raise TypeError, depending on __subclasscheck__ implementation. It may be too late to impose stricter requirements for the first argument of issubclass because 'typing' module relies on the support of non-types there. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > Isn't it just a limitation? > Most Python-implemented objects supports weakref. I don't think "requiring > weakref support implies it must be type object". Formally, there is no implication. It is the abc module authors who know the truth. But I can't imagine why anybody would impose such a limitation by design, because while instances of user-defined classes support weakrefs, built-in classes used by everybody like tuple, list and dict don't. That's why I guessed that non-types were not meant to be supported. > What "by OP" means? OP = Original poster (@jab). > I can't find `if not issubclass(cls, type): raise TypeError` in Reversible > implementation. > They do duck-typing, same to ABC. Sorry for being unclear. There is no explicit check as you say, but __mro__ is directly accessed (see msg313376). But it may probably be considered "duck typing" too. > But I don't know much about how mages use ABC. I need mages comment before > merging the pull request. Totally agree. > BTW, do you think it should be backported to 3.7, or even 3.6? 3.7 certainly has my vote -- this can hardly be considered a new feature. For 3.6, I'd listen to ABC users/experts. Might raising a TypeError instead of returning False from issubclass(user_defined_obj, ABC) break something important? Personally, I think it would mostly expose bugs and not hinder reasonable usage. > Can > https://github.com/python/cpython/commit/fc7df0e664198cb05cafd972f190a18ca422989c > be reverted? Seems like it can, but the test should survive in some form :) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: PR 5944 changes ABC.__subclasscheck__ (not issubclass) to check its first argument, so if it's merged there will be no crash even with the revert. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32903] os.chdir() may leak memory on Windows
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- pull_requests: +5715 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33030] GetLastError() may be overwritten by Py_END_ALLOW_THREADS
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > Ideally, if we don't have to do any work to reacquire the GIL, we shouldn't > do any work to preserve the error code either. Before take_gil() knows whether it has to do any work, it calls MUTEX_LOCK(_PyRuntime.ceval.gil.mutex), which is either EnterCriticalSection() or AcquireSRWLockExclusive(). So unless we can be sure that those functions can't clobber the last error (or we redesign GIL to have a fast-path like an atomic compare-exchange in non-contended case), I don't see how we can avoid the last error bookkeeping. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33030> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33030] GetLastError() may be overwritten by Py_END_ALLOW_THREADS
Alexey Izbyshev <izbys...@ispras.ru> added the comment: FWIW, GetLastError() docs[1] officially scare us: Most functions that set the thread's last-error code set it when they fail. However, some functions also set the last-error code when they succeed. If the function is not documented to set the last-error code, the value returned by this function is simply the most recent last-error code to have been set; some functions set the last-error code to 0 on success and others do not. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms679360(v=vs.85).aspx -- nosy: +izbyshev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33030> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32592] Drop support of Windows Vista in Python 3.7
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- nosy: +izbyshev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32592> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29871] Enable optimized locks on Windows
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- nosy: +izbyshev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue29871> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33030] GetLastError() may be overwritten by Py_END_ALLOW_THREADS
Alexey Izbyshev <izbys...@ispras.ru> added the comment: @eryksun Very interesting! BTW, I looked at CreateFile docs, and the fact that it may set the last error to zero is even documented there. @steve.dower > maybe we don't have to preserve errno on Windows? There are still places where errno is relied upon. PR 5784 shows one such place: os.[f]truncate(). Others are e.g. os.open() and os.lseek(). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33030> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > Because we only try VOLUME_NAME_DOS, this function always fails for a volume > that's not mounted as either a drive letter or a junction mount point. If a volume is not mounted, users have to use \\?\ or \\.\ either directly or indirectly via a symlink or a junction to get to it, right? Do you think such uses are common enough to warrant dealing with VOLUME_NAME_GUID? Also note that pathlib currently expects DOS paths only: it will strip '\\?\' prefix in resolve(), making GUID path "invalid" (and, also, precluding direct usage of '\\?\' prefix with resolve() in other cases unless users are not prepared to deal with it). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Copying the comment of @eryksun from PR 6010 for reference. Because we only try VOLUME_NAME_DOS, this function always fails for a volume that's not mounted as either a drive letter or a junction mount point. The error in this case is ERROR_PATH_NOT_FOUND. We know that the path is valid because we have a handle (in this case the file system ensures that no parent directory in the path can be unlinked or renamed). To address this, if the flags parameter isn't already VOLUME_NAME_GUID, it could switch to it and continue the while loop (no need to realloc). Otherwise if it's already VOLUME_NAME_GUID or for any other error, the call should fail. For DOS devices in paths (e.g. "C:\Temp\NUL"), CreateFile commonly succeeds, but GetFinalPathNameByHandle commonly fails with either ERROR_INVALID_FUNCTION or ERROR_INVALID_PARAMETER. What do you think about handling this failure by calling GetFullPathName instead (e.g. "C:\Temp\NUL" => "\\.\NUL")? That way most DOS device paths won't cause this function to fail (e.g. when called by pathlib's resolve method). We could do the same if CreateFile fails with ERROR_INVALID_PARAMETER, which should only occur for CON (e.g. "C:\Temp\CON") because it needs to be opened with either generic read or generic write access. CreatFile also commonly fails here with either ERROR_SHARING_VIOLATION (from a paging file) or ERROR_ACCESS_DENIED. But I think those are best handled by the caller, with a PermissionError exception handler. Currently pathlib's resolve method doesn't handle PermissionError like I think it should in non-strict mode. It only handles FileNotFoundError -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: I agree except that I'd like to see it in 3.7 too. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > We know that the path is valid because we have a handle (in this case the > file system ensures that no parent directory in the path can be unlinked or > renamed). Thank you for pointing this out. I erroneously stated that the length of the path could increase between GetFinalPathNameByHandle calls because an intermediate directory could be renamed, but actually I've only checked that the last part can be renamed (or even deleted, though it'll still linger in inaccessible state until the handle is closed). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33016] nt._getfinalpathname may use uninitialized memory
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > unless users are not prepared to deal with it ARE prepared > What do you think about handling this failure by calling GetFullPathName > instead (e.g. "C:\Temp\NUL" => "\\.\NUL")? I think it would indeed be nice if pathlib handled such paths in its resolve(), especially since os.path.abspath() does handle them, and it looks weird that even resolve(strict=False) fails. That could be an enhancement, but note that it'll expose users to '\\.\'-prefixed paths which can't be returned from resolve() now. It is not necessary a problem because users should be prepared to handle UNC-like paths anyway. > Currently pathlib's resolve method doesn't handle PermissionError like I > think it should in non-strict mode. It only handles FileNotFoundError That behavior doesn't look good, and it's inconsistent with POSIX resolve() which doesn't propagate any OSError in non-strict mode. I think this warrants an issue report. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33016> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Regarding status quo (expanding the examples of @inada.naoki and @jab): >>> import typing >>> import collections.abc as cabc >>> issubclass(typing.Mapping, cabc.Mapping) Traceback (most recent call last): File "", line 1, in File "/home/izbyshev/workspace/cpython/Lib/abc.py", line 143, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) TypeError: issubclass() arg 1 must be a class >>> from abc import ABC >>> issubclass(typing.Mapping, ABC) Traceback (most recent call last): File "", line 1, in File "/home/izbyshev/workspace/cpython/Lib/abc.py", line 143, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) File "/home/izbyshev/workspace/cpython/Lib/abc.py", line 143, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) File "/home/izbyshev/workspace/cpython/Lib/contextlib.py", line 30, in __subclasshook__ return _collections_abc._check_methods(C, "__enter__", "__exit__") File "/home/izbyshev/workspace/cpython/Lib/_collections_abc.py", line 73, in _check_methods mro = C.__mro__ File "/home/izbyshev/workspace/cpython/Lib/typing.py", line 706, in __getattr__ raise AttributeError(attr) AttributeError: __mro__ >>> ABC.register(int) >>> issubclass(typing.Mapping, ABC) Traceback (most recent call last): File "", line 1, in File "/home/izbyshev/workspace/cpython/Lib/abc.py", line 143, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) TypeError: issubclass() arg 1 must be a class >>> typing.Mapping.__mro__ = () >>> issubclass(typing.Mapping, ABC) Traceback (most recent call last): File "", line 1, in File "/home/izbyshev/workspace/cpython/Lib/abc.py", line 143, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) TypeError: issubclass() arg 1 must be a class >>> typing.Mapping.__bases__ = () >>> issubclass(typing.Mapping, ABC) False Can't say that I'm OK with it :) I'm for forbidding non-types in ABCMeta.__subclasscheck__, but if we are to add clean support for "class-likes" instead, I think that "class-like" objects should be clearly defined, for example, that they must have __mro__ and __bases__ (and probably support weakrefs unless we want to skip caching if they don't). ABCMeta.__subclasscheck__ is not standalone: it relies on issubclass() and __subclasshook__, and both of them have some expectations in practice. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Sorry for status update, this was due to a concurrent modification. -- nosy: +jab resolution: -> fixed status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: @inada.naoki: I don't question your change. My point is the same as in #33018 (I've discovered that PR only after I commented). The error message is misleading, and it's just a coincidence that a TypeError and not some other error is raised when abc attempts to add a non-type object to a WeakSet. @serhiy.storchaka: Note that an example that you requested is unlikely to be related to ABC and probably is more like my artificial __subclasscheck__ example. So, making changes just for ABC as suggested in #33018 might make sense. -- nosy: -jab resolution: fixed -> status: closed -> open ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33018] Improve issubclass() error checking and message
Alexey Izbyshev <izbys...@ispras.ru> added the comment: ABC.register() has an explicit check, and it is mentioned in PEP 3119. The point here is not to change issubclass(), but to change ABC.__subclasscheck__(). It may conceivably have stricter requirements than issubclass() has. But certainly an advice from actual ABC users would be nice. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33018> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32270] subprocess closes redirected fds even if they are in pass_fds
Alexey Izbyshev <izbys...@ispras.ru> added the comment: An alternative fix is here: https://github.com/izbyshev/cpython/commit/b89b52f28490b69142d5c061604b3a3989cec66c Feel free to use the test if you don't like the approach :) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32270> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32844] subprocess may incorrectly redirect a low fd to stderr if another low fd is closed
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32270] subprocess closes redirected fds even if they are in pass_fds
Alexey Izbyshev <izbys...@ispras.ru> added the comment: Actually I've started to work on this with #32844, but got no feedback. This issue may of course be fixed independently, but the problems with descriptor ownership for fds <= 2 will remain unless all ownership problems are fixed at once. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32270> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32871] Interrupt .communicate() on SIGTERM/INT
Alexey Izbyshev <izbys...@ispras.ru> added the comment: @ajk225 It may be useful to look at #25942 before working on this. -- nosy: +izbyshev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32871> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
New submission from Alexey Izbyshev <izbys...@ispras.ru>: Demo: >>> from abc import ABC >>> issubclass(1, ABC) Segmentation fault (core dumped) The stack trace is attached. Before reimplementation of abc in C, the result was confusing too: Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32 >>> from abc import ABC >>> issubclass(1, ABC) Traceback (most recent call last): File "", line 1, in File "abc.py", line 230, in __subclasscheck__ File "_weakrefset.py", line 84, in add TypeError: cannot create weak reference to 'int' object -- components: Extension Modules files: stack-trace.txt messages: 313259 nosy: izbyshev, levkivskyi priority: normal severity: normal status: open title: issubclass(obj, abc.ABC) causes a segfault type: crash versions: Python 3.8 Added file: https://bugs.python.org/file47470/stack-trace.txt ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > Is there any sense in accepting non-types as the first argument of > issubclass()? No, though it is not (clearly) documented. The docs mention TypeError, but only for the second argument if my reading is correct. In practice, issubclass() raises a TypeError if the first argument is not a class object: >>> issubclass(1, int) Traceback (most recent call last): File "", line 1, in TypeError: issubclass() arg 1 must be a class Though, as I mentioned above, behavior for ABCs was always weird. -- versions: -Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: In debug mode, the following assertion fails: python: ./Modules/_abc.c:642: _abc__abc_subclasscheck_impl: Assertion `((PyObject*)(mro))->ob_type))->tp_flags & ((1UL << 26))) != 0)' failed. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33001] Buffer overflow vulnerability in os.symlink on Windows
Alexey Izbyshev <izbys...@ispras.ru> added the comment: While judging by the source code it seems that bytes in 3.5 should be fine, I've got a crash with the latest binary from python.org: Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.symlink(b'x\\' * 129, b'y\\' * 129) (Windows pop-up here) -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33001] Buffer overflow vulnerability in os.symlink on Windows
Alexey Izbyshev <izbys...@ispras.ru> added the comment: > As os.symlink requires administrative privileges on most versions of Windows The current implementation requires SeCreateSymbolicLinkPrivilege on ALL versions of Windows because users must pass an additional flag to CreateSymbolicLink to enable non-privileged symlinks on recent Windows 10, which os.symlink() doesn't do (see #31512). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: I've also checked that ABC.register() doesn't allow non-classes (and PEP 3119 mentions that). Looking at PyObject_IsSubclass in Objects/abstract.c, the only case in which its check_class() could be avoided is if there is a custom __subclasscheck__: >>> class M(type): ... def __subclasscheck__(cls, c): ... return c == 1 or super().__subclasscheck__(c) ... >>> class A(metaclass=M): ... pass ... >>> issubclass(1, A) True If there is no need to support such weird __subclasscheck__, check_class() could be called earlier. Note, however, that check_class() treats anything having __bases__ as a class, so moving the check alone is not enough to avoid the crash in all cases. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31512] Add non-elevated symlink support for dev mode Windows 10
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- nosy: +izbyshev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31512> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Change by Alexey Izbyshev <izbys...@ispras.ru>: -- versions: +Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33010] os.path.isdir() returns True for broken directory symlinks or junctions
New submission from Alexey Izbyshev <izbys...@ispras.ru>:
os.path.isdir() violates its own documentation by returning True for broken
directory symlinks or junctions, for which os.path.exists() returns False:
>>> os.mkdir('b')
>>> import _winapi
>>> _winapi.CreateJunction('b', 'a')
>>> os.rmdir('b')
>>> os.path.exists('a')
False
>>> os.path.isdir('a')
True
The underlying problem is that os.path.isdir() uses GetFileAttributes, which is
documented not to follow symlinks.
Eryk, is there a cheaper way to check FILE_ATTRIBUTE_DIRECTORY on a path while
following reparse points apart from
CreateFile/GetFileInformationByHandleEx/CloseFile?
Also, does it make sense to use GetFileAttributes as a fast path and use
something like above as a fallback only if FILE_ATTRIBUTE_REPARSE_POINT is set,
or does GetFileAttributes do something equivalently expensive under the hood?
--
components: Extension Modules, Windows
messages: 313314
nosy: eryksun, izbyshev, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: os.path.isdir() returns True for broken directory symlinks or junctions
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33010>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32999] issubclass(obj, abc.ABC) causes a segfault
Alexey Izbyshev <izbys...@ispras.ru> added the comment: OK, making a new implementation behave as the old one is fine with me too. BTW, do TypeErrors related to weak references deserve any treatment? Isn't it a kind of coincidence that the error raised due to usage of WeakSet in issubclass(obj, ABC) is what we expect? (Sorry, I'm not familiar with WeakSets). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32999> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34780] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows
Alexey Izbyshev added the comment: Ping! Thanks to @eryksun for providing feedback here and for the patch review. -- ___ Python tracker <https://bugs.python.org/issue34780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32890] os: Some functions may report bogus errors on Windows
Alexey Izbyshev added the comment:
Python 2.7 doesn't have the same issue.
In os.execve(), posix_error() is used, but it is based on errno, which is
correct. (A funny bit is that os.execve('', ['a'], {}) crashes in 2.7, probably
because of some checks in CRT).
And os.truncate() is not implemented in 2.7 at all.
--
versions: -Python 2.7
___
Python tracker
<https://bugs.python.org/issue32890>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33015] Fix function cast warning in thread_pthread.h
Alexey Izbyshev added the comment: Such casts will also trigger a CFI violation if somebody tries to build Python (and the libc, in this case) with a signature-based CFI [1, 2]. It checks that the compile-time callee signature matches the signature of the actually called function in runtime. [1] https://clang.llvm.org/docs/ControlFlowIntegrity.html [2] https://android-developers.googleblog.com/2018/10/control-flow-integrity-in-android-kernel.html -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue33015> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35091] Objects/listobject.c: gallop functions rely on signed integer overflow
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9498 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35091> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35091] Objects/listobject.c: gallop functions rely on signed integer overflow
Alexey Izbyshev added the comment: > This doesn't actually matter - the code can never trigger. Yes, I considered this, and wondered why assert wasn't used in the first place, but the explicit check with a comment suggested that possibility of overflow was deemed real. I've submitted a third PR that simply removes the checks and adds asserts instead. -- ___ Python tracker <https://bugs.python.org/issue35091> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35091] Objects/listobject.c: gallop functions rely on signed integer overflow
Change by Alexey Izbyshev : -- pull_requests: +9520 ___ Python tracker <https://bugs.python.org/issue35091> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35091] Objects/listobject.c: gallop functions rely on signed integer overflow
New submission from Alexey Izbyshev : gallop_left() and gallop_right() functions explicitly rely on overflowing behavior of Py_ssize_t (https://github.com/python/cpython/blob/6015cc50bc38b9e920ce4986ee10658eaa14f561/Objects/listobject.c#L1361): ofs = (ofs << 1) + 1; if (ofs <= 0) /* int overflow */ ofs = maxofs; Signed integer overflow is undefined in C, and the above is guaranteed to work only if compiler-specific workarounds are applied, such as GCC's -fwrapv (that is what CPython does). Without such workarounds the compiler would be free to remove the if statement. -- components: Interpreter Core messages: 328688 nosy: berker.peksag, izbyshev, pitrou, serhiy.storchaka priority: normal severity: normal status: open title: Objects/listobject.c: gallop functions rely on signed integer overflow type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35091> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] bz2: Potential division by zero in BZ2_Malloc()
Alexey Izbyshev added the comment: May be we should add a new function (_PyMem_RawMallocItems?) that does the same checks as PyMem_RawCalloc, but doesn't zero-initialize memory? -- ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35091] Objects/listobject.c: gallop functions rely on signed integer overflow
Change by Alexey Izbyshev : -- pull_requests: +9507 ___ Python tracker <https://bugs.python.org/issue35091> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] bz2: Potential division by zero in BZ2_Malloc()
New submission from Alexey Izbyshev : BZ2_Malloc() checks for size < 0 at https://github.com/python/cpython/blob/6015cc50bc38b9e920ce4986ee10658eaa14f561/Modules/_bz2module.c#L278 , but doesn't check for size == 0 before dividing by it: if (items < 0 || size < 0) return NULL; if ((size_t)items > (size_t)PY_SSIZE_T_MAX / (size_t)size) return NULL; Reported by Svace static analyzer. -- components: Extension Modules messages: 328686 nosy: berker.peksag, izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: bz2: Potential division by zero in BZ2_Malloc() type: behavior versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] Potential division by zero and integer overflow in allocator wrappers
Change by Alexey Izbyshev : -- title: bz2: Potential division by zero in BZ2_Malloc() -> Potential division by zero and integer overflow in allocator wrappers ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] Potential division by zero and integer overflow in allocator wrappers
Alexey Izbyshev added the comment: Thanks, Victor. Regarding backporting, what about integer overflow? Do you think it's guaranteed that the multiple of items and size always fits in 32-bit types, in case of BZ2_Malloc and PyZlib_Malloc? -- resolution: fixed -> status: closed -> open versions: +Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] Potential division by zero and integer overflow in allocator wrappers
Alexey Izbyshev added the comment: Sorry for changing the status, it's browser caching again. -- resolution: -> fixed status: open -> closed versions: -Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35090] bz2: Potential division by zero in BZ2_Malloc()
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9497 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35090> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35147] _Py_NO_RETURN is always empty on GCC
Change by Alexey Izbyshev : -- pull_requests: +9611 ___ Python tracker <https://bugs.python.org/issue35147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35147] _Py_NO_RETURN is always empty on GCC
Change by Alexey Izbyshev : -- pull_requests: +9612 ___ Python tracker <https://bugs.python.org/issue35147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35147] _Py_NO_RETURN is always empty on GCC
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9609 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35147] _Py_NO_RETURN is always empty on GCC
New submission from Alexey Izbyshev : Non-existing __GNUC_MAJOR__ macro is used to check for GCC version at https://github.com/python/cpython/blob/b942707fc23454a998323c17e30be78ff1a4f0e7/Include/pyerrors.h#L97 . __GNUC__ should be used instead. -- components: Interpreter Core messages: 329136 nosy: benjamin.peterson, berker.peksag, brett.cannon, izbyshev, serhiy.storchaka priority: normal severity: normal status: open title: _Py_NO_RETURN is always empty on GCC type: behavior versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35147> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32256] Make patchcheck.py work for out-of-tree builds
Alexey Izbyshev added the comment: Thank you for testing and merging, Victor! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue32256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35171] test_TimeRE_recreation_timezone failure on systems with non-default posixrules
New submission from Alexey Izbyshev : I've got the following on OpenSUSE Tumbleweed (glibc 2.27): == FAIL: test_TimeRE_recreation_timezone (test.test_strptime.CacheTests) -- Traceback (most recent call last): File "/scratch2/izbyshev/cpython/Lib/test/support/__init__.py", line 1683, in inner return func(*args, **kwds) File "/scratch2/izbyshev/cpython/Lib/test/test_strptime.py", line 702, in test_TimeRE_recreation_timezone self.assertEqual(tm.tm_isdst, 1) AssertionError: 0 != 1 This test sets TZ environment variable to 'STD-1DST' and calls time.tzset() (inside support.run_with_tz decorator). This TZ value lacks rules that specify when DST transitions happen. POSIX seems to be silent what should be done if the rule is missing, thought it does specify that the rule is optional[1]. What actually happens at least on Linux/glibc[2] and FreeBSD[3] is that those rules are taken from 'posixrules' file. On OpenSUSE, it's linked to '/etc/localtime', which is linked to 'Europe/Moscow' in my case. DST transitions were cancelled in Russia in 2011, so when Python tries to get timezone names for two points of the year (https://github.com/python/cpython/blob/570e371fd6e8615ece9b9e21fbe77149ebeb172e/Modules/timemodule.c#L1603), it gets unexpected values instead. The actual values depend on the bugginess of the libc used: * Glibc seems to be confused by having 'posixrules' file with DST cancelled but a specification which includes the DST timezone part, so localtime() returns 'MSK' in tm_zone (both for January and July). * musl doesn't implement 'posixrules' fallback, so it uses "all-zero" rules instead which cause it return 'DST' in both cases. * FreeBSD 11.1 (with 'posixrules' linked to 'Europe/Moscow') returns 'STD' in both cases. With any of the above, the test fails because strptime is called with the same timezone name two times. Note that even if PyInit_timezone() didn't probe January/July and used 'tzname' global instead, it wouldn't work as expected at least on FreeBSD: it sets tzname[1] to the empty string in the test above. ISTM the best way to fix this is to remove dependence on 'posixrules' file by specifying the rules in TZ. [1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08 [2] http://man7.org/linux/man-pages/man3/tzset.3.html [3] https://www.freebsd.org/cgi/man.cgi?query=tzset=3 -- components: Tests messages: 329320 nosy: benjamin.peterson, izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: test_TimeRE_recreation_timezone failure on systems with non-default posixrules type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35171] test_TimeRE_recreation_timezone failure on systems with non-default posixrules
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9647 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35191] socket.setblocking(x) treats multiples of 2**32 as False
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9695 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35191> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35191] socket.setblocking(x) treats multiples of 2**32 as False
New submission from Alexey Izbyshev : UBSAN with -fsanitize=implicit-integer-truncation reported a suspicious case: testSetBlocking_overflow (test.test_socket.NonBlockingTCPTests) ... /scratch2/izbyshev/cpython/Modules/socketmodule.c:2688:33: runtime error: implicit conversion from type 'long' of value 4294967296 (64-bit, signed) to type 'int' changed the value to 0 (32-bit, signed) It turned out that sock_setblocking() converts its (logically boolean) argument to long, but then passes it to internal_setblocking() which accepts int (https://github.com/python/cpython/blob/fd512d76456b65c529a5bc58d8cfe73e4a10de7a/Modules/socketmodule.c#L2688). This results in unexpected truncation on platforms with 64-bit long. testSetBlocking_overflow() which is supposed to check this doesn't work because it only checks socket timeout which is updated correctly. However, the actual state of socket descriptor is changed to the opposite value (non-blocking) in this case. -- components: Extension Modules messages: 329478 nosy: izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: socket.setblocking(x) treats multiples of 2**32 as False type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35191> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35204] Disable thread and memory sanitizers for address_in_range()
Alexey Izbyshev added the comment: I've submitted a PR which implements the change with additional cleanups. GCC introduced TSan together with ASan in 4.8[1], but didn't provide a macro to test for -fsanitize=thread option until 7[2,3]. [1] https://gcc.gnu.org/gcc-4.8/changes.html [2] https://gcc.gnu.org/onlinedocs/gcc-6.4.0/cpp/Common-Predefined-Macros.html#Common-Predefined-Macros [3] https://gcc.gnu.org/onlinedocs/gcc-7.1.0/cpp/Common-Predefined-Macros.html#Common-Predefined-Macros -- ___ Python tracker <https://bugs.python.org/issue35204> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35204] Disable thread and memory sanitizers for address_in_range()
New submission from Alexey Izbyshev : address_in_range() in Objects/obmalloc.c may access memory that is mapped but is considered free by the underlying libc allocator. In #18596, address sanitizing was disabled for this function. But thread and memory sanitizers similarly trip on this function: WARNING: ThreadSanitizer: heap-use-after-free (pid=24361) Read of size 4 at 0x7b7c0020 by main thread: #0 address_in_range /scratch2/izbyshev/cpython/Objects/obmalloc.c:1312:23 (python+0x59e912) ==3515==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x9713f8 in address_in_range /scratch2/izbyshev/cpython/Objects/obmalloc.c:1313:35 I suggest to disable them for this function as well. -- components: Interpreter Core messages: 329561 nosy: benjamin.peterson, izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: Disable thread and memory sanitizers for address_in_range() type: enhancement versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35204> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35204] Disable thread and memory sanitizers for address_in_range()
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9714 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35204> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9700 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
New submission from Alexey Izbyshev : UBSan with -fsanitize=implicit-integer-truncation found a suspicious one: /scratch2/izbyshev/cpython/Modules/cjkcodecs/_codecs_jp.c:43:17: runtime error: implicit conversion from type 'unsigned int' of value 4294966013 (32-bit, unsigned) to type 'unsigned char' changed the value to 253 (8-bit, unsigned) Indeed, the wrong constant was used (the correct one is used in corresponding decoder code at https://github.com/python/cpython/blob/fd512d76456b65c529a5bc58d8cfe73e4a10de7a/Modules/cjkcodecs/_codecs_jp.c#L105). In this case the truncation was harmless because only the lowest byte of the wrong result was used, and it was correct. But it probably makes sense to fix it if only to reduce noise from UBSan. All Python versions are affected, but I've marked 3.8 only since I'm not sure what the policy for backporting such changes is. -- components: Extension Modules messages: 329489 nosy: izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: A typo in a constant in cp932 codec type: behavior versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35081] Move internal headers to Include/internal/
Alexey Izbyshev added the comment: Victor, you moved declarations of some functions to other headers, but didn't include the new headers into files that implement the functions in some cases. For example, _PyGILState_Init was moved into Include/internal/pycore_lifecycle.h in a1c249c40517917d2e0971d55aea8d14a44b2cc8, but it's implemented in Python/pystate.c, which doesn't include the new header. This may lead to subtle problems because the compiler can't check that signatures of the declaration and the implementation match. I suggest to use -Wmissing-prototypes and -Wmissing-declarations to detect such situations: ../../cpython/Python/pystate.c: At top level: ../../cpython/Python/pystate.c:968:1: warning: no previous prototype for ‘_PyGILState_Init’ [-Wmissing-prototypes] _PyGILState_Init(PyInterpreterState *i, PyThreadState *t) ^~~~ ../../cpython/Python/pystate.c:988:1: warning: no previous prototype for ‘_PyGILState_Fini’ [-Wmissing-prototypes] _PyGILState_Fini(void) ^~~~ Sadly, there are many other similar issues in Python now, but you can at least compare the number of warnings before and after your changes. -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue35081> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35214] Get the test suite passing with clang Memory Sanitizer enabled
Change by Alexey Izbyshev : -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue35214> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35081] Move internal headers to Include/internal/
Alexey Izbyshev added the comment: > Oh, I never saw this warning before. It seems to not be included in -Wall. > Would you mind to open a new issue to discuss it? Victor, I've opened #35258 as you suggested. -- ___ Python tracker <https://bugs.python.org/issue35081> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35258] Consider enabling -Wmissing-prototypes
New submission from Alexey Izbyshev :
This issue is a follow-up of msg329608 of #35081.
GCC and Clang have -Wmissing-prototypes[1] diagnostic that is reported if a
global function is defined without a previous declaration containing a
prototype. The reasons may be the following:
1) The header where it is declared is not included before the definition.
2) There is a declaration before the definition, but it doesn't contain a
prototype (e.g., "int foo()").
3) There is no separate declaration of the function in the project at all.
(1) is undesirable because the compiler can't check that signatures of the
declaration and the definition match. If they don't, subtle issues may occur
when such function is called using a wrong declaration.
(2) is undesirable too because there is usually no reason to use declarations
without prototypes. Often "int foo()" was meant to be "int foo(void)" -- these
two have different meaning in C.
(3) may mean that either the function is unused (which is probably undesirable)
or it was intentionally defined without a declaration. One case when the latter
makes sense is plugin-like files which define initialization functions or
similar external entry points. Those may be called via pointers obtained with
dlsym() or have calls generated at compile time, like Modules/config.c in
CPython.
It would be good to enable -Wmissing-prototypes to catch (1) and (2) -- such
issues exist as of time of this report. However, (3) is a problem because of
module initialization functions which would cause unwanted diagnostics.
There is no function attribute that could be put in PyMODINIT_FUNC macro to
suppress the diagnostic for a single function. The general diagnostic
suppression machinery (#pragma GCC diagnostic[2]) affects everything on the
following lines, and while the state can be saved and restored via "#pragma GCC
diagnostic push/pop", we'd either need another macro to restore the state after
the end of module initializer definitions or require that initializers always
come last in the file.
A workaround is to declare the initializer inside PyMODINIT_FUNC (or a new
macro). For example (omitting platform-specific parts of PyMODINIT_FUNC):
#define PyMODINIT_FUNC(name) \
extern PyObject *name(void); \
PyObject *name(void)
PyMODINIT_FUNC(PyInit_time)
{
...
}
I've also noticed that PyMODINIT_FUNC is used for functions with a different
signature in Modules/_testmultiphase.c, so either another macro would be needed
for that, or a general macro accepting a prototype as its variadic parameter
could be added.
Even if it's deemed infeasible to enable -Wmissing-prototypes by default,
developers may find it useful to ensure that the diagnostic is reported only
for files in Modules directory.
[1] https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
[2] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html
--
components: Build
messages: 329954
nosy: izbyshev, vstinner
priority: normal
severity: normal
status: open
title: Consider enabling -Wmissing-prototypes
type: enhancement
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue35258>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35170] 3.7.1 compile failure on CentOS 6.10; _ctypes did not build
Alexey Izbyshev added the comment: The in-tree copy of libffi was removed in #27979 (between 3.6 and 3.7) for all platforms except Windows and macOS. Therefore, you need libffi development package installed in your system to build CPython 3.7. As for PIP, personally, I think that's an overkill to require ctypes, especially at CPython build time. If you don't want to build ctypes, I believe you may use '--without-ensurepip' configure option as a workaround. -- nosy: +izbyshev, zach.ware ___ Python tracker <https://bugs.python.org/issue35170> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Alexey Izbyshev added the comment: OUTCHAR2 is a wrong example. Other examples are NEXT_IN, NEXT_OUT. -- ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Alexey Izbyshev added the comment: > Maybe test "Py_CHARMASK(c) == (c)"? This is a good alternative if multiple evaluation of 'c' is acceptable. Though I'd prefer '(unsigned char)c == c' for this style of fixing because it is bit closer to what happens in '((*outbuf)[i]) = c'. -- ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Change by Alexey Izbyshev : -- pull_requests: +9706 ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Alexey Izbyshev added the comment: I've added 'assert' to macros. Since 'typeof' seems to be disallowed in Python, I've used 'unsigned int' as the type of an intermediate variable. Another alternative is 'assert(0 <= (c) && (c) <= 255)', but 'c' will be evaluated several times. -- ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Change by Alexey Izbyshev : -- pull_requests: +9707 ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35194] A typo in a constant in cp932 codec
Alexey Izbyshev added the comment: I've checked than other macros in Modules/cjkcodecs/cjkcodecs.h don't avoid multiple argument evaluation (e.g. OUTCHAR2, _TRYMAP_ENC), so I've changed 'assert' to a variant of what Serhiy suggested. -- ___ Python tracker <https://bugs.python.org/issue35194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35161] ASAN: stack-use-after-scope in grp.getgr{nam, gid} and pwd.getpw{nam, uid}
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +9620 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35161> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35161] ASAN: stack-use-after-scope in grp.getgr{nam, gid} and pwd.getpw{nam, uid}
New submission from Alexey Izbyshev :
==24122==ERROR: AddressSanitizer: stack-use-after-scope on address
0x7fffb1c62550 at pc 0x006ec66c bp 0x7fffb1c62450 sp 0x7fffb1c62448
READ of size 8 at 0x7fffb1c62550 thread T0
#0 0x6ec66b in mkpwent /scratch2/izbyshev/cpython/Modules/pwdmodule.c:79
#1 0x6ecdc9 in pwd_getpwnam_impl
/scratch2/izbyshev/cpython/Modules/pwdmodule.c:260
#2 0x6ecfee in pwd_getpwnam
/scratch2/izbyshev/cpython/Modules/clinic/pwdmodule.c.h:39
#3 0x454146 in _PyMethodDef_RawFastCallKeywords
/scratch2/izbyshev/cpython/Objects/call.c:644
[=== snip ===]
Address 0x7fffb1c62550 is located in stack of thread T0 at offset 160 in frame
#0 0x6eca60 in pwd_getpwnam_impl
/scratch2/izbyshev/cpython/Modules/pwdmodule.c:203
This frame has 3 object(s):
[32, 40) 'name_chars'
[96, 104) 'p'
[160, 208) 'pwd' <== Memory access at offset 160 is inside this variable
Variables declared in the block scope created with
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS are referred to via a pointer
outside of that scope (i.e., after their lifetime ends). The bug was introduced
in
https://github.com/python/cpython/commit/23e65b25557f957af840cf8fe68e80659ce28629
.
--
components: Extension Modules
messages: 329230
nosy: berker.peksag, izbyshev, serhiy.storchaka, vstinner, wg
priority: normal
severity: normal
status: open
title: ASAN: stack-use-after-scope in grp.getgr{nam,gid} and pwd.getpw{nam,uid}
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue35161>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34874] Python 3.6.3 command script wrapped in single quotes produces NameError: name 'A' is not defined
Alexey Izbyshev added the comment: @Tim Is it possible that 'python3' in your command refers to some wrapper which forwards its arguments to real Python in a wrong way? -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue34874> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34760] Regression in abc in combination with passing a function to issubclass
Change by Alexey Izbyshev : -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue34760> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34780] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows
New submission from Alexey Izbyshev : In the following code inspired by a production issue I had to debug recently subprocess.call() won't return: import os import subprocess import sys import time r, w = os.pipe() p1 = subprocess.Popen([sys.executable, '-c', 'import sys; sys.stdin.read()'], stdin=r) time.sleep(1) subprocess.call([sys.executable, '-c', ''], stdin=r) os.close(w) p1.wait() The underlying reason is the same as in #22976. Python performs certain operations on stdin during it's initialization (different in 2.7 and 3.x), which block because there is an outstanding ReadFile() on the pipe end stdin refers to. Assuming that subprocess.call() runs some app that doesn't use stdin at all, if a developer doesn't control how the app is run (which was my case), I don't see any way to workaround this in pure Python. (An obvious workaround is to make a wrapper which closes stdin or redirects it to something else, but this wrapper can't be run with CPython). I propose to fix this in CPython. The details are slightly different for 2.7 and 3.x. 2.7 calls fstat(stdin) in dircheck() (Objects/fileobject.c). This hangs because msvcrt calls PeekNamedPipe() if stdin refers to a pipe. Ironically, this fstat() call is completely useless on Windows because msvcrt never sets S_IFDIR in st_mode (it can't distinguish between a file and a directory because it uses GetFileType() and doesn't perform extra checks). I've implemented a PR that skips dircheck() on Windows. (If we do want to add a proper dircheck() to 2.7, it should do something similar to 3.x). 3.x performs the dir check without relying on fstat(), but it also calls lseek() (in _buffered_init() (Modules/_io/bufferedio.c), if removed, there is another one in _io_TextIOWrapper___init___impl (Modules/_io/textio.c). mscvrt calls SetFilePointerEx(), which hangs too, which is somewhat surprising because its docs [1] say: You cannot use the SetFilePointerEx function with a handle to a nonseeking device such as a pipe or a communications device. The wording is unclear though -- it doesn't say what happens if I try. lseek() docs [2] contain the following: On devices incapable of seeking (such as terminals and printers), the return value is undefined. In practice, lseek() succeeds on pipes on Windows, but is nearly useless: Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32 >>> import os >>> r, w = os.pipe() >>> os.write(w, b'xyz') 3 >>> os.lseek(r, 0, os.SEEK_CUR) 0 >>> os.lseek(r, 0, os.SEEK_END) 3 >>> os.lseek(r, 2, os.SEEK_SET) 2 >>> os.read(r, 1) b'x' >>> os.lseek(r, 0, os.SEEK_CUR) 2 >>> os.read(r, 1) b'y' >>> os.lseek(r, 0, os.SEEK_CUR) 2 >>> os.lseek(r, 0, os.SEEK_END) 1 So lseek() can be used to check the current pipe buffer size, and that seems about it. Given the above, I suggest two solutions for the hang on Windows: 1) Make lseek() fail on pipes on Windows, as it does on Unix. A number of projects have already done that: https://referencesource.microsoft.com/#mscorlib/system/io/filestream.cs,1029 https://go.googlesource.com/go/+/ce58a39fca067a19c505220c0c907ccf32793427/src/syscall/syscall_windows.go#374 https://trac.ffmpeg.org/ticket/986 (workaround: https://lists.ffmpeg.org/pipermail/ffmpeg-cvslog/2012-June/051590.html) https://github.com/erikd/libsndfile/blob/123cb9f9a5a356b951a23e9e2ab8527f967425cc/src/file_io.c#L266 2) Delay lseek() until it's really needed. In both cases (BufferedIO and TextIO), lseek() is used to set some cached fields, so ISTM it's not necessary to do it during initialization. This would also be an optimization (skip lseek() syscall until a user really wants to tell()/seek()). This can be done as a sole fix or can be combined with the above (as an optimization). I'd like to hear other people's opinions before doing anything for Python 3. [1] https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-setfilepointerex [2] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek-lseeki64 -- components: IO, Windows messages: 326175 nosy: eryksun, izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware priority: normal severity: normal status: open title: Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue34780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34780] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +8924 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue34780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35537] use os.posix_spawn in subprocess
Alexey Izbyshev added the comment: > One of the issue that I have with using posix_spawn() is that the *exact* > behavior of subprocess is not properly defined by test_subprocess. Should we > more more tests, or document that the exact behavior is "an implementation > detail"? Regarding using PATH from "env" instead of parent's environment, it may be considered documented because subprocess docs refer to os.execvp(), where it's explicitly documented: """ The variants which include a “p” near the end (execlp(), execlpe(), execvp(), and execvpe()) will use the PATH environment variable to locate the program file. When the environment is being replaced (using one of the exec*e variants, discussed in the next paragraph), the new environment is used as the source of the PATH variable. """ The problem is that it differs from execvp() in libc (and POSIX), so I would consider such behavior a bug if it wasn't so old to become a feature. Thanks to Victor for noticing that, I missed it. So, if we can't change os.execvp() and/or current subprocess behavior, posix_spawnp seems to be ruled out. (I don't consider temporarily changing the parent environment as a solution). A naive emulation of posix_spawnp would be repeatedly calling posix_spawn for each PATH entry, but that's prohibitively expensive. Could we use a following hybrid approach instead? Iterate over PATH entries and perform a quick check for common exec errors (ENOENT, ENOTDIR, EACCES) manually (e.g. by stat()). If the check fails, exec would also fail so just skip the entry. (It's important not to get false negatives here, but the child process would have the same privileges as the parent since we don't use POSIX_SPAWN_RESETIDS, so I can't think of one). Otherwise, call posix_spawn with the absolute path. If it fails, skip the entry. Looks ugly, but are there other problems? This would seem to work just as well as posix_spawnp in the common case, but in the worst (contrived) case it would be equivalent to calling posix_spawn for each PATH entry. -- ___ Python tracker <https://bugs.python.org/issue35537> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
