[issue45767] Fix types for dev_t processing in posix module
Dmitry Marakasov added the comment: > Is device number -1 used in any context (for example as "unknown device > number")? Yes, there's NODEV macro in both Linux and FreeBSD which expands to ((dev_t)-1). -- ___ Python tracker <https://bugs.python.org/issue45767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45767] Fix types for dev_t processing in posix module
Dmitry Marakasov added the comment: In case you're curious, here are some st_dev values which are encountered on my FreeBSD: - 0xac2308de99d1f699 - ZFS - 0x7100ff00 - devfs - 0x8700ff01 - tmpfs - 0x2900ff4e - nullfs I suspect Linux uses smaller numbers which do not cause overflows so tese tests hasn't fired on Linux. -- ___ Python tracker <https://bugs.python.org/issue45767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45767] Fix types for dev_t processing in posix module
Change by Dmitry Marakasov : -- pull_requests: +27743 stage: -> patch review pull_request: https://github.com/python/cpython/pull/29494 ___ Python tracker <https://bugs.python.org/issue45767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45767] Fix types for dev_t processing in posix module
New submission from Dmitry Marakasov : So, I was investigating a test failure of python 3.11 and 3.10 on FreeBSD (but it likely applies to all python versions): == FAIL: test_makedev (test.test_posix.PosixTester) -- Traceback (most recent call last): File "/usr/ports/lang/python311/work/Python-3.11.0a1/Lib/test/test_posix.py", line 686, in test_makedev self.assertGreaterEqual(dev, 0) ^^^ AssertionError: -5228656221359548839 not greater than or equal to 0 -- The test checks that posix.stat(somepath).st_dev >= 0, but negative value was returned. Python uses PyLong_FromLongLong to convert from dev_t C type which st_dev is: https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L2410 https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L901 POSIX does not seem to define signedness of dev_t, https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/basedefs/sys_types.h.html only saying it shall be an "integer type". However on practice on both FreeBSD and Linux it's unsigned: FreeBSD: typedef __dev_t dev_t; // sys/types.h typedef __uint64_t __dev_t; // sys/_types.h Linux (Ubuntu 18.04): typedef __dev_t dev_t; // sys/stat.h __STD_TYPE __DEV_T_TYPE __dev_t; // sys/types.h #define __DEV_T_TYPE__UQUAD_TYPE; // sys/typesizes.h So I suggest the attached patch to switch _PyLong_FromDev to PyLong_FromUnsignedLongLong which also makes it consistent with _Py_Dev_Converter which converts the other way with PyLong_AsUnsignedLongLong. This change fixes the mentioned test, but another test failure is unmasked: == ERROR: test_makedev (test.test_posix.PosixTester) -- Traceback (most recent call last): File "/usr/ports/lang/python311/work/Python-3.11.0a1/Lib/test/test_posix.py", line 704, in test_makedev self.assertEqual(posix.makedev(major, minor), dev) ^^^ OverflowError: Python int too large to convert to C int -- This problem needs couple more trivial changes, but I'm not sure how to make them correctly (and I'm also confused by how this file relates with clinic/posixmodule.c). The problems are that: - os_major_impl/os_minor_impl and os_makedev_impl are inconsistent in their argument/return types for major/minor dev numbers: the former use `unsigned int`, while the latter uses `int`. - the correct type is platform dependent, for instance Linux uses `unsigned int` and FreeBSD uses `int` (from `man makedev`). I guess that to fix this failure one needs to add a macro/typedef for the type for minor/major dev numbers like #if defined(__FreeBSD__) #define DEVICE_MAJORMINOR_T int #else #define DEVICE_MAJORMINOR_T unsigned int #endif and use it in the named functions: static DEVICE_MAJORMINOR_T os_major_impl(PyObject *module, dev_t device) static DEVICE_MAJORMINOR_T os_minor_impl(PyObject *module, dev_t device) static dev_t os_makedev_impl(PyObject *module, DEVICE_MAJORMINOR_T major, DEVICE_MAJORMINOR_T minor) -- components: Extension Modules, FreeBSD files: posixmodule.c.patch keywords: patch messages: 406030 nosy: AMDmi3, koobs priority: normal severity: normal status: open title: Fix types for dev_t processing in posix module type: behavior versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file50433/posixmodule.c.patch ___ Python tracker <https://bugs.python.org/issue45767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45326] Unexpected TypeError with type alias+issubclass+ABC
New submission from Dmitry Marakasov : Here's a curious problem. issubclass() check of a type against an ABC-derived class raises TypeError claiming that type is not a class, however inspect.isclass() says it's a class, and issubclass() check against a simple class works fine: ``` from abc import ABC class C1: pass issubclass(dict[str, str], C1) # False class C2(ABC): pass issubclass(dict[str, str], C2) # TypeError: issubclass() arg 1 must be a class ``` I've ran into this problem while using `inspect` module to look for subclasses of a specific ABC in a module which may also contain type aliases, and after converting a type alias from `Dict[str, str]` to modern `dict[str, str]` I've got an unexpected crash in this code: if inspect.isclass(member) and issubclass(member, superclass): Not sure which is the culprit, ABC or how dict[]-style type aliases are implemented. -- components: Library (Lib) messages: 402914 nosy: AMDmi3 priority: normal severity: normal status: open title: Unexpected TypeError with type alias+issubclass+ABC versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue45326> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43792] Cython is not compatible with Python 3.10 yet: "SystemError: bad argument to internal function"
Dmitry Marakasov added the comment: > This issue is a bug in Cython, not in Python. I close it. This is correct, thanks! The problem is gone with git master version of cython. -- ___ Python tracker <https://bugs.python.org/issue43792> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43792] "bad argument to internal function" in cython
New submission from Dmitry Marakasov : I'm playing with adding python3.10a7 port to FreeBSD ports collection, and have run into similar problem with building multiple third party modules (for instance, yaml, yarl and pystemmer). Here's log from yaml: running build_ext cythoning yaml/_yaml.pyx to yaml/_yaml.c Traceback (most recent call last): File "", line 1, in File "setup.py", line 271, in setup( File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup return distutils.core.setup(**attrs) File "/usr/local/lib/python3.10/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/local/lib/python3.10/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/usr/local/lib/python3.10/distutils/dist.py", line 985, in run_command cmd_obj.run() File "setup.py", line 187, in run _build_ext.run(self) File "/usr/local/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/usr/local/lib/python3.10/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "setup.py", line 229, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/usr/local/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 346, in cython_sources result = cython_compile(source, options=options, File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 778, in compile return compile_single(source, options, full_module_name) File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 727, in compile_single return run_pipeline(source, options, full_module_name) File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 479, in run_pipeline context = options.create_context() File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 596, in create_context return Context(self.include_path, self.compiler_directives, File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 80, in __init__ from . import Builtin, CythonScope File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/CythonScope.py", line 5, in from .UtilityCode import CythonUtilityCode File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/UtilityCode.py", line 3, in from .TreeFragment import parse_from_strings, StringParseContext File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/TreeFragment.py", line 17, in from .Visitor import VisitorTransform File "Cython/Compiler/Visitor.py", line 17, in init Cython.Compiler.Visitor File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/ExprNodes.py", line 4742, in class SliceIndexNode(ExprNode): File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/ExprNodes.py", line 4939, in SliceIndexNode get_slice_utility_code = TempitaUtilityCode.load( File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Code.py", line 404, in load return cls(**kwargs) File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Code.py", line 645, in __init__ proto = sub_tempita(proto, context, file, name) File "/usr/local/lib/python3.10/site-packages/Cython/Compiler/Code.py", line 638, in sub_tempita return sub(s, **context) File "Cython/Tempita/_tempita.py", line 376, in Cython.Tempita._tempita.sub File "Cython/Tempita/_tempita.py", line 137, in Cython.Tempita._tempita.Template.__init__ File "Cython/Tempita/_tempita.py", line 819, in Cython.Tempita._tempita.parse File "Cython/Tempita/_tempita.py", line 661, in Cython.Tempita._tempita.lex SystemError: Python/getargs.c:2038: bad argument to internal function *** Error code 1 -- components: Extension Modules, FreeBSD messages: 390621 nosy: AMDmi3, koobs priority: normal severity: normal status: open title: "bad argument to internal function" in cython versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43792> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38744] python 3.8 hang in multiprocessing.Pool() locking on FreeBSD
Dmitry Marakasov added the comment: > What happens if you instead write: It survived more than 4000 runs, it looks like it doesn't experience the problem. -- ___ Python tracker <https://bugs.python.org/issue38744> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38744] python 3.8 hang in multiprocessing.Pool() locking on FreeBSD
New submission from Dmitry Marakasov : System: FreeBSD 12.0-RELEASE, amd64. This simple program from multiprocessing import Pool from time import sleep Pool().map(sleep, [0.01] * 10) works fine with python 3.7, but is likely (about 20-50% probability on my 4 core box) to hang with python 3.8. Example backtraces after interruption: % python3.8 1.py ^CException ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__ Process ForkPoolWorker-2: Process ForkPoolWorker-4: res = self._callback(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool cls._help_stuff_finish(inqueue, task_handler, len(pool)) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish inqueue._rlock.acquire() KeyboardInterrupt: Process ForkPoolWorker-3: Process ForkPoolWorker-1: % python3.8 1.py ^CException ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__ Process ForkPoolWorker-3: Process ForkPoolWorker-4: Process ForkPoolWorker-1: res = self._callback(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool cls._help_stuff_finish(inqueue, task_handler, len(pool)) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish inqueue._rlock.acquire() KeyboardInterrupt: Process ForkPoolWorker-2: % python3.8 1.py ^CException ignored in: Process ForkPoolWorker-2: Process ForkPoolWorker-3: Traceback (most recent call last): File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__ Process ForkPoolWorker-1: res = self._callback(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool cls._help_stuff_finish(inqueue, task_handler, len(pool)) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish Traceback (most recent call last): Traceback (most recent call last): File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker task = get() File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker task = get() File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 355, in get with self._rlock: File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 355, in get with self._rlock: File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__ return self._semlock.__enter__() KeyboardInterrupt KeyboardInterrupt inqueue._rlock.acquire() KeyboardInterrupt: Traceback (most recent call last): File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker task = get() File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 356, in get res = self._reader.recv_bytes() File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) KeyboardInterrupt Process ForkPoolWorker-4: Related issue in FreeBSD bugzilla: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241801 -- components: Library (Lib) messages: 356243 nosy: AMDmi3 priority: normal severity: normal status: open title: python 3.8 hang in multiprocessing.Pool() locking on FreeBSD versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue38744> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36527] unused parameter warnings in Include/object.h (affecting building third party code)
New submission from Dmitry Marakasov : Python 3.8 and nightly introduces unused (in some cases) parameters in object.h header. This makes compilation of third party code which includes the header fail if it's built with -Werror. Build log excerpt: --- g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -UNDEBUG -Wall -Wextra -Werror -fPIC -I/usr/include/yajl -I/opt/python/3.8-dev/include/python3.8m -c src/construct_handlers.cc -o build/temp.linux-x86_64-3.8/src/construct_handlers.o -std=c++11 -DJSONSLICER_VERSION="0.1.4" -DUSE_BYTES_INTERNALLY -fno-exceptions -fno-rtti In file included from /opt/python/3.8-dev/include/python3.8m/pytime.h:6:0, from /opt/python/3.8-dev/include/python3.8m/Python.h:85, from src/pyobjlist.hh:26, from src/jsonslicer.hh:26, from src/construct_handlers.hh:26, from src/construct_handlers.cc:23: /opt/python/3.8-dev/include/python3.8m/object.h:441:50: error: unused parameter ‘op’ [-Werror=unused-parameter] static inline void _Py_ForgetReference(PyObject *op) ^ /opt/python/3.8-dev/include/python3.8m/object.h:458:43: error: unused parameter ‘filename’ [-Werror=unused-parameter] static inline void _Py_DECREF(const char *filename, int lineno, ^ /opt/python/3.8-dev/include/python3.8m/object.h:458:57: error: unused parameter ‘lineno’ [-Werror=unused-parameter] static inline void _Py_DECREF(const char *filename, int lineno, ^ --- Full build log: https://travis-ci.org/AMDmi3/jsonslicer/jobs/515771366 Possible solutions: - Add (void)param; to silence the warning as it's already done in Include/internal/pycore_atomic.h - Make distutils list python include directory with -isystem instead of -I. This way, python include won't generate warnings for third party code. -- components: Build messages: 339452 nosy: amdmi3 priority: normal severity: normal status: open title: unused parameter warnings in Include/object.h (affecting building third party code) type: compile error versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue36527> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
