[issue28474] WinError(): Python int too large to convert to C long
Change by Adam Meily : -- keywords: +patch pull_requests: +26407 stage: -> patch review pull_request: https://github.com/python/cpython/pull/27959 ___ Python tracker <https://bugs.python.org/issue28474> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27484] Some Examples in Format String Syntax are incorrect or poorly worded
Change by Adam Meily : -- nosy: +meilyadam nosy_count: 5.0 -> 6.0 pull_requests: +26405 pull_request: https://github.com/python/cpython/pull/27959 ___ Python tracker <https://bugs.python.org/issue27484> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28474] WinError(): Python int too large to convert to C long
Adam Meily added the comment: I can potentially take a stab at writing up a PR for this. I've also seen this affecting other locations that eventually call FormatMessage, including: - ctypes.format_error() - this original issue - os.strerror() - OSError(winerror=X) I will most likely look into fixing all three. -- nosy: +meilyadam ___ Python tracker <https://bugs.python.org/issue28474> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39439] Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
Change by Adam Meily : -- pull_requests: +17546 pull_request: https://github.com/python/cpython/pull/18159 ___ Python tracker <https://bugs.python.org/issue39439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39439] Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
Change by Adam Meily : -- pull_requests: +17545 pull_request: https://github.com/python/cpython/pull/18158 ___ Python tracker <https://bugs.python.org/issue39439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39439] Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
Change by Adam Meily : -- pull_requests: +17544 pull_request: https://github.com/python/cpython/pull/18157 ___ Python tracker <https://bugs.python.org/issue39439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+
Change by Adam Meily : -- pull_requests: +17543 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/18157 ___ Python tracker <https://bugs.python.org/issue38092> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39439] Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
Adam Meily added the comment: OK, that makes sense. For 3.7, I can create a PR for that corrects the order of arguments passed into _winapi.CreateProcess For 3.8 / master, the problem appears to be that the check in popen_spawn_win32.py to set the subprocess env is failing because sys.executable != spawn.get_executable() -- spawn.get_executable() is returning sys._base_executable. So, can you confirm that the fix is to just change spawn.get_executable() to return sys.executable, like it was prior to the PR mentioned in the other ticket? -- ___ Python tracker <https://bugs.python.org/issue39439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39439] Windows Multiprocessing in Virtualenv: sys.prefix is incorrect
New submission from Adam Meily : I upgraded from Python 3.7.1 to 3.7.6 and began noticing a behavior that was breaking my code. My code detects if it's running in a virtualenv. This check worked in 3.7.1 but is broken in 3.7.6. >From the documentation, sys.prefix and sys.exec_prefix should point to the >virtualenv when one is active. However, I'm seeing that both of these >constants are pointing to the system installation directory and not my >virtualenv when I am in a multiprocessing child. Here is an example output of >a test application running in 3.7.6 (I've attached the test script to this >ticket): Parent process = sys.prefix: C:\Users\user\project\venv sys.exec_prefix: C:\Users\user\project\venv sys.base_prefix: C:\Program Files\Python37 sys.base_exec_prefix: C:\Program Files\Python37 = Subprocess = sys.prefix: C:\Program Files\Python37 sys.exec_prefix: C:\Program Files\Python37 sys.base_prefix: C:\Program Files\Python37 sys.base_exec_prefix: C:\Program Files\Python37 = I would expect that sys.prefix and sys.exec_prefix to be identical in the parent and child process. I verified that this behavior is present in 3.7.5, 3.7.6, and 3.8.1. I am on a Windows 10 x64 system. Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32 -- components: Windows files: multiproc_venv_prefix.py messages: 360581 nosy: meilyadam, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Windows Multiprocessing in Virtualenv: sys.prefix is incorrect versions: Python 3.7, Python 3.8 Added file: https://bugs.python.org/file48862/multiproc_venv_prefix.py ___ Python tracker <https://bugs.python.org/issue39439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29883] asyncio: Windows Proactor Event Loop UDP Support
Adam Meily added the comment: I've rebased onto upstream master and I fixed the CI build. -- ___ Python tracker <https://bugs.python.org/issue29883> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29883] asyncio: Windows Proactor Event Loop UDP Support
Changes by Adam Meily : -- pull_requests: +1213 ___ Python tracker <http://bugs.python.org/issue29883> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29883] asyncio: Windows Proactor Event Loop UDP Support
New submission from Adam Meily: I am working on a Python 3.5 project that uses asyncio on a Windows system to poll both UDP and TCP connections. Multiple sources online say that the Windows Proactor event loop, which uses I/O Completion Ports, is considerably faster and more efficient than the default Selector event loop. I'm using both UDP and TCP connections so I am stuck with the Selector event loop for the time being. I've seen the overhead of 128 open UDP/TCP connections on the Selector event loop to be near 85%, which I understand is entirely spent in Windows proprietary code and not the Python implementation. I'd like to take a shot at implementing UDP support in the IOCP event loop. It looks like there will be a considerable amount of code shared between TCP and UDP IOCP so I plan on implementing UDP support directly in _ProactorReadPipeTransport and _ProactorBaseWritePipeTransport. I should be able to do this by wrapping any TCP/UDP specific function calls in a check of: if sock.type == socket.SOCK_DGRAM: # Do UDP stuff elif sock.type == socket.SOCK_STREAM: # Do TCP stuff My initial implementation plan is to: - Call datagram_received() instead of data_received() when UDP data is available in _ProactorReadPipeTransport._loop_reading(). - Implement BaseProactorEventLoop._make_datagram_transport(). - Implement wrappers for WSAConnect, WSARecvFrom, and WSASendTo in _overlapped. - Implement sendto() and recvfrom() in IocpProactor, which will use the new functions in _overlapped. - Implement handling for UDP "connections" in IocpProactor.connect() to call WSAConnect(). WSAConnect() appears to always return immediately so the function not supporting IOCP shouldn't be an issue. We can't use ConnectEx() for UDP because ConnectEx() is for connection-oriented sockets only. My project is unfortunately tied to Python 3.5. So, if possible, I'd like to have UDP support merged into a v3.5 release. I can fork off of master instead of v3.5.3 if Python 3.5 support isn't an option. -- components: asyncio messages: 290010 nosy: Adam Meily, yselivanov priority: normal severity: normal status: open title: asyncio: Windows Proactor Event Loop UDP Support type: enhancement versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue29883> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24909] Windows: subprocess.Popen: race condition for leaking inheritable handles
Adam Meily added the comment: Ok, I can re-implement the patch to meet what you all are looking for. I just want to double check that I'm on the same page: I'll get rid of the lock, because the fix should really be done in the call to CreateProcessW. I imagine that I'll be editing Modules/_winapi.c:824. Essentially, the fix will include switching _winapi.c to use a STARTUPINFOEX structure, and manually passing in the three handles for stdin, stdout, and stderr to PROC_THREAD_ATTRIBUTE_HANDLE_LIST. The one potential problem that I see is that it looks like specifying PROC_THREAD_ATTRIBUTE_HANDLE_LIST means that no other handles are inherited, even if they are inheritable. Doesn't this break applications that are explicitly creating inheritable handles and expecting them to be accessible within the subprocess? I could add Windows support for pass_fds, but then existing applications would have to be updated to use the argument on Windows. -- ___ Python tracker <http://bugs.python.org/issue24909> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24909] Windows: subprocess.Popen: race condition for leaking inheritable handles
Adam Meily added the comment: @r.david.murray: Yes I could make a test. @haypo: I did not know about the PROC_THREAD_ATTRIBUTE_HANDLE_LIST structure, thanks for the heads up. You pointed me in the right direction, and I see now that you've been following this, and similar, subprocessing issues on Windows for some time. I understand the hesitancy for adding a Lock to the standard library. Unless I am mistaken, I don't think my patch could cause a deadlock. I am releasing the lock in a `finally` block. So, the way I see it, the only way a deadlock can occur is if the thread crashes while in C code, which will cause the `finally` block to not execute. The lock will stay acquired, and then another thread will deadlock trying to acquire the lock. However, if that is the case and the thread crashed in C, the whole interpreter instance would have crashed, not just the Thread, which would make the deadlock impossible. It looks like the structure you reference to, PROC_THREAD_ATTRIBUTE_HANDLE_LIST (STARTUPINFOEX), is only available in Windows Vista and after. Which means that Windows XP/Server 2003 would still have this issue. -- ___ Python tracker <http://bugs.python.org/issue24909> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24909] Windows: subprocess.Popen: race condition for leaking inheritable handles
Adam Meily added the comment: Attached is a test Python script that you can run to see the race condition in action. There are two Python scripts: pipe.py and reader.py. - pipe.py: make two subprocess.Popen() calls from two different threads. - reader.py: (its content is in the bottom comments of pipe.py, so move it to a separate file): reads lines from stdin and exits Basically, pipe.py creates a pipe between "echo Hello World!" and "python reader.py". You'll see that pipe.py will hang until ctrl+c is entered, even though the first subprocess in the pipe, reader.py, exits. Uncommenting the time.sleep() call will cause the first subprocess.Popen() time to properly close the inheritable handles before the second subprocess.Popen() call is made in the other thread, thus the inheritable handles are leaked. -- Added file: http://bugs.python.org/file40220/pipe.py ___ Python tracker <http://bugs.python.org/issue24909> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24909] Windows: subprocess.Popen: race condition for leaking inheritable handles
New submission from Adam Meily: ** This issue and attached patch only affect Windows ** Currently, the Popen constructor will duplicate any stdout, stdin, and/or stderr handle passed in and make them inheritable, by calling DuplicateHandle. If two threads call Popen at the same time, the newly created inheritable handles will leak into the subprocess that's running being created in the opposite thread. This has consequences when two or more subprocesses are piped together and executed at the time time. Example === A pipe is created using the os.pipe() function. Then, two threads are started, T1 and T2. T1 calls Popen, setting stdout to the write-end of the pipe. T2 calls Popen, setting stdin to the read-end of the pipe. Stock CPython would make T1.stdout and T2.stdin inheritable. T1's Popen will create a subprocess, which correctly inherits T1.stdout, but it will also inherit the T2.stdin handle. Likewise, T2's Popen will create a subprocess, which correctly inherits T2.stdin, but it will also inherit the T1.stdout handle. Thus, in this situation where both ends of the pipe were accidentally inherited, the pipe will not properly close and T2's subprocess will have to be forcefully killed (assuming it is reading from stdin). The patch simply enforces that the lifetime of an inheritable handle is limited to a single call to Popen. Why this should be in CPython = I believe this change should be in CPython. Tracking down this bug into the Python core took a substantial amount of time and work. Can I do this in my Python application and achieve the same effect? Absolutely, but I would argue that I shouldn't have to. The fix in CPython is very minimal: - It won't break existing code. Even if my project already has a lock around calls to Popen, my code will continue to work. Sure, it's a duplicate lock, but nothing will break or deadlock. - The performance impact is negligible. The lock is only instantiated for Windows platforms and acquiring/releasing a lock appears to have very low overhead. - Because the patch is very small, it can be easily backported to other Python versions. As far as I can tell, Python 3.5, 3.4, 3.3, and 2.7 will work with the attached patch. The one difference between the Python versions is that subprocess.mswindows was renamed to subprocess._mswindows at some point, which is very minor. - Python 3.4's new OS-specific inheritable functions [os.get_inheritable(), os.set_inheritable(), etc] will continue to work as expected. So, nothing is broken or changed there. Old code won't break, new code will benefit immediately, and existing code will work better. Similar Issues == - #19809: subprocess should warn uses on race conditions when multiple threads spawn child processes - #2320: Race condition in subprocess using stdin - #12739: read stuck with multithreading and simultaneous subprocess.Popen -- components: Library (Lib) files: win32-popen-lock.patch keywords: patch messages: 248961 nosy: Adam Meily priority: normal severity: normal status: open title: Windows: subprocess.Popen: race condition for leaking inheritable handles type: behavior versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file40219/win32-popen-lock.patch ___ Python tracker <http://bugs.python.org/issue24909> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
