Alexey Izbyshev added the comment:
Would it make sense to use os.confstr('CS_PATH') instead of a hardcoded path,
or is identical behavior on all POSIX platforms preferred to that?
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.
Alexey Izbyshev added the comment:
> Hi,
> As a disclaimer, I'm a FreeBSD developer interested in making sure we're
> doing the right thing here. =)
> May I ask what the above assessment is based on, and specifically what we
> need to address?
Hello, Kyle! That assessment
Alexey Izbyshev added the comment:
Thank you for the answers, Kyle!
> I'll be preparing a patch for our posix_spawn's signal handling.
Great!
> My mistake in my setuid assessment was pointed out to me- it doesn't seem
> like a highly likely attack vector, but it is indeed
Alexey Izbyshev added the comment:
> It should be compared to the current code. Currently, _posixsubprocess uses a
> loop calling execv(). I don't think that calling posix_spawn() in a loop
> until one doesn't fail is more inefficient.
> The worst case would be when appl
Alexey Izbyshev added the comment:
Thanks for the info on CS_PATH, Victor. IMHO it'd make sense to use the
libc-provided default PATH at least in shutil.which() since its intent is to
emulate "which" from the default shell.
--
___
Pyth
Alexey Izbyshev added the comment:
>> * pass_fds: there is not API to mark a fd as inheritable (clear O_CLOEXEC
>> flag)
> POSIX has a bug for this [5]. It's marked fixed, but the current POSIX docs
> doesn't reflect the changes. The idea is to make
> posix_spawn
Alexey Izbyshev added the comment:
> Until muscl decides to provide an "#ifdef __MUSL__"-like or any way that it's
> musl, I propose to not support musl: don't use os.posix_spawn() but
> _posixsubprocess.
FYI, I'm researching how to use vfork(), focusing on Linux, but I
Alexey Izbyshev added the comment:
Another problem with posix_spawn() on glibc: it doesn't report errors to the
parent process when run under QEMU user-space emulation and Windows Subsystem
for Linux. This is because starting with commit [1] (glibc 2.25) posix_spawn()
relies on address
Alexey Izbyshev added the comment:
Serhiy, PyOS_* functions are called only if preexec_fn != None. But it will
never be possible to implement support for preexec_fn (and some other
subprocess features, e.g. close_fds) if processes are run via posix_spawn, so I
don't see why anything should
Alexey Izbyshev added the comment:
Victor and Joannah, thanks for working on adding vfork() support to subprocess.
Regarding speedups in the real world, I can share a personal anecdote. Back at
the time when AOSP was built with make (I think it was AOSP 5) I've observed
~2x slowdown
Alexey Izbyshev added the comment:
> I'm open to experiment to use vfork() in _posixsubprocess
Are you going to do experiments? If not, I can try to do some in early January.
> Using vfork() can cause new issues: that's why there is a
> POSIX_SPAWN_USE_VFORK flag (the c
Alexey Izbyshev added the comment:
The resolution of this [1] glibc bug report effectively says that the use of
global variables tzname, timezone and daylight is not supported by glibc unless
a POSIX-style TZ setting is used (which is probably never in real world).
[1] https
Alexey Izbyshev added the comment:
> * cwd
posix_spawn_file_actions_addchdir_np() is scheduled for glibc 2.29 [1] and
exists in Solaris [2] (though its suffix indicates that it's "non-portable" --
not in POSIX). POSIX also has a bug for this [7].
> * s
Alexey Izbyshev added the comment:
You might try to check the list of DLLs loaded into the stuck python process
and find third-party ones (e.g., antivirus). If there are any, disable the
third-party software and try again.
--
nosy: +izbyshev
Alexey Izbyshev added the comment:
If I understood PR 10919 correctly, sysconfig.get_config_var('userbase') can
now return unexpanded paths containing '~'. Is it intended despite the previous
discussion starting with msg135047?
--
nosy: +izbyshev
Alexey Izbyshev added the comment:
> Would it make sense to backport this fix in 3.6 and 3.7?
I'd like to see it there, given that this bug surfaced in many use cases not
involving any modern features or systemd at all.
--
___
Python trac
Alexey Izbyshev added the comment:
> I prefer to stick to the initial bug report which hasn't been fixed in 8 years
I'm interested in fixing this bug too since it bit me in SCons and I had to use
a local patch for it. I welcome the upstream fix and don't object to PR 10919.
Alexey Izbyshev added the comment:
How is it possible to use faulthandler if the interpreter hasn't even started
yet?
--
___
Python tracker
<https://bugs.python.org/issue35
Alexey Izbyshev added the comment:
See #28108 and https://sourceware.org/bugzilla/show_bug.cgi?id=23859 (for
msg276123).
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue35
Alexey Izbyshev added the comment:
argparse.SUPPRESS is an opaque value to be used by argparse clients. It could
be anything, it just happens to be a string. So the code doesn't compare
strings but checks whether a supplied object *is* the opaque value. I do not
see any problem
Alexey Izbyshev added the comment:
The added test exposed a leak in unicode_encode_locale(). See msg330534.
--
___
Python tracker
<https://bugs.python.org/issue34
Alexey Izbyshev added the comment:
Correction: the fall-through in "else if (res == -3)" clause doesn't cause a
memory leak, but still results in usage of uninitialized 'str'.
--
___
Python tracker
<https://bugs.python.o
Alexey Izbyshev added the comment:
This is because of a leak of 'wstr' at
https://github.com/python/cpython/blob/1005c84535191a72ebb7587d8c5636a065b7ed79/Objects/unicodeobject.c#L3476
.
There is another leak and usage of uninitialized 'str' because the following
"else if" clau
Change by Alexey Izbyshev :
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue35674>
___
___
Python-bugs-list mailing list
Unsubscribe:
Alexey Izbyshev added the comment:
> * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn()
> "file actions" fails, posix_spawn() succeed but the child process exits
> immediately with exit code 127 without trying to call execv(). If
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +8656
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue34649>
___
___
Py
New submission from Alexey Izbyshev :
The return value of PyUnicode_InternFromString() is not checked for NULL and
then dereferenced at three places in
https://github.com/python/cpython/blob/0bd1a2dcfdf36b181385ae61361e7692f4ebb0fd/Modules/_json.c#L1369
Reported by Svace static analyzer
Alexey Izbyshev added the comment:
What are the drawbacks of moving all code from __init__ to __new__, at least in
master branch (3.8)? IMO it's much more robust than playing whack-a-mole both
with ref/memleaks caused by repeated calls to __init__ and absence of
initialization checks in all
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +8754
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue34695>
___
___
Py
New submission from Alexey Izbyshev :
It is possible to manually create an instance of private CField type which is
used by ctypes to represent fields of Structure and Union types. This instance
will be uninitialized because it's normally initialized when instances of
Structure/Union
Alexey Izbyshev added the comment:
Some additional motivation for removing Cache: it may be used to crash the
interpreter (#31734).
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue30
New submission from Alexey Izbyshev :
sqlite3.Cache allows users to create uninitialized instances because it relies
on __init__() instead of __new__() for initialization, which can be skipped.
Cache.get() crashes if called on an uninitialized instance:
>>> from sqlite3 imp
Alexey Izbyshev added the comment:
bz2 in 2.7 is also affected.
Victor, do we want to fix the crash at all in stable branches? If yes, IMHO
taking the slight risk of __init__ -> __new__ change is preferable to taking
the trouble to implement the alternative backwards-compatible fix (i
Alexey Izbyshev added the comment:
Reclosing (browser cache problem).
--
resolution: -> duplicate
stage: needs patch -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
New submission from Alexey Izbyshev :
The compressor/decompressor classes from bz2 and lzma modules rely on
__init__() for initialization, but it is not guaranteed to be called. Method
calls on an uninitialized object crash:
>>> from bz2 import BZ2Compressor as C
>>&g
Alexey Izbyshev added the comment:
I somehow failed to notice #23224 when I searched the tracker. You're right,
it's the same, and, moreover, PR 7822 fixes problem with both compressors and
decompressors (though it includes tests only for the latter for some reason).
I think
Change by Alexey Izbyshev :
--
nosy: +gregory.p.smith, izbyshev
___
Python tracker
<https://bugs.python.org/issue35537>
___
___
Python-bugs-list mailin
New submission from Alexey Izbyshev :
This issue is to propose a (complementary) alternative to the usage of
posix_spawn() in subprocess (see bpo-35537).
As mentioned by Victor Stinner in msg332236, posix_spawn() has the potential of
being faster and safer than fork()/exec() approach
Change by Alexey Izbyshev :
--
keywords: +patch, patch
pull_requests: +11484, 11485
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issu
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +11484
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue35823>
___
_
Change by Alexey Izbyshev :
--
keywords: +patch, patch, patch
pull_requests: +11484, 11485, 11486
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issu
Alexey Izbyshev added the comment:
> Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's
> equal, is it possible to explicitly exclude WSL in the subprocess test,
> _use_posix_spawn()?
I don't have immediate access to WSL right now, but I'll try t
Alexey Izbyshev added the comment:
I've checked subprocess.Popen() error reporting in QEMU user-mode and WSL and
confirm that it works both with my patch (vfork/exec) and the traditional
fork/exec, but doesn't work with glibc's posix_spawn.
The first command below uses posix_spawn
Change by Alexey Izbyshev :
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue31904>
___
___
Python-bugs-list mailing list
Unsubscribe:
Alexey Izbyshev added the comment:
> Interesting. Because both errors/conditions are mapped to
> ERROR_INVALID_HANDLE we need the creation time. I can work on a patch for
> that.
I don't understand why any patch for CPython is needed at all. Using invalid
handles is a serious pr
Alexey Izbyshev added the comment:
Patrick, could you provide more background that would explain your choice of
setreuid/setregid functions and the desired handling of supplementary groups?
I'm not a security expert, so I may not have sufficient expertise to judge on
that, but maybe my
Change by Alexey Izbyshev :
--
assignee: -> gregory.p.smith
___
Python tracker
<https://bugs.python.org/issue36046>
___
___
Python-bugs-list mailing list
Un
Alexey Izbyshev added the comment:
> 1) This is intentional, this is for dropping privileges before running some
> (possibly untrusted) command, we do not want to leave a path for the
> subprocess to gain root back. If there is a subprocess that needs root for
> some operatio
Change by Alexey Izbyshev :
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue36046>
___
___
Python-bugs-list mailing list
Unsubscribe:
Alexey Izbyshev added the comment:
I'll look into it later today. An obvious guess is that my test simply exposed
an existing leak because the exception code path wasn't tested before AFAIK,
but I need to check it.
--
assignee: -> izbys
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +11876
stage: test needed -> patch review
___
Python tracker
<https://bugs.python.org/issu
Alexey Izbyshev added the comment:
Thank you for your introduction about _xxsubinterpreters, Eric.
This particular leak is easy: it's right in _channel_send(). I've submitted a
PR.
I've also done a quick scan of neighboring code, and it seems there are other
leaks as well, e.g
New submission from Alexey Izbyshev :
Victor Stinner pointed out that on x86 Gentoo Installed with X 3.x buildbot,
there is a compiler warning:
Python/pystate.c:1483:18: warning: cast to pointer from integer of
different size [-Wint-to-pointer-cast]
(https://buildbot.python.org/all
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +11852
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue35972>
___
_
Alexey Izbyshev added the comment:
"long long" is mandated to be at least 64-bit by C99 (5.2.4.2.1 Sizes of
integer types). If it were 32-bit, no warnings would have been issued.
--
___
Python tracker
<https://bugs.python.o
Alexey Izbyshev added the comment:
I don't know what you mean by "in-line" pre-processing output, but you can use
-E option to get the normal preprocessor output. Line directives will tell you
where those functions come from on a system where there is no compilation error.
Alexey Izbyshev added the comment:
Thank you for the review and your thoughts, Gregory.
> With this in place we may want to make the _use_posix_spawn() logic in
> subprocess.py stricter? That could be its own followup PR.
Yes, I think we can always use vfork() on Linux unless we fin
Alexey Izbyshev added the comment:
> W.r.t. closing all file descriptors > 2: posix_spawn_file_actions_addclose
> can do this when using posix_spawn. That would have a performance cost, you'd
> basically have to resort to closing all possible file descriptors and cannot
>
Alexey Izbyshev added the comment:
I've been struggling with fixing spurious -Wclobbered GCC warnings. Originally,
I've got the following:
/scratch2/izbyshev/cpython/Modules/_posixsubprocess.c: In function
‘subprocess_fork_exec’:
/scratch2/izbyshev/cpython/Modules/_posixsubprocess.c:612:15
Change by Alexey Izbyshev :
--
pull_requests: +16379
pull_request: https://github.com/python/cpython/pull/5812
___
Python tracker
<https://bugs.python.org/issue34
Alexey Izbyshev added the comment:
Well, much later than promised, but I'm picking it up. Since in the meantime
support for setting uid/gid/groups was merged, and I'm aware about potential
issues with calling corresponding C library functions in a vfork()-child, I
asked a question on musl
Change by Alexey Izbyshev :
--
resolution: -> not a bug
___
Python tracker
<https://bugs.python.org/issue36034>
___
___
Python-bugs-list mailing list
Un
Alexey Izbyshev added the comment:
I've updated my PR.
* After a discussion with Alexander Monakov (a GCC developer), moved vfork()
into a small function to isolate it from both subprocess_fork_exec() and
child_exec(). This appears to be the best strategy to avoid -Wclobbered false
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +21882
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/22966
___
Python tracker
<https://bugs.python.org/issu
Change by Alexey Izbyshev :
--
pull_requests: +21885
pull_request: https://github.com/python/cpython/pull/22970
___
Python tracker
<https://bugs.python.org/issue42
Change by Alexey Izbyshev :
--
type: behavior -> resource usage
___
Python tracker
<https://bugs.python.org/issue42146>
___
___
Python-bugs-list mai
Change by Alexey Izbyshev :
--
keywords: +patch
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue42146>
___
___
Python-bugs-list mai
New submission from Alexey Izbyshev :
The following test demonstrates the leak:
```
import subprocess
cwd = 'x' * 10**6
for __ in range(100):
try:
subprocess.call(['/xxx'], cwd=cwd, user=2**64)
except OverflowError:
pass
from resource import *
print(getrusage
Alexey Izbyshev added the comment:
I've submitted both PRs.
Regarding PR 22970:
* I made it a draft since we'd probably want to fix the leak first, but then it
will have to be rebased.
* It fixes a bug with _enable_gc(): if it failed after fork(), we'd raise
OSError instead. Additionally
Alexey Izbyshev added the comment:
Thanks for merging! I've rebased PR 22970.
--
___
Python tracker
<https://bugs.python.org/issue42146>
___
___
Python-bug
Change by Alexey Izbyshev :
Added file: https://bugs.python.org/file49531/test.py
___
Python tracker
<https://bugs.python.org/issue42097>
___
___
Python-bugs-list mailin
Alexey Izbyshev added the comment:
(Restored test.py attachment)
The issue happens due to an incorrect usage of `multiprocessing.Pool`.
```
# Set up multiprocessing pool, initialising logging in each subprocess
with multiprocessing.Pool(initializer=process_setup,
initargs=(get_queue
Alexey Izbyshev added the comment:
By the way, I don't see a direct relation between `test.py` (which doesn't use
`subprocess` directly) and your comment describing `subprocess` usage with
threads. So if you think that the bug in `test.py` is unrelated to the problem
you face, feel free
Change by Alexey Izbyshev :
--
pull_requests: +21862
pull_request: https://github.com/python/cpython/pull/22944
___
Python tracker
<https://bugs.python.org/issue35
Alexey Izbyshev added the comment:
> regarding excluding the setsid() case: I was being conservative as I couldn't
> find a reference of what was and wasn't allowed after vfork.
Yes, there is no list of functions allowed after vfork(), except for the
conservative POSIX.1 list cons
Alexey Izbyshev added the comment:
> Thank you for taking this on! I'm calling it fixed for now as the buildbots
> are looking happy with it. If issues with it arise we can address them.
Thank you for reviewing and merging!
Using POSIX_CALL for pthread_sigmask() is incorrect, h
Alexey Izbyshev added the comment:
@ronaldoussoren
> I'd prefer to not use vfork on macOS. For one I don't particularly trust that
> vfork would work reliably when using higher level APIs, but more importantly
> posix_spawn on macOS has some options that are hard to achieve
Alexey Izbyshev added the comment:
I'd really like to get this merged eventually because vfork()-based solution is
fundamentally more generic than posix_spawn(). Apart from having no issue with
close_fds=True, it will also continue to allow subprocess to add any process
context tweaks
Alexey Izbyshev added the comment:
I've encountered this issue too. My use case was a 32-bit Python on a 64-bit
CentOS system, and my understanding of the issue was that 64-bit libgcc_s is
somehow counted as a "provider" of libgcc_s for 32-bit libc by the package
manager, so 32-bi
Alexey Izbyshev added the comment:
> Using close_fds=False, subprocess can use posix_spawn() which is safer and
> faster than fork+exec. For example, on Linux, the glibc implements it as a
> function using vfork which is faster than fork if the parent allocated a lot
> of memo
Change by Alexey Izbyshev :
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue38435>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Alexey Izbyshev :
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue42736>
___
___
Python-bugs-list mailing list
Unsubscribe:
Alexey Izbyshev added the comment:
> So it should be, "if they fail and you're in a context where exceptions are
> allowed, raise an exception" (which will chain back to the one raised from an
> audit hook".
What exception should be raised if _Py_fopen() fails (retur
Change by Alexey Izbyshev :
--
components: +Library (Lib)
nosy: +davin, pitrou
versions: -Python 3.6
___
Python tracker
<https://bugs.python.org/issue42
Alexey Izbyshev added the comment:
> It seems like PyErr_ProgramText() is no longer used in Python.
Isn't it a part of the public API? I can't find it in the docs, but it seems to
be declared in the public header.
--
___
Python tracker
<
Alexey Izbyshev added the comment:
Thanks, Eryk, for catching the dup, I missed it somehow.
@ZackerySpytz: do you plan to proceed with your PR? If not, I can pick it up --
this issue broke the software I develop after upgrade to 3.8.
I filed issue 42569 to hopefully clarify the status
Alexey Izbyshev added the comment:
> To implement PEP 446: create non-inheritable file descriptors.
Yes, I understand that was the original role. But currently there is no easy
way to deal with errors from the helpers because of exception vs. errno
conundrum. Maybe they should be sp
Alexey Izbyshev added the comment:
Thanks for the patch, Victor, it looks good.
Just so it doesn't get lost: the problem with the contract of
PyErr_ProgramText() which I mentioned in my dup 42568 is still there.
--
___
Python tracker
<ht
Change by Alexey Izbyshev :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
versions: -Python 3.7
___
Python tracker
<https://bugs.python.or
Alexey Izbyshev added the comment:
Thanks for the fix and backports!
--
resolution: fixed ->
stage: resolved -> patch review
status: closed -> open
versions: +Python 3.7
___
Python tracker
<https://bugs.python.or
Alexey Izbyshev added the comment:
Yes, despite that MSVCRT knows the type of the file descriptor because it calls
GetFileType() on its creation, it doesn't check it in lseek() implementation
and simply calls SetFilePointer(), which spuriously succeeds for pipes. MSDN
says the following[1
Alexey Izbyshev added the comment:
Great approach :)
--
___
Python tracker
<https://bugs.python.org/issue42569>
___
___
Python-bugs-list mailing list
Unsub
New submission from Alexey Izbyshev :
On POSIX-conforming systems, O_APPEND flag for open() must ensure that no
intervening file modification occurs between changing the file offset and the
write operation[1]. In effect, two processes that independently opened the same
file with O_APPEND
Change by Alexey Izbyshev :
--
keywords: +patch
pull_requests: +22575
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/23712
___
Python tracker
<https://bugs.python.org/issu
Change by Alexey Izbyshev :
--
nosy: +rhettinger
versions: -Python 3.6, Python 3.7
___
Python tracker
<https://bugs.python.org/issue42457>
___
___
Python-bug
New submission from Alexey Izbyshev :
> python тест.pyc
python: Can't reopen .pyc file
The issue is caused by _Py_fopen() being used as though it can deal with paths
encoded in FS-default encoding (UTF-8 by default on Windows), but in fact it's
just a simple wrapper around fopen() from th
New submission from Alexey Izbyshev :
Before addition of audit hooks in 3.8, _Py_fopen() and _Py_wfopen() were simple
wrappers around corresponding C runtime functions. They didn't require GIL,
reported errors via errno and could be safely called during early interpreter
initialization
Alexey Izbyshev added the comment:
> I've been struggling to understand today why a simple file redirection
> couldn't work properly today (encoding issues)
The core issue is that "working properly" is not defined in general when we're
talking about piping/redirec
Alexey Izbyshev added the comment:
This bug would have been caught at compile time if `_Py_Gid_Converter()` used
`gid_t *` instead of `void *`. I couldn't find any call sites where `void *`
would be needed, so probably `_Py_Gid_Converter()` should be fixed too (in a
separate PR/issue
Alexey Izbyshev added the comment:
> (probably can't even limit that to the case when `text` is used, since it was
> added in 3.7)
Well, actually, we can, since we probably don't need to preserve compatibility
with the AttributeError currently caused by `text=True` with `inpu
201 - 300 of 331 matches
Mail list logo
