Re: Passing File Descriptors To Subprocesses

[Terry Reedy]

On 5/11/2018 11:23 AM, pevo...@gmail.com wrote:

I encountered the same issue with Python 3.4 on CentOS 7 when using only the 
close_fds argument. Since I am passing a lot of dynamically obtained file 
descriptors, using the pass_fds argument is impossible for my case. Setting 
close_fds to False *but* also explicitly making the fds inheritable upon 
generation solved the issue. I would be really grateful if there was at least a 
note in the subprocess module explaining that one might need to explicitly make 
file descriptors inheritable since if it wasn't this post I would have lost 
even more time than I already did.


Make specific proposal on the tracker bugs.python.org.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Passing File Descriptors To Subprocesses

[pevogam]

I encountered the same issue with Python 3.4 on CentOS 7 when using only the 
close_fds argument. Since I am passing a lot of dynamically obtained file 
descriptors, using the pass_fds argument is impossible for my case. Setting 
close_fds to False *but* also explicitly making the fds inheritable upon 
generation solved the issue. I would be really grateful if there was at least a 
note in the subprocess module explaining that one might need to explicitly make 
file descriptors inheritable since if it wasn't this post I would have lost 
even more time than I already did.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Passing File Descriptors To Subprocesses

[Lawrence D’Oliveiro]

On Sunday, July 17, 2016 at 3:01:36 PM UTC+12, eryk sun wrote:

> It works correctly in 3.4+ ...

Yup, confirmed it happens only in 3.3.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Passing File Descriptors To Subprocesses

[eryk sun]

On Sun, Jul 17, 2016 at 1:59 AM, Lawrence D’Oliveiro
 wrote:
> The various subprocess functions 
>  have
> arguments called “close_fds” and “pass_fds”, which specify which file 
> descriptors are to be
> left open in the child process. Yet no matter what I set these to, it seemed 
> I could not pass
> my pipes to a subprocess.
>
> What the docs *don’t* tell you is that these arguments do not control what 
> happens after the
> exec. The file descriptors that are kept open are only those which do not 
> have the
> FD_CLOEXEC flags set in their fcntl settings.

It works correctly in 3.4+, which makes the pass_fds file descriptors
inheritable in the child, after fork. See issue 18571 and PEP 446,
section "Other Changes":

http://bugs.python.org/issue18571
https://www.python.org/dev/peps/pep-0446/#other-changes

For example:

Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
[GCC 5.3.1 20160330] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, subprocess
>>> fdr, fdw = os.pipe()
>>> fdw
4
>>> os.get_inheritable(fdw)
False
>>> subprocess.call(['python3'], pass_fds=[fdw])

child:

Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
[GCC 5.3.1 20160330] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.write(4, b'spam')
4
>>> exit()

parent:

0
>>> os.read(fdr, 4)
b'spam'
-- 
https://mail.python.org/mailman/listinfo/python-list

Passing File Descriptors To Subprocesses

[Lawrence D’Oliveiro]

A few years ago I wrote a tool  to do 
comparisons between different versions of the Android API. Then one day, after 
a Python update (from the dates, it had to have been some version of 3.2), it 
stopped working. It took quite a while--over a year--until I figured out what 
was happening, and I could move my code from Python 2 back to Python 3 again.

The various subprocess functions 
 have arguments called 
“close_fds” and “pass_fds”, which specify which file descriptors are to be left 
open in the child process. Yet no matter what I set these to, it seemed I could 
not pass my pipes to a subprocess.

What the docs *don’t* tell you is that these arguments do not control what 
happens after the exec. The file descriptors that are kept open are only those 
which do not have the FD_CLOEXEC flags set in their fcntl settings.

Remember how the subprocess functions work:
* first, a fork(2) call is executed, then
* the child process does execve(2) (or some convenience variant of this) to 
actually execute the command or program that you specified.

The “close_fds” and “pass_fds” args only matter in the first step. You need to 
specify these, *and* have the right CLOEXEC settings on those file descriptors 
for the second step. Leave out either one, and your child process does not get 
the file descriptors.

Python 3.4 has added special calls 
 to 
manage this CLOEXEC setting. My code uses fcntl 
.

The docs for the subprocess module need to make this requirement clear.
-- 
https://mail.python.org/mailman/listinfo/python-list