John Nagle wrote:
> I have working code from Python 2 which uses "pickle"
> to talk to a subprocess via stdin/stdio. I'm trying to
> make that work in Python 3.
>
> First, the subprocess Python is invoked with the "-d' option,
> so stdin and stdio are supposed to be unbuffered binary streams.
> That was enough in Python 2, but it's not enough in Python 3.
>
> The subprocess and its connections are set up with
>
> proc = subprocess.Popen(launchargs,stdin=subprocess.PIPE,
> stdout=subprocess.PIPE, env=env)
>
> ...
> self.reader = pickle.Unpickler(self.proc.stdout)
> self.writer = pickle.Pickler(self.proc.stdin, 2)
>
> after which I get
>
> result = self.reader.load()
> TypeError: 'str' does not support the buffer interface
>
> That's as far as traceback goes, so I assume this is
> disappearing into C code.
>
> OK, I know I need a byte stream. I tried
>
> self.reader = pickle.Unpickler(self.proc.stdout.buffer)
> self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)
>
> That's not allowed. The "stdin" and "stdout" that are
> fields of "proc" do not have "buffer". So I can't do that
> in the parent process. In the child, though, where
> stdin and stdout come from "sys", "sys.stdin.buffer" is valid.
> That fixes the ""str" does not support the buffer interface
> error." But now I get the pickle error "Ran out of input"
> on the process child side. Probably because there's a
> str/bytes incompatibility somewhere.
>
> So how do I get clean binary byte streams between parent
> and child process?
I don't know what you have to do to rule out deadlocks when you use pipes
for both stdin and stdout, but binary streams are the default for
subprocess. Can you provide a complete example?
Anyway, here is a demo for two-way communication using the communicate()
method:
$ cat parent.py
import pickle
import subprocess
data = (5, 4.3, "üblich ähnlich nötig")
p = subprocess.Popen(
["python3", "child.py"],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
result = p.communicate(pickle.dumps(data, protocol=2))[0]
print(pickle.loads(result))
$ cat child.py
import sys
import pickle
a, b, c = pickle.load(sys.stdin.buffer)
pickle.dump((a, b, c.upper()), sys.stdout.buffer)
$ python3 parent.py
(5, 4.3, 'ÜBLICH ÄHNLICH NÖTIG')
This is likely not what you want because here everything is buffered so that
continuous interaction is not possible.
--
https://mail.python.org/mailman/listinfo/python-list
