I asked:
I am developing a Python program that submits a command to each node of a cluster and consumes the stdout and stderr from each. I want all the processes to run in parallel, so I start a thread for each node. There could be a lot of output from a node, so I have a thread reading each stream, for a total of three threads per node. (I could probably reduce to two threads per node by having the process thread handle stdout or stderr.)
Simon Wittber said:
> In the past, I have used the select module to manage asynchronous > IO operations. > > I pass the select.select function a list of file-like objects, and it > returns a list of file-like objects which are ready for reading and > writing.
Donn Cave said:
As I see another followup has already mentioned, the classic "pre threads" solution to multiple I/O sources is the select(2) function, ...
Thanks for your replies. The streams that I need to read contain pickled data. The select call returns files that have available input, and I can use read(file_descriptor, max) to read some of the input data. But then how can I convert the bytes just read into a stream for unpickling? I somehow need to take the bytes arriving for a given file descriptor and buffer them until the unpickler has enough data to return a complete unpickled object.
(It would be nice to do this without copying the bytes from one place to another, but I don't even see how do solve the problem with copying.)
Jack
-- http://mail.python.org/mailman/listinfo/python-list
