Re: Implementing "tee" in python asyncio
On Thu, Jul 27, 2023 at 12:49:10PM -0400, John Snow wrote: > On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé > wrote: > > > On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote: > > > Hi folks, > > > > > > I'm currently wondering how to take a StreamReader as found on > > > > > https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process > > > and to consume the data while optionally re-streaming it to a > > > secondary consumer. > > > > > > What I'd like to do is create a StreamWatcher class that consumes > > > console data while optionally logging to python logging and/or a file; > > > but re-buffers the data into an async stream where an additional > > > consumer is free to use the "standard asyncio API" to consume console > > > data at their leisure in a way that's unsurprising. > > > > > > What I'd like this *for* is to be able to do aggressive logging of > > > stdout/stderr and console data without denying tests the ability to > > > consume the data as they see fit for their testing purposes. I want to > > > have my cake and eat it too, and we don't do a good job of managing > > > this consistently across the board. > > > > > > I am wondering if there's any way around creating a literal socketpair > > > and suffering the creation of a full four StreamReader/StreamWriter > > > instances (one pair per socket...) and then just hanging on to the > > > "unused" reader/writer per each. It seems kind of foolishly excessive. > > > It also seems like it might be a pain in the butt if I want > > > cross-platform compatibility with windows for the machine appliance. > > > > > > Anyone got any bright ideas? > > > > Don't bother with any of the above, just add 'logfile=/path/to/log' > > to the -chardev argument. > > > > Part of me feels like it'd be a shame to miss out on the chance to do the > logging in the library, but it *would* be a lot easier to just not worry > about it. > > anyway, this is easy for console, what about stdio? is there a way to run > that through a chardev that's just connected to stdio but we attach a > logfile? When you say 'stdio' are you refering to a '-chardev stdio' backend, or simply whatever the python code dup()s onto stdio when spawnnig QEMU ? The former case can use ',logfile=...', and in the latter case we could open a log file and dup() its FD onto stdout/stderr when spawning QEMU. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: Implementing "tee" in python asyncio
On Thu, Jul 27, 2023, 12:53 PM Daniel P. Berrangé wrote: > On Thu, Jul 27, 2023 at 12:49:10PM -0400, John Snow wrote: > > On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé > > wrote: > > > > > On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote: > > > > Hi folks, > > > > > > > > I'm currently wondering how to take a StreamReader as found on > > > > > > > > https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process > > > > and to consume the data while optionally re-streaming it to a > > > > secondary consumer. > > > > > > > > What I'd like to do is create a StreamWatcher class that consumes > > > > console data while optionally logging to python logging and/or a > file; > > > > but re-buffers the data into an async stream where an additional > > > > consumer is free to use the "standard asyncio API" to consume console > > > > data at their leisure in a way that's unsurprising. > > > > > > > > What I'd like this *for* is to be able to do aggressive logging of > > > > stdout/stderr and console data without denying tests the ability to > > > > consume the data as they see fit for their testing purposes. I want > to > > > > have my cake and eat it too, and we don't do a good job of managing > > > > this consistently across the board. > > > > > > > > I am wondering if there's any way around creating a literal > socketpair > > > > and suffering the creation of a full four StreamReader/StreamWriter > > > > instances (one pair per socket...) and then just hanging on to the > > > > "unused" reader/writer per each. It seems kind of foolishly > excessive. > > > > It also seems like it might be a pain in the butt if I want > > > > cross-platform compatibility with windows for the machine appliance. > > > > > > > > Anyone got any bright ideas? > > > > > > Don't bother with any of the above, just add 'logfile=/path/to/log' > > > to the -chardev argument. > > > > > > > Part of me feels like it'd be a shame to miss out on the chance to do the > > logging in the library, but it *would* be a lot easier to just not worry > > about it. > > > > anyway, this is easy for console, what about stdio? is there a way to run > > that through a chardev that's just connected to stdio but we attach a > > logfile? > > When you say 'stdio' are you refering to a '-chardev stdio' backend, or > simply whatever the python code dup()s onto stdio when spawnnig QEMU ? > The former case can use ',logfile=...', and in the latter case we could > open a log file and dup() its FD onto stdout/stderr when spawning QEMU. > I mean stdout and stderr for the QEMU process itself. Some tests like to watch the output for various strings, but we also need to capture and log that output to be able to show it during unhandled exceptions, etc. > > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org-o- > https://www.instagram.com/dberrange :| > >
Re: Implementing "tee" in python asyncio
On Thu, Jul 27, 2023, 6:47 AM Daniel P. Berrangé wrote: > On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote: > > Hi folks, > > > > I'm currently wondering how to take a StreamReader as found on > > > https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process > > and to consume the data while optionally re-streaming it to a > > secondary consumer. > > > > What I'd like to do is create a StreamWatcher class that consumes > > console data while optionally logging to python logging and/or a file; > > but re-buffers the data into an async stream where an additional > > consumer is free to use the "standard asyncio API" to consume console > > data at their leisure in a way that's unsurprising. > > > > What I'd like this *for* is to be able to do aggressive logging of > > stdout/stderr and console data without denying tests the ability to > > consume the data as they see fit for their testing purposes. I want to > > have my cake and eat it too, and we don't do a good job of managing > > this consistently across the board. > > > > I am wondering if there's any way around creating a literal socketpair > > and suffering the creation of a full four StreamReader/StreamWriter > > instances (one pair per socket...) and then just hanging on to the > > "unused" reader/writer per each. It seems kind of foolishly excessive. > > It also seems like it might be a pain in the butt if I want > > cross-platform compatibility with windows for the machine appliance. > > > > Anyone got any bright ideas? > > Don't bother with any of the above, just add 'logfile=/path/to/log' > to the -chardev argument. > Part of me feels like it'd be a shame to miss out on the chance to do the logging in the library, but it *would* be a lot easier to just not worry about it. > anyway, this is easy for console, what about stdio? is there a way to run that through a chardev that's just connected to stdio but we attach a logfile? > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org-o- > https://www.instagram.com/dberrange :| > >
Re: Implementing "tee" in python asyncio
On Wed, Jul 26, 2023 at 04:25:34PM -0400, John Snow wrote: > Hi folks, > > I'm currently wondering how to take a StreamReader as found on > https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process > and to consume the data while optionally re-streaming it to a > secondary consumer. > > What I'd like to do is create a StreamWatcher class that consumes > console data while optionally logging to python logging and/or a file; > but re-buffers the data into an async stream where an additional > consumer is free to use the "standard asyncio API" to consume console > data at their leisure in a way that's unsurprising. > > What I'd like this *for* is to be able to do aggressive logging of > stdout/stderr and console data without denying tests the ability to > consume the data as they see fit for their testing purposes. I want to > have my cake and eat it too, and we don't do a good job of managing > this consistently across the board. > > I am wondering if there's any way around creating a literal socketpair > and suffering the creation of a full four StreamReader/StreamWriter > instances (one pair per socket...) and then just hanging on to the > "unused" reader/writer per each. It seems kind of foolishly excessive. > It also seems like it might be a pain in the butt if I want > cross-platform compatibility with windows for the machine appliance. > > Anyone got any bright ideas? Don't bother with any of the above, just add 'logfile=/path/to/log' to the -chardev argument. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Implementing "tee" in python asyncio
Hi folks, I'm currently wondering how to take a StreamReader as found on https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process and to consume the data while optionally re-streaming it to a secondary consumer. What I'd like to do is create a StreamWatcher class that consumes console data while optionally logging to python logging and/or a file; but re-buffers the data into an async stream where an additional consumer is free to use the "standard asyncio API" to consume console data at their leisure in a way that's unsurprising. What I'd like this *for* is to be able to do aggressive logging of stdout/stderr and console data without denying tests the ability to consume the data as they see fit for their testing purposes. I want to have my cake and eat it too, and we don't do a good job of managing this consistently across the board. I am wondering if there's any way around creating a literal socketpair and suffering the creation of a full four StreamReader/StreamWriter instances (one pair per socket...) and then just hanging on to the "unused" reader/writer per each. It seems kind of foolishly excessive. It also seems like it might be a pain in the butt if I want cross-platform compatibility with windows for the machine appliance. Anyone got any bright ideas? --js
