Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
Thanks to all for the thorough replies!
Of course I'm not really interested in interaction with od; I just used it
as an example of a program that reads input and produces output accordingly.
I wanted to interface to a program (gap) that flushes its output on
terminals, but seemingly not on pipes.
That leaves 2 options: either change gap so that it doesn't buffer its
output, or run the process on a pseudo-tty.
In all cases, the problem isn't with Julia :)
On Sunday, 28 June 2015 22:39:36 UTC+2, Jameson wrote:
yes, it is the fault of `od`. you can see this by using `cat` instead and
observing that it works. if you play around with the thresholds in `od`,
you will observe that it is doing block-buffering of 4096 byte chunks when
the input is a pipe.
Unless you turn on write buffering (with `buffer_writes(si)`), the `write`
function implicitly calls `flush`, and will block the task until the write
completes (the actual `flush` function is a no-op). Therefore, it's best to
call write in a separate task if you are processing both ends of the pipe:
julia (so,si,pr) = readandwrite(`od`)
(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`od`,
ProcessRunning))
julia @async write(si,repeat(test\n,1))
Task (done) @0x7fe37aa0daa0
(note, the line_buffered property has no effect)
Note that Base.readavailable returns a random, non-zero number of bytes,
which is presumably not what you want?
julia Base.readavailable(so)
135168-element Array{UInt8,1}:
...
julia Base.readavailable(so)
61440-element Array{UInt8,1}:
...
In order to get the full output, you need to instead call:
julia close(si)
julia readall(so)
On Sun, Jun 28, 2015 at 9:08 AM Jeff Waller trut...@gmail.com
javascript: wrote:
On Saturday, June 27, 2015 at 5:03:53 PM UTC-4, ele...@gmail.com wrote:
Is your od program flushing its output? Maybe its stuck in its internal
buffers, not in the pipe.
nah man, if it's it's od, it's not going to do that, the only thing
simpler is cat; I suppose you could try cat.
What happens when you do something like this?
*julia **fromStream,toStream,p=readandwrite(`cat`)*
*(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`,
ProcessRunning))*
*julia **toStream.line_buffered*
*true*
*julia **toStream.line_buffered = false*
*false*
*julia **fromStream.line_buffered = false*
*false*
*julia **write(toStream,x)*
*1*
*julia **readavailable(fromStream)*
*1-element Array{UInt8,1}:*
* 0x78*
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
On Saturday, June 27, 2015 at 5:03:53 PM UTC-4, ele...@gmail.com wrote:
Is your od program flushing its output? Maybe its stuck in its internal
buffers, not in the pipe.
nah man, if it's it's od, it's not going to do that, the only thing simpler
is cat; I suppose you could try cat.
What happens when you do something like this?
*julia **fromStream,toStream,p=readandwrite(`cat`)*
*(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`,
ProcessRunning))*
*julia **toStream.line_buffered*
*true*
*julia **toStream.line_buffered = false*
*false*
*julia **fromStream.line_buffered = false*
*false*
*julia **write(toStream,x)*
*1*
*julia **readavailable(fromStream)*
*1-element Array{UInt8,1}:*
* 0x78*
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
yes, it is the fault of `od`. you can see this by using `cat` instead and
observing that it works. if you play around with the thresholds in `od`,
you will observe that it is doing block-buffering of 4096 byte chunks when
the input is a pipe.
Unless you turn on write buffering (with `buffer_writes(si)`), the `write`
function implicitly calls `flush`, and will block the task until the write
completes (the actual `flush` function is a no-op). Therefore, it's best to
call write in a separate task if you are processing both ends of the pipe:
julia (so,si,pr) = readandwrite(`od`)
(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`od`,
ProcessRunning))
julia @async write(si,repeat(test\n,1))
Task (done) @0x7fe37aa0daa0
(note, the line_buffered property has no effect)
Note that Base.readavailable returns a random, non-zero number of bytes,
which is presumably not what you want?
julia Base.readavailable(so)
135168-element Array{UInt8,1}:
...
julia Base.readavailable(so)
61440-element Array{UInt8,1}:
...
In order to get the full output, you need to instead call:
julia close(si)
julia readall(so)
On Sun, Jun 28, 2015 at 9:08 AM Jeff Waller truth...@gmail.com wrote:
On Saturday, June 27, 2015 at 5:03:53 PM UTC-4, ele...@gmail.com wrote:
Is your od program flushing its output? Maybe its stuck in its internal
buffers, not in the pipe.
nah man, if it's it's od, it's not going to do that, the only thing
simpler is cat; I suppose you could try cat.
What happens when you do something like this?
*julia **fromStream,toStream,p=readandwrite(`cat`)*
*(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`,
ProcessRunning))*
*julia **toStream.line_buffered*
*true*
*julia **toStream.line_buffered = false*
*false*
*julia **fromStream.line_buffered = false*
*false*
*julia **write(toStream,x)*
*1*
*julia **readavailable(fromStream)*
*1-element Array{UInt8,1}:*
* 0x78*
[julia-users] readandwrite: how can I read a line as soon as it's written by the process?
I'm trying to interact with an exterior program, and can't get it to work as I want. Let's say the program is od for example. I try: julia (so,si,pr) = readandwrite(`od`) (Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`od`, ProcessRunning)) julia write(si,repeat(test\n,100)) 500 julia flush(si) Pipe(open, 0 bytes waiting) julia readline(so) ^C # nothing happens, the process is blocking output How do I manage to get Julia to give me the output of od? A line was already ready after the first 16 characters of input. This is related to buffering: if I stuff the buffer, then I do get lines asynchronously: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,1)); julia readline(so) 000062564 072163 072012 071545 005164 062564 072163 072012\n However, there also seems to be a bug in Julia here: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,10)); # and Julia is stuck ^CERROR: InterruptException: ERROR (unhandled task failure): write: broken pipe (EPIPE) in yieldto at ./task.jl:21 in wait at ./task.jl:309 in wait at ./task.jl:225 in wait_full at ./multi.jl:573 in take! at ./multi.jl:744 in take_ref at ./multi.jl:752 in call_on_owner at ./multi.jl:718 in anonymous at task.jl:86 julia pr # hmmm... I wonder in which state the process is ^CERROR: InterruptException: bash$ # Julia crashed
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
You can make sure that there is data in `so` by looking at its 'bytes waiting' field. If there is indeed data in the pipe, maybe there is no newline character in it? Have you tried `readall` instead of `readline`? -- mb On Sat, Jun 27, 2015 at 8:45 AM, Laurent Bartholdi laurent.bartho...@gmail.com wrote: I'm trying to interact with an exterior program, and can't get it to work as I want. Let's say the program is od for example. I try: julia (so,si,pr) = readandwrite(`od`) (Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`od`, ProcessRunning)) julia write(si,repeat(test\n,100)) 500 julia flush(si) Pipe(open, 0 bytes waiting) julia readline(so) ^C # nothing happens, the process is blocking output How do I manage to get Julia to give me the output of od? A line was already ready after the first 16 characters of input. This is related to buffering: if I stuff the buffer, then I do get lines asynchronously: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,1)); julia readline(so) 000062564 072163 072012 071545 005164 062564 072163 072012\n However, there also seems to be a bug in Julia here: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,10)); # and Julia is stuck ^CERROR: InterruptException: ERROR (unhandled task failure): write: broken pipe (EPIPE) in yieldto at ./task.jl:21 in wait at ./task.jl:309 in wait at ./task.jl:225 in wait_full at ./multi.jl:573 in take! at ./multi.jl:744 in take_ref at ./multi.jl:752 in call_on_owner at ./multi.jl:718 in anonymous at task.jl:86 julia pr # hmmm... I wonder in which state the process is ^CERROR: InterruptException: bash$ # Julia crashed
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
Dear Miguel: Indeed, data is not made available to the pipe; though it should be there, because od prints lines as soon as they're available. I tried readall, but it also blocks. I should have added that I tested this with the latest, 0.4 release from github. I also tried just reading one character, with read(so,UInt8), and this also blocks. I notice that you are the author of the gnuplot package Gaston; so you are certainly familiar with the issue. Looking at Gaston's code, I see that you directly called :popen from the C library. Is there a reason not to use the higher-level interface of Julia? I got more crashes by feeding large amounts of data to a pipe: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,10)); ^CERROR: InterruptException:Assertion failed: (req-handle == stream), function uv__write, file src/unix/stream.c, line 741. signal (6): Abort trap: 6 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line) Abort trap: 6 bash$ Thanks in advance! Laurent
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
To answer your question about Gaston first, when I wrote that code nearly 3 years ago, there was no infrastructure in julia to create pipes to external processes. That's why I went with popen from the C standard library. I will update that code, but I want to read from both Gnuplot's STDOUT and STDERR while writing to STDIN, which is not supported by Julia at the moment. I'm trying to come up with a solution to that. Are you sure that your process `od` is printing to STDOUT and not STDERR? The crashes you're seeing look like worth opening an issue for, especially if they're easy to reproduce. -- mb On Sat, Jun 27, 2015 at 11:49 AM, Laurent Bartholdi laurent.bartho...@gmail.com wrote: Dear Miguel: Indeed, data is not made available to the pipe; though it should be there, because od prints lines as soon as they're available. I tried readall, but it also blocks. I should have added that I tested this with the latest, 0.4 release from github. I also tried just reading one character, with read(so,UInt8), and this also blocks. I notice that you are the author of the gnuplot package Gaston; so you are certainly familiar with the issue. Looking at Gaston's code, I see that you directly called :popen from the C library. Is there a reason not to use the higher-level interface of Julia? I got more crashes by feeding large amounts of data to a pipe: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,10)); ^CERROR: InterruptException:Assertion failed: (req-handle == stream), function uv__write, file src/unix/stream.c, line 741. signal (6): Abort trap: 6 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line) Abort trap: 6 bash$ Thanks in advance! Laurent
Re: [julia-users] readandwrite: how can I read a line as soon as it's written by the process?
Is your od program flushing its output? Maybe its stuck in its internal buffers, not in the pipe. On Sunday, June 28, 2015 at 4:34:20 AM UTC+10, Miguel Bazdresch wrote: To answer your question about Gaston first, when I wrote that code nearly 3 years ago, there was no infrastructure in julia to create pipes to external processes. That's why I went with popen from the C standard library. I will update that code, but I want to read from both Gnuplot's STDOUT and STDERR while writing to STDIN, which is not supported by Julia at the moment. I'm trying to come up with a solution to that. Are you sure that your process `od` is printing to STDOUT and not STDERR? The crashes you're seeing look like worth opening an issue for, especially if they're easy to reproduce. -- mb On Sat, Jun 27, 2015 at 11:49 AM, Laurent Bartholdi laurent@gmail.com javascript: wrote: Dear Miguel: Indeed, data is not made available to the pipe; though it should be there, because od prints lines as soon as they're available. I tried readall, but it also blocks. I should have added that I tested this with the latest, 0.4 release from github. I also tried just reading one character, with read(so,UInt8), and this also blocks. I notice that you are the author of the gnuplot package Gaston; so you are certainly familiar with the issue. Looking at Gaston's code, I see that you directly called :popen from the C library. Is there a reason not to use the higher-level interface of Julia? I got more crashes by feeding large amounts of data to a pipe: julia (so,si,pr) = readandwrite(`od`); julia write(si,repeat(test\n,10)); ^CERROR: InterruptException:Assertion failed: (req-handle == stream), function uv__write, file src/unix/stream.c, line 741. signal (6): Abort trap: 6 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line) Abort trap: 6 bash$ Thanks in advance! Laurent
