Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?
Sat, Apr 20, 2024 at 16:45 Carl Edquist wrote:However, stdout and stderr are still separate streams even if they refer to the same output file/pipe/device, so partial lines are not interleaved in the order that they were printed. will output abc\n123\n instead of a1b2c3\n\n, even if you run it as $ ./abc123 21 | cat It seems that its interleaved when buffer is written to a file or pipe, and because stdout is buffered it waits until buffer is full or flushed, while stderr is not and it doesnt wait and write immediately.
Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?
On Sat, 20 Apr 2024, Zachary Santer wrote: This was actually in RHEL 7. Oh. In that case it might be worth looking into ... I don't know how buffering works when stdout and stderr get redirected to the same pipe. You'd think, whatever it is, it would have to be smart enough to keep them interleaved in the same order they were printed to in. That in mind, I would assume they both get placed into the same block buffer by default. My take is always to try it and find out. Though in this case I think the default (without using stdbuf) is that the program's stderr is output to the pipe immediately (ie, unbuffered) on each library call (fprintf(3), fputs(3), putc(3), fwrite(3)), while stdout is written to the pipe at block boundaries - even though fd 1 and 2 refer to the same pipe. If you force line buffering for stdout and stderr, that is likely what you want, and it will interleave _lines_ in the order that they were printed. However, stdout and stderr are still separate streams even if they refer to the same output file/pipe/device, so partial lines are not interleaved in the order that they were printed. For example: #include int main() { putc('a', stderr); putc('1', stdout); putc('b', stderr); putc('2', stdout); putc('c', stderr); putc('3', stdout); putc('\n', stderr); putc('\n', stdout); return 0; } will output "abc\n123\n" instead of "a1b2c3\n\n", even if you run it as $ ./abc123 2>&1 | cat or $ stdbuf -oL -eL ./abc123 2>&1 | cat ... Not that that's relevant for what you're doing :) Carl
Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?
On Sat, Apr 20, 2024 at 11:58 AM Carl Edquist wrote: > > On Thu, 18 Apr 2024, Zachary Santer wrote: > > > > Finally had a chance to try to build with 'stdbuf --output=L --error=L > > --' in front of the build script, and it caused some crazy problems. > > For what it's worth, when I was trying that out msys2 (since that's what > you said you were using), I also ran into some very weird errors when just > trying to export LD_PRELOAD and _STDBUF_O to what stdbuf -oL sets. It was > weird because I didn't see issues when just running a command (including > bash) directly under stdbuf. I didn't get to the bottom of it though and > I don't have access to a windows laptop any more to experiment. This was actually in RHEL 7. stdbuf --output=L --error=L -- "${@}" 2>&1 | tee log-file | while IFS='' read -r line; do # do stuff done # And then obviously the arguments to this script give the command I want it to run. > Also I might ask, why are you setting "--error=L" ? > > Not that this is the problem you're seeing, but in any case stderr is > unbuffered by default, and you might mess up the output a bit by line > buffering it, if it's expecting to output partial lines for progress or > whatever. I don't know how buffering works when stdout and stderr get redirected to the same pipe. You'd think, whatever it is, it would have to be smart enough to keep them interleaved in the same order they were printed to in. That in mind, I would assume they both get placed into the same block buffer by default.
Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?
On Thu, 18 Apr 2024, Zachary Santer wrote: On Wed, Mar 20, 2024 at 4:54 AM Carl Edquist wrote: However, if stdbuf's magic env vars are exported in your shell (either by doing a trick like 'export $(env -i stdbuf -oL env)', or else more simply by first starting a new shell with 'stdbuf -oL bash'), then every command in your pipelines will start with the new default line-buffered stdout. That way your line-items from build.sh should get passed all the way through the pipeline as they are produced. Finally had a chance to try to build with 'stdbuf --output=L --error=L --' in front of the build script, and it caused some crazy problems. For what it's worth, when I was trying that out msys2 (since that's what you said you were using), I also ran into some very weird errors when just trying to export LD_PRELOAD and _STDBUF_O to what stdbuf -oL sets. It was weird because I didn't see issues when just running a command (including bash) directly under stdbuf. I didn't get to the bottom of it though and I don't have access to a windows laptop any more to experiment. Also I might ask, why are you setting "--error=L" ? Not that this is the problem you're seeing, but in any case stderr is unbuffered by default, and you might mess up the output a bit by line buffering it, if it's expecting to output partial lines for progress or whatever. Carl