Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?

2024-04-20 Thread wrotycz
 Sat, Apr 20, 2024 at 16:45 Carl Edquist wrote:However, stdout 
and stderr are still separate streams even if they refer to the same output 
file/pipe/device, so partial lines are not interleaved in the order that they 
were printed. will output abc\n123\n instead of 
a1b2c3\n\n, even if you run it as   $ ./abc123 21 | cat  
 It seems that its interleaved when buffer is written to a file 
or pipe, and because stdout is buffered it waits until buffer is full or 
flushed, while stderr is not and it doesnt wait and write immediately.


Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?

2024-04-20 Thread Carl Edquist

On Sat, 20 Apr 2024, Zachary Santer wrote:


This was actually in RHEL 7.


Oh.  In that case it might be worth looking into ...


I don't know how buffering works when stdout and stderr get redirected 
to the same pipe. You'd think, whatever it is, it would have to be smart 
enough to keep them interleaved in the same order they were printed to 
in. That in mind, I would assume they both get placed into the same 
block buffer by default.


My take is always to try it and find out.  Though in this case I think the 
default (without using stdbuf) is that the program's stderr is output to 
the pipe immediately (ie, unbuffered) on each library call (fprintf(3), 
fputs(3), putc(3), fwrite(3)), while stdout is written to the pipe at 
block boundaries - even though fd 1 and 2 refer to the same pipe.


If you force line buffering for stdout and stderr, that is likely what you 
want, and it will interleave _lines_ in the order that they were printed.


However, stdout and stderr are still separate streams even if they refer 
to the same output file/pipe/device, so partial lines are not interleaved 
in the order that they were printed.


For example:

#include 

int main()
{
putc('a', stderr);
putc('1', stdout);
putc('b', stderr);
putc('2', stdout);
putc('c', stderr);
putc('3', stdout);
putc('\n', stderr);
putc('\n', stdout);

return 0;
}

will output "abc\n123\n" instead of "a1b2c3\n\n", even if you run it as

$ ./abc123 2>&1 | cat
or
$ stdbuf -oL -eL ./abc123 2>&1 | cat


...

Not that that's relevant for what you're doing :)

Carl




Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?

2024-04-20 Thread Zachary Santer
On Sat, Apr 20, 2024 at 11:58 AM Carl Edquist  wrote:
>
> On Thu, 18 Apr 2024, Zachary Santer wrote:
> >
> > Finally had a chance to try to build with 'stdbuf --output=L --error=L
> > --' in front of the build script, and it caused some crazy problems.
>
> For what it's worth, when I was trying that out msys2 (since that's what
> you said you were using), I also ran into some very weird errors when just
> trying to export LD_PRELOAD and _STDBUF_O to what stdbuf -oL sets.  It was
> weird because I didn't see issues when just running a command (including
> bash) directly under stdbuf.  I didn't get to the bottom of it though and
> I don't have access to a windows laptop any more to experiment.

This was actually in RHEL 7.

stdbuf --output=L --error=L -- "${@}" 2>&1 |
  tee log-file |
while IFS='' read -r line; do
  # do stuff
done
#

And then obviously the arguments to this script give the command I
want it to run.

> Also I might ask, why are you setting "--error=L" ?
>
> Not that this is the problem you're seeing, but in any case stderr is
> unbuffered by default, and you might mess up the output a bit by line
> buffering it, if it's expecting to output partial lines for progress or
> whatever.

I don't know how buffering works when stdout and stderr get redirected
to the same pipe. You'd think, whatever it is, it would have to be
smart enough to keep them interleaved in the same order they were
printed to in. That in mind, I would assume they both get placed into
the same block buffer by default.



Re: Modify buffering of standard streams via environment variables (not LD_PRELOAD)?

2024-04-20 Thread Carl Edquist via GNU coreutils General Discussion

On Thu, 18 Apr 2024, Zachary Santer wrote:


On Wed, Mar 20, 2024 at 4:54 AM Carl Edquist  wrote:


However, if stdbuf's magic env vars are exported in your shell (either 
by doing a trick like 'export $(env -i stdbuf -oL env)', or else more 
simply by first starting a new shell with 'stdbuf -oL bash'), then 
every command in your pipelines will start with the new default 
line-buffered stdout. That way your line-items from build.sh should get 
passed all the way through the pipeline as they are produced.


Finally had a chance to try to build with 'stdbuf --output=L --error=L 
--' in front of the build script, and it caused some crazy problems.


For what it's worth, when I was trying that out msys2 (since that's what 
you said you were using), I also ran into some very weird errors when just 
trying to export LD_PRELOAD and _STDBUF_O to what stdbuf -oL sets.  It was 
weird because I didn't see issues when just running a command (including 
bash) directly under stdbuf.  I didn't get to the bottom of it though and 
I don't have access to a windows laptop any more to experiment.


Also I might ask, why are you setting "--error=L" ?

Not that this is the problem you're seeing, but in any case stderr is 
unbuffered by default, and you might mess up the output a bit by line 
buffering it, if it's expecting to output partial lines for progress or 
whatever.


Carl