It may seem intemperate to propose that a program with an 82-page manual "page" could benefit from yet another feature, but that is what this is doing: Bash's users would benefit from a feature that suppresses buffering of output sent to particular pipes (designated by the user).
At the bottom of this message is a list of a dozen references spanning 20 years requesting a general mechanism to obtain this behavior. At least the following programs have options or default behaviors to suppress buffering stdout in situations where a generic program would buffer: python -u (or env. var. PHTHONUNBUFFERED) grep --line-buffered sed -u cat -u (unbuffered output by default in Gnu) tcpdump -l tee (unbuffered output by default) tail -f awk/mawk (fflush function or unbuffered output by default) The following commands have been devised to suppress buffering: unbuffer (probably == expect_unbuffer) stdbuf -oL script -q env. var. NSUnbufferedIO=YES (on MacOS) Each runs into one or more of these issues: does not work with statically-linked executables security issues involving env. var. LD_PRELOAD allocation of a pty combines stderr and stdout the effect does not apply to subprocesses of command, particularly commands within shell scripts, or applies to subprocesses whose output is not the pipe This message describes a proposed solution to this problem which is easier to invoke and works in a much broader range of circumstances than the previous solutions. The associated messages provide a proof-of-concept demonstration implementation of this proposal. This command demonstrates the issue: for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A | cat Currently, when this command is run, after 10 seconds it outputs 10 lines reading "ABCDE". That is because the "grep" command's stdout is to a pipe, and so is buffered. The buffer accumulates all 70 characters of output and outputs them to the pipe when "grep" exits. The demo proposed change defines: for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A >|> cat The "unbuffered pipe" symbol ">|>" causes Bash to set in the environment of the "grep" process a variable "STDOUT_UNBUFFERED" with a value that contains the dev and ino values for the pipe which the "grep" process sees as fd 1. The stdio library of the "grep" process, when it initializes processing of stdout (== fd 1), observes that the fd has the dev and ino values specified in the env. var. STDOUT_UNBUFFERED, and sets stdout's buffering state to unbuffered. (This can be overridden later if the process calls setvbuf.) The result is that the above command outputs one line "ABCDE" each second for 10 seconds. In addition to ">|>" for an unbuffered pipe, the demo also defines ">|>&", which is the unbuffered parallel of "|&", i.e., it is equivalent to "2>&1 >|>" and redirects both stdout and stderr from one command into an unbuffered pipe. There are a large number of open issues regarding this proposal, and in particular, the code only works for the simplest usages. But the first issue to be addressed is whether this functionality is worth pursuing. Dale ---------------------------------------------------------------------- References 14 Sep 1999 https://marc.info/?l=glibc-bug&m=98313957306295&w=4 "[REMINDER] stdio buffer flushing control environment variable" 26 May 2006 http://www.pixelbeat.org/programming/stdio_buffering/ "buffering in standard streams" 19 May 2009 https://stackoverflow.com/questions/881696/unbuffered-stdout-in-python-as-in-python-u-from-within-the-program "unbuffered stdout in python (as in python -u) from within the program [duplicate]" upvoted 57 times 16 Jun 2009 https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe "Turn off buffering in pipe" upvoted 415 times 20 Jan 2010 https://lists.gnu.org/archive/html/bug-bash/2010-01/msg00069.html "weird bash pipe behavior" 26 Jul 2010 https://stackoverflow.com/questions/3332045/bash-force-execd-process-to-have-unbuffered-stdout "bash: force exec'd process to have unbuffered stdout" upvoted 21 times 12 Aug 2010 https://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered "How to make output of any shell command unbuffered?" upvoted 66 times 23 Jun 2011 https://perkin.org.uk/posts/how-to-fix-stdio-buffering.html "How to fix stdio buffering" 26 Jul 2011 https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux/589614#589614 "Is there a way to redirect output to a file without buffering on unix/linux?" upvoted 50 times 30 Nov 2012 https://stackoverflow.com/questions/13644024/stdbuf-with-setuid-capabilities/18624182 "stdbuf with setuid/capabilities" upvoted 3 times 2 Nov 2013 https://blog.jpalardy.com/posts/grep-and-output-buffering/ "Grep and Output Buffering" 31 Jan 2015 https://unix.stackexchange.com/questions/182232/pipes-how-do-data-flow-in-a-pipeline "Pipes, how do data flow in a pipeline?" upvoted 22 times 4 May 2015 https://unix.stackexchange.com/questions/200235/how-to-use-sed-to-manipulate-continuously-streaming-output "How to use sed to manipulate continuously streaming output?" upvoted 13 times