Scraping through this - thanks for the lessons aka explanations.
On 18/03/2021 16:08, Chet Ramey wrote:
If I understand correctly, the commands are generated by gmake as it processes targets.On 3/18/21 5:53 AM, Michael Felt wrote:Yes, something to test. Thx. The ojdk scenario is: /usr/bin/printf > >(tee -a stdout.log) 2> >(tee -a stderr.log).So, yes, in this case it is working because printf is the parent - (which I never seemed to find actually calling open() of the file. It seems to be using the fd opened by the child - in a magical way).It's the redirection. The shell does the open, since the filename resulting from process substitution is the target of a redirection operator. This isa common idiom -- so common, in fact, that people have interpreted it to mean that the entire `> >(xxx)' is a single operator. However, the shell expands redirections in the child process it forks to exec printf, so that child shell is what does the process substitution. That might be the problem here. The command itself doesn't do anything, though. `tee' just sits there waiting for data to write to log files. It has no purpose. I'm not sure what the intent is.If you wrapped that command into a script, it's unlikely that either `tee'would exit (why would they?) before `printf' exits and the script completes. In bash-5.0, there would be nothing to remove the FIFOs.
Got it: must remember - initially it is bash busy with word expansion (a new bash child for each 'process substitution'But what I thoght I was seeing is that diff is the PARENT calling substitute_process() that create(s) a child process that reads/writes to a fifo file.This is defined to provide `diff' with two arguments. Let's call them /var/tmp/sh-np12345 and /var/tmp/sh-np67890 So diff runs, sees two arguments, opens both files, and does its thing. Diff has to see two filenames when it runs, otherwise it's an error.Yes and no. Process substitution is a word expansion that results in a filename. The stuff between the parens defines a command that writes toor reads from a pipe expressed as a filename (/dev/fd/NN or a FIFO) that is the result of the word expansion. In this case, the process substitution isthe target of a redirection, so the shell performs that word expansion before it execs diff.a) the child process never returns - it `exits` via, iirc, sh_exit(result) and the end of the routineIt executes the specified command and exits.
Yes - for me at least it is much easier to fathom as input - that ends and behaves/looks/feels like EOF.b) the parent gets the filename (pathname) - but I never see it actually opening it - only (when using bash -x) seeing the name in the -x command expansion.It doesn't have to. The filename itself is the expansion: it's an object you can use to communicate with an asynchronous process. This is how youcan have programs that expect a filename use program-generated output, forinstance, without using a temp file.
Nods: just as it would if it was the output from a program than had just run 'moments' before.In this case, it opens the FIFO because it is the target of a redirection.
No I cannot - and for now it is a `hack` to solve a bigger issue. With 3500 calls in a single build I hope the race occurs - and I'll finally see where the PARENT actually uses the name returned.Now, let's say your change is there. The shell still runs diff /var/tmp/sh-np12345 /var/tmp/sh-np67890 but, depending on how processes get scheduled, the shell forked to run the process substitutions has already unlinked those FIFOs. Diff will error out. The user will be unhappy. I will get bug reports. You have introduced a race condition. You may not get hit by it, but you cannot guarantee that no one will.You mean in terms of using the filename as an argument to a shell builtin?Otherwise you'll have to trace into other child process execution.
/usr/bin/printf is not a built-n (afaik) If I understand correctly - from printf perspective we have/usr/bin/printf "Some formatted message" > /tmp/sh-np.123456 2> /tmp/sh-np.9876543 &
And, if for ease of discussion we say program1 is PID-123456 and program is PID-987654 - these programs have no way of knowing their stdin is named /tmp/sh-np-something? BING: as the dutch (used to) say - the quarter drops - the other programs (e.g., tee) have no fifo knowledge - they are who/what they are. What maybe needed for this situation - is rather than directly execev() the program - yet another fork (for the execve() - and wait for that program to hit it's EOF on input and then the sleeping 'word expansion child' cleans up the fifo file it created for the communication path.
* Am I getting closer? :)
Below - not helping yet. And, in the case of the process I am trying to resolve the first command is:Again, my issue was with >(command) substitution - where the `files` get written to by the parent - rather than reading them.The shell can't unlink the FIFO until it can guarantee that the processes that need to open it have opened it, and it can't guarantee that in the general case. It has to wait until the process completes, at least, and even that might not be correct.It doesn't matter. Let's try that scenario. A FIFO reader can live forever;just waiting for someone to open the FIFO to write to it. In this case, the child process opens the FIFO for read, and blocks until another process opens it for write. That's the shell, since it's the target of a redirection, but it doesn't have to be (the filename could just be passed to another process as an argument). The file descriptor gets passed toprintf as its stdout (and printf apparently does nothing with it) and thenclosed as part of the process exiting. When that happens, the tee shouldget EOF and exit. The shell notices that tee exits and cleans up the FIFO. If the shell exits, for instance, before the tee exits, nothing cleans up the FIFO.p.s. it is not my call to ask why they do not use regular redirection or pipes. Feels much simpler - but some people cannot miss the opportunity to use something blinky and shiney.p.p.s. - If you have `words of wisdom` re: why this approach is much better than `standard` redirection - I am all ears!If you want to send the output to the terminal (or wherever) as well as alog file, something like `tee' is required. If you want to keep stderr andstdout logs separate, two separate redirections are required. If you want this to happen asynchronously, process substitution is required.Otherwise, you introduce temporary files and synchronous behavior when youwant neither.
/usr/bin/printf "Pretty message" >tmpout 2>tmpout " That too, makes my head hurt. why not /usr/bin/printf "Pretty message\n" 2>&1 | tee -a /my/special/logfile (This is what I am also asking in an issue at AdoptOpenJDK)
If it helps, the above printf command is syntactic sugar for something like(with no error checking): { /usr/bin/printf >tmpout 2>tmperr cat tmperr tmpout cat tmpout >> stdout.log cat tmperr >> stderr.log rm -f tmpout tmperr }
OpenPGP_0x722BFDB61F396FC2.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature