Re: 'wait' in command group in pipeline fails to recognize child process
On 6/23/21 6:17 PM, Martin Jambon wrote: On 6/23/21 6:24 AM, Chet Ramey wrote: On 6/22/21 9:54 PM, Martin Jambon wrote: In the posix definition, a subshell - is not necessarily implemented as a separate process - belongs to a unique shell - is not a shell Why is it not "a shell?" Because '$$' doesn't match its process ID. OK. That's a rather unique definition, but it's a definition. I'm not sure why I have to fight over this. It's clearly my misunderstanding. That's why I suggest clarifications in the documentation, if you're interested in creating a better experience for users like me. I don't think we're fighting here. We're discussing whether or not we need to invent new nomenclature to clarify an obscure point, or whether a clearer explanation of the concept of a subshell will suffice. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: 'wait' in command group in pipeline fails to recognize child process
On Wed, Jun 23, 2021 at 03:17:48PM -0700, Martin Jambon wrote: > > What is the magic quality that imparts shellness? > > '$$' matching the process ID. That just signifies that you're in the original shell, or the main shell, or the root shell, or the fundamental shell, or make up whatever term you like for it. Every bash (or sh) process is a shell, regardless of whether it's a subshell or not. That includes subshells, and non-sub-shells. Here's a brief demonstration: I have two terminals opened. The first one is pts/3: unicorn:~$ ps PID TTY TIME CMD 966 pts/300:00:00 bash 1363991 pts/300:00:00 ps unicorn:~$ (sleep 10; true) In the second terminal, while the sleep is running: unicorn:~$ ps f -t pts/3 PID TTY STAT TIME COMMAND 966 pts/3Ss 0:00 bash 1364016 pts/3S+ 0:00 \_ bash 1364017 pts/3S+ 0:00 \_ sleep 10 Process 966 is the interactive bash shell, which is not a subshell. It's the other kind of shell. Process 1364016 is the subshell created by the (parentheses) to run the command which was specified inside them. They're both instances of bash. They're both shells. > I'm not sure why I have to fight over this. It's clearly my > misunderstanding. That's why I suggest clarifications in the documentation, > if you're interested in creating a better experience for users like me. Nobody knows what the hell to CALL it. That's the problem. It doesn't have a clearly defined name, because nobody ever NEEDED one.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/23/21 6:24 AM, Chet Ramey wrote: On 6/22/21 9:54 PM, Martin Jambon wrote: In the posix definition, a subshell - is not necessarily implemented as a separate process - belongs to a unique shell - is not a shell Why is it not "a shell?" Because '$$' doesn't match its process ID. What is the magic quality that imparts shellness? '$$' matching the process ID. Posix: '$' shall expand to the same value as that of the current shell. (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02) Bash: ($$) Expands to the process ID of the shell. (https://www.gnu.org/software/bash/manual/bash.html#Special-Parameters) I'm not sure why I have to fight over this. It's clearly my misunderstanding. That's why I suggest clarifications in the documentation, if you're interested in creating a better experience for users like me.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 9:54 PM, Martin Jambon wrote: In the posix definition, a subshell - is not necessarily implemented as a separate process - belongs to a unique shell - is not a shell Why is it not "a shell?" What is the magic quality that imparts shellness? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 8:00 PM, Lawrence Velázquez wrote: On Tue, Jun 22, 2021, at 6:32 PM, Martin Jambon wrote: I would also mention pipelines here, since these are more commonly used than () subshells. I don't know if there are other ways of creating subshells. If that's all, I think it would be valuable to mention those two cases rather than just one. There are also command substitutions and any commands run asynchronously. The documentation specifies that command substitutions, asynchronous commands, () subshells, and pipeline elements are all run in subshell environments. The description makes it clear that a subshell is a duplicate of its parent shell, with signal dispositions as the exception. If the description applies to all subshells, not just ones inside (...), then perhaps the documentation should just omit "()" instead of spelling out every possible way a subshell can be created. Agreed. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: 'wait' in command group in pipeline fails to recognize child process
On Tue, Jun 22, 2021 at 06:54:18PM -0700, Martin Jambon wrote: > This is confirmed by this definition from posix: > > A subshell environment shall be created as a duplicate of the shell > environment, except [...] > In the posix definition, a subshell > - is not necessarily implemented as a separate process > - belongs to a unique shell > - is not a shell I have no idea how you came to the conclusion that it's "not a shell". It's a DUPLICATE OF A SHELL, and therefore a shell. The point is that it's a DIFFERENT shell. Not the original one. I mentioned earlier that there's a formal definition you can largely ignore, and just go with "a subshell is the direct result of a fork()".[1] That's all you really need to know. The formal language exists to allow for implementations in some theoretical non-Unix-like environment where fork() is not a thing. It's simply describing what a fork() does, without using that term. [1] Except for that bit about resetting signal handlers.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 6:11 PM, Lawrence Velázquez wrote: On Tue, Jun 22, 2021, at 8:52 PM, Martin Jambon wrote: It's better. However, the reader is still left wondering what "the shell" is referring to in first sentence. Subshells aside, I have a hard time believing that "the process ID of the shell" confuses anybody in practice. Even POSIX doesn't overcomplicate this: $ Expands to the decimal process ID of the invoked shell. In a subshell (see Shell Execution Environment), '$' shall expand to the same value as that of the current shell. (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02) This language implies that a subshell is part of the shell but not a shell in itself. This is what I pointed out earlier but I received a conflicting answer. This is confirmed by this definition from posix: A subshell environment shall be created as a duplicate of the shell environment, except that signal traps that are not being ignored shall be set to the default action. Changes made to the subshell environment shall not affect the shell environment. Command substitution, commands that are grouped with parentheses, and asynchronous lists shall be executed in a subshell environment. Additionally, each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment. All other commands shall be executed in the current shell environment. (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12) In the posix definition, a subshell - is not necessarily implemented as a separate process - belongs to a unique shell - is not a shell This makes the term "the shell" unambiguous to a reader who's aware of this - but it's obvious to me that "subshell" should be clarified in the context of '$$'. A subset is a set, a subgroup is a group, a subtype is a type, a subtree is a tree, a subprocess is a process. However a submarine is not a marine, and a subshell is not a shell (sea stuff!). Anyway, I left a suggestion to revise the documentation for '$$' in my previous email. That's the best I can produce at this time. Martin
Re: 'wait' in command group in pipeline fails to recognize child process
> It's not the brace command; it's the pipeline.
>
>> Minimal repro:
>>
>>$ sleep 1 & { wait $!; } | cat
>>[1] 665454
>>bash: wait: pid 665454 is not a child of this shell
Interestingly, this is almost trivial to clairfy:
$ sleep 5 & { wait $!; }
[1] 19069
[1]+ Donesleep 5
$
> I was also misled by the term "subshell" which is not a proper shell
> like a subprocess is just another process.
The criticial point is that "being a subshell" is a relationship between
the child bash process and its parent. Intuitively, a subshell is
created whenever a bash child process is created "implicitly". The
manual page et al. enumerate all the ways a subshell can be created;
search for the word "subshell".
Dale
Re: 'wait' in command group in pipeline fails to recognize child process
On Tue, Jun 22, 2021, at 8:52 PM, Martin Jambon wrote: > It's better. However, the reader is still left wondering what "the > shell" is referring to in first sentence. Subshells aside, I have a hard time believing that "the process ID of the shell" confuses anybody in practice. Even POSIX doesn't overcomplicate this: $ Expands to the decimal process ID of the invoked shell. In a subshell (see Shell Execution Environment), '$' shall expand to the same value as that of the current shell. (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02) -- vq
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 5:00 PM, Lawrence Velázquez wrote: Maybe something like this would get the point across: ($$) Expands to the process ID of the shell. In a subshell, it expands to the value that $$ has in the invoking shell. It's better. However, the reader is still left wondering what "the shell" is referring to in first sentence. Here's my next suggestion, which borrows some language from the documentation for BASHPID: ($$) Expands to the ID of the process that initialized Bash. The value of $$ is inherited in subshells, such as pipelines, regardless of their own process ID. See also the Bash variable BASHPID. For reference, the documentation for 'BASHPID' is this: Expands to the process ID of the current Bash process. This differs from $$ under certain circumstances, such as subshells that do not require Bash to be re-initialized. Assignments to BASHPID have no effect. If BASHPID is unset, it loses its special properties, even if it is subsequently reset.
Re: 'wait' in command group in pipeline fails to recognize child process
On Tue, Jun 22, 2021, at 6:32 PM, Martin Jambon wrote: > I would also mention pipelines here, since these are more commonly > used than () subshells. I don't know if there are other ways of > creating subshells. If that's all, I think it would be valuable > to mention those two cases rather than just one. There are also command substitutions and any commands run asynchronously. If the description applies to all subshells, not just ones inside (...), then perhaps the documentation should just omit "()" instead of spelling out every possible way a subshell can be created. > Well, I suppose "root shell" could be misunderstood as a shell run by > the 'root' user but this doesn't make much sense in this context. Yes, but then the reader is left to reverse-engineer the meaning of this term you just made up. If the documentation is to be changed, it should be more clear than before. "Root shell" makes it less clear and should not be used. There wouldn't be anything wrong with adding a more verbose but more precise explanation. > I also changed "invoking shell" to "root shell" because the invoking > shell or parent shell is not necessarily the same as the root shell e.g. > > $ echo $$; (echo $$; (echo $$)) > 688918 > 688918 > 688918 Maybe something like this would get the point across: ($$) Expands to the process ID of the shell. In a subshell, it expands to the value that $$ has in the invoking shell. -- vq
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 1:37 PM, Greg Wooledge wrote: On Tue, Jun 22, 2021 at 01:12:10PM -0700, Martin Jambon wrote: On 6/22/21 4:31 AM, Greg Wooledge wrote: A pipeline creates two or more subshells, one for each command in the pipeline. Therefore, your wait command is running in a different process than the one which created the sleep background job. The curly braces are irrelevant here. unicorn:~$ sleep 1 & wait "$!"|cat [1] 1290127 bash: wait: pid 1290127 is not a child of this shell Thank you! Now I know that a subshell is not a shell, $$ being the process ID of the shell, not of the subshell. It's a forked child process of bash, and as such, it's still a shell. Specifically, it's bash. There's some formal definition that you can ignore. The real definition of a subshell is "the direct result of a fork()". It inherits all of the shell variables, functions, aliases, and so on that the parent shell process had at the time of the fork. This is different from a direct call to "bash", which would not inherit shell variables and so on -- only environment variables. Great, then I would change the documentation of '$$'. Currently it says this: ($$) Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the invoking shell, not the subshell. This first mentions just "the shell". I suggest calling it the "root shell" instead, i.e. the process where "exec bash" took place. I would also mention pipelines here, since these are more commonly used than () subshells. I don't know if there are other ways of creating subshells. If that's all, I think it would be valuable to mention those two cases rather than just one. Well, I suppose "root shell" could be misunderstood as a shell run by the 'root' user but this doesn't make much sense in this context. I can't find a better name. Here's how we might change the description of '$$': ($$) Expands to the process ID of the root shell. In a pipeline or in a () subshell, it expands to the process ID of the root shell, not the subshell. I also changed "invoking shell" to "root shell" because the invoking shell or parent shell is not necessarily the same as the root shell e.g. $ echo $$; (echo $$; (echo $$)) 688918 688918 688918 In the pipeline, there are two subshells created. One of them runs the command wait "$!", and the other runs the command cat. The special parameter $! is inherited from the parent shell, and contains the PID of the sleep process that the parent shell created. But the child subshell can't wait for that process, because it belongs to someone else. Hence the error message. Yup. I think the error message is clear. No complaint about that.
Re: 'wait' in command group in pipeline fails to recognize child process
On Tue, Jun 22, 2021 at 01:12:10PM -0700, Martin Jambon wrote: > On 6/22/21 4:31 AM, Greg Wooledge wrote: > > A pipeline creates two or more subshells, one for each command in the > > pipeline. Therefore, your wait command is running in a different > > process than the one which created the sleep background job. > > > > The curly braces are irrelevant here. > > > > unicorn:~$ sleep 1 & wait "$!"|cat > > [1] 1290127 > > bash: wait: pid 1290127 is not a child of this shell > > Thank you! Now I know that a subshell is not a shell, $$ being the process > ID of the shell, not of the subshell. It's a forked child process of bash, and as such, it's still a shell. Specifically, it's bash. There's some formal definition that you can ignore. The real definition of a subshell is "the direct result of a fork()". It inherits all of the shell variables, functions, aliases, and so on that the parent shell process had at the time of the fork. This is different from a direct call to "bash", which would not inherit shell variables and so on -- only environment variables. In the pipeline, there are two subshells created. One of them runs the command wait "$!", and the other runs the command cat. The special parameter $! is inherited from the parent shell, and contains the PID of the sleep process that the parent shell created. But the child subshell can't wait for that process, because it belongs to someone else. Hence the error message.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 4:31 AM, Greg Wooledge wrote:
On Tue, Jun 22, 2021 at 02:42:40AM -0700, Martin Jambon wrote:
I ran into something that looks like a bug to me, although I'm not super
familiar curly-brace command groups.
Bash version: latest from GitHub mirror (commit ce23728: Bash-5.1 patch 7)
Minimal repro:
$ sleep 1 & { wait $!; } | cat
[1] 665454
bash: wait: pid 665454 is not a child of this shell
I was expecting a success, just like we get without the pipeline:
A pipeline creates two or more subshells, one for each command in the
pipeline. Therefore, your wait command is running in a different
process than the one which created the sleep background job.
The curly braces are irrelevant here.
unicorn:~$ sleep 1 & wait "$!"|cat
[1] 1290127
bash: wait: pid 1290127 is not a child of this shell
Thank you! Now I know that a subshell is not a shell, $$ being the
process ID of the shell, not of the subshell.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 6:15 AM, Chet Ramey wrote: On 6/22/21 5:42 AM, Martin Jambon wrote: Hello! I ran into something that looks like a bug to me, although I'm not super familiar curly-brace command groups. It's not the brace command; it's the pipeline. Thank you! It's the $$ that confused me since I thought incorrectly that it would print the current process ID. I was also misled by the term "subshell" which is not a proper shell like a subprocess is just another process.
Re: 'wait' in command group in pipeline fails to recognize child process
On 6/22/21 5:42 AM, Martin Jambon wrote:
Hello!
I ran into something that looks like a bug to me, although I'm not super
familiar curly-brace command groups.
It's not the brace command; it's the pipeline.
Minimal repro:
$ sleep 1 & { wait $!; } | cat
[1] 665454
bash: wait: pid 665454 is not a child of this shell
I was expecting a success, just like we get without the pipeline:
The pipeline does, in fact, make the difference. Each process in a pipeline
is run in a subshell, and that subshell isn't the parent of the previously-
created asynchronous job. You can't wait for a process that isn't one of
your children.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: 'wait' in command group in pipeline fails to recognize child process
On Tue, Jun 22, 2021 at 02:42:40AM -0700, Martin Jambon wrote:
> I ran into something that looks like a bug to me, although I'm not super
> familiar curly-brace command groups.
>
> Bash version: latest from GitHub mirror (commit ce23728: Bash-5.1 patch 7)
>
> Minimal repro:
>
> $ sleep 1 & { wait $!; } | cat
> [1] 665454
> bash: wait: pid 665454 is not a child of this shell
>
> I was expecting a success, just like we get without the pipeline:
A pipeline creates two or more subshells, one for each command in the
pipeline. Therefore, your wait command is running in a different
process than the one which created the sleep background job.
The curly braces are irrelevant here.
unicorn:~$ sleep 1 & wait "$!"|cat
[1] 1290127
bash: wait: pid 1290127 is not a child of this shell
'wait' in command group in pipeline fails to recognize child process
Hello!
I ran into something that looks like a bug to me, although I'm not super
familiar curly-brace command groups.
Bash version: latest from GitHub mirror (commit ce23728: Bash-5.1 patch 7)
Minimal repro:
$ sleep 1 & { wait $!; } | cat
[1] 665454
bash: wait: pid 665454 is not a child of this shell
I was expecting a success, just like we get without the pipeline:
$ sleep 1 & { wait $!; }
[1] 665591
[1]+ Donesleep 1
The following scripts are similar but print PIDs along the way:
failing script
$ cat wait-for-child
echo "main pid: $$"
/usr/bin/sleep 1 &
job_pid=$!
echo "job pid: $job_pid"
{
echo "command group pid: $$"
wait "$job_pid" || echo "wait failed!"
} | cat
Output:
$ ./bash wait-for-child
main pid: 664755
job pid: 664756
wait-for-child: line 7: wait: pid 664756 is not a child of this shell
command group pid: 664755
wait failed!
^ We see that the current PID in the command group is the same as in the
rest of the shell, as it should be. It is unclear to me why 'wait'
believes that the background job's PID is not of a child.
successful script
Compare to the same script without having the command group in a
pipeline (I removed '| cat'):
$ cat wait-for-child-ok
echo "main pid: $$"
/usr/bin/sleep 1 &
job_pid=$!
echo "job pid: $job_pid"
{
echo "command group pid: $$"
wait "$job_pid" || echo "wait failed!"
}
$ ./bash wait-for-child-ok
main pid: 664705
job pid: 664706
command group pid: 664705
^ OK
Martin
