Re: Checking executability for asynchronous commands
Date:Tue, 29 Dec 2020 10:23:07 -0500 From:Chet Ramey Message-ID: | Bash will save some number of exit statuses corresponding to terminated | asynchronous child processes, and those statuses are retrievable using | `wait'. That number varies -- POSIX says you need to save at least | CHILD_MAX statuses, but bash will look at its nproc resource limit (using | sysconf(3)) and try to save that many, up to a max of 32K in bash-5.1. This | will be sufficient in most cases. The interesting case (for shell authors, in practice this rarely, if ever, actually happens) is how to handle the case where the shell has collected (waited for) the status from a child process, but the script has yet to fetch it (that status is one of those being saved). Then, during that interval it is possible for the same pid to be assigned by the kernel for some other process - and that other process might be a child of the same shell. If that child is a fg process (never moved to bg) there's no real problem, but if it is a bg process, and its pid is exported via $! then there can be a real problem, as the script no longer has any safe documented and simple way to distinguish the two processes (they will have different %n job numbers, but scripts don't usually even consider such things). I have considered have the shell (the one I maintain, not bash) check (both parent and child) after a fork, and abort the child immediately, and simply try the fork again, if the system assigns a pid which would cause a problem like that - but never bothered to actually write the code (both parent and child after the fork can run the same test, in parallel, so there's no issue knowing what happened or what to do next .. the vfork case is similarly easy). It has never been a high priority as, in practice, the issue just doesn't arise. kre
Re: Checking executability for asynchronous commands
On 12/29/20 10:28 AM, Chet Ramey wrote: On 12/28/20 5:30 PM, Eli Schwartz wrote: (Though I have to wonder at these amazing AWOL commands that get uninstalled on people all the time right in the middle of their scripts. It's a potential security concern, though that class of vulnerabilities mostly involves executables being changed between testing and execution. Right, the race condition / security concern is specifically based on the idea that one is checking for permission / authority to run a program, possibly as setuid, and it gets replaced by something malicious before being used. If you were going to blindly run the program either way, then having it be *uninstalled* (i.e. does not exist, period) is... probably not going to result in security concerns. It will just fail to run. And it would do so even without the race condition. By all means, let people be concerned about their commands being replaced by attack code. Not about them being rm'ed. -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User OpenPGP_signature Description: OpenPGP digital signature
Re: Checking executability for asynchronous commands
On 12/28/20 5:30 PM, Eli Schwartz wrote: (Though I have to wonder at these amazing AWOL commands that get uninstalled on people all the time right in the middle of their scripts. It's a potential security concern, though that class of vulnerabilities mostly involves executables being changed between testing and execution. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Checking executability for asynchronous commands
On 12/28/20 1:23 AM, Markus Elfring wrote: I got another programming concern: Process identifications are a system resource. The exit status is preserved until this information will eventually be used for corresponding data processing. How many processes can a selected system configuration handle? This is a hard question to answer. It depends on the system, its configuration, the PID allocation rate, and the allocation strategy. PIDs are indeed allocated by the system, using different strategies. Many systems use random allocation through a 16- or 32-bit space (whatever a pid_t is). It's possible to receive the same PID for two different spawned child processes before enumerating the entire PID space. One guarantee is that the system won't reallocate a given PID until it's been reaped, and its parent gets its exit status, so a script will usually have the opportunity to obtain the status information it wants. Bash will save some number of exit statuses corresponding to terminated asynchronous child processes, and those statuses are retrievable using `wait'. That number varies -- POSIX says you need to save at least CHILD_MAX statuses, but bash will look at its nproc resource limit (using sysconf(3)) and try to save that many, up to a max of 32K in bash-5.1. This will be sufficient in most cases. If a script saves exit status information it retrieves using `wait' into shell variables, it can obviously save as many statuses as it wants. So the answer is, as it usually is, "it depends." -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Checking executability for asynchronous commands
>…, is not a winning argument. I got further development ideas according to the usage of wait function calls and better error reporting for asynchronous commands. >The OP seems to think that "people will occasionally forget to run `wait`", This happens for various reasons. > and wants to know if we "care" that people will forget … The attention varies for this implementation aspect. > Why should we care? Would you like to care for proper management of system resources (including process identifications)? >The official advice is to run `wait` How do you think about any additional guidance for such a programming interface? Regards, Markus
Re: Checking executability for asynchronous commands
Markus Elfring writes: > I imagine that it can be occasionally helpful to determine the execution > failure > in the synchronous way. > Would it make sense to configure the error reporting for selected asynchronous > commands so that they would become background processes only after the > required > check for executability? In many situations, you can check whether a command is executable with [[ -x name ]] As a general rule, Bash provides functionality which matches the direct way to implement an operation using Unix. In the case of "command &", the direct way is "(1) fork a subprocess, (2) the subprocess does an exec() of command". The consequences are that (a) the parent process has no information about the subprocess until it executes a wait() for the subprocess, and (b) the subprocess, while executing bash, has no information about whether "command" is executable until it executes exec() -- if the command is not executable, the exec() will fail and bash will print an error message and exit with an error, but if the command is executable, the subprocess bash is immediately replaced by the command. In particular, there is no way for it to report to the parent process that the exec() has been successfully executed. With this sequence of operations, there is no way for the parent bash to know in advance whether "command" is executable. Of course, with further operations, it could first test whether "command" is executable, but that is effectively the same as performing "[[ -x command ]]" in the bash script before performing "command &". (And it has the same difficulty regarding race conditions.) Dale
Re: Checking executability for asynchronous commands
On 12/28/20 4:45 PM, Léa Gris wrote: When you handle such logic within Bash you already lost on a race condition when foo is readable and executable when you test it, but when it reaches the actual execution, it is no longer the case. Bash is full of race conditions, because bash was never meant as a General Purpose language. Bash is a good command sequencer. Now if ppl forget to use wait PID after launching a sub-shell background command, then shame on them. The race condition doesn't matter if they anyways need to check the status by waiting on the PID, in order to handle "it executed, but resulted in failure". So it's not very tragic if the race condition resulted in users being unable to see the *enhanced* error message describing the missing dependency executable condition... this entire thread seems to just be about error reporting, right? (Though I have to wonder at these amazing AWOL commands that get uninstalled on people all the time right in the middle of their scripts. Maybe if they use a package manager for both the scripts and the dependency executables, they could fully prevent race conditions *and* not even need to check if their dependencies are installed.) -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User OpenPGP_signature Description: OpenPGP digital signature
Re: Checking executability for asynchronous commands
On 28/12/2020 at 21:18, Eli Schwartz wrote: if cmd=$(type -P foo) && test -x "$foo"; then foo & else echo "error: foo could not be found or is not executable" fi When you handle such logic within Bash you already lost on a race condition when foo is readable and executable when you test it, but when it reaches the actual execution, it is no longer the case. Bash is full of race conditions, because bash was never meant as a General Purpose language. Bash is a good command sequencer. Now if ppl forget to use wait PID after launching a sub-shell background command, then shame on them. -- Léa Gris
Re: Checking executability for asynchronous commands
On 12/28/20 8:15 AM, Greg Wooledge wrote: On Sun, Dec 27, 2020 at 08:02:49AM -0500, Eli Schwartz wrote: I'm not sure I understand the question? My interpretation is that for an async *simple* command of the form cmd args & where cmd cannot be executed at all (due to lack of permissions, perhaps, or because the external program is not installed), they want bash to set $? to a nonzero value to indicate that the command couldn't even be started. I've seen similar requests several times over the years. The problem is that the parent bash (the script) doesn't know, and cannot know, that the command was stillborn. Only the child bash process can know this, and by the time this information has become available, the parent bash process has already moved on. The only way the parent can obtain this information is to wait until that information becomes available. Actually, I don't see why one could not circumvent the entire process, and do this instead if cmd=$(type -P foo) && test -x "$foo"; then foo & else echo "error: foo could not be found or is not executable" fi But I do get the initial premise of the thread. I don't get the *defense* being offered though. The logic here seems to be completely bankrupt -- saying bash needs new features (that are not well thought out) so you don't "need" to include code to handle your intentions, is not a winning argument. The OP seems to think that "people will occasionally forget to run `wait`", and wants to know if we "care" that people will forget and if Chet will add new features to bash in order to cater to these forgetful people. This is what I don't understand. Why should we care? The official advice is to run `wait` (or perform executability checks upfront, or whatever). -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User OpenPGP_signature Description: OpenPGP digital signature
Re: Checking executability for asynchronous commands
On Mon, Dec 28, 2020 at 3:16 PM Greg Wooledge wrote: > The problem is that the parent bash (the script) doesn't know, and > cannot know, that the command was stillborn. Only the child bash > process can know this, and by the time this information has become > available, the parent bash process has already moved on. > In principle, if the parent and child were to cooperate, I think the status of the final execve() could be communicated to the parent like this: Set up a pipe between the parent and the child, with the write side set to close-on-exec, and have the parent block on the read side. If the execve() call fails, the child can send an error message via the pipe, and if it succeeds, the parent will see the pipe being closed without a message. Polling the child after some fraction of a second might not be able to tell a failed execve() apart from the exec'ed process exiting after the exec.
Re: Checking executability for asynchronous commands
On Sun, Dec 27, 2020 at 08:02:49AM -0500, Eli Schwartz wrote: > I'm not sure I understand the question? My interpretation is that for an async *simple* command of the form cmd args & where cmd cannot be executed at all (due to lack of permissions, perhaps, or because the external program is not installed), they want bash to set $? to a nonzero value to indicate that the command couldn't even be started. I've seen similar requests several times over the years. The problem is that the parent bash (the script) doesn't know, and cannot know, that the command was stillborn. Only the child bash process can know this, and by the time this information has become available, the parent bash process has already moved on. The only way the parent can obtain this information is to wait until that information becomes available. The obvious problem here is that the parent does not know when that information will become available. So, one is stuck choosing from among the following strategies: 1) After launching the async command, sleep for some fraction of a second, and then check whether the child is still running. If it isn't running, retrieve its exit status. 2) Set up a SIGCHLD handler (trap), and process the child's exit status whenever the trap fires. 3) Poll "kill -0" on the child's PID during the script's main loop. Each of these strategies has its advantages and flaws. None of them is correct for every script.
Re: Checking executability for asynchronous commands
>> Would you care if waiting on such identifications for background processes >> will occasionally be forgotten? >> >> How many efforts would you invest to add potentially missing wait function >> calls? > > It's axiomatic: if you want to make a decision based on the exit status of > any asynchronous process, you need to use `wait' to obtain the status of > that process. > > I don't think "I don't want to do it that way" is a good reason to provide > a different method. I got another programming concern: Process identifications are a system resource. The exit status is preserved until this information will eventually be used for corresponding data processing. How many processes can a selected system configuration handle? Regards, Markus
Re: Checking executability for asynchronous commands
On 12/27/20 5:01 AM, Markus Elfring wrote: If you have the pid of an asynchronous command -- and the easiest way to get that pid is by referencing $! after it was started -- you can call `wait' with that pid to retrieve the status, even if it's already terminated. Would you care if waiting on such identifications for background processes will occasionally be forgotten? How many efforts would you invest to add potentially missing wait function calls? It's axiomatic: if you want to make a decision based on the exit status of any asynchronous process, you need to use `wait' to obtain the status of that process. I don't think "I don't want to do it that way" is a good reason to provide a different method. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Checking executability for asynchronous commands
On 12/27/20 5:01 AM, Markus Elfring wrote: If you have the pid of an asynchronous command -- and the easiest way to get that pid is by referencing $! after it was started -- you can call `wait' with that pid to retrieve the status, even if it's already terminated. Would you care if waiting on such identifications for background processes will occasionally be forgotten? How many efforts would you invest to add potentially missing wait function calls? Would you care if configuring bash to wait on identification of background processes will occasionally be forgotten? Would you care if checking the status of foreground processes and doing different things based on success or failure will occasionally be forgotten? Would you care if will occasionally be forgotten? I'm not sure I understand the question? Writing programs in *any* programming language requires attention to detail and effectively conveying your need to the programming language. bash is no exception, even if people have a terrible habit of treating bash like it should be special or different merely because it uses subprocesses a lot, and is popular. A stronger argument must be made for new features rather than merely "sometimes people are extremely forgetful, we need a new language feature that doesn't fit in well and doesn't behave consistently, so they can be forgetful about that instead". -- Eli Schwartz Arch Linux Bug Wrangler and Trusted User OpenPGP_signature Description: OpenPGP digital signature
Re: Checking executability for asynchronous commands
> If you have the pid of an asynchronous command -- and the easiest way to get > that pid > is by referencing $! after it was started -- you can call `wait' with that pid > to retrieve the status, even if it's already terminated. Would you care if waiting on such identifications for background processes will occasionally be forgotten? How many efforts would you invest to add potentially missing wait function calls? Regards, Markus
Re: Checking executability for asynchronous commands
On 12/26/20 4:10 AM, Markus Elfring wrote: If you want the exit status of the child process, add `wait $!' Thanks for this information. Is there a need then to point the aspect out that exit values can be determined for child processes even if they terminated before a wait command would be performed? Is this what you mean? If you have the pid of an asynchronous command -- and the easiest way to get that pid is by referencing $! after it was started -- you can call `wait' with that pid to retrieve the status, even if it's already terminated. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Checking executability for asynchronous commands
> If you want the exit status of the child process, add `wait $!' Thanks for this information. Is there a need then to point the aspect out that exit values can be determined for child processes even if they terminated before a wait command would be performed? Regards, Markus
Re: Checking executability for asynchronous commands
On 12/25/20 4:40 AM, Markus Elfring wrote: Hello, I am looking for another bit of clarification according to an implementation detail. The manual is providing the following information. https://git.savannah.gnu.org/cgit/bash.git/tree/doc/bash.1?id=76404c85d492c001f59f2644074333ffb7608532#n627 “… If a command is terminated by the control operator &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0. …” Thus I observe a behaviour like the following for a simple test with the software “bash 5.0.18-3.1” according to a “program” which does not exist here. elfring@Sonne:~> xy & [1] 4063 xy: Befehl nicht gefunden [1]+ Exit 127xy Yes, the child process exits with status 127, but $? is set to 0, as POSIX requires. If you want the exit status of the child process, add `wait $!' when you are ready to use the status, which will set $? to 127. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Checking executability for asynchronous commands
> Are you maybe saying > > cmd & > > should be a special case? I got special imaginations according to such a command variant. Regards, Markus
Re: Checking executability for asynchronous commands
On Fri, Dec 25, 2020 at 10:40:17AM +0100, Markus Elfring wrote: > Hello, > > I am looking for another bit of clarification according to an implementation > detail. > > The manual is providing the following information. > https://git.savannah.gnu.org/cgit/bash.git/tree/doc/bash.1?id=76404c85d492c001f59f2644074333ffb7608532#n627 > > “… >If a command is terminated by the control operator &, the shell > executes the command in the background in a subshell. The shell does not > wait for the command to finish, and the >return status is 0. > …” > > > Thus I observe a behaviour like the following for a simple test with the > software “bash 5.0.18-3.1” according to a “program” which does not exist here. > > elfring@Sonne:~> xy & > [1] 4063 > xy: Befehl nicht gefunden > [1]+ Exit 127xy > > > I imagine that it can be occasionally helpful to determine the execution > failure > in the synchronous way. > Would it make sense to configure the error reporting for selected asynchronous > commands so that they would become background processes only after the > required > check for executability? How would you propose that the shell handle something like the following? ( cmd1 && cmd2 ) & Are you maybe saying cmd & should be a special case? -- Andreas (Kusalananda) Kähäri SciLifeLab, NBIS, ICM Uppsala University, Sweden .
