[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #24 from Christophe Lyon  ---
(In reply to Thomas Koenig from comment #21)
> [...]
> but I am not sure that waitpid is available on all systems;
> this is more likely to break things than fix things.
> 
I tried simpler things (just calling wait()), but since you have this concern,
I agree it's not worth pursuing.

> What you could to to reduce the amount of noise that you see is
> sleep for a second; just insert
> 
>   CALL SLEEP(1)
> 
> before the END statement of the test case, locally in your tree.
> This could help. I don't want to do this for everybody (test times
> being long enough as they are), but for most cases, this should
> be enough.

For the time being, it's much simpler for me to handle the 127 return code in
my testcase execution wrapper.

Thanks for the help.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Thomas Koenig  changed:

   What|Removed |Added

 Status|ASSIGNED|RESOLVED
 Resolution|--- |FIXED

--- Comment #23 from Thomas Koenig  ---
Fixed on all open branches, closing.

Abut the failing test case - this is really a problem with
the test system, this would work on real hardware. If anybody has
a solution, please implement.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #22 from Thomas Koenig  ---
Author: tkoenig
Date: Sun Oct 22 14:21:16 2017
New Revision: 253983

URL: https://gcc.gnu.org/viewcvs?rev=253983=gcc=rev
Log:
2017-10-22  Thomas Koenig  

Backport from trunk
PR libfortran/82233
* intrinsics/execute_command_line.c (execute_command_line):
No call to runtime_error if cmdstat is present.

2017-10-22  Thomas Koenig  

Backport from trunk
PR libfortran/82233
* gfortran.dg/execute_command_line_3.f90: New test.


Added:
branches/gcc-6-branch/gcc/testsuite/gfortran.dg/execute_command_line_3.f90
Modified:
branches/gcc-6-branch/gcc/testsuite/ChangeLog
branches/gcc-6-branch/libgfortran/ChangeLog
branches/gcc-6-branch/libgfortran/intrinsics/execute_command_line.c

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #21 from Thomas Koenig  ---
We could do something like

program boom
implicit none
  interface
 subroutine mywait() bind(c)
 end subroutine mywait
  end interface
integer   :: i,j 
character(len=256):: msg
character(len=:), allocatable :: command
   command='notthere'
   msg='' ! seems to only be defined if exitstatus.ne.0
   ! ok -- these work
   call execute_command_line(command , wait=.false., exitstat=i, cmdstat=j,
cmdmsg=msg)
   if (j /= 0 .or. msg /= '') call abort
   call execute_command_line(command ,   exitstat=i, cmdstat=j,
cmdmsg=msg )
   if (j /= 3 .or. msg /= "Invalid command line" ) call abort
   msg = ''
   call execute_command_line(command , wait=.false., exitstat=i,   
cmdmsg=msg )
   print *,msg
   if (msg /= '') call abort
   call execute_command_line(command ,   exitstat=i, cmdstat=j 
   )
   if (j /= 3) call abort
   call execute_command_line(command , wait=.false., exitstat=i
   )
   call mywait()
end program boom

and

#include 
#include 
#include 
#include 


/* Wait in a busy loop for any processes that may still be around. */
void mywait()
{
  errno = 0;
  while (waitpid (-1, NULL, WNOHANG) >= 0)
{
  n++;
  if (errno != ECHILD)
break;
}
}

but I am not sure that waitpid is available on all systems;
this is more likely to break things than fix things.

What you could to to reduce the amount of noise that you see is
sleep for a second; just insert

  CALL SLEEP(1)

before the END statement of the test case, locally in your tree.
This could help. I don't want to do this for everybody (test times
being long enough as they are), but for most cases, this should
be enough.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #20 from Thomas Koenig  ---
(In reply to Jerry DeLisle from comment #19)
> (In reply to Christophe Lyon from comment #17)
> > Thanks for your effort; I'm still seeing noise though.
> > 
> > Sorry, I'm not fluent in fortran: is there a way to call wait() from 
> > fortran?
> > I could try adding it at the end of the testcase, to see if it reliably
> > fixes the problem I observed.
> 
> You can use ISO_C_Binding and call a C function.  See
> https://gcc.gnu.org/onlinedocs/gfortran/Interoperable-Subroutines-and-
> Functions.html

It might be possible to call wait() from Fortran using ISO C binding,
but I don't know how to get the type of pid_t from Fortran.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Jerry DeLisle  changed:

   What|Removed |Added

 CC||jvdelisle at gcc dot gnu.org

--- Comment #19 from Jerry DeLisle  ---
(In reply to Christophe Lyon from comment #17)
> Thanks for your effort; I'm still seeing noise though.
> 
> Sorry, I'm not fluent in fortran: is there a way to call wait() from fortran?
> I could try adding it at the end of the testcase, to see if it reliably
> fixes the problem I observed.

You can use ISO_C_Binding and call a C function.  See
https://gcc.gnu.org/onlinedocs/gfortran/Interoperable-Subroutines-and-Functions.html

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #17 from Christophe Lyon  ---
Thanks for your effort; I'm still seeing noise though.

Sorry, I'm not fluent in fortran: is there a way to call wait() from fortran?
I could try adding it at the end of the testcase, to see if it reliably fixes
the problem I observed.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #18 from Thomas Koenig  ---
Author: tkoenig
Date: Thu Oct 19 17:49:24 2017
New Revision: 253907

URL: https://gcc.gnu.org/viewcvs?rev=253907=gcc=rev
Log:
2017-10-19  Thomas Koenig  

Backport from trunk
PR libfortran/82233
* intrinsics/execute_command_line.c (execute_command_line):
No call to runtime_error if cmdstat is present.

2017-10-19  Thomas Koenig  

Backport from trunk
PR libfortran/82233
* gfortran.dg/execute_command_line_3.f90: New test.


Added:
branches/gcc-7-branch/gcc/testsuite/gfortran.dg/execute_command_line_3.f90
Modified:
branches/gcc-7-branch/gcc/testsuite/ChangeLog
branches/gcc-7-branch/libgfortran/ChangeLog
branches/gcc-7-branch/libgfortran/intrinsics/execute_command_line.c

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #16 from Thomas Koenig  ---
(In reply to Christophe Lyon from comment #14)

> Removing the last
>call execute_command_line(command , wait=.false., exitstat=i)
> or moving it before
>call execute_command_line(command ,   exitstat=i, cmdstat=j)
> seems to workaround the issue for me.

I have done the latter in the commit just now. Hopefully, this
will the reduce the amount of noise in your test log.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #15 from Thomas Koenig  ---
Author: tkoenig
Date: Wed Oct 18 17:54:18 2017
New Revision: 253865

URL: https://gcc.gnu.org/viewcvs?rev=253865=gcc=rev
Log:
2017-10-18  Thomas Koenig  

PR libfortran/82233
* gfortran.dg/execute_command_line_3.f90:  Remove unneeded output.
Move test with wait=.false. before the last test.


Modified:
trunk/gcc/testsuite/ChangeLog
trunk/gcc/testsuite/gfortran.dg/execute_command_line_3.f90

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #14 from Christophe Lyon  ---
I think I understand the problem better now, after a few experiments, and a
discussion with proot's author.

The testcase finishes with a call to execute_command_line with wait=.false, so
the main process may complete before the child (execute_command_line) does.

In such a case, proot returns the exit code of the last living process, the
child in our case, so 127 (command not found). And sometimes it can return 0,
depending on the process scheduling on the host.

Forcing qemu to dump traces seems to hide the problem, probably because it does
change the process schedule.

FWIW, I can live with this by flagging this test as 'random' in my
GCC-regression testing.

Removing the last
   call execute_command_line(command , wait=.false., exitstat=i)
or moving it before
   call execute_command_line(command ,   exitstat=i, cmdstat=j)
seems to workaround the issue for me.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #13 from Thomas Koenig  ---
I am not sure what to do with this failing test case.
To me, this sounds more like a problem with your test setup.

execute_command_line calls the C system() function.  What
does that do on your system if there is no /bin/sh?  Anything useful?

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #12 from Christophe Lyon  ---
I am running the GCC tests under proot + qemu.
proot is similar to chroot but does not require root privileges,
qemu is a well known emulator that enables me to run validations for
arm/aarch64 on x86 hosts.

I use proot to make sure the simulated process only sees the filesystem part it
needs; in particular this is mandatory when simulating armeb (big-endian) to
prevent the test from trying to parse the host's /etc/ld.so.cache and
crashing...

Back to the problem: if I run the testcase using qemu only (no proot), I get:
$ qemu-wrapper.sh ./execute_command_line_3.exe
sh: 1: notthere: not found
sh: 1: notthere: not found
sh: 1: notthere: not found

sh: 1: notthere: not found
sh: 1: notthere: not found
$ echo $?
0

If I use proot, I get:
$ qemu-wrapper.sh ./execute_command_line_3.exe

$ echo $?
0

In both cases, stdout contains 257 white spaces followed by a newline.

So it seems the output is the same. Does the test care about stderr?

When I use proot, command-line execution fails earlier because there is no
/bin/sh, but the return-code of execve is still 127 (command not found), so the
behaviour should still be the same.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #11 from Thomas Koenig  ---
Another idea: Could there be a network drive in the PATH
that does not respond in time?

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #10 from Thomas Koenig  ---
(In reply to Christophe Lyon from comment #9)
> Indeed, it passes/fails randomly in subsequent builds. It is killed, maybe
> because of a timeout, but there is nothing in the standard logs.
> 
> I've been trying to patch dejagnu to be more verbose in such cases, to avoid
> reporting noise; it's still wip.

There could be one issue with the test case: Possibly, there
could be output to standard error. This didn't cause any problem
during regression testing.

What happens if you run the test case by hand?

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #9 from Christophe Lyon  ---
Indeed, it passes/fails randomly in subsequent builds. It is killed, maybe
because of a timeout, but there is nothing in the standard logs.

I've been trying to patch dejagnu to be more verbose in such cases, to avoid
reporting noise; it's still wip.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #8 from Andrew Pinski  ---
(In reply to Christophe Lyon from comment #7)
> The new test fails at execution on ARM and AArch64:
> FAIL:gfortran.dg/execute_command_line_3.f90   -O  execution test

Does not fail for me with either of my builds of r253621 (ed8e4f5) on
aarch64-linux-gnu:
https://gcc.gnu.org/ml/gcc-testresults/2017-10/msg00859.html
https://gcc.gnu.org/ml/gcc-testresults/2017-10/msg00847.html

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[clyon at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Christophe Lyon  changed:

   What|Removed |Added

 CC||clyon at gcc dot gnu.org

--- Comment #7 from Christophe Lyon  ---
The new test fails at execution on ARM and AArch64:
FAIL:gfortran.dg/execute_command_line_3.f90   -O  execution test

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #6 from Thomas Koenig  ---
Author: tkoenig
Date: Tue Oct 10 16:49:32 2017
New Revision: 253593

URL: https://gcc.gnu.org/viewcvs?rev=253593=gcc=rev
Log:
2017-10-10  Thomas Koenig  

PR libfortran/82233
* intrinsics/execute_command_line.c (execute_command_line):
No call to runtime_error if cmdstat is present.

2017-10-10  Thomas Koenig  

PR libfortran/82233
* gfortran.dg/execute_command_line_3.f90: New test.


Added:
trunk/gcc/testsuite/gfortran.dg/execute_command_line_3.f90
Modified:
trunk/gcc/testsuite/ChangeLog
trunk/libgfortran/ChangeLog
trunk/libgfortran/intrinsics/execute_command_line.c

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #5 from Thomas Koenig  ---
This looks good:

Index: intrinsics/execute_command_line.c
===
--- intrinsics/execute_command_line.c   (Revision 253525)
+++ intrinsics/execute_command_line.c   (Arbeitskopie)
@@ -125,15 +125,9 @@
   free (cmd);

   /* Now copy back to the Fortran string if needed.  */
-  if (cmdstat && *cmdstat > EXEC_NOERROR)
-{
-  if (cmdmsg)
-   fstrcpy (cmdmsg, cmdmsg_len, cmdmsg_values[*cmdstat],
+  if (cmdstat && *cmdstat > EXEC_NOERROR && cmdmsg)
+fstrcpy (cmdmsg, cmdmsg_len, cmdmsg_values[*cmdstat],
strlen (cmdmsg_values[*cmdstat]));
-  else
-   runtime_error ("Failure in EXECUTE_COMMAND_LINE: %s",
-  cmdmsg_values[*cmdstat]);
-}
 }

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Thomas Koenig  changed:

   What|Removed |Added

 Status|NEW |ASSIGNED
 CC||tkoenig at gcc dot gnu.org
   Assignee|unassigned at gcc dot gnu.org  |tkoenig at gcc dot 
gnu.org

--- Comment #4 from Thomas Koenig  ---
Let's take a look here.. this sounds doable.

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Richard Biener  changed:

   What|Removed |Added

   Target Milestone|--- |6.5

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[urbanjost at comcast dot net]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

--- Comment #3 from urbanjost at comcast dot net ---
Created attachment 42193
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=42193=edit
specifically even if cmdstat option present, must have cmdmsg or get failure,
and man(1) page needs changed

After looking over the F2008 standard and the recent gfortran changes indicated
in the previous comment it looks like part of the changes between 5.0.4 and 6+
actually made EXECUTE_COMMAND_LINE(3f) more standard-conforming; so (although I
prefer the 5.0.4 version's behavior) I'd like to change the problem description
to two parts:

The man page for gfortran should mention that if CMDSTAT is not present and the
command cannot be executed that the program should terminate. It currently does
not:
  https://gcc.gnu.org/onlinedocs/gfortran/EXECUTE_005fCOMMAND_005fLINE.html
and that is a significant change from 5.0.4 (and is apparently what it should
do to be f2008 standard conforming).

Then, as the second attachment shows, supplying CMDSTAT should be sufficient,
but even when CMDSTAT is present if CMDMSG is not the program fails.

And I think a simple termination equivalent to "STOP 'message'" is probably the
expected behavior, where I am getting what looks to be an abort with
traceback/segfault.

Terminating the program is so unlike the de-factor standard SYSTEM(3f) or the C
routine system(3c) the 5.0.4- version behavior was far more intuitive, but the
f2008 standard does say command execution failure without the CMDSTAT option
being present terminates the program; although it is very arguable as to
exactly what that means on different systems (requiring unavailable resources,
file permissions, not being present, getting a "system error" when executing,
...???)

[Bug libfortran/82233] [6/7/8 Regression] execute_command_line causes program to stop when command fails (or does not exist)

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82233

Dominique d'Humieres  changed:

   What|Removed |Added

   Priority|P3  |P4
 Status|UNCONFIRMED |NEW
  Known to work||5.4.0
   Keywords||wrong-code
   Last reconfirmed||2017-09-17
  Component|fortran |libfortran
 CC||fxcoudert at gcc dot gnu.org
 Ever confirmed|0   |1
Summary|execute_command_line causes |[6/7/8 Regression]
   |program to stop when|execute_command_line causes
   |command fails (or does not  |program to stop when
   |exist)  |command fails (or does not
   ||exist)
  Known to fail||6.4.0, 7.2.0, 8.0

--- Comment #2 from Dominique d'Humieres  ---
Likely caused by r227105 (pr54572).