pipe hang issue when running as SYSTEM
This issue was reported to MSYS2 as a hang when trying to build libxml2 in a Windows docker container. Another user was able to come up with a simple reproducer and a reasonable theory as to why it happens. The msys2 issue is https://github.com/msys2/msys2-runtime/issues/77, and I will quote the reproducer and analysis here: I am having the same issue when building mingw. I managed to reduce the issue to the following shell script: #!/bin/sh seq 1 9 > big_file eval '$(eval cmd.exe /c "type big_file" | : )' When running as a normal user this completes immediately, but when run as a system service it hangs forever. The issue appears to be that when running under the SYSTEM account, the third sh process holds open the read end of the pipe. Since cmd has too much output to write all at once, it waits until the pipe's buffer has room to write more, but since sh isn't actually reading from the pipe, this hangs forever. When running as a normal user the read end of the pipe is not kept open, and so cmd.exe gets an error when attempting to write and exits immediately. My suspicion is that this is caused by f79a461 (which keeps the read end of the pipe open) and b531d6b (which changes the behavior depending on whether or not the program is running as a service). -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Mon, 21 Mar 2022 11:45:57 -0700 (PDT) Jeremy Drake wrote: > This issue was reported to MSYS2 as a hang when trying to build libxml2 in > a Windows docker container. Another user was able to come up with a > simple reproducer and a reasonable theory as to why it happens. The msys2 > issue is https://github.com/msys2/msys2-runtime/issues/77, and I will > quote the reproducer and analysis here: > > I am having the same issue when building mingw. I managed to reduce the > issue to the following shell script: > > #!/bin/sh > seq 1 9 > big_file > eval '$(eval cmd.exe /c "type big_file" | : )' > > When running as a normal user this completes immediately, but when run as > a system service it hangs forever. The issue appears to be that when > running under the SYSTEM account, the third sh process holds open the read > end of the pipe. Since cmd has too much output to write all at once, it > waits until the pipe's buffer has room to write more, but since sh isn't > actually reading from the pipe, this hangs forever. When running as a > normal user the read end of the pipe is not kept open, and so cmd.exe gets > an error when attempting to write and exits immediately. > > My suspicion is that this is caused by f79a461 (which keeps the read end > of the pipe open) and b531d6b (which changes the behavior depending on > whether or not the program is running as a service). Thanks for the report. This is expected problem as mentioned in b531d6b commit message. However, I could not imagin the situation that the service has multiple writer for the pipe and one of them is a non-cygwin app. Question is: Does the docker invoke the command using SYSTEM account? Or is the processes in docker determined as running as a service? -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Tue, 22 Mar 2022 19:18:07 +0900 Takashi Yano wrote: > On Mon, 21 Mar 2022 11:45:57 -0700 (PDT) > Jeremy Drake wrote: > > This issue was reported to MSYS2 as a hang when trying to build libxml2 in > > a Windows docker container. Another user was able to come up with a > > simple reproducer and a reasonable theory as to why it happens. The msys2 > > issue is https://github.com/msys2/msys2-runtime/issues/77, and I will > > quote the reproducer and analysis here: > > > > I am having the same issue when building mingw. I managed to reduce the > > issue to the following shell script: > > > > #!/bin/sh > > seq 1 9 > big_file > > eval '$(eval cmd.exe /c "type big_file" | : )' > > > > When running as a normal user this completes immediately, but when run as > > a system service it hangs forever. The issue appears to be that when > > running under the SYSTEM account, the third sh process holds open the read > > end of the pipe. Since cmd has too much output to write all at once, it > > waits until the pipe's buffer has room to write more, but since sh isn't > > actually reading from the pipe, this hangs forever. When running as a > > normal user the read end of the pipe is not kept open, and so cmd.exe gets > > an error when attempting to write and exits immediately. > > > > My suspicion is that this is caused by f79a461 (which keeps the read end > > of the pipe open) and b531d6b (which changes the behavior depending on > > whether or not the program is running as a service). > > Thanks for the report. This is expected problem as mentioned > in b531d6b commit message. However, I could not imagin the > situation that the service has multiple writer for the pipe > and one of them is a non-cygwin app. > > Question is: Does the docker invoke the command using SYSTEM > account? Or is the processes in docker determined as running > as a service? I confirmed the processes in Windows docker are running as well_known_service_sid. Let me consider a while. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Tue, 22 Mar 2022, Takashi Yano wrote: > > Thanks for the report. This is expected problem as mentioned > > in b531d6b commit message. However, I could not imagin the > > situation that the service has multiple writer for the pipe > > and one of them is a non-cygwin app. > > > > Question is: Does the docker invoke the command using SYSTEM > > account? Or is the processes in docker determined as running > > as a service? > > I confirmed the processes in Windows docker are running as > well_known_service_sid. Let me consider a while. > Just got another report to MSYS2 of this behavior, this time from a Gitlab CI runner that is running as a service and trying to pipe output from Microsoft cl.exe while running configure. This user reports that 3.3.4 with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer." applied to it does not solve the problem. https://github.com/msys2/MSYS2-packages/issues/2893 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Thu, 24 Mar 2022 20:42:50 -0700 (PDT) Jeremy Drake wrote: > On Tue, 22 Mar 2022, Takashi Yano wrote: > > > > Thanks for the report. This is expected problem as mentioned > > > in b531d6b commit message. However, I could not imagin the > > > situation that the service has multiple writer for the pipe > > > and one of them is a non-cygwin app. > > > > > > Question is: Does the docker invoke the command using SYSTEM > > > account? Or is the processes in docker determined as running > > > as a service? > > > > I confirmed the processes in Windows docker are running as > > well_known_service_sid. Let me consider a while. > > > > Just got another report to MSYS2 of this behavior, this time from a Gitlab > CI runner that is running as a service and trying to pipe output from > Microsoft cl.exe while running configure. This user reports that 3.3.4 > with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer." > applied to it does not solve the problem. > > https://github.com/msys2/MSYS2-packages/issues/2893 I can confirm the sample script hangs indeed if the script is running as SYSTEM account. However, in my test, the patch solves the issue. It would be very helpfull if a simple github repository with github actions, which can reproduce the issue, is provided. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Fri, 25 Mar 2022, Takashi Yano wrote: > I can confirm the sample script hangs indeed if the script is > running as SYSTEM account. However, in my test, the patch solves > the issue. > > It would be very helpfull if a simple github repository with > github actions, which can reproduce the issue, is provided. After much fighting with docker on windows, https://github.com/jeremyd2019/msys2-pipe-hang-test/runs/5688176578?check_suite_focus=true -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
Hi, Takashi Yano wrote: [SNIP] Just got another report to MSYS2 of this behavior, this time from a Gitlab CI runner that is running as a service and trying to pipe output from Microsoft cl.exe while running configure. This user reports that 3.3.4 with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer." applied to it does not solve the problem. https://github.com/msys2/MSYS2-packages/issues/2893 I can confirm the sample script hangs indeed if the script is running as SYSTEM account. However, in my test, the patch solves the issue. It would be very helpfull if a simple github repository with github actions, which can reproduce the issue, is provided. In my environment with AD user account: $ cmd.exe /c "type big_file" | : The process tried to write to a nonexistent pipe. $ cat big_file | : *** starting debugger for pid NNN tid K $ uname -a CYGWIN_NT-10.0 ZZZ 3.3.4(0.341/5/3) 2022-01-31 19:35 x86_64 Cygwin Remarks: - no way to debug - no error if run as administrator Regards, Roumen Petrov -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Fri, 25 Mar 2022 00:09:59 -0700 (PDT) Jeremy Drake wrote: > On Fri, 25 Mar 2022, Takashi Yano wrote: > > > I can confirm the sample script hangs indeed if the script is > > running as SYSTEM account. However, in my test, the patch solves > > the issue. > > > > It would be very helpfull if a simple github repository with > > github actions, which can reproduce the issue, is provided. > > After much fighting with docker on windows, > https://github.com/jeremyd2019/msys2-pipe-hang-test/runs/5688176578?check_suite_focus=true Thanks much! I could reproduce the issue even with the patch. I also noticed what is the cause. The problem is that the command: docker run msys2-base c:\msys64\usr\bin\bash -lc "/c/test.sh" does not allocate tty. Therefore, kill_pgrp(__SIGNONCYGCHLD) is not called. If you use: docker run -t msys2-base c:\msys64\usr\bin\bash -lc "/c/test.sh" the problem does not occur. I will submit v3 patch shortly. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Fri, 25 Mar 2022, Takashi Yano wrote: > I will submit v3 patch shortly. I applied your v3 patch to 3.3.4 in https://github.com/msys2/msys2-runtime/pull/88 and re-ran my test action, and it worked as expected this time. I've put out a call to test on the msys2-runtime issue to confirm that it also solves the real-world cases. -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: pipe hang issue when running as SYSTEM
On Fri, 25 Mar 2022 10:01:53 -0700 (PDT) Jeremy Drake wrote: > On Fri, 25 Mar 2022, Takashi Yano wrote: > > > I will submit v3 patch shortly. > > I applied your v3 patch to 3.3.4 in > https://github.com/msys2/msys2-runtime/pull/88 and re-ran my test action, > and it worked as expected this time. > > I've put out a call to test on the msys2-runtime issue to confirm that it > also solves the real-world cases. I have submitted v4 patch, which fixes the issue caused when multiple pipes in the same process group exist. -- Takashi Yano -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple