[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17865715#comment-17865715 ] Gary D. Gregory commented on DAEMON-460: Hi [~markt] Did you get a chance for another look? > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png, test-windows-service.zip > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856812#comment-17856812 ] Mark Thomas commented on DAEMON-460: Thanks. That all makes sense. I'll take a look and see what our options are. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png, test-windows-service.zip > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[
https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856781#comment-17856781
]
Mark Linley commented on DAEMON-460:
[~markt],
I had a deeper look at this yesterday. I ran remote debug sessions with both
"Tomcat 10" and our problematic Java windows service that uses 100% of a CPU
core. The "Tomcat 10" service did not suffer from the same high CPU core usage
problem. Both services were using the exact same JVM (openjdk version
"11.0.14.1" 2022-02-08) and prunsrv.exe v1.3.4.
Something that caught my attention was a difference in the service log between
each of the two services. Shortly after service startup, our problematic Java
service has this line in the service log:
{quote}[2024-06-21 10:00:29] [debug] ( javajni.c:1092) [11744] Java worker
thread finished ourservice/util/OurService:main with status = 0
{quote}
This didn't make sense to me because the app was still running and functioning
seemingly normally despite the single CPU core being at 100%. This log
statement is not present in the "Tomcat 10" service log until you stop the
Tomcat10 windows service.
Then I thought about it, and it occurred to me that in our case our startup
'main' method will spawn some application worker threads that our app needs to
function and it then returns, i.e. it doesn't block. When stopping the windows
service, The OurService::stop() method will then signal to the app threads to
shutdown and our application then terminates normally.
The problem I think is that on the prunsrv.exe native C java worker thread side
it likely detects that the JNI method call to OurService::main has terminated
and therefore our application has terminated, so it places the prunsrv.exe
native C java worker thread into state 0, which will trigger the tight loop we
are seeing. I highlighted the line in a code snippet above (line 1165 above).
I tested this using a simple java application and it appears to confirm my
suspicions.
When the java start main() method blocks, processing in a while loop waiting on
a boolean condition and doing work every 5 seconds, there is
low/normal/expected CPU usage. The stop() method will then change a static
boolean variable which the loop is monitoring, and it will then terminate
normally.
However, when the java service's main() start method spawns a thread to do work
asynchronously and immediately returns, then the CPU core immediately sits at
100%. You then see prunsrv.exe detecting that and putting it's native java
worker thread in state=0, triggering the tight loop:
{quote}[2024-06-21 11:13:39] [debug] ( javajni.c:1067) [ 7928] Java worker
thread started for test/commons/daemon/TestWindowsService:main
[2024-06-21 11:13:39] [debug] ( javajni.c:1073) [ 7928] JNI calling static void
method test/commons/daemon/TestWindowsService:main
*[2024-06-21 11:13:39] [debug] ( javajni.c:1092) [ 7928] Java worker thread
finished test/commons/daemon/TestWindowsService:main with status = 0*
[2024-06-21 11:13:40] [debug] ( prunsrv.c:1490) [ 2084] Java started
'test/commons/daemon/TestWindowsService'.
[2024-06-21 11:13:40] [info] ( prunsrv.c:1589) [ 2084] Service started in 1096
milliseconds.
{quote}
I have a test app you can unzip to c:\temp attached [^test-windows-service.zip].
You can then run this command to register the Windows service and start it to
test:
{quote}prunsrv.exe //IS//TestService --Startup=auto --DisplayName="Test
Service" --Description="This is a test Java service"
--Install="C:\temp\test-windows-service\prunsrv.exe"
--Classpath="C:\temp\test-windows-service"
--Jvm="C:\java11jdk\bin\client\jvm.dll" --StartMode=jvm --StopTimeout=30
--StopMode=jvm --StartClass=test/commons/daemon/TestWindowsService
--StopMethod=stop --StopClass=test/commons/daemon/TestWindowsService
--StartMethod=main --LogPath="C:\temp\test-windows-service"
--LogPrefix=Test-Service-Log --LogLevel=Debug --JvmMs=128 --JvmMx=256
--JvmOptions=-Dfile.encoding=UTF8
{quote}
Hopefully this helps recreate the issue on your side. Let me know if you need
anything else.
Thanks
Mark
> High CPU usage in prunsrv.exe since Daemon 1.3.3
>
>
> Key: DAEMON-460
> URL: https://issues.apache.org/jira/browse/DAEMON-460
> Project: Commons Daemon
> Issue Type: Bug
> Components: prunsrv
>Affects Versions: 1.3.3
>Reporter: Japie vd Linde
>Priority: Major
> Attachments: EspRun-Service-Log.2023-06-05.log,
> image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png,
> image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png,
> image-2024-05-31-10-00-10-916.png, test-windows-service.zip
>
>
> When using the --StopTimeout=30 parameter on service using prunsrv the CPU
> usage is reported as very high on Windows. Rolling back to older
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856488#comment-17856488 ] Mark Thomas commented on DAEMON-460: I've reviewed the code and I don't see a way for the code can enter the tight loop while the service is running. We really need a reproducible test case to investigate this further. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856288#comment-17856288 ] Mark Thomas commented on DAEMON-460: I'd prefer to see if we can fix this issue and keep the force stop capability. I'll try and find some time to look at this in the next few weeks. As per a previous comment, a simple test case that demonstrates the issue would help as the last time I looked at this using Tomcat, I was unable to recreate the issue. A combination of settings that triggers the issue with Tomcat would be sufficient. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856255#comment-17856255 ] Gary D. Gregory commented on DAEMON-460: [~plasm0r], Thank you for your analysis. [~markt], Should we revert the PR? Which is worse, Daemon with the PR or without it? > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856254#comment-17856254 ] Gary D. Gregory commented on DAEMON-460: A big picture item is that we don't have unit tests for C code. If we had at least one, we could say "No main changes without a matching unit test", which is what I usually say to Java changes. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png, > image-2024-05-31-10-00-10-916.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850467#comment-17850467 ] Mark Linley commented on DAEMON-460: Hi Our Java application is managed as a Windows service by Apache Commons Daemon. Upgrading to v1.3.4 showed what others have been seeing. One of the CPU cores constantly sits at close to 100% CPU utilization. I was able to successfully configure Visual Studio 2022 to do remote debugging of prunsrv.exe v1.3.4. I recorded the CPU profile in Visual Studio once the debugger was connected to the remote prunsrv.exe process. I could see one of the threads of prunsrv.exe consuming almost 100% of the CPU. I looked at the top function, based on CPU usage, and it was prunsrv.c::apxJavaWait. With a breakpoint activated, my attention was drawn to this code in prunsrv.c::serviceMain : !image-2024-05-29-15-56-35-585.png! I could see in my debug session that the wait of 2 seconds is definitely not happening, resulting in the loop iterating as fast as the CPU core will allow it to, which likely explains the 100% CPU utilization issue. Pull request 64, mentioned above, did add this loop so it could be related to the problem we are seeing. Stepping down into the callstack in apxHandlerWait you arrive at the method prunsrv.c::apxJavaWait that received the timeout value of 2000 milliseconds but the timeout value is effectively never used because the code keeps returning here: !image-2024-05-29-15-57-37-665.png! If someone more experienced in C or one of the maintainers could comment, I'd appreciate it. I'm a Java developer :) Thanks! Mark > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png, > image-2024-05-29-15-56-35-585.png, image-2024-05-29-15-57-37-665.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846325#comment-17846325 ] Mark Thomas commented on DAEMON-460: This is likely caused by this PR [https://github.com/apache/commons-daemon/pull/64] but I don't see any obvious cause. I can't repeat this issue with Tomcat - my default test application for any Daemon issue. Please provide the simplest possible test case that demonstrates this issue. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833172#comment-17833172 ] Daniel commented on DAEMON-460: --- Hi, we noticed the same problem with v1.3.4 and downgraded to v1.3.2. One CPU core is fully used by prunsrv.exe. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793663#comment-17793663 ] Japie vd Linde commented on DAEMON-460: --- Hi As an interim solution we used the prunsrv.exe v1.3.2 instead of prunsrv.exe v1.3.3 with the apache-daemon.jar from v1.3.3. > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793624#comment-17793624 ] Magnus Palmér commented on DAEMON-460: -- We have noticed this as well, high CPU usage after upgrading to 1.3.4 (from latest 1.2.x) > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739918#comment-17739918 ] Gary D. Gregory commented on DAEMON-460: I don't have anything. You might want to try and see what is happening for your particular setup through a debugger or monitoring tool. Our sources are on GitHub ;) > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739913#comment-17739913 ] Japie vd Linde commented on DAEMON-460: --- Hi, just wanted to check if there is any additional information required to do an assessment of the issue? > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAEMON-460) High CPU usage in prunsrv.exe since Daemon 1.3.3
[ https://issues.apache.org/jira/browse/DAEMON-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737669#comment-17737669 ] Gary D. Gregory commented on DAEMON-460: Where are we on this ticket? > High CPU usage in prunsrv.exe since Daemon 1.3.3 > > > Key: DAEMON-460 > URL: https://issues.apache.org/jira/browse/DAEMON-460 > Project: Commons Daemon > Issue Type: Bug > Components: prunsrv >Affects Versions: 1.3.3 >Reporter: Japie vd Linde >Priority: Major > Attachments: EspRun-Service-Log.2023-06-05.log, > image-2023-05-31-09-31-21-485.png, image-2023-06-05-13-38-38-435.png > > > When using the --StopTimeout=30 parameter on service using prunsrv the CPU > usage is reported as very high on Windows. Rolling back to older prunsrv > seems to resolve the problem. > !image-2023-05-31-09-31-21-485.png! > What could be the possible causes for this problem? -- This message was sent by Atlassian Jira (v8.20.10#820010)
