[I] [BUG] Shenyu-Bootstrap Pod Restart Issue: Port 9195 Already in Use and JVM Not Exiting Properly [shenyu]</span></a></span> </h1> <p class="darkgray font13"> <span class="sender pipe"><a href="/search?l=notifications@shenyu.apache.org&q=from:%22via+GitHub%22" rel="nofollow"><span itemprop="author" itemscope itemtype="http://schema.org/Person"><span itemprop="name">via GitHub</span></span></a></span> <span class="date"><a href="/search?l=notifications@shenyu.apache.org&q=date:20250326" rel="nofollow">Wed, 26 Mar 2025 20:41:25 -0700</a></span> </p> </div> <div itemprop="articleBody" class="msgBody"> <!--X-Body-of-Message--> <pre> fddOs opened a new issue, #5978: URL: <a rel="nofollow" href="https://github.com/apache/shenyu/issues/5978">https://github.com/apache/shenyu/issues/5978</a></pre><pre> ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior ### Description: When deploying `shenyu-bootstrap` on Kubernetes, after accessing the liveness probe `/actuator/health/liveness` multiple times and injecting errors to simulate a pod restart, there is a probability of encountering the following error: ```java org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.PortInUseException: Port 9195 is already in use ``` Additionally, when simulating a shutdown of `shenyu-bootstrap` locally using `/actuator/shutdown`, the JVM does not exit properly. A `jstack` output reveals several threads in a `WAITING` or `TIMED_WAITING` state, indicating that the JVM is not shutting down cleanly. ### Steps to Reproduce: 1. Deploy `shenyu-bootstrap` on Kubernetes. 2. Access the liveness probe `/actuator/health/liveness` multiple times. 3. Inject errors to simulate a pod restart. 4. Observe the error `Port 9195 is already in use` in the logs. 5. Locally, simulate a shutdown using `/actuator/shutdown`. 6. Check the JVM status using `jstack` and observe that the JVM does not exit properly. ### Expected Behavior: - The pod should restart without encountering the `PortInUseException`. - The JVM should exit cleanly when the `/actuator/shutdown` endpoint is called. ### Actual Behavior: - There is a probability of encountering the `PortInUseException` during pod restart. - The JVM does not exit properly, and several threads remain in a `WAITING` or `TIMED_WAITING` state. ### Environment: - **Shenyu Version**: 2.6.0 ### Additional Information: - **Thread Dump**: Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode): ```java "logback-2" #69 daemon prio=5 os_prio=0 tid=0x000000002dfe0000 nid=0x6acc waiting on condition [0x000000004244e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c2215a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "boundedElastic-1" #47 daemon prio=5 os_prio=0 tid=0x000000002dfe6000 nid=0x3094 waiting on condition [0x000000004083e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000074b6ede68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "DestroyJavaVM" #45 prio=5 os_prio=0 tid=0x000000002dfe2000 nid=0x686c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Thread-12" #34 daemon prio=5 os_prio=0 tid=0x000000003798c800 nid=0x663c runnable [0x00000000359bf000] java.lang.Thread.State: RUNNABLE at sun.net.dns.ResolverConfigurationImpl.notifyAddrChange0(Native Method) at sun.net.dns.ResolverConfigurationImpl$AddressChangeListener.run(ResolverConfigurationImpl.java:144) "shenyu-plugin-ext-loader-3" #33 daemon prio=5 os_prio=0 tid=0x000000003798e000 nid=0x3f40 waiting on condition [0x00000000356be000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c3996f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "shenyu-shared_wheel_timer-2" #31 prio=5 os_prio=0 tid=0x000000003798f800 nid=0x34ec waiting on condition [0x00000000355be000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c39c0158> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at org.apache.shenyu.common.timer.HierarchicalWheelTimer.advanceClock(HierarchicalWheelTimer.java:116) at org.apache.shenyu.common.timer.HierarchicalWheelTimer$Worker.run(HierarchicalWheelTimer.java:168) at java.lang.Thread.run(Thread.java:748) "boundedElastic-evictor-1" #29 daemon prio=5 os_prio=0 tid=0x000000002c66b000 nid=0x4184 waiting on condition [0x00000000300ee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c36b8f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "shenyu-Shenyu-Memory-Calculator--1" #27 prio=5 os_prio=0 tid=0x000000002c68d800 nid=0x6740 waiting on condition [0x000000002ffee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c39c06e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "logback-1" #24 daemon prio=5 os_prio=0 tid=0x0000000029b93000 nid=0x3740 waiting on condition [0x000000002da4e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c2215a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) "AsyncAppender-Worker-ASYNC_ERROR_FILE" #22 daemon prio=5 os_prio=0 tid=0x000000002bea5000 nid=0x43b0 waiting on condition [0x000000002d6be000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c2269e30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289) "AsyncAppender-Worker-ASYNC_FILE" #21 daemon prio=5 os_prio=0 tid=0x000000002bf71800 nid=0x15bc waiting on condition [0x000000002d5bf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c226f260> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289) "AsyncAppender-Worker-ASYNC_STDOUT" #20 daemon prio=5 os_prio=0 tid=0x0000000029cd6800 nid=0x6318 waiting on condition [0x000000002d4bf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c2233cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289) "Service Thread" #12 daemon prio=9 os_prio=0 tid=0x0000000029749000 nid=0x6b54 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread2" #11 daemon prio=9 os_prio=2 tid=0x00000000296ea800 nid=0x5bd8 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #10 daemon prio=9 os_prio=2 tid=0x00000000296e6000 nid=0x57dc waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #9 daemon prio=9 os_prio=2 tid=0x00000000296e3000 nid=0x3594 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Command Reader" #8 daemon prio=10 os_prio=0 tid=0x0000000027ed3000 nid=0x444c runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x0000000027ed1000 nid=0x3454 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x0000000027ec7000 nid=0x5ef4 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000026515000 nid=0x2e30 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000027ec6800 nid=0x3110 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000264f2000 nid=0x45f8 in Object.wait() [0x0000000027e6e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000005c222c488> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000005c222c488> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000264ca800 nid=0x43cc in Object.wait() [0x0000000027d6f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000005c23d3ce0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000005c23d3ce0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "VM Thread" os_prio=2 tid=0x00000000264c9000 nid=0x69a4 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000003668000 nid=0x2b78 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000003669800 nid=0x1e24 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000000000366b000 nid=0x6824 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000000000366e000 nid=0x5254 runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000003670000 nid=0x2294 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000003671000 nid=0x59d4 runnable "VM Periodic Task Thread" os_prio=2 tid=0x000000002974e000 nid=0x608c waiting on condition JNI global references: 32619 ``` ### Expected Behavior _No response_ ### Steps To Reproduce _No response_ ### Environment ```markdown ShenYu version(s): ``` ### Debug logs _No response_ ### Anything else? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@shenyu.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org </pre> </div> <div class="msgButtons margintopdouble"> <ul class="overflow"> <li class="msgButtonItems"><a class="button buttonleft " accesskey="p" href="msg32838.html">Previous message</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="c" href="index.html#32841">View by thread</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="i" href="maillist.html#32841">View by date</a></li> <li class="msgButtonItems textalignright"><a class="button buttonright " accesskey="n" href="msg32847.html">Next message</a></li> </ul> </div> <a name="tslice"></a> <div class="tSliceList margintopdouble"> <ul class="icons monospace"> <li class="icons-email tSliceCur"><span class="subject">[I] [BUG] <title>Shenyu-Bootstrap Pod Restart Issue: Port...</span> <span class="sender italic">via GitHub</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg32847.html">Re: [I] [BUG] <title>Shenyu-Bootstrap Pod Restart Is...</a></span> <span class="sender italic">via GitHub</span></li> <li class="icons-email"><span class="subject"><a href="msg32895.html">Re: [I] [BUG] <title>Shenyu-Bootstrap Pod Restart Is...</a></span> <span class="sender italic">via GitHub</span></li> <li class="icons-email"><span class="subject"><a href="msg32913.html">Re: [I] [BUG] <title>Shenyu-Bootstrap Pod Restart Is...</a></span> <span class="sender italic">via GitHub</span></li> </ul> </ul> </div> <div class="overflow msgActions margintopdouble"> <div class="msgReply" > <h2> Reply via email to </h2> <form method="POST" action="/mailto.php"> <input type="hidden" name="subject" value="[I] [BUG] <title>Shenyu-Bootstrap Pod Restart Issue: Port 9195 Already in Use and JVM Not Exiting Properly [shenyu]"> <input type="hidden" name="msgid" value="I_kwDOCGCHjs6v6nDW@gitbox.apache.org"> <input type="hidden" name="relpath" value="notifications@shenyu.apache.org/msg32841.html"> <input type="submit" value=" via GitHub "> </form> </div> </div> </div> <div class="aside" role="complementary"> <div class="logo"> <a href="/"><img src="/logo.png" width=247 height=88 alt="The Mail Archive"></a> </div> <form class="overflow" action="/search" method="get"> <input type="hidden" name="l" value="notifications@shenyu.apache.org"> <label class="hidden" for="q">Search the site</label> <input class="submittext" type="text" id="q" name="q" placeholder="Search notifications"> <input class="submitbutton" name="submit" type="image" src="/submit.png" alt="Submit"> </form> <div class="nav margintop" id="nav" role="navigation"> <ul class="icons font16"> <li class="icons-home"><a href="/">The Mail Archive home</a></li> <li class="icons-list"><a href="/notifications@shenyu.apache.org/">notifications - all messages</a></li> <li class="icons-about"><a href="/notifications@shenyu.apache.org/info.html">notifications - about the list</a></li> <li class="icons-expand"><a href="/search?l=notifications@shenyu.apache.org&q=subject:%22%5C%5BI%5C%5D+%5C%5BBUG%5C%5D+%3Ctitle%3EShenyu%5C-Bootstrap+Pod+Restart+Issue%5C%3A+Port+9195+Already+in+Use+and+JVM+Not+Exiting+Properly+%5C%5Bshenyu%5C%5D%22&o=newest&f=1" title="e" id="e">Expand</a></li> <li class="icons-prev"><a href="msg32838.html" title="p">Previous message</a></li> <li class="icons-next"><a href="msg32847.html" title="n">Next message</a></li> </ul> </div> <div class="listlogo margintopdouble"> </div> <div class="margintopdouble"> </div> </div> </div> <div class="footer" role="contentinfo"> <ul> <li><a href="/">The Mail Archive home</a></li> <li><a href="/faq.html#newlist">Add your mailing list</a></li> <li><a href="/faq.html">FAQ</a></li> <li><a href="/faq.html#support">Support</a></li> <li><a href="/faq.html#privacy">Privacy</a></li> <li class="darkgray">I_kwDOCGCHjs6v6nDW@gitbox.apache.org</li> </ul> </div> </body> </html> <script>(function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9ecadbe64897cf88',t:'MTc3NjI1NTE1MQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();</script>