[ https://issues.apache.org/jira/browse/HBASE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258038#comment-16258038 ]
Chia-Ping Tsai edited comment on HBASE-19204 at 11/18/17 11:55 AM: ------------------------------------------------------------------- It is blocked when closing {{HttpServer2}}. Zoom in on the {{HttpServer2}} {code} @Test public void testHttpServerHang() throws Exception { for (int i = 0; i != 500; ++i) { HttpServer2 server = null; try { Configuration conf = new Configuration(); conf.setInt("hadoop.http.max.threads", 10); server = createTestServer(conf); server.addServlet("echo", "/echo", TestHttpServer.EchoServlet.class); server.addServlet("echomap", "/echomap", TestHttpServer.EchoMapServlet.class); server.addServlet("htmlcontent", "/htmlcontent", TestHttpServer.HtmlContentServlet.class); server.addServlet("longheader", "/longheader", TestHttpServer.LongHeaderServlet.class); server.addJerseyResourcePackage( JerseyResource.class.getPackage().getName(), "/jersey/*"); server.start(); } finally { if (server != null) { server.stop(); } } } } {code} And then i got the following stack dump. {quote} "2020688994@qtp-1069352689-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39279" daemon prio=10 tid=0x00007fbd8c7a4000 nid=0x1c3e waiting for monitor entry [0x00007fbd740e1000] java.lang.Thread.State: BLOCKED (on object monitor) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:464) - waiting to lock <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) "main" prio=10 tid=0x00007fbd8c009000 nid=0x1bdc runnable [0x00007fbd93ad5000] java.lang.Thread.State: RUNNABLE at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:878) - locked <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.io.nio.SelectorManager.doStop(SelectorManager.java:240) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000ec07da28> (a java.lang.Object) at org.mortbay.jetty.nio.SelectChannelConnector.close(SelectChannelConnector.java:136) {quote} And then i noticed the same observation is already in the HBASE-14430 and HBASE-14505. [~stack] Do you still look for the cryptic JVM version? :) was (Author: chia7712): It is blocked when closing {{HttpServer2}}. Zoom in on the {{HttpServer2}} {code} @Test public void testHttpServerHang() throws Exception { for (int i = 0; i != 500; ++i) { HttpServer2 server = null; try { Configuration conf = new Configuration(); conf.setInt("hadoop.http.max.threads", 10); server = createTestServer(conf); server.addServlet("echo", "/echo", TestHttpServer.EchoServlet.class); server.addServlet("echomap", "/echomap", TestHttpServer.EchoMapServlet.class); server.addServlet("htmlcontent", "/htmlcontent", TestHttpServer.HtmlContentServlet.class); server.addServlet("longheader", "/longheader", TestHttpServer.LongHeaderServlet.class); server.addJerseyResourcePackage( JerseyResource.class.getPackage().getName(), "/jersey/*"); server.start(); } finally { if (server != null) { server.stop(); } } } } {code} And then i got the following stack dump. {quote} "2020688994@qtp-1069352689-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39279" daemon prio=10 tid=0x00007fbd8c7a4000 nid=0x1c3e waiting for monitor entry [0x00007fbd740e1000] java.lang.Thread.State: BLOCKED (on object monitor) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:464) - waiting to lock <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) "main" prio=10 tid=0x00007fbd8c009000 nid=0x1bdc runnable [0x00007fbd93ad5000] java.lang.Thread.State: RUNNABLE at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:878) - locked <0x00000000ec24c798> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.io.nio.SelectorManager.doStop(SelectorManager.java:240) at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76) - locked <0x00000000ec07da28> (a java.lang.Object) at org.mortbay.jetty.nio.SelectChannelConnector.close(SelectChannelConnector.java:136) {quote} And then i noticed the same observation is already in the HBASE-14430 and HBASE-14505. [~stack] Are you still look for the cryptic JVM version? :) > branch-1.2 times out and is taking 6-7 hours to complete > -------------------------------------------------------- > > Key: HBASE-19204 > URL: https://issues.apache.org/jira/browse/HBASE-19204 > Project: HBase > Issue Type: Umbrella > Components: test > Reporter: stack > > Sean has been looking at tooling and infra. This Umbrellas is about looking > at actual tests. For example, running locally on dedicated machine I picked a > random test, TestPerColumnFamilyFlush. In my test run, it wrote 16M lines. It > seems to be having zk issues but it is catching interrupts and ignoring them > ([~carp84] fixed this in later versions over in HBASE-18441). > Let me try and do some fixup under this umbrella so we can get a 1.2.7 out > the door. -- This message was sent by Atlassian JIRA (v6.4.14#64029)