Same happened again: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4733/console
Should I kill -3? ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: u...@thetaphi.de From: dawid.we...@gmail.com [mailto:dawid.we...@gmail.com] On Behalf Of Dawid Weiss Sent: Sunday, June 22, 2014 10:03 PM To: dev@lucene.apache.org Subject: Re: SSLMigration test hangs on FreeBSD blackhole This looks very weird because there's no actual "test" thread running. And the runner's thread is hung on readBytes (?)... This is very suspicious. Dawid On Sun, Jun 22, 2014 at 4:46 PM, Uwe Schindler <u...@thetaphi.de> wrote: I requested a stack dump: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/565/console [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp/junit4-J0-20140621_125257_468.sysout [junit4] >>> JVM J0: stdout (verbatim) ---- [junit4] 2014-06-22 14:44:03 [junit4] Full thread dump OpenJDK 64-Bit Server VM (24.60-b09 mixed mode): [junit4] [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e6800 nid=0x846b95c00 runnable [0x00007ffff6e70000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] at java.net.PlainSocketImpl.socketAccept(Native Method) [junit4] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) [junit4] at java.net.ServerSocket.implAccept(ServerSocket.java:530) [junit4] at java.net.ServerSocket.accept(ServerSocket.java:498) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "RMI RenewClean-[127.0.0.1:40915]" daemon prio=5 tid=0x00000008517df000 nid=0x83c049800 in Object.wait() [0x00007ffff2426000] [junit4] java.lang.Thread.State: TIMED_WAITING (on object monitor) [junit4] at java.lang.Object.wait(Native Method) [junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) [junit4] - locked <0x000000080de40760> (a java.lang.ref.ReferenceQueue$Lock) [junit4] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "RMI Scheduler(0)" daemon prio=5 tid=0x00000008033e2800 nid=0x84abf0c00 waiting on condition [0x00007fffec5c8000] [junit4] java.lang.Thread.State: TIMED_WAITING (parking) [junit4] at sun.misc.Unsafe.park(Native Method) [junit4] - parking to wait for <0x000000080de3e008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) [junit4] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) [junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) [junit4] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) [junit4] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) [junit4] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "GC Daemon" daemon prio=5 tid=0x00000008033e4000 nid=0x837d98400 in Object.wait() [0x00007ffff090b000] [junit4] java.lang.Thread.State: TIMED_WAITING (on object monitor) [junit4] at java.lang.Object.wait(Native Method) [junit4] at sun.misc.GC$Daemon.run(GC.java:117) [junit4] - locked <0x000000080de434a8> (a sun.misc.GC$LatencyLock) [junit4] [junit4] "RMI Reaper" prio=5 tid=0x00000008033e3800 nid=0x83c050400 in Object.wait() [0x00007ffff3638000] [junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4] at java.lang.Object.wait(Native Method) [junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) [junit4] - locked <0x000000080de36d38> (a java.lang.ref.ReferenceQueue$Lock) [junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) [junit4] at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e2000 nid=0x850ca1800 runnable [0x00007ffff5557000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] at java.net.PlainSocketImpl.socketAccept(Native Method) [junit4] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) [junit4] at java.net.ServerSocket.implAccept(ServerSocket.java:530) [junit4] at java.net.ServerSocket.accept(ServerSocket.java:498) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e1000 nid=0x8454da800 runnable [0x00007fffddde1000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] at java.net.PlainSocketImpl.socketAccept(Native Method) [junit4] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) [junit4] at java.net.ServerSocket.implAccept(ServerSocket.java:530) [junit4] at java.net.ServerSocket.accept(ServerSocket.java:498) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388) [junit4] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360) [junit4] at java.lang.Thread.run(Thread.java:745) [junit4] [junit4] "Service Thread" daemon prio=5 tid=0x0000000801256800 nid=0x801183800 runnable [0x0000000000000000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] [junit4] "C2 CompilerThread1" daemon prio=5 tid=0x0000000801255800 nid=0x801182c00 waiting on condition [0x0000000000000000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] [junit4] "C2 CompilerThread0" daemon prio=5 tid=0x0000000801255000 nid=0x801049c00 waiting on condition [0x0000000000000000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] [junit4] "Signal Dispatcher" daemon prio=5 tid=0x0000000801254000 nid=0x801049000 waiting on condition [0x0000000000000000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] [junit4] "Finalizer" daemon prio=5 tid=0x0000000801253800 nid=0x801047c00 in Object.wait() [0x00007ffffebed000] [junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4] at java.lang.Object.wait(Native Method) [junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) [junit4] - locked <0x000000080be6bee0> (a java.lang.ref.ReferenceQueue$Lock) [junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) [junit4] at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) [junit4] [junit4] "Reference Handler" daemon prio=5 tid=0x0000000801252800 nid=0x801047000 in Object.wait() [0x00007ffffecee000] [junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4] at java.lang.Object.wait(Native Method) [junit4] at java.lang.Object.wait(Object.java:503) [junit4] at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) [junit4] - locked <0x000000080be84ac8> (a java.lang.ref.Reference$Lock) [junit4] [junit4] "main" prio=5 tid=0x0000000801252000 nid=0x801007c00 runnable [0x00007fffffbfd000] [junit4] java.lang.Thread.State: RUNNABLE [junit4] at java.io.FileInputStream.readBytes(Native Method) [junit4] at java.io.FileInputStream.read(FileInputStream.java:272) [junit4] at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) [junit4] at java.io.BufferedInputStream.read(BufferedInputStream.java:334) [junit4] - locked <0x000000080be8cd38> (a java.io.BufferedInputStream) [junit4] at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) [junit4] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) [junit4] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) [junit4] - locked <0x000000080bef6c00> (a java.io.InputStreamReader) [junit4] at java.io.InputStreamReader.read(InputStreamReader.java:184) [junit4] at java.io.BufferedReader.fill(BufferedReader.java:154) [junit4] at java.io.BufferedReader.readLine(BufferedReader.java:317) [junit4] - locked <0x000000080bef6c00> (a java.io.InputStreamReader) [junit4] at java.io.BufferedReader.readLine(BufferedReader.java:382) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.StdInLineIterator.computeNext(StdInLineIterator.java:31) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.StdInLineIterator.computeNext(StdInLineIterator.java:13) [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) [junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.Iterators$5.hasNext(Iterators.java:542) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:169) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:276) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12) [junit4] [junit4] "VM Thread" prio=5 tid=0x00000008013d1800 nid=0x801046400 runnable [junit4] [junit4] "GC task thread#0 (ParallelGC)" prio=5 tid=0x00000008013c8800 nid=0x80103c000 runnable [junit4] [junit4] "GC task thread#1 (ParallelGC)" prio=5 tid=0x00000008013c9800 nid=0x80103cc00 runnable [junit4] [junit4] "GC task thread#2 (ParallelGC)" prio=5 tid=0x00000008013ca000 nid=0x80103d800 runnable [junit4] [junit4] "GC task thread#3 (ParallelGC)" prio=5 tid=0x00000008013ca800 nid=0x80103e400 runnable [junit4] [junit4] "GC task thread#4 (ParallelGC)" prio=5 tid=0x00000008013cb000 nid=0x80103f000 runnable [junit4] [junit4] "GC task thread#5 (ParallelGC)" prio=5 tid=0x00000008013cc000 nid=0x80103fc00 runnable [junit4] [junit4] "GC task thread#6 (ParallelGC)" prio=5 tid=0x00000008013cc800 nid=0x801040800 runnable [junit4] [junit4] "GC task thread#7 (ParallelGC)" prio=5 tid=0x00000008013cd000 nid=0x801041400 runnable [junit4] [junit4] "GC task thread#8 (ParallelGC)" prio=5 tid=0x00000008013cd800 nid=0x801042000 runnable [junit4] [junit4] "GC task thread#9 (ParallelGC)" prio=5 tid=0x00000008013ce800 nid=0x801042c00 runnable [junit4] [junit4] "GC task thread#10 (ParallelGC)" prio=5 tid=0x00000008013cf000 nid=0x801043800 runnable [junit4] [junit4] "GC task thread#11 (ParallelGC)" prio=5 tid=0x00000008013cf800 nid=0x801044400 runnable [junit4] [junit4] "GC task thread#12 (ParallelGC)" prio=5 tid=0x00000008013d0000 nid=0x801045000 runnable [junit4] [junit4] "VM Periodic Task Thread" prio=5 tid=0x00000008013d2000 nid=0x801184800 waiting on condition [junit4] [junit4] JNI global references: 367 [junit4] [junit4] Heap [junit4] PSYoungGen total 166400K, used 3708K [0x0000000820e00000, 0x000000082b300000, 0x000000082b900000) [junit4] eden space 163840K, 2% used [0x0000000820e00000,0x000000082119f1d8,0x000000082ae00000) [junit4] from space 2560K, 0% used [0x000000082b080000,0x000000082b080000,0x000000082b300000) [junit4] to space 2560K, 0% used [0x000000082ae00000,0x000000082ae00000,0x000000082b080000) [junit4] ParOldGen total 302592K, used 130308K [0x000000080b900000, 0x000000081e080000, 0x0000000820e00000) [junit4] object space 302592K, 43% used [0x000000080b900000,0x0000000813841278,0x000000081e080000) [junit4] PSPermGen total 55808K, used 55612K [0x0000000806700000, 0x0000000809d80000, 0x000000080b900000) [junit4] object space 55808K, 99% used [0x0000000806700000,0x0000000809d4f160,0x0000000809d80000) [junit4] <<< JVM J0: EOF ---- [junit4] JVM J1: 1.21 .. 6848.88 = 6847.67s [junit4] Execution time total: 1 day 1 hour 51 minutes 34 seconds ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Uwe Schindler [mailto:u...@thetaphi.de] > Sent: Sunday, June 22, 2014 11:51 AM > To: dev@lucene.apache.org > Subject: SSLMigration test hangs on FreeBSD blackhole > > Hi, > > in every run the SSLMigrationTest hung on ASF Jenkins, where we have the > FreeBSD blackhole. It is not even cancelled by the Test Runner, it hangs > forever: https://builds.apache.org/job/Lucene-Solr-NightlyTests- > 4.x/565/console > We did not see this error before, because the ASF Jenkins was offline for > about a month. > It looks like there is a timeout missing. For those who don't know: On > FreeBSD the blackhole causes even for connection to localhost: if the port is > not bound it timeouts when connecting (instead of connection error). If no > timeout is set, it takes forever. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional > commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
