Not sure of the right place to bring this up, so figured I'd start here with the mailing list. At the very least this information may prove useful to anyone else that may encounter these issues. (Solution is to use Tomcat 8)
I've been experiencing frequent Tomcat core dumps during heavy re-indexes of SOLR running on Tomcat 7 servers. Approx. 815k documents, comprised mostly of the Enron email data set. This issue has been present across the past few releases of Java 7, Tomcat 7 & SOLR. Current version details: Java: 1.7.0_67 Tomcat: 7.0.55 SOLR: 4.10.0 ( also 4.9.0) Essentially the JRE frequently crashes during a full re-index within the tcnative-1.dll frame under org.apache.tomcat.jni.Poll.poll. We're using a messaging queue set up with multiple threads, so the re-index process is slamming the SOLR server with many requests. This high load may be contributing to the crashes seen in the Poller. I'll include some of the core dump log at the bottom of this message. The good news, is that using the latest version of Tomcat 8 (8.0.12) seems to resolve this issue and a re-index runs without any core dumps. ========================= Core DUMP log (Partial): ========================= # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000180007e73, pid=4268, tid=1612 # # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode windows-amd64 compressed oops) # Problematic frame: # C [tcnative-1.dll+0x7e73] # # Failed to write core dump. Call to MiniDumpWriteDump() failed # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- T H R E A D --------------- Current thread (0x000000000d94b000): JavaThread "http-apr-8092-Poller" daemon [_thread_in_native, id=1612, stack(0x000000000e960000,0x000000000ea60000)] siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000 Stack: [0x000000000e960000,0x000000000ea60000], sp=0x000000000ea5f670, free space=1021k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [tcnative-1.dll+0x7e73] [error occurred during error reporting (printing native stack), id 0xc0000005] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 1752 org.apache.tomcat.jni.Poll.poll(JJ[JZ)I (0 bytes) @ 0x0000000001a3a182 [0x0000000001a3a120+0x62] J 2451% C2 org.apache.tomcat.util.net.AprEndpoint$Poller.run()V (2291 bytes) @ 0x0000000001dcc3ec [0x0000000001dcc220+0x1cc] j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x000000000d859800 JavaThread "commitScheduler-20-thread-1" [_thread_blocked, id=3300, stack(0x0000000039720000,0x0000000039820000)] 0x000000000d857800 JavaThread "commitScheduler-14-thread-1" [_thread_blocked, id=556, stack(0x0000000010760000,0x0000000010860000)] 0x000000000d856800 JavaThread "commitScheduler-13-thread-1" [_thread_blocked, id=1116, stack(0x0000000010660000,0x0000000010760000)] 0x000000000d855800 JavaThread "commitScheduler-36-thread-1" [_thread_blocked, id=1108, stack(0x0000000010560000,0x0000000010660000)] 0x000000000d948800 JavaThread "commitScheduler-25-thread-1" [_thread_blocked, id=3456, stack(0x0000000015370000,0x0000000015470000)] 0x000000000d855000 JavaThread "commitScheduler-29-thread-1" [_thread_blocked, id=1220, stack(0x0000000010cb0000,0x0000000010db0000)] 0x000000000d854000 JavaThread "http-apr-8092-exec-12" daemon [_thread_blocked, id=3240, stack(0x0000000019f90000,0x000000001a090000)] 0x000000000d853800 JavaThread "commitScheduler-33-thread-1" [_thread_blocked, id=252, stack(0x0000000019e90000,0x0000000019f90000)] 0x000000000d852800 JavaThread "http-apr-8092-exec-11" daemon [_thread_blocked, id=3584, stack(0x0000000019d90000,0x0000000019e90000)] 0x000000000d852000 JavaThread "http-apr-8092-exec-10" daemon [_thread_blocked, id=1812, stack(0x0000000019c90000,0x0000000019d90000)] 0x000000000d851000 JavaThread "http-apr-8092-exec-9" daemon [_thread_blocked, id=2860, stack(0x0000000019b90000,0x0000000019c90000)] 0x000000000d850800 JavaThread "http-apr-8092-exec-8" daemon [_thread_blocked, id=4584, stack(0x0000000019a90000,0x0000000019b90000)] 0x000000000d84f800 JavaThread "http-apr-8092-exec-7" daemon [_thread_blocked, id=1736, stack(0x0000000019990000,0x0000000019a90000)] 0x000000000d84f000 JavaThread "http-apr-8092-exec-6" daemon [_thread_blocked, id=1324, stack(0x0000000019890000,0x0000000019990000)] 0x000000000d84e000 JavaThread "http-apr-8092-exec-5" daemon [_thread_blocked, id=4036, stack(0x0000000019790000,0x0000000019890000)] 0x000000000d84d800 JavaThread "http-apr-8092-exec-4" daemon [_thread_blocked, id=2344, stack(0x0000000019690000,0x0000000019790000)] 0x000000000d84c800 JavaThread "http-apr-8092-exec-3" daemon [_thread_blocked, id=2368, stack(0x0000000019590000,0x0000000019690000)] 0x000000000d84c000 JavaThread "http-apr-8092-exec-2" daemon [_thread_blocked, id=5028, stack(0x0000000019490000,0x0000000019590000)] 0x000000000d94e000 JavaThread "http-apr-8092-exec-1" daemon [_thread_blocked, id=3304, stack(0x0000000019390000,0x0000000019490000)] 0x000000000d94d000 JavaThread "http-apr-8092-AsyncTimeout" daemon [_thread_blocked, id=1228, stack(0x0000000019290000,0x0000000019390000)] 0x000000000d94c800 JavaThread "http-apr-8092-Acceptor-0" daemon [_thread_in_native, id=3048, stack(0x0000000019190000,0x0000000019290000)] 0x000000000d94b800 JavaThread "http-apr-8092-Sendfile" daemon [_thread_blocked, id=3916, stack(0x000000000fb20000,0x000000000fc20000)] =>0x000000000d94b000 JavaThread "http-apr-8092-Poller" daemon [_thread_in_native, id=1612, stack(0x000000000e960000,0x000000000ea60000)] 0x000000000d94a000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=3552, stack(0x000000000e860000,0x000000000e960000)] 0x000000000d949800 JavaThread "Thread-2" daemon [_thread_blocked, id=5080, stack(0x0000000018e90000,0x0000000018f90000)] 0x000000000d948000 JavaThread "searcherExecutor-38-thread-1" [_thread_blocked, id=736, stack(0x0000000010120000,0x0000000010220000)] 0x000000000d947000 JavaThread "searcherExecutor-31-thread-1" [_thread_blocked, id=1720, stack(0x0000000010460000,0x0000000010560000)] 0x000000000d6ef000 JavaThread "searcherExecutor-23-thread-1" [_thread_blocked, id=2332, stack(0x0000000010360000,0x0000000010460000)] 0x000000000f5d4000 JavaThread "searcherExecutor-27-thread-1" [_thread_blocked, id=4340, stack(0x0000000010260000,0x0000000010360000)] 0x000000000dda4000 JavaThread "searcherExecutor-22-thread-1" [_thread_blocked, id=3868, stack(0x0000000010020000,0x0000000010120000)] 0x000000000f271800 JavaThread "searcherExecutor-18-thread-1" [_thread_blocked, id=4192, stack(0x000000000ff20000,0x0000000010020000)] 0x000000000d362800 JavaThread "searcherExecutor-7-thread-1" [_thread_blocked, id=5100, stack(0x000000000fe20000,0x000000000ff20000)] 0x000000000e280000 JavaThread "searcherExecutor-6-thread-1" [_thread_blocked, id=4608, stack(0x000000000fd20000,0x000000000fe20000)] 0x000000000db5c000 JavaThread "searcherExecutor-8-thread-1" [_thread_blocked, id=1256, stack(0x000000000fc20000,0x000000000fd20000)] 0x000000000cc80000 JavaThread "GC Daemon" daemon [_thread_blocked, id=1276, stack(0x000000000e2d0000,0x000000000e3d0000)] 0x000000000ac4d000 JavaThread "Service Thread" daemon [_thread_blocked, id=992, stack(0x000000000c350000,0x000000000c450000)] 0x000000000ac4c000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=1536, stack(0x000000000c250000,0x000000000c350000)] 0x000000000ac47000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3248, stack(0x000000000c150000,0x000000000c250000)] 0x000000000ac45800 JavaThread "Attach Listener" daemon [_thread_blocked, id=4364, stack(0x000000000c050000,0x000000000c150000)] 0x000000000ac3e000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=4464, stack(0x000000000bf50000,0x000000000c050000)] 0x000000000abe7000 JavaThread "Finalizer" daemon [_thread_blocked, id=1388, stack(0x000000000be50000,0x000000000bf50000)] 0x000000000abe0000 JavaThread "Reference Handler" daemon [_thread_blocked, id=4948, stack(0x000000000bd50000,0x000000000be50000)] 0x000000000108e800 JavaThread "main" [_thread_in_native, id=1360, stack(0x0000000001240000,0x0000000001340000)] Compilation events (10 events): Event: 754.499 Thread 0x000000000ac47000 2450 org.apache.lucene.analysis.DelegatingAnalyzerWrapper::wrapReader (7 bytes) Event: 754.500 Thread 0x000000000ac47000 nmethod 2450 0x0000000001db2b50 code [0x0000000001db2c80, 0x0000000001db2cd8] Event: 765.302 Thread 0x000000000ac4c000 2451 % ! org.apache.tomcat.util.net.AprEndpoint$Poller::run @ 703 (2291 bytes) Event: 765.451 Thread 0x000000000ac4c000 nmethod 2451% 0x0000000001dcbad0 code [0x0000000001dcc220, 0x0000000001dd25e0] Event: 778.839 Thread 0x000000000ac47000 2452 java.util.concurrent.ConcurrentSkipListMap$Node::casNext (13 bytes) Event: 778.839 Thread 0x000000000ac47000 nmethod 2452 0x0000000001db33d0 code [0x0000000001db3500, 0x0000000001db3598] Event: 783.215 Thread 0x000000000ac4c000 2453 org.apache.lucene.analysis.Analyzer::getReuseStrategy (5 bytes) Event: 783.215 Thread 0x000000000ac4c000 nmethod 2453 0x0000000001db31d0 code [0x0000000001db3300, 0x0000000001db3378] Event: 789.177 Thread 0x000000000ac47000 2454 org.apache.lucene.codecs.lucene41.ForUtil::<init> (171 bytes) Event: 789.211 Thread 0x000000000ac47000 nmethod 2454 0x0000000001dc8f10 code [0x0000000001dc9260, 0x0000000001dca7b0] --------------- S Y S T E M --------------- OS: Windows Server 2008 R2 , 64 bit Build 7601 Service Pack 1 CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, tsc, tscinvbit, tscinv Memory: 4k page, physical 33553976k(14424816k free), swap 67106088k(48924104k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (24.65-b04) for windows-amd64 JRE (1.7.0_67-b01), built on Jul 25 2014 08:55:00 by "java_re" with unknown MS VC++:1600 time: Wed Sep 10 14:22:44 2014 elapsed time: 806 seconds Jeremy Anderson Solutions Engineer [Description: Description: Description: Description: Logo]Eastport Analytics, Inc.
