[ https://issues.apache.org/jira/browse/CASSANDRA-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009583#comment-16009583 ]
ZhaoYang commented on CASSANDRA-13523: -------------------------------------- are you using 32-bit jvm ? could you share you jvm setting, etc. > StreamReceiveTask: java.lang.OutOfMemoryError: Map failed > --------------------------------------------------------- > > Key: CASSANDRA-13523 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13523 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Ubuntu 14.04.5 LTS, Docker version 1.9.1, run as a > container, 4 core server with 16GB memory. > Reporter: Matthew O'Riordan > Labels: bug, crash > Fix For: 2.1.13 > > > During a nodetool repair -par on one of our keyspaces, Cassandra crashed due > to what seems like memory exhaustion within the JVM. The machine itself had > plenty of available memory at the time and did not appear to be under any > significant load. > In the system log, before the crash, the following was logged: > {code} > ... > INFO [AntiEntropySessions:55] 2017-05-10 18:18:20,627 RepairJob.java:163 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] requesting merkle trees for > stats_day_aggregates (to [/54.162.66.114, /54.236.226.76, /52.221.228.170, > /54.154.35.144, /54.154.96.213, /52.221.217.27]) > INFO [ValidationExecutor:54] 2017-05-10 18:18:20,628 > ColumnFamilyStore.java:905 - Enqueuing flush of stats_day_aggregates: 7018 > (0%) on-heap, 0 (0%) off-heap > INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,628 Memtable.java:347 > - Writing Memtable-stats_day_aggregates@3469792(2.734KiB serialized bytes, 14 > ops, 0%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,629 Memtable.java:382 > - Completed flushing > /var/lib/cassandra/data/ably_production_0_stats/stats_day_aggregates-b6e29201e3d111e5bbf3091830ac5256/ably_production_0_stats-stats_day_aggregates-tmp-ka-43026-Data.db > (0.000KiB) for commitlog position ReplayPosition(segmentId=1491420635101, > position=24224955) > INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220 > StreamResultFuture.java:180 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256] > Session with /52.203.21.193 is complete > INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220 > StreamResultFuture.java:212 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256] > All sessions completed > INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,221 > StreamingRepairTask.java:96 - [repair #fe7e3320-35ac-11e7-b7e4-091830ac5256] > streaming task succeed, returning response to /52.221.217.27 > INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,230 > CqlSlowLogWriter.java:151 - Recording statements with duration of 4844 in > slow log > INFO [Service Thread] 2017-05-10 18:18:26,233 GCInspector.java:258 - G1 Old > Generation GC in 4781ms. G1 Eden Space: 1310720000 -> 0; G1 Old Gen: > 2774539816 -> 1830851216; G1 Survivor Space: 37748736 -> 0; > INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,237 RepairSession.java:171 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for > stats_day_aggregates from /54.162.66.114 > INFO [StreamConnectionEstablisher:1] 2017-05-10 18:18:26,239 > StreamCoordinator.java:209 - [Stream #0293bb60-35ad-11e7-b7e4-091830ac5256, > ID#0] Beginning stream session with /54.154.226.20 > INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,294 RepairSession.java:171 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for > stats_day_aggregates from /54.236.226.76 > INFO [Service Thread] 2017-05-10 18:18:26,298 StatusLogger.java:51 - Pool > Name Active Pending Completed Blocked All Time > Blocked > INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for > stats_day_aggregates from /54.154.35.144 > INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,344 > CqlSlowLogWriter.java:151 - Recording statements with duration of 5035 in > slow log > WARN [GossipTasks:1] 2017-05-10 18:18:26,344 FailureDetector.java:258 - Not > marking nodes down due to local pause of 5109502584 > 5000000000 > INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for > stats_day_aggregates from /54.154.96.213 > INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 - > [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for > stats_day_aggregates from /52.221.228.170 > INFO [Service Thread] 2017-05-10 18:18:26,345 StatusLogger.java:66 - > MutationStage 0 0 1199223432 0 > 0 > INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,345 > CqlSlowLogWriter.java:151 - Recording statements with duration of 5095 in > slow log > ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,346 > StreamReceiveTask.java:183 - Error applying streamed data: > org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed > at > org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:399) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Caused by: java.io.IOException: Map failed > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) > ~[na:1.8.0_101] > at > org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > ... 11 common frames omitted > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_101] > ... 12 common frames omitted > ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,349 > JVMStabilityInspector.java:117 - JVM state determined to be unstable. > Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_101] > at > org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141) > ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > {code} > The JVM then exited completely and logged the following: > {code} > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (mmap) failed to map 12288 bytes for committing > reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # In 32 bit mode, the process size limit was hit > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Use 64 bit Java on a 64 bit OS > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2627), pid=79, tid=0x00007f06b6e56700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build > 1.8.0_101-b13) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode > linux-amd64 compressed oops) > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > --------------- T H R E A D --------------- > Current thread (0x00007f03d5054000): JavaThread "Thread-1" [_thread_new, > id=23656, stack(0x00007f06b6e16000,0x00007f06b6e57000)] > Stack: [0x00007f06b6e16000,0x00007f06b6e57000], sp=0x00007f06b6e55720, free > space=253k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [libjvm.so+0xac3f0a] VMError::report_and_die()+0x2ba > V [libjvm.so+0x4fbf9b] report_vm_out_of_memory(char const*, int, unsigned > long, VMErrorType, char const*)+0x8b > V [libjvm.so+0x9222d3] os::Linux::commit_memory_impl(char*, unsigned long, > bool)+0x103 > V [libjvm.so+0x92239c] os::pd_commit_memory(char*, unsigned long, bool)+0xc > V [libjvm.so+0x91be4a] os::commit_memory(char*, unsigned long, bool)+0x2a > V [libjvm.so+0x92056f] os::pd_create_stack_guard_pages(char*, unsigned > long)+0x7f > V [libjvm.so+0xa69ace] JavaThread::create_stack_guard_pages()+0x5e > V [libjvm.so+0xa73484] JavaThread::run()+0x34 > V [libjvm.so+0x924698] java_start(Thread*)+0x108 > C [libpthread.so.0+0x8184] start_thread+0xc4 > --------------- P R O C E S S --------------- > Java Threads: ( => current thread ) > 0x00007f03d578c000 JavaThread "Daemon shutdown" [_thread_new, id=23658, > stack(0x00007f074a62a000,0x00007f074a66b000)] > 0x00007f03d5278000 JavaThread "Thread-0" [_thread_new, id=23657, > stack(0x00007f074422f000,0x00007f0744270000)] > =>0x00007f03d5054000 JavaThread "Thread-1" [_thread_new, id=23656, > stack(0x00007f06b6e16000,0x00007f06b6e57000)] > 0x00007f03d5187800 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23644, stack(0x00007f0749704000,0x00007f0749745000)] > 0x00007f03d465d800 JavaThread "STREAM-OUT-/54.169.103.14" daemon > [_thread_blocked, id=23627, stack(0x00007f07493f7000,0x00007f0749438000)] > 0x00007f05ac99d000 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23625, stack(0x00007f074ab13000,0x00007f074ab54000)] > 0x00007f03d4620800 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23591, stack(0x00007f07253cb000,0x00007f072540c000)] > 0x00007f03d5086000 JavaThread "STREAM-IN-/54.154.226.20" daemon > [_thread_in_native, id=23511, stack(0x00007f074986b000,0x00007f07498ac000)] > 0x00007f0449a77000 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23508, stack(0x00007f0747758000,0x00007f0747799000)] > 0x00007f03d5185000 JavaThread "STREAM-IN-/54.154.226.20" daemon > [_thread_in_native, id=23488, stack(0x00007f07491bc000,0x00007f07491fd000)] > 0x00007f03d5104000 JavaThread "STREAM-IN-/54.154.226.20" daemon > [_thread_in_native, id=23473, stack(0x00007f0743e40000,0x00007f0743e81000)] > 0x00007f0448b64000 JavaThread "STREAM-IN-/54.154.226.20" daemon > [_thread_in_native, id=23393, stack(0x00007f074917b000,0x00007f07491bc000)] > 0x00007f03d4f7d800 JavaThread "STREAM-OUT-/52.203.21.193" daemon > [_thread_blocked, id=23369, stack(0x00007f0749966000,0x00007f07499a7000)] > 0x00007f03d4fed800 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23367, stack(0x00007f074416c000,0x00007f07441ad000)] > 0x00007f03d5302000 JavaThread "STREAM-OUT-/54.154.226.20" daemon > [_thread_blocked, id=23303, stack(0x00007f0722044000,0x00007f0722085000)] > 0x00007f044a921000 JavaThread "RepairJobTask:15" daemon [_thread_blocked, > id=23131, stack(0x00007f074ae05000,0x00007f074ae46000)] > 0x00007f044b9ae000 JavaThread "RepairJobTask:14" daemon [_thread_blocked, > id=23121, stack(0x00007f074982a000,0x00007f074986b000)] > 0x00007f044a920000 JavaThread "RepairJobTask:13" daemon [_thread_blocked, > id=23116, stack(0x00007f074366e000,0x00007f07436af000)] > 0x00007f03d45eb000 JavaThread "RepairJobTask:12" daemon [_thread_blocked, > id=23115, stack(0x00007f0717d50000,0x00007f0717d91000)] > 0x00007f03d45ea800 JavaThread "RepairJobTask:11" daemon [_thread_blocked, > id=23103, stack(0x00007f0743f64000,0x00007f0743fa5000)] > 0x00007f03d4fce000 JavaThread "RepairJobTask:10" daemon [_thread_blocked, > id=23102, stack(0x00007f074ae6f000,0x00007f074aeb0000)] > 0x00007f03d4e78800 JavaThread "RepairJobTask:9" daemon [_thread_blocked, > id=23101, stack(0x00007f0720fa2000,0x00007f0720fe3000)] > 0x00007f03d50bc000 JavaThread "RepairJobTask:8" daemon [_thread_blocked, > id=23100, stack(0x00007f07210a6000,0x00007f07210e7000)] > 0x00007f03d506f000 JavaThread "RepairJobTask:7" daemon [_thread_blocked, > id=23099, stack(0x00007f074ad2d000,0x00007f074ad6e000)] > 0x00007f03d5146000 JavaThread "RepairJobTask:6" daemon [_thread_blocked, > id=23098, stack(0x00007f0747c0b000,0x00007f0747c4c000)] > 0x00007f03d4f6d800 JavaThread "RepairJobTask:5" daemon [_thread_blocked, > id=23097, stack(0x00007f0750043000,0x00007f0750084000)] > 0x00007f03d5079800 JavaThread "RepairJobTask:4" daemon [_thread_blocked, > id=23096, stack(0x00007f0742dde000,0x00007f0742e1f000)] > 0x00007f03d4f94000 JavaThread "RepairJobTask:3" daemon [_thread_blocked, > id=23095, stack(0x00007f074a5c4000,0x00007f074a605000)] > 0x00007f03d52a6000 JavaThread "RepairJobTask:2" daemon [_thread_blocked, > id=23094, stack(0x00007f072122c000,0x00007f072126d000)] > 0x00007f03d52ee800 JavaThread "RepairJobTask:1" daemon [_thread_blocked, > id=23093, stack(0x00007f07436f0000,0x00007f0743731000)] > 0x00007f057f042000 JavaThread "RepairJobTask:15" daemon [_thread_blocked, > id=23078, stack(0x00007f074a799000,0x00007f074a7da000)] > 0x00007f03d57a4800 JavaThread "RepairJobTask:14" daemon [_thread_blocked, > id=23077, stack(0x00007f0683bd4000,0x00007f0683c15000)] > 0x00007f044ba0f800 JavaThread "RepairJobTask:13" daemon [_thread_blocked, > id=23035, stack(0x00007f074aa41000,0x00007f074aa82000)] > 0x00007f03d50b9000 JavaThread "RepairJobTask:12" daemon [_thread_blocked, > id=23010, stack(0x00007f074384e000,0x00007f074388f000)] > 0x00007f03d50b7800 JavaThread "RepairJobTask:11" daemon [_thread_blocked, > id=23009, stack(0x00007f0749380000,0x00007f07493c1000)] > 0x00007f03d50b6000 JavaThread "RepairJobTask:10" daemon [_thread_blocked, > id=23008, stack(0x00007f07438d0000,0x00007f0743911000)] > 0x00007f03d50b5800 JavaThread "RepairJobTask:9" daemon [_thread_blocked, > id=23007, stack(0x00007f0744027000,0x00007f0744068000)] > 0x00007f03d50af000 JavaThread "RepairJobTask:8" daemon [_thread_blocked, > id=23006, stack(0x00007f074388f000,0x00007f07438d0000)] > 0x00007f03d5097800 JavaThread "RepairJobTask:7" daemon [_thread_blocked, > id=22989, stack(0x00007f074ba63000,0x00007f074baa4000)] > 0x00007f03d5096800 JavaThread "RepairJobTask:6" daemon [_thread_blocked, > id=22988, stack(0x00007f0742b98000,0x00007f0742bd9000)] > 0x00007f03d5095000 JavaThread "RepairJobTask:5" daemon [_thread_blocked, > id=22987, stack(0x00007f0721fc2000,0x00007f0722003000)] > 0x00007f03d5093800 JavaThread "RepairJobTask:4" daemon [_thread_blocked, > id=22986, stack(0x00007f0743f23000,0x00007f0743f64000)] > 0x00007f03d5092800 JavaThread "RepairJobTask:3" daemon [_thread_blocked, > id=22985, stack(0x00007f074a99e000,0x00007f074a9df000)] > 0x00007f03d5091000 JavaThread "RepairJobTask:2" daemon [_thread_blocked, > id=22984, stack(0x00007f074a6ac000,0x00007f074a6ed000)] > 0x00007f03d5090800 JavaThread "RepairJobTask:1" daemon [_thread_blocked, > id=22983, stack(0x00007f0749745000,0x00007f0749786000)] > 0x00007f03d4f7a000 JavaThread "ReadRepairStage:73775" daemon > [_thread_blocked, id=22227, stack(0x00007f074aabc000,0x00007f074aafd000)] > 0x00007f044a94f800 JavaThread "ReadRepairStage:73774" daemon > [_thread_blocked, id=22226, stack(0x00007f0744068000,0x00007f07440a9000)] > 0x00007f03d437d000 JavaThread "ReadRepairStage:73773" daemon > [_thread_blocked, id=21837, stack(0x00007f0743772000,0x00007f07437b3000)] > 0x00007f03d4471000 JavaThread "ReadRepairStage:73770" daemon > [_thread_blocked, id=18261, stack(0x00007f07491fd000,0x00007f074923e000)] > 0x00007f060f438800 JavaThread "RMI TCP Connection(idle)" daemon > [_thread_blocked, id=15398, stack(0x00007f0747011000,0x00007f0747052000)] > 0x00007f060fa59000 JavaThread "RMI TCP Connection(5238)-172.17.0.5" daemon > [_thread_blocked, id=10756, stack(0x00007f0743fa5000,0x00007f0743fe6000)] > 0x00007f044b9aa000 JavaThread "ValidationExecutor:54" daemon > [_thread_in_Java, id=9233, stack(0x00007f0748b0a000,0x00007f0748b4b000)] > 0x00007f04494ef800 JavaThread "SharedPool-Worker-407" daemon > [_thread_blocked, id=3652, stack(0x00007f074907e000,0x00007f07490bf000)] > 0x00007f057e393000 JavaThread "SharedPool-Worker-406" daemon > [_thread_blocked, id=3651, stack(0x00007f0744270000,0x00007f07442b1000)] > 0x00007f044a847000 JavaThread "ValidationExecutor:51" daemon > [_thread_blocked, id=442, stack(0x00007f07498ed000,0x00007f074992e000)] > 0x00007f04b147b000 JavaThread "AntiEntropySessions:56" daemon > [_thread_blocked, id=1481, stack(0x00007f0742b16000,0x00007f0742b57000)] > 0x00007f0693cd4800 JavaThread "StreamReceiveTask:640" daemon > [_thread_blocked, id=1467, stack(0x00007f074acec000,0x00007f074ad2d000)] > 0x00007f05d9c8d800 JavaThread "StreamReceiveTask:639" daemon > [_thread_blocked, id=1458, stack(0x00007f0749635000,0x00007f0749676000)] > 0x00007f04b3e37000 JavaThread "StreamReceiveTask:638" daemon > [_thread_blocked, id=1434, stack(0x00007f074a583000,0x00007f074a5c4000)] > 0x00007f06924f7800 JavaThread "AntiEntropySessions:55" daemon > [_thread_blocked, id=1412, stack(0x00007f074781b000,0x00007f074785c000)] > 0x00007f074ceaf000 JavaThread "StreamReceiveTask:637" daemon > [_thread_blocked, id=1401, stack(0x00007f0722085000,0x00007f07220c6000)] > 0x00007f069398e800 JavaThread "MemtableFlushWriter:13609" daemon > [_thread_blocked, id=1380, stack(0x00007f0748ac9000,0x00007f0748b0a000)] > 0x00007f069374e800 JavaThread "MemtableReclaimMemory:9889" daemon > [_thread_blocked, id=1379, stack(0x00007f07440ea000,0x00007f074412b000)] > 0x00007f0787c63800 JavaThread "MemtableFlushWriter:13608" daemon > [_thread_blocked, id=1378, stack(0x00007f074412b000,0x00007f074416c000)] > 0x00007f0693cc6000 JavaThread "AntiEntropySessions:54" daemon > [_thread_blocked, id=1376, stack(0x00007f07436af000,0x00007f07436f0000)] > 0x00007f06936ff000 JavaThread "MemtablePostFlush:36174" daemon > [_thread_blocked, id=1374, stack(0x00007f07442b1000,0x00007f07442f2000)] > 0x00007f0787c19000 JavaThread "Thread-280182" daemon [_thread_blocked, > id=1371, stack(0x00007f07498ac000,0x00007f07498ed000)] > 0x00007f074f9da800 JavaThread "JMX server connection timeout 768070" daemon > [_thread_blocked, id=1369, stack(0x00007f0743ee2000,0x00007f0743f23000)] > 0x0000000001fed800 JavaThread "CompactionExecutor:16697" daemon > [_thread_in_Java, id=1365, stack(0x00007f074b8ec000,0x00007f074b92d000)] > 0x0000000001367800 JavaThread "CompactionExecutor:16696" daemon > [_thread_in_Java, id=1364, stack(0x00007f0747695000,0x00007f07476d6000)] > 0x00007f0690975800 JavaThread "MessagingService-Incoming-/54.174.252.207" > [_thread_in_native, id=30389, stack(0x00007f07499e8000,0x00007f0749a29000)] > 0x00007f074cd88800 JavaThread "MessagingService-Incoming-/54.174.252.207" > [_thread_in_native, id=30381, stack(0x00007f0743911000,0x00007f0743952000)] > 0x00007f0690a6a000 JavaThread "AntiEntropySessions:50" daemon > [_thread_blocked, id=24703, stack(0x00007f07441ee000,0x00007f074422f000)] > 0x00007f06924e9000 JavaThread "Thread-169959" daemon [_thread_blocked, > id=24700, stack(0x00007f07476d6000,0x00007f0747717000)] > 0x00007f074e24a000 JavaThread "JMX server connection timeout 174925" daemon > [_thread_blocked, id=15333, stack(0x00007f04b808f000,0x00007f04b80d0000)] > 0x00007f060cc21000 JavaThread "RMI TCP Connection(364)-172.17.0.4" daemon > [_thread_blocked, id=15332, stack(0x00007f0683d57000,0x00007f0683d98000)] > 0x00007f074f3e5800 JavaThread "StreamConnectionEstablisher:4" daemon > [_thread_in_native, id=32671, stack(0x00007f074a6fb000,0x00007f074a73c000)] > 0x00007f069044a800 JavaThread "StreamingTransferTaskTimeouts:1" daemon > [_thread_blocked, id=32667, stack(0x00007f0747799000,0x00007f07477da000)] > 0x00007f074d19d800 JavaThread "StreamConnectionEstablisher:3" daemon > [_thread_in_native, id=32663, stack(0x00007f0747bca000,0x00007f0747c0b000)] > 0x00007f078770d800 JavaThread "StreamConnectionEstablisher:2" daemon > [_thread_blocked, id=32660, stack(0x00007f074a53e000,0x00007f074a57f000)] > 0x00007f04b070d000 JavaThread "StreamConnectionEstablisher:1" daemon > [_thread_blocked, id=32655, stack(0x00007f074ac70000,0x00007f074acb1000)] > 0x00007f0693fd0800 JavaThread "MessagingService-Incoming-/172.17.0.2" > [_thread_in_native, id=32641, stack(0x00007f0747c4c000,0x00007f0747c8d000)] > 0x00007f06931c5800 JavaThread > "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked, > id=32639, stack(0x00007f07495ef000,0x00007f0749630000)] > 0x00007f04b028d000 JavaThread > "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked, > id=32638, stack(0x00007f07497cd000,0x00007f074980e000)] > 0x00007f07876a6800 JavaThread "MessagingService-Incoming-/54.162.66.114" > [_thread_in_native, id=13194, stack(0x00007f074a66b000,0x00007f074a6ac000)] > 0x00007f0693644800 JavaThread "MessagingService-Incoming-/54.236.226.76" > [_thread_in_native, id=13192, stack(0x00007f0747b1b000,0x00007f0747b5c000)] > 0x00007f04b02cf800 JavaThread "MessagingService-Incoming-/54.236.226.76" > [_thread_in_native, id=13191, stack(0x00007f074a7da000,0x00007f074a81b000)] > 0x00007f0693f5d800 JavaThread "MessagingService-Incoming-/54.154.35.144" > [_thread_in_native, id=13189, stack(0x00007f074a9fd000,0x00007f074aa3e000)] > 0x00007f0690909800 JavaThread "MessagingService-Incoming-/54.171.109.9" > [_thread_in_native, id=13184, stack(0x00007f07499a7000,0x00007f07499e8000)] > 0x00007f04b0728000 JavaThread "MessagingService-Incoming-/54.162.66.114" > [_thread_in_native, id=13182, stack(0x00007f0683c15000,0x00007f0683c56000)] > 0x00007f0690e07000 JavaThread "MessagingService-Incoming-/54.154.96.213" > [_thread_in_native, id=13180, stack(0x00007f0707ec2000,0x00007f0707f03000)] > 0x00007f05aca67800 JavaThread "MessagingService-Incoming-/54.154.226.20" > [_thread_blocked, id=13177, stack(0x00007f0747052000,0x00007f0747093000)] > 0x00007f04b10c6000 JavaThread "MessagingService-Incoming-/54.154.35.144" > [_thread_in_native, id=13174, stack(0x00007f0747b5c000,0x00007f0747b9d000)] > 0x00007f04b0e58800 JavaThread "MessagingService-Incoming-/54.154.96.213" > [_thread_in_native, id=13173, stack(0x00007f0749bf9000,0x00007f0749c3a000)] > 0x00007f04b07a1800 JavaThread "MessagingService-Incoming-/54.171.109.9" > [_thread_in_native, id=13172, stack(0x00007f0750170000,0x00007f07501b1000)] > 0x00007f0690e1a800 JavaThread "MessagingService-Incoming-/54.154.226.20" > [_thread_in_native, id=13170, stack(0x00007f0707f85000,0x00007f0707fc6000)] > 0x00007f04b02d0800 JavaThread "MessagingService-Incoming-/52.203.21.193" > [_thread_in_native, id=13159, stack(0x00007f0717e13000,0x00007f0717e54000)] > 0x00007f04b1bd2000 JavaThread "MessagingService-Incoming-/52.203.21.193" > [_thread_in_native, id=13155, stack(0x00007f0747188000,0x00007f07471c9000)] > 0x00007f04b0090000 JavaThread "SharedPool-Worker-398" daemon > [_thread_blocked, id=2502, stack(0x00007f0683d98000,0x00007f0683dd9000)] > 0x00007f0786dc5000 JavaThread "SharedPool-Worker-397" daemon > [_thread_blocked, id=2501, stack(0x00007f0683dd9000,0x00007f0683e1a000)] > 0x00007f06938cc000 JavaThread "SharedPool-Worker-399" daemon > [_thread_blocked, id=2500, stack(0x00007f0683e1a000,0x00007f0683e5b000)] > 0x00007f06937eb800 JavaThread "SharedPool-Worker-400" daemon > [_thread_blocked, id=2499, stack(0x00007f0683e5b000,0x00007f0683e9c000)] > 0x00007f0693686800 JavaThread "SharedPool-Worker-401" daemon > [_thread_blocked, id=2498, stack(0x00007f0683e9c000,0x00007f0683edd000)] > 0x00007f07877f8800 JavaThread "SharedPool-Worker-402" daemon > [_thread_blocked, id=2497, stack(0x00007f0683edd000,0x00007f0683f1e000)] > 0x00007f074c101800 JavaThread "SharedPool-Worker-403" daemon > [_thread_blocked, id=2496, stack(0x00007f0683f1e000,0x00007f0683f5f000)] > 0x00007f069397f000 JavaThread "SharedPool-Worker-404" daemon > [_thread_blocked, id=2495, stack(0x00007f068566d000,0x00007f06856ae000)] > 0x00007f0693572000 JavaThread "SharedPool-Worker-392" daemon > [_thread_blocked, id=2494, stack(0x00007f06856ae000,0x00007f06856ef000)] > 0x00007f0786f0f000 JavaThread "SharedPool-Worker-405" daemon > [_thread_blocked, id=2493, stack(0x00007f0685995000,0x00007f06859d6000)] > 0x00007f0786a7b000 JavaThread "SharedPool-Worker-376" daemon > [_thread_blocked, id=2492, stack(0x00007f06859d6000,0x00007f0685a17000)] > 0x00007f0693ef4800 JavaThread "SharedPool-Worker-379" daemon > [_thread_blocked, id=2491, stack(0x00007f0685a17000,0x00007f0685a58000)] > 0x00007f04b0281000 JavaThread "SharedPool-Worker-378" daemon > [_thread_blocked, id=2490, stack(0x00007f0685a58000,0x00007f0685a99000)] > 0x00007f04b0280000 JavaThread "SharedPool-Worker-381" daemon > [_thread_blocked, id=2489, stack(0x00007f0685a99000,0x00007f0685ada000)] > 0x00007f069046b800 JavaThread "SharedPool-Worker-382" daemon > [_thread_blocked, id=2488, stack(0x00007f0685ada000,0x00007f0685b1b000)] > 0x00007f069046b000 JavaThread "SharedPool-Worker-383" daemon > [_thread_blocked, id=2487, stack(0x00007f0685b1b000,0x00007f0685b5c000)] > 0x00007f0693a5f000 JavaThread "SharedPool-Worker-385" daemon > [_thread_blocked, id=2486, stack(0x00007f0685b5c000,0x00007f0685b9d000)] > 0x00007f0693a5e800 JavaThread "SharedPool-Worker-370" daemon > [_thread_blocked, id=2485, stack(0x00007f0685b9d000,0x00007f0685bde000)] > 0x00007f04b0649000 JavaThread "SharedPool-Worker-373" daemon > [_thread_blocked, id=2484, stack(0x00007f0685bde000,0x00007f0685c1f000)] > 0x00007f04b0648800 JavaThread "SharedPool-Worker-386" daemon > [_thread_blocked, id=2483, stack(0x00007f0685c1f000,0x00007f0685c60000)] > 0x00007f07876a5000 JavaThread "SharedPool-Worker-387" daemon > [_thread_blocked, id=2482, stack(0x00007f0685c60000,0x00007f0685ca1000)] > 0x00007f07876a4800 JavaThread "SharedPool-Worker-388" daemon > [_thread_blocked, id=2481, stack(0x00007f0685ca1000,0x00007f0685ce2000)] > 0x00007f04b0c91000 JavaThread "SharedPool-Worker-391" daemon > [_thread_blocked, id=2480, stack(0x00007f0685ce2000,0x00007f0685d23000)] > 0x00007f04b0c90000 JavaThread "SharedPool-Worker-394" daemon > [_thread_blocked, id=2479, stack(0x00007f069404a000,0x00007f069408b000)] > 0x00007f04b0752800 JavaThread "SharedPool-Worker-393" daemon > [_thread_blocked, id=2478, stack(0x00007f069408b000,0x00007f06940cc000)] > 0x00007f04b0751800 JavaThread "SharedPool-Worker-396" daemon > [_thread_blocked, id=2477, stack(0x00007f06940cc000,0x00007f069410d000)] > 0x00007f074caf4800 JavaThread "SharedPool-Worker-362" daemon > [_thread_blocked, id=2476, stack(0x00007f069410d000,0x00007f069414e000)] > 0x00007f074caf3800 JavaThread "SharedPool-Worker-395" daemon > [_thread_blocked, id=2475, stack(0x00007f069414e000,0x00007f069418f000)] > 0x00007f069393a800 JavaThread "SharedPool-Worker-364" daemon > [_thread_blocked, id=2474, stack(0x00007f069418f000,0x00007f06941d0000)] > 0x00007f069393a000 JavaThread "SharedPool-Worker-365" daemon > [_thread_blocked, id=2473, stack(0x00007f06941d0000,0x00007f0694211000)] > 0x00007f078765a000 JavaThread "SharedPool-Worker-390" daemon > [_thread_blocked, id=2472, stack(0x00007f0694211000,0x00007f0694252000)] > 0x00007f0787659800 JavaThread "SharedPool-Worker-389" daemon > [_thread_blocked, id=2471, stack(0x00007f0694252000,0x00007f0694293000)] > 0x00007f06938b2000 JavaThread "SharedPool-Worker-366" daemon > [_thread_blocked, id=2470, stack(0x00007f0694293000,0x00007f06942d4000)] > 0x00007f06938b1000 JavaThread "SharedPool-Worker-368" daemon > [_thread_blocked, id=2469, stack(0x00007f06942d4000,0x00007f0694315000)] > 0x00007f04b090e800 JavaThread "SharedPool-Worker-369" daemon > [_thread_blocked, id=2468, stack(0x00007f0694315000,0x00007f0694356000)] > 0x00007f04b090e000 JavaThread "SharedPool-Worker-384" daemon > [_thread_blocked, id=2467, stack(0x00007f0694356000,0x00007f0694397000)] > 0x00007f04b0866000 JavaThread "SharedPool-Worker-374" daemon > [_thread_blocked, id=2466, stack(0x00007f06946ce000,0x00007f069470f000)] > 0x00007f04b0865000 JavaThread "SharedPool-Worker-380" daemon > [_thread_blocked, id=2465, stack(0x00007f069470f000,0x00007f0694750000)] > 0x00007f074cde4800 JavaThread "SharedPool-Worker-377" daemon > [_thread_blocked, id=2464, stack(0x00007f0694750000,0x00007f0694791000)] > 0x00007f074cde3800 JavaThread "SharedPool-Worker-375" daemon > [_thread_blocked, id=2463, stack(0x00007f0695812000,0x00007f0695853000)] > 0x00007f074ce69000 JavaThread "SharedPool-Worker-359" daemon > [_thread_blocked, id=2462, stack(0x00007f06b60da000,0x00007f06b611b000)] > 0x00007f074ce68800 JavaThread "SharedPool-Worker-372" daemon > [_thread_blocked, id=2461, stack(0x00007f06b611b000,0x00007f06b615c000)] > 0x00007f07871f8800 JavaThread "SharedPool-Worker-371" daemon > [_thread_blocked, id=2460, stack(0x00007f06b7cc5000,0x00007f06b7d06000)] > 0x00007f07871f8000 JavaThread "SharedPool-Worker-361" daemon > [_thread_blocked, id=2459, stack(0x00007f06c081e000,0x00007f06c085f000)] > 0x00007f04b0c11000 JavaThread "SharedPool-Worker-367" daemon > [_thread_blocked, id=2458, stack(0x00007f06c085f000,0x00007f06c08a0000)] > 0x00007f04b0c10800 JavaThread "SharedPool-Worker-355" daemon > [_thread_blocked, id=2457, stack(0x00007f06c0ae6000,0x00007f06c0b27000)] > 0x00007f074cb38000 JavaThread "SharedPool-Worker-363" daemon > [_thread_blocked, id=2456, stack(0x00007f06c0b27000,0x00007f06c0b68000)] > 0x00007f074cb37800 JavaThread "SharedPool-Worker-353" daemon > [_thread_blocked, id=2455, stack(0x00007f06c0b68000,0x00007f06c0ba9000)] > 0x00007f0787de6800 JavaThread "SharedPool-Worker-360" daemon > [_thread_blocked, id=2454, stack(0x00007f06c0ba9000,0x00007f06c0bea000)] > 0x00007f0787de6000 JavaThread "SharedPool-Worker-356" daemon > [_thread_blocked, id=2453, stack(0x00007f06c0bea000,0x00007f06c0c2b000)] > 0x00007f04b0146000 JavaThread "SharedPool-Worker-358" daemon > [_thread_blocked, id=2452, stack(0x00007f06c0c2b000,0x00007f06c0c6c000)] > 0x00007f04b0145000 JavaThread "SharedPool-Worker-348" daemon > [_thread_blocked, id=2451, stack(0x00007f06d5e0c000,0x00007f06d5e4d000)] > 0x00007f0787ea3000 JavaThread "SharedPool-Worker-357" daemon > [_thread_blocked, id=2450, stack(0x00007f06d5e4d000,0x00007f06d5e8e000)] > 0x00007f0787ea2000 JavaThread "SharedPool-Worker-349" daemon > [_thread_blocked, id=2449, stack(0x00007f06d5e8e000,0x00007f06d5ecf000)] > 0x00007f04b0f5e000 JavaThread "SharedPool-Worker-354" daemon > [_thread_blocked, id=2448, stack(0x00007f06d5ecf000,0x00007f06d5f10000)] > 0x00007f04b0f5d000 JavaThread "SharedPool-Worker-350" daemon > [_thread_blocked, id=2447, stack(0x00007f06d68fc000,0x00007f06d693d000)] > 0x00007f074e06f800 JavaThread "SharedPool-Worker-352" daemon > [_thread_blocked, id=2446, stack(0x00007f06d693d000,0x00007f06d697e000)] > 0x00007f074e06f000 JavaThread "SharedPool-Worker-346" daemon > [_thread_blocked, id=2445, stack(0x00007f06d697e000,0x00007f06d69bf000)] > 0x00007f0693f77000 JavaThread "SharedPool-Worker-351" daemon > [_thread_blocked, id=2444, stack(0x00007f06d69bf000,0x00007f06d6a00000)] > 0x00007f0693f76000 JavaThread "SharedPool-Worker-347" daemon > [_thread_blocked, id=2443, stack(0x00007f06d6a00000,0x00007f06d6a41000)] > 0x00007f0787e23000 JavaThread "SharedPool-Worker-344" daemon > [_thread_blocked, id=2442, stack(0x00007f06d6a41000,0x00007f06d6a82000)] > 0x00007f0787e22800 JavaThread "SharedPool-Worker-345" daemon > [_thread_blocked, id=2441, stack(0x00007f06d6a82000,0x00007f06d6ac3000)] > 0x00007f06909f2000 JavaThread "SharedPool-Worker-343" daemon > [_thread_blocked, id=2440, stack(0x00007f06df3f4000,0x00007f06df435000)] > 0x00007f04b0ebb800 JavaThread "SharedPool-Worker-342" daemon > [_thread_blocked, id=2439, stack(0x00007f0701182000,0x00007f07011c3000)] > 0x00007f07877d1800 JavaThread "SharedPool-Worker-341" daemon > [_thread_blocked, id=2438, stack(0x00007f07011c3000,0x00007f0701204000)] > 0x00007f04b0233800 JavaThread "SharedPool-Worker-340" daemon > [_thread_blocked, id=2437, stack(0x00007f0701204000,0x00007f0701245000)] > 0x00007f074c302000 JavaThread "SharedPool-Worker-339" daemon > [_thread_blocked, id=2436, stack(0x00007f0701245000,0x00007f0701286000)] > 0x00007f04b08eb800 JavaThread "SharedPool-Worker-336" daemon > [_thread_blocked, id=2435, stack(0x00007f0701286000,0x00007f07012c7000)] > 0x00007f074d101000 JavaThread "SharedPool-Worker-337" daemon > [_thread_blocked, id=2434, stack(0x00007f07012c7000,0x00007f0701308000)] > 0x00007f04b08ed000 JavaThread "SharedPool-Worker-338" daemon > [_thread_blocked, id=2433, stack(0x00007f0701308000,0x00007f0701349000)] > 0x00007f0787e4e000 JavaThread "SharedPool-Worker-334" daemon > [_thread_blocked, id=2432, stack(0x00007f0701349000,0x00007f070138a000)] > 0x00007f04b0231800 JavaThread "SharedPool-Worker-324" daemon > [_thread_blocked, id=2431, stack(0x00007f070138a000,0x00007f07013cb000)] > 0x00007f06935f7800 JavaThread "SharedPool-Worker-321" daemon > [_thread_blocked, id=2430, stack(0x00007f07013cb000,0x00007f070140c000)] > 0x00007f04b00b0800 JavaThread "SharedPool-Worker-322" daemon > [_thread_blocked, id=2429, stack(0x00007f070140c000,0x00007f070144d000)] > 0x00007f04b0a37800 JavaThread "SharedPool-Worker-320" daemon > [_thread_blocked, id=2428, stack(0x00007f070144d000,0x00007f070148e000)] > 0x00007f04b0e91000 JavaThread "SharedPool-Worker-323" daemon > [_thread_blocked, id=2427, stack(0x00007f0707a34000,0x00007f0707a75000)] > 0x00007f0693474800 JavaThread "SharedPool-Worker-326" daemon > [_thread_blocked, id=2426, stack(0x00007f0707d0c000,0x00007f0707d4d000)] > 0x00007f04b0a29000 JavaThread "SharedPool-Worker-327" daemon > [_thread_blocked, id=2425, stack(0x00007f0707d4d000,0x00007f0707d8e000)] > 0x00007f0691394800 JavaThread "SharedPool-Worker-328" daemon > [_thread_blocked, id=2424, stack(0x00007f0711b2c000,0x00007f0711b6d000)] > 0x00007f0693f9d800 JavaThread "SharedPool-Worker-329" daemon > [_thread_blocked, id=2423, stack(0x00007f0711b6d000,0x00007f0711bae000)] > 0x00007f06935dd800 JavaThread "SharedPool-Worker-331" daemon > [_thread_blocked, id=2422, stack(0x00007f0711bae000,0x00007f0711bef000)] > 0x00007f0786de3000 JavaThread "SharedPool-Worker-332" daemon > [_thread_blocked, id=2421, stack(0x00007f0711bef000,0x00007f0711c30000)] > 0x00007f07877e2800 JavaThread "SharedPool-Worker-313" daemon > [_thread_blocked, id=2420, stack(0x00007f0711c30000,0x00007f0711c71000)] > 0x00007f04b00c6000 JavaThread "SharedPool-Worker-333" daemon > [_thread_blocked, id=2419, stack(0x00007f0711eda000,0x00007f0711f1b000)] > 0x00007f04b0928800 JavaThread "SharedPool-Worker-335" daemon > [_thread_blocked, id=2418, stack(0x00007f0711f1b000,0x00007f0711f5c000)] > 0x00007f0691507800 JavaThread "SharedPool-Worker-312" daemon > [_thread_blocked, id=2417, stack(0x00007f0711f5c000,0x00007f0711f9d000)] > 0x00007f074da87800 JavaThread "SharedPool-Worker-315" daemon > [_thread_blocked, id=2416, stack(0x00007f0711f9d000,0x00007f0711fde000)] > 0x00007f0787ea4800 JavaThread "SharedPool-Worker-330" daemon > [_thread_blocked, id=2415, stack(0x00007f0711fde000,0x00007f071201f000)] > 0x00007f074cb01000 JavaThread "SharedPool-Worker-317" daemon > [_thread_blocked, id=2414, stack(0x00007f071201f000,0x00007f0712060000)] > 0x00007f04b07ec800 JavaThread "SharedPool-Worker-325" daemon > [_thread_blocked, id=2413, stack(0x00007f0712060000,0x00007f07120a1000)] > 0x00007f04b035b800 JavaThread "SharedPool-Worker-318" daemon > [_thread_blocked, id=2412, stack(0x00007f07120a1000,0x00007f07120e2000)] > 0x00007f074d0ab000 JavaThread "SharedPool-Worker-319" daemon > [_thread_blocked, id=2411, stack(0x00007f07120e2000,0x00007f0712123000)] > 0x00007f0693ba2000 JavaThread "SharedPool-Worker-310" daemon > [_thread_blocked, id=2410, stack(0x00007f0712123000,0x00007f0712164000)] > 0x00007f074c978000 JavaThread "SharedPool-Worker-311" daemon > [_thread_blocked, id=2409, stack(0x00007f0712164000,0x00007f07121a5000)] > 0x00007f0691527800 JavaThread "SharedPool-Worker-316" daemon > [_thread_blocked, id=2408, stack(0x00007f07121a5000,0x00007f07121e6000)] > 0x00007f04b009e000 JavaThread "SharedPool-Worker-314" daemon > [_thread_blocked, id=2407, stack(0x00007f07121e6000,0x00007f0712227000)] > 0x00007f0787da9800 JavaThread "SharedPool-Worker-308" daemon > [_thread_blocked, id=2406, stack(0x00007f0712227000,0x00007f0712268000)] > 0x00007f0690943000 JavaThread "SharedPool-Worker-309" daemon > [_thread_blocked, id=2405, stack(0x00007f0712268000,0x00007f07122a9000)] > 0x00007f074c663800 JavaThread "SharedPool-Worker-307" daemon > [_thread_blocked, id=2404, stack(0x00007f07122a9000,0x00007f07122ea000)] > 0x00007f0693536000 JavaThread "SharedPool-Worker-306" daemon > [_thread_blocked, id=2403, stack(0x00007f07122ea000,0x00007f071232b000)] > 0x00007f06935ab800 JavaThread "SharedPool-Worker-305" daemon > [_thread_blocked, id=2402, stack(0x00007f071232b000,0x00007f071236c000)] > 0x00007f06906c0000 JavaThread "SharedPool-Worker-304" daemon > [_thread_blocked, id=2401, stack(0x00007f071236c000,0x00007f07123ad000)] > 0x00007f0787053800 JavaThread "SharedPool-Worker-303" daemon > [_thread_blocked, id=2400, stack(0x00007f07123ad000,0x00007f07123ee000)] > 0x00007f0787064000 JavaThread "SharedPool-Worker-301" daemon > [_thread_blocked, id=2399, stack(0x00007f071f396000,0x00007f071f3d7000)] > 0x00007f074d6bf800 JavaThread "SharedPool-Worker-302" daemon > [_thread_blocked, id=2398, stack(0x00007f071f3d7000,0x00007f071f418000)] > 0x00007f0693605800 JavaThread "SharedPool-Worker-300" daemon > [_thread_blocked, id=2397, stack(0x00007f07202eb000,0x00007f072032c000)] > 0x00007f04b0d1f000 JavaThread "SharedPool-Worker-298" daemon > [_thread_blocked, id=2396, stack(0x00007f07208a9000,0x00007f07208ea000)] > 0x00007f04b07bf000 JavaThread "SharedPool-Worker-299" daemon > [_thread_blocked, id=2395, stack(0x00007f07208ea000,0x00007f072092b000)] > 0x00007f0693506800 JavaThread "SharedPool-Worker-297" daemon > [_thread_blocked, id=2394, stack(0x00007f0720ab4000,0x00007f0720af5000)] > 0x00007f074c167800 JavaThread "SharedPool-Worker-296" daemon > [_thread_blocked, id=2393, stack(0x00007f0707e81000,0x00007f0707ec2000)] > 0x00007f04b1334800 JavaThread "SharedPool-Worker-295" daemon > [_thread_blocked, id=2392, stack(0x00007f074565e000,0x00007f074569f000)] > 0x00007f0787c5a000 JavaThread "SharedPool-Worker-294" daemon > [_thread_blocked, id=2391, stack(0x00007f072538a000,0x00007f07253cb000)] > 0x00007f04b0867000 JavaThread "SharedPool-Worker-292" daemon > [_thread_blocked, id=1300, stack(0x00007f072540c000,0x00007f072544d000)] > 0x00007f07860c1800 JavaThread "SharedPool-Worker-293" daemon > [_thread_blocked, id=1299, stack(0x00007f0708007000,0x00007f0708048000)] > 0x00007f0787d19000 JavaThread "SharedPool-Worker-291" daemon > [_thread_blocked, id=1298, stack(0x00007f0707f44000,0x00007f0707f85000)] > 0x00007f04b0338000 JavaThread "SharedPool-Worker-290" daemon > [_thread_blocked, id=1297, stack(0x00007f07210e7000,0x00007f0721128000)] > 0x00007f04b0595000 JavaThread "MessagingService-Incoming-/54.179.183.26" > [_thread_in_native, id=1127, stack(0x00007f0707f03000,0x00007f0707f44000)] > 0x00007f06938af800 JavaThread "MessagingService-Incoming-/54.179.183.26" > [_thread_in_native, id=1122, stack(0x00007f0707fc6000,0x00007f0708007000)] > 0x00007f06909da000 JavaThread "SharedPool-Worker-286" daemon > [_thread_blocked, id=1119, stack(0x00007f0717c4c000,0x00007f0717c8d000)] > 0x00007f0693833000 JavaThread "SharedPool-Worker-288" daemon > [_thread_blocked, id=1118, stack(0x00007f0717c8d000,0x00007f0717cce000)] > 0x00007f04b014a800 JavaThread "SharedPool-Worker-289" daemon > [_thread_blocked, id=1117, stack(0x00007f0717cce000,0x00007f0717d0f000)] > 0x00007f0787e53000 JavaThread "SharedPool-Worker-284" daemon > [_thread_blocked, id=1116, stack(0x00007f0717d0f000,0x00007f0717d50000)] > 0x00007f078767f800 JavaThread "SharedPool-Worker-283" daemon > [_thread_blocked, id=1115, stack(0x00007f0717e95000,0x00007f0717ed6000)] > 0x00007f074c3af800 JavaThread "SharedPool-Worker-285" daemon > [_thread_blocked, id=1114, stack(0x00007f0717d91000,0x00007f0717dd2000)] > 0x00007f04b020a000 JavaThread "SharedPool-Worker-287" daemon > [_thread_blocked, id=1113, stack(0x00007f0717e54000,0x00007f0717e95000)] > 0x00007f0693fcf800 JavaThread "SharedPool-Worker-281" daemon > [_thread_blocked, id=1112, stack(0x00007f04afa23000,0x00007f04afa64000)] > 0x00007f0787210800 JavaThread "SharedPool-Worker-282" daemon > [_thread_blocked, id=1111, stack(0x00007f0749d3c000,0x00007f0749d7d000)] > 0x00007f04b010a800 JavaThread "SharedPool-Worker-280" daemon > [_thread_blocked, id=1110, stack(0x00007f0722003000,0x00007f0722044000)] > 0x00007f0693832000 JavaThread "MessagingService-Incoming-/52.221.228.170" > [_thread_in_native, id=1101, stack(0x00007f0717dd2000,0x00007f0717e13000)] > 0x00007f06934f6000 JavaThread "MessagingService-Incoming-/52.221.228.170" > [_thread_in_native, id=1096, stack(0x00007f0717ed6000,0x00007f0717f17000)] > 0x00007f04b00d5000 JavaThread "SharedPool-Worker-279" daemon > [_thread_blocked, id=1093, stack(0x00007f0717f17000,0x00007f0717f58000)] > 0x00007f04b00d4000 JavaThread "SharedPool-Worker-278" daemon > [_thread_blocked, id=1092, stack(0x00007f0717f58000,0x00007f0717f99000)] > 0x00007f04b01df000 JavaThread "SharedPool-Worker-271" daemon > [_thread_blocked, id=1091, stack(0x00007f07205b4000,0x00007f07205f5000)] > 0x00007f04b01de000 JavaThread "SharedPool-Worker-274" daemon > [_thread_blocked, id=1090, stack(0x00007f07205f5000,0x00007f0720636000)] > 0x00007f074c1c7000 JavaThread "SharedPool-Worker-273" daemon > [_thread_blocked, id=1089, stack(0x00007f0720636000,0x00007f0720677000)] > 0x00007f074c1c6800 JavaThread "SharedPool-Worker-276" daemon > [_thread_blocked, id=1088, stack(0x00007f0720677000,0x00007f07206b8000)] > 0x00007f04b02a4800 JavaThread "SharedPool-Worker-277" daemon > [_thread_blocked, id=1087, stack(0x00007f07206b8000,0x00007f07206f9000)] > 0x00007f04b02a4000 JavaThread "SharedPool-Worker-269" daemon > [_thread_blocked, id=1086, stack(0x00007f07206f9000,0x00007f072073a000)] > 0x00007f074c2ae000 JavaThread "SharedPool-Worker-268" daemon > [_thread_blocked, id=1085, stack(0x00007f072073a000,0x00007f072077b000)] > 0x00007f074c2ad000 JavaThread "SharedPool-Worker-275" daemon > [_thread_blocked, id=1084, stack(0x00007f072077b000,0x00007f07207bc000)] > 0x00007f0693ec3000 JavaThread "SharedPool-Worker-272" daemon > [_thread_blocked, id=1083, stack(0x00007f07207bc000,0x00007f07207fd000)] > 0x00007f0693ec2000 JavaThread "SharedPool-Worker-270" daemon > [_thread_blocked, id=1082, stack(0x00007f07207fd000,0x00007f072083e000)] > 0x00007f0690983800 JavaThread "SharedPool-Worker-267" daemon > [_thread_blocked, id=1081, stack(0x00007f072083e000,0x00007f072087f000)] > 0x00007f0690983000 JavaThread "SharedPool-Worker-265" daemon > [_thread_blocked, id=1080, stack(0x00007f0720be6000,0x00007f0720c27000)] > 0x00007f04b0358000 JavaThread "SharedPool-Worker-266" daemon > [_thread_blocked, id=1079, stack(0x00007f0720c27000,0x00007f0720c68000)] > 0x00007f04b0357000 JavaThread "SharedPool-Worker-263" daemon > [_thread_blocked, id=1078, stack(0x00007f0720c68000,0x00007f0720ca9000)] > 0x00007f04b05e6800 JavaThread "SharedPool-Worker-264" daemon > [_thread_blocked, id=1077, stack(0x00007f0720ca9000,0x00007f0720cea000)] > 0x00007f04b05e5800 JavaThread "SharedPool-Worker-262" daemon > [_thread_blocked, id=1076, stack(0x00007f0720cea000,0x00007f0720d2b000)] > 0x00007f0786e27800 JavaThread "SharedPool-Worker-261" daemon > [_thread_blocked, id=1074, stack(0x00007f0720d2b000,0x00007f0720d6c000)] > 0x00007f0786e27000 JavaThread "SharedPool-Worker-258" daemon > [_thread_blocked, id=1073, stack(0x00007f0720d6c000,0x00007f0720dad000)] > 0x00007f069366c800 JavaThread "SharedPool-Worker-259" daemon > [_thread_blocked, id=1072, stack(0x00007f0720dad000,0x00007f0720dee000)] > 0x00007f069366c000 JavaThread "SharedPool-Worker-260" daemon > [_thread_blocked, id=1071, stack(0x00007f0720dee000,0x00007f0720e2f000)] > 0x00007f0693661000 JavaThread "SharedPool-Worker-257" daemon > [_thread_blocked, id=1070, stack(0x00007f0720e2f000,0x00007f0720e70000)] > 0x00007f0693660000 JavaThread "SharedPool-Worker-256" daemon > [_thread_blocked, id=1069, stack(0x00007f0720e70000,0x00007f0720eb1000)] > 0x00007f0693603000 JavaThread "SharedPool-Worker-255" daemon > [_thread_blocked, id=1068, stack(0x00007f0720f61000,0x00007f0720fa2000)] > 0x00007f0693602800 JavaThread "SharedPool-Worker-252" daemon > [_thread_blocked, id=1067, stack(0x00007f0720eb1000,0x00007f0720ef2000)] > 0x00007f0693fd7000 JavaThread "SharedPool-Worker-251" daemon > [_thread_blocked, id=1066, stack(0x00007f0749d7d000,0x00007f0749dbe000)] > 0x00007f04b0109800 JavaThread "SharedPool-Worker-254" daemon > [_thread_blocked, id=1065, stack(0x00007f04af508000,0x00007f04af549000)] > 0x00007f0787c49800 JavaThread "SharedPool-Worker-253" daemon > [_thread_blocked, id=1064, stack(0x00007f04b4030000,0x00007f04b4071000)] > 0x00007f04b00d9000 JavaThread "SharedPool-Worker-249" daemon > [_thread_blocked, id=1063, stack(0x00007f04b772a000,0x00007f04b776b000)] > ... > {code} > Complete error log is at > https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/error.log > A screenshot load, memory usage etc. is at > https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/Voila_Capture%202017-05-11_09-08-53_am.png -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org