[
https://issues.apache.org/jira/browse/CASSANDRA-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-13523:
---------------------------------------
Environment: Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version 1.9.1,
run as a container, 4 core server with 16GB memory. (was: Ubuntu 14.04.5 LTS,
Docker version 1.9.1, run as a container, 4 core server with 16GB memory.)
> StreamReceiveTask: java.lang.OutOfMemoryError: Map failed
> ---------------------------------------------------------
>
> Key: CASSANDRA-13523
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13523
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version
> 1.9.1, run as a container, 4 core server with 16GB memory.
> Reporter: Matthew O'Riordan
> Priority: Major
> Labels: bug, crash
> Fix For: 2.1.13
>
>
> During a nodetool repair -par on one of our keyspaces, Cassandra crashed due
> to what seems like memory exhaustion within the JVM. The machine itself had
> plenty of available memory at the time and did not appear to be under any
> significant load.
> In the system log, before the crash, the following was logged:
> {code}
> ...
> INFO [AntiEntropySessions:55] 2017-05-10 18:18:20,627 RepairJob.java:163 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] requesting merkle trees for
> stats_day_aggregates (to [/54.162.66.114, /54.236.226.76, /52.221.228.170,
> /54.154.35.144, /54.154.96.213, /52.221.217.27])
> INFO [ValidationExecutor:54] 2017-05-10 18:18:20,628
> ColumnFamilyStore.java:905 - Enqueuing flush of stats_day_aggregates: 7018
> (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,628 Memtable.java:347
> - Writing Memtable-stats_day_aggregates@3469792(2.734KiB serialized bytes, 14
> ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,629 Memtable.java:382
> - Completed flushing
> /var/lib/cassandra/data/ably_production_0_stats/stats_day_aggregates-b6e29201e3d111e5bbf3091830ac5256/ably_production_0_stats-stats_day_aggregates-tmp-ka-43026-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1491420635101,
> position=24224955)
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:180 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> Session with /52.203.21.193 is complete
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:212 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> All sessions completed
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,221
> StreamingRepairTask.java:96 - [repair #fe7e3320-35ac-11e7-b7e4-091830ac5256]
> streaming task succeed, returning response to /52.221.217.27
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,230
> CqlSlowLogWriter.java:151 - Recording statements with duration of 4844 in
> slow log
> INFO [Service Thread] 2017-05-10 18:18:26,233 GCInspector.java:258 - G1 Old
> Generation GC in 4781ms. G1 Eden Space: 1310720000 -> 0; G1 Old Gen:
> 2774539816 -> 1830851216; G1 Survivor Space: 37748736 -> 0;
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,237 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.162.66.114
> INFO [StreamConnectionEstablisher:1] 2017-05-10 18:18:26,239
> StreamCoordinator.java:209 - [Stream #0293bb60-35ad-11e7-b7e4-091830ac5256,
> ID#0] Beginning stream session with /54.154.226.20
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,294 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.236.226.76
> INFO [Service Thread] 2017-05-10 18:18:26,298 StatusLogger.java:51 - Pool
> Name Active Pending Completed Blocked All Time
> Blocked
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.35.144
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,344
> CqlSlowLogWriter.java:151 - Recording statements with duration of 5035 in
> slow log
> WARN [GossipTasks:1] 2017-05-10 18:18:26,344 FailureDetector.java:258 - Not
> marking nodes down due to local pause of 5109502584 > 5000000000
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.96.213
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /52.221.228.170
> INFO [Service Thread] 2017-05-10 18:18:26,345 StatusLogger.java:66 -
> MutationStage 0 0 1199223432 0
> 0
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,345
> CqlSlowLogWriter.java:151 - Recording statements with duration of 5095 in
> slow log
> ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,346
> StreamReceiveTask.java:183 - Error applying streamed data:
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:399)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_101]
> at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> ... 11 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_101]
> ... 12 common frames omitted
> ERROR [StreamReceiveTask:640] 2017-05-10 18:18:26,349
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_101]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_101]
> at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:391)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:365)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:174)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:463)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:141)
> ~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}
> The JVM then exited completely and logged the following:
> {code}
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 12288 bytes for committing
> reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # In 32 bit mode, the process size limit was hit
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Use 64 bit Java on a 64 bit OS
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2627), pid=79, tid=0x00007f06b6e56700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> --------------- T H R E A D ---------------
> Current thread (0x00007f03d5054000): JavaThread "Thread-1" [_thread_new,
> id=23656, stack(0x00007f06b6e16000,0x00007f06b6e57000)]
> Stack: [0x00007f06b6e16000,0x00007f06b6e57000], sp=0x00007f06b6e55720, free
> space=253k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0xac3f0a] VMError::report_and_die()+0x2ba
> V [libjvm.so+0x4fbf9b] report_vm_out_of_memory(char const*, int, unsigned
> long, VMErrorType, char const*)+0x8b
> V [libjvm.so+0x9222d3] os::Linux::commit_memory_impl(char*, unsigned long,
> bool)+0x103
> V [libjvm.so+0x92239c] os::pd_commit_memory(char*, unsigned long, bool)+0xc
> V [libjvm.so+0x91be4a] os::commit_memory(char*, unsigned long, bool)+0x2a
> V [libjvm.so+0x92056f] os::pd_create_stack_guard_pages(char*, unsigned
> long)+0x7f
> V [libjvm.so+0xa69ace] JavaThread::create_stack_guard_pages()+0x5e
> V [libjvm.so+0xa73484] JavaThread::run()+0x34
> V [libjvm.so+0x924698] java_start(Thread*)+0x108
> C [libpthread.so.0+0x8184] start_thread+0xc4
> --------------- P R O C E S S ---------------
> Java Threads: ( => current thread )
> 0x00007f03d578c000 JavaThread "Daemon shutdown" [_thread_new, id=23658,
> stack(0x00007f074a62a000,0x00007f074a66b000)]
> 0x00007f03d5278000 JavaThread "Thread-0" [_thread_new, id=23657,
> stack(0x00007f074422f000,0x00007f0744270000)]
> =>0x00007f03d5054000 JavaThread "Thread-1" [_thread_new, id=23656,
> stack(0x00007f06b6e16000,0x00007f06b6e57000)]
> 0x00007f03d5187800 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23644, stack(0x00007f0749704000,0x00007f0749745000)]
> 0x00007f03d465d800 JavaThread "STREAM-OUT-/54.169.103.14" daemon
> [_thread_blocked, id=23627, stack(0x00007f07493f7000,0x00007f0749438000)]
> 0x00007f05ac99d000 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23625, stack(0x00007f074ab13000,0x00007f074ab54000)]
> 0x00007f03d4620800 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23591, stack(0x00007f07253cb000,0x00007f072540c000)]
> 0x00007f03d5086000 JavaThread "STREAM-IN-/54.154.226.20" daemon
> [_thread_in_native, id=23511, stack(0x00007f074986b000,0x00007f07498ac000)]
> 0x00007f0449a77000 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23508, stack(0x00007f0747758000,0x00007f0747799000)]
> 0x00007f03d5185000 JavaThread "STREAM-IN-/54.154.226.20" daemon
> [_thread_in_native, id=23488, stack(0x00007f07491bc000,0x00007f07491fd000)]
> 0x00007f03d5104000 JavaThread "STREAM-IN-/54.154.226.20" daemon
> [_thread_in_native, id=23473, stack(0x00007f0743e40000,0x00007f0743e81000)]
> 0x00007f0448b64000 JavaThread "STREAM-IN-/54.154.226.20" daemon
> [_thread_in_native, id=23393, stack(0x00007f074917b000,0x00007f07491bc000)]
> 0x00007f03d4f7d800 JavaThread "STREAM-OUT-/52.203.21.193" daemon
> [_thread_blocked, id=23369, stack(0x00007f0749966000,0x00007f07499a7000)]
> 0x00007f03d4fed800 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23367, stack(0x00007f074416c000,0x00007f07441ad000)]
> 0x00007f03d5302000 JavaThread "STREAM-OUT-/54.154.226.20" daemon
> [_thread_blocked, id=23303, stack(0x00007f0722044000,0x00007f0722085000)]
> 0x00007f044a921000 JavaThread "RepairJobTask:15" daemon [_thread_blocked,
> id=23131, stack(0x00007f074ae05000,0x00007f074ae46000)]
> 0x00007f044b9ae000 JavaThread "RepairJobTask:14" daemon [_thread_blocked,
> id=23121, stack(0x00007f074982a000,0x00007f074986b000)]
> 0x00007f044a920000 JavaThread "RepairJobTask:13" daemon [_thread_blocked,
> id=23116, stack(0x00007f074366e000,0x00007f07436af000)]
> 0x00007f03d45eb000 JavaThread "RepairJobTask:12" daemon [_thread_blocked,
> id=23115, stack(0x00007f0717d50000,0x00007f0717d91000)]
> 0x00007f03d45ea800 JavaThread "RepairJobTask:11" daemon [_thread_blocked,
> id=23103, stack(0x00007f0743f64000,0x00007f0743fa5000)]
> 0x00007f03d4fce000 JavaThread "RepairJobTask:10" daemon [_thread_blocked,
> id=23102, stack(0x00007f074ae6f000,0x00007f074aeb0000)]
> 0x00007f03d4e78800 JavaThread "RepairJobTask:9" daemon [_thread_blocked,
> id=23101, stack(0x00007f0720fa2000,0x00007f0720fe3000)]
> 0x00007f03d50bc000 JavaThread "RepairJobTask:8" daemon [_thread_blocked,
> id=23100, stack(0x00007f07210a6000,0x00007f07210e7000)]
> 0x00007f03d506f000 JavaThread "RepairJobTask:7" daemon [_thread_blocked,
> id=23099, stack(0x00007f074ad2d000,0x00007f074ad6e000)]
> 0x00007f03d5146000 JavaThread "RepairJobTask:6" daemon [_thread_blocked,
> id=23098, stack(0x00007f0747c0b000,0x00007f0747c4c000)]
> 0x00007f03d4f6d800 JavaThread "RepairJobTask:5" daemon [_thread_blocked,
> id=23097, stack(0x00007f0750043000,0x00007f0750084000)]
> 0x00007f03d5079800 JavaThread "RepairJobTask:4" daemon [_thread_blocked,
> id=23096, stack(0x00007f0742dde000,0x00007f0742e1f000)]
> 0x00007f03d4f94000 JavaThread "RepairJobTask:3" daemon [_thread_blocked,
> id=23095, stack(0x00007f074a5c4000,0x00007f074a605000)]
> 0x00007f03d52a6000 JavaThread "RepairJobTask:2" daemon [_thread_blocked,
> id=23094, stack(0x00007f072122c000,0x00007f072126d000)]
> 0x00007f03d52ee800 JavaThread "RepairJobTask:1" daemon [_thread_blocked,
> id=23093, stack(0x00007f07436f0000,0x00007f0743731000)]
> 0x00007f057f042000 JavaThread "RepairJobTask:15" daemon [_thread_blocked,
> id=23078, stack(0x00007f074a799000,0x00007f074a7da000)]
> 0x00007f03d57a4800 JavaThread "RepairJobTask:14" daemon [_thread_blocked,
> id=23077, stack(0x00007f0683bd4000,0x00007f0683c15000)]
> 0x00007f044ba0f800 JavaThread "RepairJobTask:13" daemon [_thread_blocked,
> id=23035, stack(0x00007f074aa41000,0x00007f074aa82000)]
> 0x00007f03d50b9000 JavaThread "RepairJobTask:12" daemon [_thread_blocked,
> id=23010, stack(0x00007f074384e000,0x00007f074388f000)]
> 0x00007f03d50b7800 JavaThread "RepairJobTask:11" daemon [_thread_blocked,
> id=23009, stack(0x00007f0749380000,0x00007f07493c1000)]
> 0x00007f03d50b6000 JavaThread "RepairJobTask:10" daemon [_thread_blocked,
> id=23008, stack(0x00007f07438d0000,0x00007f0743911000)]
> 0x00007f03d50b5800 JavaThread "RepairJobTask:9" daemon [_thread_blocked,
> id=23007, stack(0x00007f0744027000,0x00007f0744068000)]
> 0x00007f03d50af000 JavaThread "RepairJobTask:8" daemon [_thread_blocked,
> id=23006, stack(0x00007f074388f000,0x00007f07438d0000)]
> 0x00007f03d5097800 JavaThread "RepairJobTask:7" daemon [_thread_blocked,
> id=22989, stack(0x00007f074ba63000,0x00007f074baa4000)]
> 0x00007f03d5096800 JavaThread "RepairJobTask:6" daemon [_thread_blocked,
> id=22988, stack(0x00007f0742b98000,0x00007f0742bd9000)]
> 0x00007f03d5095000 JavaThread "RepairJobTask:5" daemon [_thread_blocked,
> id=22987, stack(0x00007f0721fc2000,0x00007f0722003000)]
> 0x00007f03d5093800 JavaThread "RepairJobTask:4" daemon [_thread_blocked,
> id=22986, stack(0x00007f0743f23000,0x00007f0743f64000)]
> 0x00007f03d5092800 JavaThread "RepairJobTask:3" daemon [_thread_blocked,
> id=22985, stack(0x00007f074a99e000,0x00007f074a9df000)]
> 0x00007f03d5091000 JavaThread "RepairJobTask:2" daemon [_thread_blocked,
> id=22984, stack(0x00007f074a6ac000,0x00007f074a6ed000)]
> 0x00007f03d5090800 JavaThread "RepairJobTask:1" daemon [_thread_blocked,
> id=22983, stack(0x00007f0749745000,0x00007f0749786000)]
> 0x00007f03d4f7a000 JavaThread "ReadRepairStage:73775" daemon
> [_thread_blocked, id=22227, stack(0x00007f074aabc000,0x00007f074aafd000)]
> 0x00007f044a94f800 JavaThread "ReadRepairStage:73774" daemon
> [_thread_blocked, id=22226, stack(0x00007f0744068000,0x00007f07440a9000)]
> 0x00007f03d437d000 JavaThread "ReadRepairStage:73773" daemon
> [_thread_blocked, id=21837, stack(0x00007f0743772000,0x00007f07437b3000)]
> 0x00007f03d4471000 JavaThread "ReadRepairStage:73770" daemon
> [_thread_blocked, id=18261, stack(0x00007f07491fd000,0x00007f074923e000)]
> 0x00007f060f438800 JavaThread "RMI TCP Connection(idle)" daemon
> [_thread_blocked, id=15398, stack(0x00007f0747011000,0x00007f0747052000)]
> 0x00007f060fa59000 JavaThread "RMI TCP Connection(5238)-172.17.0.5" daemon
> [_thread_blocked, id=10756, stack(0x00007f0743fa5000,0x00007f0743fe6000)]
> 0x00007f044b9aa000 JavaThread "ValidationExecutor:54" daemon
> [_thread_in_Java, id=9233, stack(0x00007f0748b0a000,0x00007f0748b4b000)]
> 0x00007f04494ef800 JavaThread "SharedPool-Worker-407" daemon
> [_thread_blocked, id=3652, stack(0x00007f074907e000,0x00007f07490bf000)]
> 0x00007f057e393000 JavaThread "SharedPool-Worker-406" daemon
> [_thread_blocked, id=3651, stack(0x00007f0744270000,0x00007f07442b1000)]
> 0x00007f044a847000 JavaThread "ValidationExecutor:51" daemon
> [_thread_blocked, id=442, stack(0x00007f07498ed000,0x00007f074992e000)]
> 0x00007f04b147b000 JavaThread "AntiEntropySessions:56" daemon
> [_thread_blocked, id=1481, stack(0x00007f0742b16000,0x00007f0742b57000)]
> 0x00007f0693cd4800 JavaThread "StreamReceiveTask:640" daemon
> [_thread_blocked, id=1467, stack(0x00007f074acec000,0x00007f074ad2d000)]
> 0x00007f05d9c8d800 JavaThread "StreamReceiveTask:639" daemon
> [_thread_blocked, id=1458, stack(0x00007f0749635000,0x00007f0749676000)]
> 0x00007f04b3e37000 JavaThread "StreamReceiveTask:638" daemon
> [_thread_blocked, id=1434, stack(0x00007f074a583000,0x00007f074a5c4000)]
> 0x00007f06924f7800 JavaThread "AntiEntropySessions:55" daemon
> [_thread_blocked, id=1412, stack(0x00007f074781b000,0x00007f074785c000)]
> 0x00007f074ceaf000 JavaThread "StreamReceiveTask:637" daemon
> [_thread_blocked, id=1401, stack(0x00007f0722085000,0x00007f07220c6000)]
> 0x00007f069398e800 JavaThread "MemtableFlushWriter:13609" daemon
> [_thread_blocked, id=1380, stack(0x00007f0748ac9000,0x00007f0748b0a000)]
> 0x00007f069374e800 JavaThread "MemtableReclaimMemory:9889" daemon
> [_thread_blocked, id=1379, stack(0x00007f07440ea000,0x00007f074412b000)]
> 0x00007f0787c63800 JavaThread "MemtableFlushWriter:13608" daemon
> [_thread_blocked, id=1378, stack(0x00007f074412b000,0x00007f074416c000)]
> 0x00007f0693cc6000 JavaThread "AntiEntropySessions:54" daemon
> [_thread_blocked, id=1376, stack(0x00007f07436af000,0x00007f07436f0000)]
> 0x00007f06936ff000 JavaThread "MemtablePostFlush:36174" daemon
> [_thread_blocked, id=1374, stack(0x00007f07442b1000,0x00007f07442f2000)]
> 0x00007f0787c19000 JavaThread "Thread-280182" daemon [_thread_blocked,
> id=1371, stack(0x00007f07498ac000,0x00007f07498ed000)]
> 0x00007f074f9da800 JavaThread "JMX server connection timeout 768070" daemon
> [_thread_blocked, id=1369, stack(0x00007f0743ee2000,0x00007f0743f23000)]
> 0x0000000001fed800 JavaThread "CompactionExecutor:16697" daemon
> [_thread_in_Java, id=1365, stack(0x00007f074b8ec000,0x00007f074b92d000)]
> 0x0000000001367800 JavaThread "CompactionExecutor:16696" daemon
> [_thread_in_Java, id=1364, stack(0x00007f0747695000,0x00007f07476d6000)]
> 0x00007f0690975800 JavaThread "MessagingService-Incoming-/54.174.252.207"
> [_thread_in_native, id=30389, stack(0x00007f07499e8000,0x00007f0749a29000)]
> 0x00007f074cd88800 JavaThread "MessagingService-Incoming-/54.174.252.207"
> [_thread_in_native, id=30381, stack(0x00007f0743911000,0x00007f0743952000)]
> 0x00007f0690a6a000 JavaThread "AntiEntropySessions:50" daemon
> [_thread_blocked, id=24703, stack(0x00007f07441ee000,0x00007f074422f000)]
> 0x00007f06924e9000 JavaThread "Thread-169959" daemon [_thread_blocked,
> id=24700, stack(0x00007f07476d6000,0x00007f0747717000)]
> 0x00007f074e24a000 JavaThread "JMX server connection timeout 174925" daemon
> [_thread_blocked, id=15333, stack(0x00007f04b808f000,0x00007f04b80d0000)]
> 0x00007f060cc21000 JavaThread "RMI TCP Connection(364)-172.17.0.4" daemon
> [_thread_blocked, id=15332, stack(0x00007f0683d57000,0x00007f0683d98000)]
> 0x00007f074f3e5800 JavaThread "StreamConnectionEstablisher:4" daemon
> [_thread_in_native, id=32671, stack(0x00007f074a6fb000,0x00007f074a73c000)]
> 0x00007f069044a800 JavaThread "StreamingTransferTaskTimeouts:1" daemon
> [_thread_blocked, id=32667, stack(0x00007f0747799000,0x00007f07477da000)]
> 0x00007f074d19d800 JavaThread "StreamConnectionEstablisher:3" daemon
> [_thread_in_native, id=32663, stack(0x00007f0747bca000,0x00007f0747c0b000)]
> 0x00007f078770d800 JavaThread "StreamConnectionEstablisher:2" daemon
> [_thread_blocked, id=32660, stack(0x00007f074a53e000,0x00007f074a57f000)]
> 0x00007f04b070d000 JavaThread "StreamConnectionEstablisher:1" daemon
> [_thread_blocked, id=32655, stack(0x00007f074ac70000,0x00007f074acb1000)]
> 0x00007f0693fd0800 JavaThread "MessagingService-Incoming-/172.17.0.2"
> [_thread_in_native, id=32641, stack(0x00007f0747c4c000,0x00007f0747c8d000)]
> 0x00007f06931c5800 JavaThread
> "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked,
> id=32639, stack(0x00007f07495ef000,0x00007f0749630000)]
> 0x00007f04b028d000 JavaThread
> "MessagingService-Outgoing-9b57701d1728/172.17.0.2" daemon [_thread_blocked,
> id=32638, stack(0x00007f07497cd000,0x00007f074980e000)]
> 0x00007f07876a6800 JavaThread "MessagingService-Incoming-/54.162.66.114"
> [_thread_in_native, id=13194, stack(0x00007f074a66b000,0x00007f074a6ac000)]
> 0x00007f0693644800 JavaThread "MessagingService-Incoming-/54.236.226.76"
> [_thread_in_native, id=13192, stack(0x00007f0747b1b000,0x00007f0747b5c000)]
> 0x00007f04b02cf800 JavaThread "MessagingService-Incoming-/54.236.226.76"
> [_thread_in_native, id=13191, stack(0x00007f074a7da000,0x00007f074a81b000)]
> 0x00007f0693f5d800 JavaThread "MessagingService-Incoming-/54.154.35.144"
> [_thread_in_native, id=13189, stack(0x00007f074a9fd000,0x00007f074aa3e000)]
> 0x00007f0690909800 JavaThread "MessagingService-Incoming-/54.171.109.9"
> [_thread_in_native, id=13184, stack(0x00007f07499a7000,0x00007f07499e8000)]
> 0x00007f04b0728000 JavaThread "MessagingService-Incoming-/54.162.66.114"
> [_thread_in_native, id=13182, stack(0x00007f0683c15000,0x00007f0683c56000)]
> 0x00007f0690e07000 JavaThread "MessagingService-Incoming-/54.154.96.213"
> [_thread_in_native, id=13180, stack(0x00007f0707ec2000,0x00007f0707f03000)]
> 0x00007f05aca67800 JavaThread "MessagingService-Incoming-/54.154.226.20"
> [_thread_blocked, id=13177, stack(0x00007f0747052000,0x00007f0747093000)]
> 0x00007f04b10c6000 JavaThread "MessagingService-Incoming-/54.154.35.144"
> [_thread_in_native, id=13174, stack(0x00007f0747b5c000,0x00007f0747b9d000)]
> 0x00007f04b0e58800 JavaThread "MessagingService-Incoming-/54.154.96.213"
> [_thread_in_native, id=13173, stack(0x00007f0749bf9000,0x00007f0749c3a000)]
> 0x00007f04b07a1800 JavaThread "MessagingService-Incoming-/54.171.109.9"
> [_thread_in_native, id=13172, stack(0x00007f0750170000,0x00007f07501b1000)]
> 0x00007f0690e1a800 JavaThread "MessagingService-Incoming-/54.154.226.20"
> [_thread_in_native, id=13170, stack(0x00007f0707f85000,0x00007f0707fc6000)]
> 0x00007f04b02d0800 JavaThread "MessagingService-Incoming-/52.203.21.193"
> [_thread_in_native, id=13159, stack(0x00007f0717e13000,0x00007f0717e54000)]
> 0x00007f04b1bd2000 JavaThread "MessagingService-Incoming-/52.203.21.193"
> [_thread_in_native, id=13155, stack(0x00007f0747188000,0x00007f07471c9000)]
> 0x00007f04b0090000 JavaThread "SharedPool-Worker-398" daemon
> [_thread_blocked, id=2502, stack(0x00007f0683d98000,0x00007f0683dd9000)]
> 0x00007f0786dc5000 JavaThread "SharedPool-Worker-397" daemon
> [_thread_blocked, id=2501, stack(0x00007f0683dd9000,0x00007f0683e1a000)]
> 0x00007f06938cc000 JavaThread "SharedPool-Worker-399" daemon
> [_thread_blocked, id=2500, stack(0x00007f0683e1a000,0x00007f0683e5b000)]
> 0x00007f06937eb800 JavaThread "SharedPool-Worker-400" daemon
> [_thread_blocked, id=2499, stack(0x00007f0683e5b000,0x00007f0683e9c000)]
> 0x00007f0693686800 JavaThread "SharedPool-Worker-401" daemon
> [_thread_blocked, id=2498, stack(0x00007f0683e9c000,0x00007f0683edd000)]
> 0x00007f07877f8800 JavaThread "SharedPool-Worker-402" daemon
> [_thread_blocked, id=2497, stack(0x00007f0683edd000,0x00007f0683f1e000)]
> 0x00007f074c101800 JavaThread "SharedPool-Worker-403" daemon
> [_thread_blocked, id=2496, stack(0x00007f0683f1e000,0x00007f0683f5f000)]
> 0x00007f069397f000 JavaThread "SharedPool-Worker-404" daemon
> [_thread_blocked, id=2495, stack(0x00007f068566d000,0x00007f06856ae000)]
> 0x00007f0693572000 JavaThread "SharedPool-Worker-392" daemon
> [_thread_blocked, id=2494, stack(0x00007f06856ae000,0x00007f06856ef000)]
> 0x00007f0786f0f000 JavaThread "SharedPool-Worker-405" daemon
> [_thread_blocked, id=2493, stack(0x00007f0685995000,0x00007f06859d6000)]
> 0x00007f0786a7b000 JavaThread "SharedPool-Worker-376" daemon
> [_thread_blocked, id=2492, stack(0x00007f06859d6000,0x00007f0685a17000)]
> 0x00007f0693ef4800 JavaThread "SharedPool-Worker-379" daemon
> [_thread_blocked, id=2491, stack(0x00007f0685a17000,0x00007f0685a58000)]
> 0x00007f04b0281000 JavaThread "SharedPool-Worker-378" daemon
> [_thread_blocked, id=2490, stack(0x00007f0685a58000,0x00007f0685a99000)]
> 0x00007f04b0280000 JavaThread "SharedPool-Worker-381" daemon
> [_thread_blocked, id=2489, stack(0x00007f0685a99000,0x00007f0685ada000)]
> 0x00007f069046b800 JavaThread "SharedPool-Worker-382" daemon
> [_thread_blocked, id=2488, stack(0x00007f0685ada000,0x00007f0685b1b000)]
> 0x00007f069046b000 JavaThread "SharedPool-Worker-383" daemon
> [_thread_blocked, id=2487, stack(0x00007f0685b1b000,0x00007f0685b5c000)]
> 0x00007f0693a5f000 JavaThread "SharedPool-Worker-385" daemon
> [_thread_blocked, id=2486, stack(0x00007f0685b5c000,0x00007f0685b9d000)]
> 0x00007f0693a5e800 JavaThread "SharedPool-Worker-370" daemon
> [_thread_blocked, id=2485, stack(0x00007f0685b9d000,0x00007f0685bde000)]
> 0x00007f04b0649000 JavaThread "SharedPool-Worker-373" daemon
> [_thread_blocked, id=2484, stack(0x00007f0685bde000,0x00007f0685c1f000)]
> 0x00007f04b0648800 JavaThread "SharedPool-Worker-386" daemon
> [_thread_blocked, id=2483, stack(0x00007f0685c1f000,0x00007f0685c60000)]
> 0x00007f07876a5000 JavaThread "SharedPool-Worker-387" daemon
> [_thread_blocked, id=2482, stack(0x00007f0685c60000,0x00007f0685ca1000)]
> 0x00007f07876a4800 JavaThread "SharedPool-Worker-388" daemon
> [_thread_blocked, id=2481, stack(0x00007f0685ca1000,0x00007f0685ce2000)]
> 0x00007f04b0c91000 JavaThread "SharedPool-Worker-391" daemon
> [_thread_blocked, id=2480, stack(0x00007f0685ce2000,0x00007f0685d23000)]
> 0x00007f04b0c90000 JavaThread "SharedPool-Worker-394" daemon
> [_thread_blocked, id=2479, stack(0x00007f069404a000,0x00007f069408b000)]
> 0x00007f04b0752800 JavaThread "SharedPool-Worker-393" daemon
> [_thread_blocked, id=2478, stack(0x00007f069408b000,0x00007f06940cc000)]
> 0x00007f04b0751800 JavaThread "SharedPool-Worker-396" daemon
> [_thread_blocked, id=2477, stack(0x00007f06940cc000,0x00007f069410d000)]
> 0x00007f074caf4800 JavaThread "SharedPool-Worker-362" daemon
> [_thread_blocked, id=2476, stack(0x00007f069410d000,0x00007f069414e000)]
> 0x00007f074caf3800 JavaThread "SharedPool-Worker-395" daemon
> [_thread_blocked, id=2475, stack(0x00007f069414e000,0x00007f069418f000)]
> 0x00007f069393a800 JavaThread "SharedPool-Worker-364" daemon
> [_thread_blocked, id=2474, stack(0x00007f069418f000,0x00007f06941d0000)]
> 0x00007f069393a000 JavaThread "SharedPool-Worker-365" daemon
> [_thread_blocked, id=2473, stack(0x00007f06941d0000,0x00007f0694211000)]
> 0x00007f078765a000 JavaThread "SharedPool-Worker-390" daemon
> [_thread_blocked, id=2472, stack(0x00007f0694211000,0x00007f0694252000)]
> 0x00007f0787659800 JavaThread "SharedPool-Worker-389" daemon
> [_thread_blocked, id=2471, stack(0x00007f0694252000,0x00007f0694293000)]
> 0x00007f06938b2000 JavaThread "SharedPool-Worker-366" daemon
> [_thread_blocked, id=2470, stack(0x00007f0694293000,0x00007f06942d4000)]
> 0x00007f06938b1000 JavaThread "SharedPool-Worker-368" daemon
> [_thread_blocked, id=2469, stack(0x00007f06942d4000,0x00007f0694315000)]
> 0x00007f04b090e800 JavaThread "SharedPool-Worker-369" daemon
> [_thread_blocked, id=2468, stack(0x00007f0694315000,0x00007f0694356000)]
> 0x00007f04b090e000 JavaThread "SharedPool-Worker-384" daemon
> [_thread_blocked, id=2467, stack(0x00007f0694356000,0x00007f0694397000)]
> 0x00007f04b0866000 JavaThread "SharedPool-Worker-374" daemon
> [_thread_blocked, id=2466, stack(0x00007f06946ce000,0x00007f069470f000)]
> 0x00007f04b0865000 JavaThread "SharedPool-Worker-380" daemon
> [_thread_blocked, id=2465, stack(0x00007f069470f000,0x00007f0694750000)]
> 0x00007f074cde4800 JavaThread "SharedPool-Worker-377" daemon
> [_thread_blocked, id=2464, stack(0x00007f0694750000,0x00007f0694791000)]
> 0x00007f074cde3800 JavaThread "SharedPool-Worker-375" daemon
> [_thread_blocked, id=2463, stack(0x00007f0695812000,0x00007f0695853000)]
> 0x00007f074ce69000 JavaThread "SharedPool-Worker-359" daemon
> [_thread_blocked, id=2462, stack(0x00007f06b60da000,0x00007f06b611b000)]
> 0x00007f074ce68800 JavaThread "SharedPool-Worker-372" daemon
> [_thread_blocked, id=2461, stack(0x00007f06b611b000,0x00007f06b615c000)]
> 0x00007f07871f8800 JavaThread "SharedPool-Worker-371" daemon
> [_thread_blocked, id=2460, stack(0x00007f06b7cc5000,0x00007f06b7d06000)]
> 0x00007f07871f8000 JavaThread "SharedPool-Worker-361" daemon
> [_thread_blocked, id=2459, stack(0x00007f06c081e000,0x00007f06c085f000)]
> 0x00007f04b0c11000 JavaThread "SharedPool-Worker-367" daemon
> [_thread_blocked, id=2458, stack(0x00007f06c085f000,0x00007f06c08a0000)]
> 0x00007f04b0c10800 JavaThread "SharedPool-Worker-355" daemon
> [_thread_blocked, id=2457, stack(0x00007f06c0ae6000,0x00007f06c0b27000)]
> 0x00007f074cb38000 JavaThread "SharedPool-Worker-363" daemon
> [_thread_blocked, id=2456, stack(0x00007f06c0b27000,0x00007f06c0b68000)]
> 0x00007f074cb37800 JavaThread "SharedPool-Worker-353" daemon
> [_thread_blocked, id=2455, stack(0x00007f06c0b68000,0x00007f06c0ba9000)]
> 0x00007f0787de6800 JavaThread "SharedPool-Worker-360" daemon
> [_thread_blocked, id=2454, stack(0x00007f06c0ba9000,0x00007f06c0bea000)]
> 0x00007f0787de6000 JavaThread "SharedPool-Worker-356" daemon
> [_thread_blocked, id=2453, stack(0x00007f06c0bea000,0x00007f06c0c2b000)]
> 0x00007f04b0146000 JavaThread "SharedPool-Worker-358" daemon
> [_thread_blocked, id=2452, stack(0x00007f06c0c2b000,0x00007f06c0c6c000)]
> 0x00007f04b0145000 JavaThread "SharedPool-Worker-348" daemon
> [_thread_blocked, id=2451, stack(0x00007f06d5e0c000,0x00007f06d5e4d000)]
> 0x00007f0787ea3000 JavaThread "SharedPool-Worker-357" daemon
> [_thread_blocked, id=2450, stack(0x00007f06d5e4d000,0x00007f06d5e8e000)]
> 0x00007f0787ea2000 JavaThread "SharedPool-Worker-349" daemon
> [_thread_blocked, id=2449, stack(0x00007f06d5e8e000,0x00007f06d5ecf000)]
> 0x00007f04b0f5e000 JavaThread "SharedPool-Worker-354" daemon
> [_thread_blocked, id=2448, stack(0x00007f06d5ecf000,0x00007f06d5f10000)]
> 0x00007f04b0f5d000 JavaThread "SharedPool-Worker-350" daemon
> [_thread_blocked, id=2447, stack(0x00007f06d68fc000,0x00007f06d693d000)]
> 0x00007f074e06f800 JavaThread "SharedPool-Worker-352" daemon
> [_thread_blocked, id=2446, stack(0x00007f06d693d000,0x00007f06d697e000)]
> 0x00007f074e06f000 JavaThread "SharedPool-Worker-346" daemon
> [_thread_blocked, id=2445, stack(0x00007f06d697e000,0x00007f06d69bf000)]
> 0x00007f0693f77000 JavaThread "SharedPool-Worker-351" daemon
> [_thread_blocked, id=2444, stack(0x00007f06d69bf000,0x00007f06d6a00000)]
> 0x00007f0693f76000 JavaThread "SharedPool-Worker-347" daemon
> [_thread_blocked, id=2443, stack(0x00007f06d6a00000,0x00007f06d6a41000)]
> 0x00007f0787e23000 JavaThread "SharedPool-Worker-344" daemon
> [_thread_blocked, id=2442, stack(0x00007f06d6a41000,0x00007f06d6a82000)]
> 0x00007f0787e22800 JavaThread "SharedPool-Worker-345" daemon
> [_thread_blocked, id=2441, stack(0x00007f06d6a82000,0x00007f06d6ac3000)]
> 0x00007f06909f2000 JavaThread "SharedPool-Worker-343" daemon
> [_thread_blocked, id=2440, stack(0x00007f06df3f4000,0x00007f06df435000)]
> 0x00007f04b0ebb800 JavaThread "SharedPool-Worker-342" daemon
> [_thread_blocked, id=2439, stack(0x00007f0701182000,0x00007f07011c3000)]
> 0x00007f07877d1800 JavaThread "SharedPool-Worker-341" daemon
> [_thread_blocked, id=2438, stack(0x00007f07011c3000,0x00007f0701204000)]
> 0x00007f04b0233800 JavaThread "SharedPool-Worker-340" daemon
> [_thread_blocked, id=2437, stack(0x00007f0701204000,0x00007f0701245000)]
> 0x00007f074c302000 JavaThread "SharedPool-Worker-339" daemon
> [_thread_blocked, id=2436, stack(0x00007f0701245000,0x00007f0701286000)]
> 0x00007f04b08eb800 JavaThread "SharedPool-Worker-336" daemon
> [_thread_blocked, id=2435, stack(0x00007f0701286000,0x00007f07012c7000)]
> 0x00007f074d101000 JavaThread "SharedPool-Worker-337" daemon
> [_thread_blocked, id=2434, stack(0x00007f07012c7000,0x00007f0701308000)]
> 0x00007f04b08ed000 JavaThread "SharedPool-Worker-338" daemon
> [_thread_blocked, id=2433, stack(0x00007f0701308000,0x00007f0701349000)]
> 0x00007f0787e4e000 JavaThread "SharedPool-Worker-334" daemon
> [_thread_blocked, id=2432, stack(0x00007f0701349000,0x00007f070138a000)]
> 0x00007f04b0231800 JavaThread "SharedPool-Worker-324" daemon
> [_thread_blocked, id=2431, stack(0x00007f070138a000,0x00007f07013cb000)]
> 0x00007f06935f7800 JavaThread "SharedPool-Worker-321" daemon
> [_thread_blocked, id=2430, stack(0x00007f07013cb000,0x00007f070140c000)]
> 0x00007f04b00b0800 JavaThread "SharedPool-Worker-322" daemon
> [_thread_blocked, id=2429, stack(0x00007f070140c000,0x00007f070144d000)]
> 0x00007f04b0a37800 JavaThread "SharedPool-Worker-320" daemon
> [_thread_blocked, id=2428, stack(0x00007f070144d000,0x00007f070148e000)]
> 0x00007f04b0e91000 JavaThread "SharedPool-Worker-323" daemon
> [_thread_blocked, id=2427, stack(0x00007f0707a34000,0x00007f0707a75000)]
> 0x00007f0693474800 JavaThread "SharedPool-Worker-326" daemon
> [_thread_blocked, id=2426, stack(0x00007f0707d0c000,0x00007f0707d4d000)]
> 0x00007f04b0a29000 JavaThread "SharedPool-Worker-327" daemon
> [_thread_blocked, id=2425, stack(0x00007f0707d4d000,0x00007f0707d8e000)]
> 0x00007f0691394800 JavaThread "SharedPool-Worker-328" daemon
> [_thread_blocked, id=2424, stack(0x00007f0711b2c000,0x00007f0711b6d000)]
> 0x00007f0693f9d800 JavaThread "SharedPool-Worker-329" daemon
> [_thread_blocked, id=2423, stack(0x00007f0711b6d000,0x00007f0711bae000)]
> 0x00007f06935dd800 JavaThread "SharedPool-Worker-331" daemon
> [_thread_blocked, id=2422, stack(0x00007f0711bae000,0x00007f0711bef000)]
> 0x00007f0786de3000 JavaThread "SharedPool-Worker-332" daemon
> [_thread_blocked, id=2421, stack(0x00007f0711bef000,0x00007f0711c30000)]
> 0x00007f07877e2800 JavaThread "SharedPool-Worker-313" daemon
> [_thread_blocked, id=2420, stack(0x00007f0711c30000,0x00007f0711c71000)]
> 0x00007f04b00c6000 JavaThread "SharedPool-Worker-333" daemon
> [_thread_blocked, id=2419, stack(0x00007f0711eda000,0x00007f0711f1b000)]
> 0x00007f04b0928800 JavaThread "SharedPool-Worker-335" daemon
> [_thread_blocked, id=2418, stack(0x00007f0711f1b000,0x00007f0711f5c000)]
> 0x00007f0691507800 JavaThread "SharedPool-Worker-312" daemon
> [_thread_blocked, id=2417, stack(0x00007f0711f5c000,0x00007f0711f9d000)]
> 0x00007f074da87800 JavaThread "SharedPool-Worker-315" daemon
> [_thread_blocked, id=2416, stack(0x00007f0711f9d000,0x00007f0711fde000)]
> 0x00007f0787ea4800 JavaThread "SharedPool-Worker-330" daemon
> [_thread_blocked, id=2415, stack(0x00007f0711fde000,0x00007f071201f000)]
> 0x00007f074cb01000 JavaThread "SharedPool-Worker-317" daemon
> [_thread_blocked, id=2414, stack(0x00007f071201f000,0x00007f0712060000)]
> 0x00007f04b07ec800 JavaThread "SharedPool-Worker-325" daemon
> [_thread_blocked, id=2413, stack(0x00007f0712060000,0x00007f07120a1000)]
> 0x00007f04b035b800 JavaThread "SharedPool-Worker-318" daemon
> [_thread_blocked, id=2412, stack(0x00007f07120a1000,0x00007f07120e2000)]
> 0x00007f074d0ab000 JavaThread "SharedPool-Worker-319" daemon
> [_thread_blocked, id=2411, stack(0x00007f07120e2000,0x00007f0712123000)]
> 0x00007f0693ba2000 JavaThread "SharedPool-Worker-310" daemon
> [_thread_blocked, id=2410, stack(0x00007f0712123000,0x00007f0712164000)]
> 0x00007f074c978000 JavaThread "SharedPool-Worker-311" daemon
> [_thread_blocked, id=2409, stack(0x00007f0712164000,0x00007f07121a5000)]
> 0x00007f0691527800 JavaThread "SharedPool-Worker-316" daemon
> [_thread_blocked, id=2408, stack(0x00007f07121a5000,0x00007f07121e6000)]
> 0x00007f04b009e000 JavaThread "SharedPool-Worker-314" daemon
> [_thread_blocked, id=2407, stack(0x00007f07121e6000,0x00007f0712227000)]
> 0x00007f0787da9800 JavaThread "SharedPool-Worker-308" daemon
> [_thread_blocked, id=2406, stack(0x00007f0712227000,0x00007f0712268000)]
> 0x00007f0690943000 JavaThread "SharedPool-Worker-309" daemon
> [_thread_blocked, id=2405, stack(0x00007f0712268000,0x00007f07122a9000)]
> 0x00007f074c663800 JavaThread "SharedPool-Worker-307" daemon
> [_thread_blocked, id=2404, stack(0x00007f07122a9000,0x00007f07122ea000)]
> 0x00007f0693536000 JavaThread "SharedPool-Worker-306" daemon
> [_thread_blocked, id=2403, stack(0x00007f07122ea000,0x00007f071232b000)]
> 0x00007f06935ab800 JavaThread "SharedPool-Worker-305" daemon
> [_thread_blocked, id=2402, stack(0x00007f071232b000,0x00007f071236c000)]
> 0x00007f06906c0000 JavaThread "SharedPool-Worker-304" daemon
> [_thread_blocked, id=2401, stack(0x00007f071236c000,0x00007f07123ad000)]
> 0x00007f0787053800 JavaThread "SharedPool-Worker-303" daemon
> [_thread_blocked, id=2400, stack(0x00007f07123ad000,0x00007f07123ee000)]
> 0x00007f0787064000 JavaThread "SharedPool-Worker-301" daemon
> [_thread_blocked, id=2399, stack(0x00007f071f396000,0x00007f071f3d7000)]
> 0x00007f074d6bf800 JavaThread "SharedPool-Worker-302" daemon
> [_thread_blocked, id=2398, stack(0x00007f071f3d7000,0x00007f071f418000)]
> 0x00007f0693605800 JavaThread "SharedPool-Worker-300" daemon
> [_thread_blocked, id=2397, stack(0x00007f07202eb000,0x00007f072032c000)]
> 0x00007f04b0d1f000 JavaThread "SharedPool-Worker-298" daemon
> [_thread_blocked, id=2396, stack(0x00007f07208a9000,0x00007f07208ea000)]
> 0x00007f04b07bf000 JavaThread "SharedPool-Worker-299" daemon
> [_thread_blocked, id=2395, stack(0x00007f07208ea000,0x00007f072092b000)]
> 0x00007f0693506800 JavaThread "SharedPool-Worker-297" daemon
> [_thread_blocked, id=2394, stack(0x00007f0720ab4000,0x00007f0720af5000)]
> 0x00007f074c167800 JavaThread "SharedPool-Worker-296" daemon
> [_thread_blocked, id=2393, stack(0x00007f0707e81000,0x00007f0707ec2000)]
> 0x00007f04b1334800 JavaThread "SharedPool-Worker-295" daemon
> [_thread_blocked, id=2392, stack(0x00007f074565e000,0x00007f074569f000)]
> 0x00007f0787c5a000 JavaThread "SharedPool-Worker-294" daemon
> [_thread_blocked, id=2391, stack(0x00007f072538a000,0x00007f07253cb000)]
> 0x00007f04b0867000 JavaThread "SharedPool-Worker-292" daemon
> [_thread_blocked, id=1300, stack(0x00007f072540c000,0x00007f072544d000)]
> 0x00007f07860c1800 JavaThread "SharedPool-Worker-293" daemon
> [_thread_blocked, id=1299, stack(0x00007f0708007000,0x00007f0708048000)]
> 0x00007f0787d19000 JavaThread "SharedPool-Worker-291" daemon
> [_thread_blocked, id=1298, stack(0x00007f0707f44000,0x00007f0707f85000)]
> 0x00007f04b0338000 JavaThread "SharedPool-Worker-290" daemon
> [_thread_blocked, id=1297, stack(0x00007f07210e7000,0x00007f0721128000)]
> 0x00007f04b0595000 JavaThread "MessagingService-Incoming-/54.179.183.26"
> [_thread_in_native, id=1127, stack(0x00007f0707f03000,0x00007f0707f44000)]
> 0x00007f06938af800 JavaThread "MessagingService-Incoming-/54.179.183.26"
> [_thread_in_native, id=1122, stack(0x00007f0707fc6000,0x00007f0708007000)]
> 0x00007f06909da000 JavaThread "SharedPool-Worker-286" daemon
> [_thread_blocked, id=1119, stack(0x00007f0717c4c000,0x00007f0717c8d000)]
> 0x00007f0693833000 JavaThread "SharedPool-Worker-288" daemon
> [_thread_blocked, id=1118, stack(0x00007f0717c8d000,0x00007f0717cce000)]
> 0x00007f04b014a800 JavaThread "SharedPool-Worker-289" daemon
> [_thread_blocked, id=1117, stack(0x00007f0717cce000,0x00007f0717d0f000)]
> 0x00007f0787e53000 JavaThread "SharedPool-Worker-284" daemon
> [_thread_blocked, id=1116, stack(0x00007f0717d0f000,0x00007f0717d50000)]
> 0x00007f078767f800 JavaThread "SharedPool-Worker-283" daemon
> [_thread_blocked, id=1115, stack(0x00007f0717e95000,0x00007f0717ed6000)]
> 0x00007f074c3af800 JavaThread "SharedPool-Worker-285" daemon
> [_thread_blocked, id=1114, stack(0x00007f0717d91000,0x00007f0717dd2000)]
> 0x00007f04b020a000 JavaThread "SharedPool-Worker-287" daemon
> [_thread_blocked, id=1113, stack(0x00007f0717e54000,0x00007f0717e95000)]
> 0x00007f0693fcf800 JavaThread "SharedPool-Worker-281" daemon
> [_thread_blocked, id=1112, stack(0x00007f04afa23000,0x00007f04afa64000)]
> 0x00007f0787210800 JavaThread "SharedPool-Worker-282" daemon
> [_thread_blocked, id=1111, stack(0x00007f0749d3c000,0x00007f0749d7d000)]
> 0x00007f04b010a800 JavaThread "SharedPool-Worker-280" daemon
> [_thread_blocked, id=1110, stack(0x00007f0722003000,0x00007f0722044000)]
> 0x00007f0693832000 JavaThread "MessagingService-Incoming-/52.221.228.170"
> [_thread_in_native, id=1101, stack(0x00007f0717dd2000,0x00007f0717e13000)]
> 0x00007f06934f6000 JavaThread "MessagingService-Incoming-/52.221.228.170"
> [_thread_in_native, id=1096, stack(0x00007f0717ed6000,0x00007f0717f17000)]
> 0x00007f04b00d5000 JavaThread "SharedPool-Worker-279" daemon
> [_thread_blocked, id=1093, stack(0x00007f0717f17000,0x00007f0717f58000)]
> 0x00007f04b00d4000 JavaThread "SharedPool-Worker-278" daemon
> [_thread_blocked, id=1092, stack(0x00007f0717f58000,0x00007f0717f99000)]
> 0x00007f04b01df000 JavaThread "SharedPool-Worker-271" daemon
> [_thread_blocked, id=1091, stack(0x00007f07205b4000,0x00007f07205f5000)]
> 0x00007f04b01de000 JavaThread "SharedPool-Worker-274" daemon
> [_thread_blocked, id=1090, stack(0x00007f07205f5000,0x00007f0720636000)]
> 0x00007f074c1c7000 JavaThread "SharedPool-Worker-273" daemon
> [_thread_blocked, id=1089, stack(0x00007f0720636000,0x00007f0720677000)]
> 0x00007f074c1c6800 JavaThread "SharedPool-Worker-276" daemon
> [_thread_blocked, id=1088, stack(0x00007f0720677000,0x00007f07206b8000)]
> 0x00007f04b02a4800 JavaThread "SharedPool-Worker-277" daemon
> [_thread_blocked, id=1087, stack(0x00007f07206b8000,0x00007f07206f9000)]
> 0x00007f04b02a4000 JavaThread "SharedPool-Worker-269" daemon
> [_thread_blocked, id=1086, stack(0x00007f07206f9000,0x00007f072073a000)]
> 0x00007f074c2ae000 JavaThread "SharedPool-Worker-268" daemon
> [_thread_blocked, id=1085, stack(0x00007f072073a000,0x00007f072077b000)]
> 0x00007f074c2ad000 JavaThread "SharedPool-Worker-275" daemon
> [_thread_blocked, id=1084, stack(0x00007f072077b000,0x00007f07207bc000)]
> 0x00007f0693ec3000 JavaThread "SharedPool-Worker-272" daemon
> [_thread_blocked, id=1083, stack(0x00007f07207bc000,0x00007f07207fd000)]
> 0x00007f0693ec2000 JavaThread "SharedPool-Worker-270" daemon
> [_thread_blocked, id=1082, stack(0x00007f07207fd000,0x00007f072083e000)]
> 0x00007f0690983800 JavaThread "SharedPool-Worker-267" daemon
> [_thread_blocked, id=1081, stack(0x00007f072083e000,0x00007f072087f000)]
> 0x00007f0690983000 JavaThread "SharedPool-Worker-265" daemon
> [_thread_blocked, id=1080, stack(0x00007f0720be6000,0x00007f0720c27000)]
> 0x00007f04b0358000 JavaThread "SharedPool-Worker-266" daemon
> [_thread_blocked, id=1079, stack(0x00007f0720c27000,0x00007f0720c68000)]
> 0x00007f04b0357000 JavaThread "SharedPool-Worker-263" daemon
> [_thread_blocked, id=1078, stack(0x00007f0720c68000,0x00007f0720ca9000)]
> 0x00007f04b05e6800 JavaThread "SharedPool-Worker-264" daemon
> [_thread_blocked, id=1077, stack(0x00007f0720ca9000,0x00007f0720cea000)]
> 0x00007f04b05e5800 JavaThread "SharedPool-Worker-262" daemon
> [_thread_blocked, id=1076, stack(0x00007f0720cea000,0x00007f0720d2b000)]
> 0x00007f0786e27800 JavaThread "SharedPool-Worker-261" daemon
> [_thread_blocked, id=1074, stack(0x00007f0720d2b000,0x00007f0720d6c000)]
> 0x00007f0786e27000 JavaThread "SharedPool-Worker-258" daemon
> [_thread_blocked, id=1073, stack(0x00007f0720d6c000,0x00007f0720dad000)]
> 0x00007f069366c800 JavaThread "SharedPool-Worker-259" daemon
> [_thread_blocked, id=1072, stack(0x00007f0720dad000,0x00007f0720dee000)]
> 0x00007f069366c000 JavaThread "SharedPool-Worker-260" daemon
> [_thread_blocked, id=1071, stack(0x00007f0720dee000,0x00007f0720e2f000)]
> 0x00007f0693661000 JavaThread "SharedPool-Worker-257" daemon
> [_thread_blocked, id=1070, stack(0x00007f0720e2f000,0x00007f0720e70000)]
> 0x00007f0693660000 JavaThread "SharedPool-Worker-256" daemon
> [_thread_blocked, id=1069, stack(0x00007f0720e70000,0x00007f0720eb1000)]
> 0x00007f0693603000 JavaThread "SharedPool-Worker-255" daemon
> [_thread_blocked, id=1068, stack(0x00007f0720f61000,0x00007f0720fa2000)]
> 0x00007f0693602800 JavaThread "SharedPool-Worker-252" daemon
> [_thread_blocked, id=1067, stack(0x00007f0720eb1000,0x00007f0720ef2000)]
> 0x00007f0693fd7000 JavaThread "SharedPool-Worker-251" daemon
> [_thread_blocked, id=1066, stack(0x00007f0749d7d000,0x00007f0749dbe000)]
> 0x00007f04b0109800 JavaThread "SharedPool-Worker-254" daemon
> [_thread_blocked, id=1065, stack(0x00007f04af508000,0x00007f04af549000)]
> 0x00007f0787c49800 JavaThread "SharedPool-Worker-253" daemon
> [_thread_blocked, id=1064, stack(0x00007f04b4030000,0x00007f04b4071000)]
> 0x00007f04b00d9000 JavaThread "SharedPool-Worker-249" daemon
> [_thread_blocked, id=1063, stack(0x00007f04b772a000,0x00007f04b776b000)]
> ...
> {code}
> Complete error log is at
> https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/error.log
> A screenshot load, memory usage etc. is at
> https://dl.dropboxusercontent.com/u/1575409/Ably/logs/2017-05-10-cassandra-crash/ap-southeast-1/Voila_Capture%202017-05-11_09-08-53_am.png
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
