[ https://issues.apache.org/jira/browse/CASSANDRA-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382664#comment-16382664 ]
Gil Tene edited comment on CASSANDRA-13757 at 3/1/18 9:26 PM: -------------------------------------------------------------- See CASSANDRA-14284 for analysis of how/why this can happen, and what needs to be fixed to prevent it. was (Author: giltene): See https://issues.apache.org/jira/browse/CASSANDRA-14284 for analysis of how/why this can happen, and what needs to be fixed to prevent it. > Cassandra 3.5.0 JVM Segfault Problem While Repair Job is Running > ---------------------------------------------------------------- > > Key: CASSANDRA-13757 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13757 > Project: Cassandra > Issue Type: Bug > Environment: Operation System: Debian Jessie > Java: Oracle JDK 1.8.0_131 > Cassandra: 3.5.0 > Reporter: Serhat Rıfat Demircan > Priority: Major > > We got following error while repair job running on our cluster. One of the > nodes stop due to segmantation fault in JVM and repair job fails. > We could not reproduce this problem on our test and staging enviroment (main > difference is data size). > {code:java} > # > # SIGSEGV (0xb) at pc=0x00007fd80a399e70, pid=1305, tid=0x00007fd7ee7c4700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build > 1.8.0_131-b11) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode > linux-amd64 compressed oops) > # Problematic frame: > # C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > --------------- T H R E A D --------------- > Current thread (0x00007fce32dad1b0): JavaThread "CompactionExecutor:9798" > daemon [_thread_in_native, id=16879, > stack(0x00007fd7ee784000,0x00007fd7ee7c5000)] > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: > 0x00007fd450c4d000 > Registers: > RAX=0x00007fcde6560d32, RBX=0x00007fd450c4cff9, RCX=0x00007fcde6560c7a, > RDX=0x00007fcde6560d3e > RSP=0x00007fd7ee7c3160, RBP=0x00007fd450c44ae6, RSI=0x00007fcde6562ff8, > RDI=0x00000000000000c2 > R8 =0x00007fcde6562ff4, R9 =0x00007fcde6563000, R10=0x0000000000000000, > R11=0x0000000000000000 > R12=0x000000000000000c, R13=0x00007fd4501cd000, R14=0x00007fcde6562ff7, > R15=0x00007fcde6562ffb > RIP=0x00007fd80a399e70, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, > ERR=0x0000000000000004 > TRAPNO=0x000000000000000e > Top of Stack: (sp=0x00007fd7ee7c3160) > 0x00007fd7ee7c3160: 0000000000000008 00007fd81e21c3d0 > 0x00007fd7ee7c3170: 0000000000000004 0000000000000001 > 0x00007fd7ee7c3180: 0000000000000002 0000000000000001 > 0x00007fd7ee7c3190: 0000000000000004 0000000000000004 > 0x00007fd7ee7c31a0: 0000000000000004 0000000000000004 > 0x00007fd7ee7c31b0: 0000000000000000 0000000000000000 > 0x00007fd7ee7c31c0: 0000000000000000 ffffffffffffffff > 0x00007fd7ee7c31d0: 0000000000000000 0000000000000001 > 0x00007fd7ee7c31e0: 0000000000000002 0000000000000003 > 0x00007fd7ee7c31f0: 00007fd7ee7c32b8 00007fce32dad3a8 > 0x00007fd7ee7c3200: 0000000000000000 0000000000000000 > 0x00007fd7ee7c3210: 00007fd4501cd000 00007fcde6553000 > 0x00007fd7ee7c3220: 0000000000a77ae6 00007fd80a39659d > 0x00007fd7ee7c3230: 0000000000000000 00000000dcb8fc9b > 0x00007fd7ee7c3240: 00007fd7ee7c32d0 0000000000000000 > 0x00007fd7ee7c3250: 00000006e5c7e4d8 00007fd7ee7c32b8 > 0x00007fd7ee7c3260: 00007fce32dad1b0 00007fd81df2099d > 0x00007fd7ee7c3270: 00007fd7ee7c32a8 0000000000000000 > 0x00007fd7ee7c3280: 0000000000010000 0000000000000000 > 0x00007fd7ee7c3290: 00000006e5c7e528 00007fd81d74df10 > 0x00007fd7ee7c32a0: 0000000000000000 00000006e5c7e4d8 > 0x00007fd7ee7c32b0: 00000006f6c7fbf8 00000006f6e957f0 > 0x00007fd7ee7c32c0: 00000006e5c7e350 00007fd87fffffff > 0x00007fd7ee7c32d0: 00000006e5c7e528 00007fd81fa867e0 > 0x00007fd7ee7c32e0: 00a77ae200010000 0000000000a77ae2 > 0x00007fd7ee7c32f0: 00000006e5c7e488 000000000112d5f1 > 0x00007fd7ee7c3300: dcb8fc9b000099ce 0001000000a77ae6 > 0x00007fd7ee7c3310: 00a814b000a814b4 00000006e5c7e4d8 > 0x00007fd7ee7c3320: 00000006e5c7e4d8 00000006f6a4df38 > 0x00007fd7ee7c3330: 0000000600010000 000000067fffffff > 0x00007fd7ee7c3340: 0089715800002c8a 0006189d87852057 > 0x00007fd7ee7c3350: 0000000000000000 00000000e5244e71 > Instructions: (pc=0x00007fd80a399e70) > 0x00007fd80a399e50: e4 0f 49 83 fc 0f 0f 84 94 00 00 00 4a 8d 14 20 > 0x00007fd80a399e60: 48 39 f2 0f 87 c0 00 00 00 0f 1f 80 00 00 00 00 > 0x00007fd80a399e70: 48 8b 0b 48 83 c3 08 48 89 08 48 83 c0 08 48 39 > 0x00007fd80a399e80: c2 77 ed 48 29 d0 48 89 d1 48 29 c3 0f b7 03 48 > Register to memory mapping: > RAX=0x00007fcde6560d32 is an unknown value > RBX=0x00007fd450c4cff9 is an unknown value > RCX=0x00007fcde6560c7a is an unknown value > RDX=0x00007fcde6560d3e is an unknown value > RSP=0x00007fd7ee7c3160 is pointing into the stack for thread: > 0x00007fce32dad1b0 > RBP=0x00007fd450c44ae6 is an unknown value > RSI=0x00007fcde6562ff8 is an unknown value > RDI=0x00000000000000c2 is an unknown value > R8 =0x00007fcde6562ff4 is an unknown value > R9 =0x00007fcde6563000 is an unknown value > R10=0x0000000000000000 is an unknown value > R11=0x0000000000000000 is an unknown value > R12=0x000000000000000c is an unknown value > R13=0x00007fd4501cd000 is an unknown value > R14=0x00007fcde6562ff7 is an unknown value > R15=0x00007fcde6562ffb is an unknown value > Stack: [0x00007fd7ee784000,0x00007fd7ee7c5000], sp=0x00007fd7ee7c3160, free > space=252k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [liblz4-java3580121503903465201.so+0x5e70] LZ4_decompress_fast+0xd0 > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > J 5084 > net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast([BLjava/nio/ByteBuffer;I[BLjava/nio/ByteBuffer;II)I > (0 bytes) @ 0x00007fd81df20923 [0x00007fd81df20880+0xa3] > J 20311 C2 > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V > (354 bytes) @ 0x00007fd81fa867e0 [0x00007fd81fa86160+0x680] > J 10706 C2 org.apache.cassandra.io.util.RandomAccessReader.reBuffer()V (78 > bytes) @ 0x00007fd81ea59c58 [0x00007fd81ea59b20+0x138] > J 19996 C2 > org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput; > (12 bytes) @ 0x00007fd81fa32934 [0x00007fd81fa32540+0x3f4] > J 20645 C2 > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(Lorg/apache/cassandra/io/sstable/format/SSTableReader;Lorg/apache/cassandra/io/util/FileDataInput;Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/RowIndexEntry;Lorg/apache/cassandra/db/Slices;Lorg/apache/cassandra/db/filter/ColumnFilter;Z)V > (423 bytes) @ 0x00007fd81fbf8cec [0x00007fd81fbf8a20+0x2cc] > J 20644 C2 > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; > (120 bytes) @ 0x00007fd81e61b5d0 [0x00007fd81e61b140+0x490] > J 9597 C2 > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext()Ljava/lang/Object; > (5 bytes) @ 0x00007fd81da8dad4 [0x00007fd81da8daa0+0x34] > J 3432 C2 org.apache.cassandra.utils.AbstractIterator.hasNext()Z (80 bytes) @ > 0x00007fd81da28f20 [0x00007fd81da28e80+0xa0] > J 19755 C2 > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext()Ljava/lang/Object; > (9 bytes) @ 0x00007fd81f81d00c [0x00007fd81f81cb60+0x4ac] > J 21810 C2 > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Lorg/apache/cassandra/db/rows/Unfiltered; > (42 bytes) @ 0x00007fd81ef929fc [0x00007fd81ef927a0+0x25c] > J 15842 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(Ljava/util/List;Lorg/apache/cassandra/metrics/TableMetrics;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (71 bytes) @ 0x00007fd81ece33b4 [0x00007fd81ece3200+0x1b4] > J 21162 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (635 bytes) @ 0x00007fd81fdb2ff0 [0x00007fd81fdb2100+0xef0] > J 27194 C2 > org.apache.cassandra.db.Keyspace.indexPartition(Lorg/apache/cassandra/db/DecoratedKey;Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Set;)V > (274 bytes) @ 0x00007fd820c8b044 [0x00007fd820c8aae0+0x564] > J 27422 C2 > org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build()V (73 > bytes) @ 0x00007fd81f0b5a00 [0x00007fd81f0b5940+0xc0] > J 27619 C1 org.apache.cassandra.db.compaction.CompactionManager$12.run()V (56 > bytes) @ 0x00007fd82051313c [0x00007fd820512a00+0x73c] > J 24483 C2 java.util.concurrent.FutureTask.run()V (126 bytes) @ > 0x00007fd81d50faf4 [0x00007fd81d50fa20+0xd4] > J 23783 C2 > java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V > (225 bytes) @ 0x00007fd81fca71a8 [0x00007fd81fca70a0+0x108] > J 21390 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fd81e6d3cb8 > [0x00007fd81e6d3c60+0x58] > v ~StubRoutines::call_stub > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org