[
https://issues.apache.org/jira/browse/CASSANDRA-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Ravn updated CASSANDRA-13503:
-----------------------------------
Description:
One of our cassandra nodes segfaulted. I've attached the hs_err.log. It looks
like it was loading sstables from disk:
{noformat}
Current thread (0x00007f3df4e7b930): JavaThread "CompactionExecutor:2" daemon
[_thread_in_Java, id=30023, stack(0x00007f3dce922000,0x00007f3dce963000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
0xffffffffbd49a8b6
Stack: [0x00007f3dce922000,0x00007f3dce963000], sp=0x00007f3dce961430, free
space=253k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 11168 C2
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext()Ljava/lang/Object;
(48 bytes) @ 0x00007f3de6097df4 [0x00007f3de6097cc0+0x134]
J 2042 C2 com.google.common.collect.AbstractIterator.hasNext()Z (65 bytes) @
0x00007f3de560fbc0 [0x00007f3de560fb00+0xc0]
J 14662 C2
org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(Ljava/util/Iterator;)Lorg/apache/cassandra/db/ColumnIndex;
(36 bytes) @ 0x00007f3de7143a08 [0x00007f3de7143440+0x5c8]
J 13756 C2
org.apache.cassandra.io.sstable.SSTableRewriter.append(Lorg/apache/cassandra/db/compaction/AbstractCompactedRow;)Lorg/apache/cassandra/db/RowIndexEntry;
(119 bytes) @ 0x00007f3de6a98b88 [0x00007f3de6a985c0+0x5c8]
J 14627 C2 org.apache.cassandra.db.compaction.CompactionTask.runMayThrow()V
(1622 bytes) @ 0x00007f3de710eed4 [0x00007f3de710daa0+0x1434]
J 18294 C1
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
(19 bytes) @ 0x00007f3de7981aa4 [0x00007f3de79818c0+0x1e4]
J 18503 C1
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
(39 bytes) @ 0x00007f3de7a4cf2c [0x00007f3de7a4ce20+0x10c]
J 17908 C2
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()V
(182 bytes) @ 0x00007f3de789e0a4 [0x00007f3de789aca0+0x3404]
{noformat}
No errors in kernel logs, and no other noticeable issues on the node.
was:
One of our cassandra nodes segfaulted. I've attached the hs_err.log. It looks
like it was loading sstables from disk:
{noformat}
Current thread (0x00007f3df4e7b930): JavaThread "CompactionExecutor:2" daemon
[_thread_in_Java, id=30023, stack(0x00007f3dce922000,0x00007f3dce963000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
0xffffffffbd49a8b6
Stack: [0x00007f3dce922000,0x00007f3dce963000], sp=0x00007f3dce961430, free
space=253k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 11168 C2
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext()Ljava/lang/Object;
(48 bytes) @ 0x00007f3de6097df4 [0x00007f3de6097cc0+0x134]
J 2042 C2 com.google.common.collect.AbstractIterator.hasNext()Z (65 bytes) @
0x00007f3de560fbc0 [0x00007f3de560fb00+0xc0]
J 14662 C2
org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(Ljava/util/Iterator;)Lorg/apache/cassandra/db/ColumnIndex;
(36 bytes) @ 0x00007f3de7143a08 [0x00007f3de7143440+0x5c8]
J 13756 C2
org.apache.cassandra.io.sstable.SSTableRewriter.append(Lorg/apache/cassandra/db/compaction/AbstractCompactedRow;)Lorg/apache/cassandra/db/RowIndexEntry;
(119 bytes) @ 0x00007f3de6a98b88 [0x00007f3de6a985c0+0x5c8]
J 14627 C2 org.apache.cassandra.db.compaction.CompactionTask.runMayThrow()V
(1622 bytes) @ 0x00007f3de710eed4 [0x00007f3de710daa0+0x1434]
J 18294 C1
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
(19 bytes) @ 0x00007f3de7981aa4 [0x00007f3de79818c0+0x1e4]
J 18503 C1
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
(39 bytes) @ 0x00007f3de7a4cf2c [0x00007f3de7a4ce20+0x10c]
J 17908 C2
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()V
(182 bytes) @ 0x00007f3de789e0a4 [0x00007f3de789aca0+0x3404]
{noformat}
No errors in kernel logs, and no other noticeable issues on the node (lots of
ram, not under heavy load).
> Segfault during compaction
> --------------------------
>
> Key: CASSANDRA-13503
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13503
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: {noformat}
> Cassandra 2.1.17
> java -version:
> java version "1.8.0_112"
> Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
> Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)
> uname -a (Ubuntu 16.04.2):
> Linux ip-10-50-194-251 4.8.0-51-generic #54~16.04.1-Ubuntu SMP Wed Apr 26
> 16:00:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> lstopo:
> Machine (16GB total)
> NUMANode L#0 (P#0 7998MB) + Package L#0 + L3 L#0 (45MB) + L2 L#0 (256KB) +
> L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
> PU L#0 (P#0)
> PU L#1 (P#1)
> NUMANode L#1 (P#1 8047MB) + Package L#1 + L3 L#1 (45MB) + L2 L#1 (256KB) +
> L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
> PU L#2 (P#2)
> PU L#3 (P#3)
> {noformat}
> Reporter: James Ravn
> Attachments: hs_err_pid29774.log
>
>
> One of our cassandra nodes segfaulted. I've attached the hs_err.log. It looks
> like it was loading sstables from disk:
> {noformat}
> Current thread (0x00007f3df4e7b930): JavaThread "CompactionExecutor:2"
> daemon [_thread_in_Java, id=30023,
> stack(0x00007f3dce922000,0x00007f3dce963000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
> 0xffffffffbd49a8b6
> Stack: [0x00007f3dce922000,0x00007f3dce963000], sp=0x00007f3dce961430, free
> space=253k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> J 11168 C2
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext()Ljava/lang/Object;
> (48 bytes) @ 0x00007f3de6097df4 [0x00007f3de6097cc0+0x134]
> J 2042 C2 com.google.common.collect.AbstractIterator.hasNext()Z (65 bytes) @
> 0x00007f3de560fbc0 [0x00007f3de560fb00+0xc0]
> J 14662 C2
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(Ljava/util/Iterator;)Lorg/apache/cassandra/db/ColumnIndex;
> (36 bytes) @ 0x00007f3de7143a08 [0x00007f3de7143440+0x5c8]
> J 13756 C2
> org.apache.cassandra.io.sstable.SSTableRewriter.append(Lorg/apache/cassandra/db/compaction/AbstractCompactedRow;)Lorg/apache/cassandra/db/RowIndexEntry;
> (119 bytes) @ 0x00007f3de6a98b88 [0x00007f3de6a985c0+0x5c8]
> J 14627 C2 org.apache.cassandra.db.compaction.CompactionTask.runMayThrow()V
> (1622 bytes) @ 0x00007f3de710eed4 [0x00007f3de710daa0+0x1434]
> J 18294 C1
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
> (19 bytes) @ 0x00007f3de7981aa4 [0x00007f3de79818c0+0x1e4]
> J 18503 C1
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(Lorg/apache/cassandra/db/compaction/CompactionManager$CompactionExecutorStatsCollector;)I
> (39 bytes) @ 0x00007f3de7a4cf2c [0x00007f3de7a4ce20+0x10c]
> J 17908 C2
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run()V
> (182 bytes) @ 0x00007f3de789e0a4 [0x00007f3de789aca0+0x3404]
> {noformat}
> No errors in kernel logs, and no other noticeable issues on the node.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
