[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503263#comment-14503263
]
Justin Poole commented on CASSANDRA-8718:
-
Sorry for late comment; but I confirm that upgrading from 2.0.12 to 2.0.13
fixes the issue. Thank you!
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Assignee: Joshua McKenzie
>Priority: Minor
> Fix For: 2.0.15
>
> Attachments: java_hs_err.log
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500122#comment-14500122
]
Benedict commented on CASSANDRA-8718:
-
This is most likely a resource cleanup issue, with accessing offheap memory
that has been freed as you suggest (the error printed in the stdout is more
helpful, since it's clearly in the middle of the offheap binarySearch). Which
means most likely either a double decrement of refcounts, or not taking a
reference somewhere. Since this is being thrown in Compaction, the latter is
actually always true (ie we never take a separate reference), so I would
suspect what is happening is cleanup releases references even if a compaction
is operating on the sstables, or perhaps doesn't properly mark the sstable
compacting first, or something along those lines.
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Assignee: Joshua McKenzie
>Priority: Minor
> Fix For: 2.0.15
>
> Attachments: java_hs_err.log
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500089#comment-14500089
]
Joshua McKenzie commented on CASSANDRA-8718:
[~philipthompson]
Looks like the crash occurs while getting information for our index scan
position, likely during access of off-heap memory since the rest of
getIndexScanPosition is pretty innocuous. I believe a full memory dump would be
necessary to get more visibility into what's gone wrong, though JDK crash dumps
aren't my forte ([~benedict] - care to sanity check?)
[~srspnda] [~rossmohax]: Were either of you able to get more information about
this error? Updates to JDK / C* have any impact on this error presenting?
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Priority: Minor
> Fix For: 2.0.15
>
> Attachments: java_hs_err.log
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499932#comment-14499932
]
Philip Thompson commented on CASSANDRA-8718:
We've tested cleanup on both of those JDK's, and not have encountered a similar
issue. [~JoshuaMcKenzie], does the attached log give you any info that may help
us reproduce the issue?
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Priority: Minor
> Fix For: 2.0.15
>
> Attachments: java_hs_err.log
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328630#comment-14328630
]
Justin Poole commented on CASSANDRA-8718:
-
I'm also seeing the same issue.
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
ReleaseVersion: 2.0.12
Linux 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
It's happened consistently now around the ring after I've added nodes to the
cluster. There were no clear signs in the logs as to why it happened, however,
a log of the Java SIGSEGV by Cassandra is attached. I'm going to enable more
verbose logging and see if I can get more insight from the running process in
the morning.
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Priority: Minor
> Fix For: 2.0.13
>
> Attachments: java_hs_err.log
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303056#comment-14303056
]
Jon Pyle commented on CASSANDRA-8718:
-
Maxim's colleague here.
Oracle JRE version :-
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
OS :-
3.11.0-26-generic #45~precise1-Ubuntu SMP Tue Jul 15 04:02:35 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Priority: Minor
> Fix For: 2.0.13
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8718) nodetool cleanup causes segfault
[
https://issues.apache.org/jira/browse/CASSANDRA-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302721#comment-14302721
]
Philip Thompson commented on CASSANDRA-8718:
You are using JDK 1.7.0_71-b14 correct? What operating system are you running
on?
> nodetool cleanup causes segfault
>
>
> Key: CASSANDRA-8718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8718
> Project: Cassandra
> Issue Type: Bug
>Reporter: Maxim Ivanov
>Priority: Minor
> Fix For: 2.0.13
>
>
> When doing cleanup on C* 2.0.12 following error crashes the java process:
> {code}
> INFO 17:59:02,800 Cleaning up
> SSTableReader(path='/data/sdd/cassandra_prod/vdna/analytics/vdna-analytics-jb-21670-Data.db')
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f750890268e, pid=28039, tid=140130222446336
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build
> 1.7.0_71-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 2655 C2
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(Lorg/apache/cassandra/db/RowPosition;)I
> (88 bytes) @ 0x7f750890268e [0x7f7508902580+0x10e]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /var/lib/cassandra_prod/hs_err_pid28039.log
> Compiled method (c2) 913167265 4849
> org.apache.cassandra.dht.Token::maxKeyBound (24 bytes)
> total in heap [0x7f7508572450,0x7f7508573318] = 3784
> relocation [0x7f7508572570,0x7f7508572618] = 168
> main code [0x7f7508572620,0x7f7508572cc0] = 1696
> stub code [0x7f7508572cc0,0x7f7508572cf8] = 56
> oops [0x7f7508572cf8,0x7f7508572d90] = 152
> scopes data[0x7f7508572d90,0x7f7508573118] = 904
> scopes pcs [0x7f7508573118,0x7f7508573268] = 336
> dependencies [0x7f7508573268,0x7f7508573280] = 24
> handler table [0x7f7508573280,0x7f75085732e0] = 96
> nul chk table [0x7f75085732e0,0x7f7508573318] = 56
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
