[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343329#comment-14343329
]
T Jake Luciani commented on CASSANDRA-8689:
---
Ah I.C. I tried cleaning up the tp and compactions and forced the test to run
last and that fixed the issue. Attached new test patch.
So this in conjunction with CASSANDRA-8757 looks good +1
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.4
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341734#comment-14341734
]
Benedict commented on CASSANDRA-8689:
-
Since you're running 6 tests, I assume you're running with your test (as
stands) included? This breaks the other tests for the two reasons I gave above:
CASSANDRA-8805, and because the threads are still running mucking around with
the state even after the new test finishes successfully.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.4
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339748#comment-14339748
]
T Jake Luciani commented on CASSANDRA-8689:
---
I'm now seeing many other failures related to index summaries
{code}
[echo] running test bucket 0 tests
[junit] WARNING: multiple versions of ant detected in path for junit
[junit]
jar:file:/usr/share/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit] and
jar:file:/home/jake/workspace/cassandra/build/lib/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
[junit] Testsuite: org.apache.cassandra.io.sstable.IndexSummaryManagerTest
[junit] Tests run: 6, Failures: 5, Errors: 0, Skipped: 0, Time elapsed:
10.138 sec
[junit]
[junit] - Standard Output ---
[junit] ERROR 05:25:52 LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@9f7cd4c) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1638131854:build/test/cassandra/data:0/Keyspace1/StandardLowIndexInterval-15bf0780be4111e4869db7d84572b531/Keyspace1-StandardLowIndexInterval-ka-2
was not released before the reference was garbage collected
[junit] ERROR 05:25:52 LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@9f7cd4c) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1638131854:build/test/cassandra/data:0/Keyspace1/StandardLowIndexInterval-15bf0780be4111e4869db7d84572b531/Keyspace1-StandardLowIndexInterval-ka-2
was not released before the reference was garbage collected
[junit] ERROR 05:25:52 Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@9f7cd4c:
[junit] Thread[main,5,main]
[junit] at java.lang.Thread.getStackTrace(Thread.java:1589)
[junit] at
org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:196)
[junit] at
org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
[junit] at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
[junit] at
org.apache.cassandra.io.sstable.SSTableReader.init(SSTableReader.java:242)
[junit] at
org.apache.cassandra.io.sstable.SSTableReader.init(SSTableReader.java:605)
[junit] at
org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:991)
[junit] at
org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:410)
[junit] at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:288)
[junit] at
org.apache.cassandra.io.sstable.IndexSummaryManagerTest.testChangeMaxIndexInterval(IndexSummaryManagerTest.java:319)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:606)
[junit] at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
[junit] at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit] at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
[junit] at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
[junit] at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
[junit] at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
[junit] at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
[junit] at
org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
[junit] at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit] at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
[junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
[junit] at
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
[junit]
[junit] ERROR 05:25:52
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327723#comment-14327723
]
Benedict commented on CASSANDRA-8689:
-
I've rebased both, but you should be able to just merge from one into the other.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.4
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326440#comment-14326440
]
T Jake Luciani commented on CASSANDRA-8689:
---
I'm not able to apply the patch for 8757 to this one (can you rebase them all?)
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.4
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320482#comment-14320482
]
Benedict commented on CASSANDRA-8689:
-
It looks like that is something else entirely, caused by that test leaving an
infinite loop of summary redistribution running. I've filed CASSANDRA-8805 to
deal with that. So, those two patches together fixes the problem, although why
that is I am not certain.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320016#comment-14320016
]
Benedict commented on CASSANDRA-8689:
-
If I run this against a merge of my latest branch for 8757 with my branch here,
the test passes. With either one it does not. However with both of these
branches, this test causes the _other_ tests to fail, whilst passing itself
(and they passing together if this test is excluded), so clearly there's still
some problematic state
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319199#comment-14319199
]
Benedict commented on CASSANDRA-8689:
-
Well that's annoying. Nice work reproducing though.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
Attachments: 8689-test.txt
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317933#comment-14317933
]
Benedict commented on CASSANDRA-8689:
-
Rebased and pushed with the comment included above the sstablesMap property
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316974#comment-14316974
]
T Jake Luciani commented on CASSANDRA-8689:
---
Apologies for the delay, could you rebase this?
It would be good to put your comment above as a code comment explaining any why
we need sstable map.
I've been trying to trigger this issue with stress by re summarizing every 1
minute. I agree with your guess that this is the problem causing this error
but it would be great to repro.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307404#comment-14307404
]
Benedict commented on CASSANDRA-8689:
-
Can we get a bisect run on this?
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308004#comment-14308004
]
Benedict commented on CASSANDRA-8689:
-
Patch available
[here|https://github.com/belliottsmith/cassandra/tree/8689-racemarkcompact]
This converts the Set of live sstables in DataTracker.View to a Map, so that we
can easily perform identity checks as well as equality checks. When marking
compacting, we now indicate if we expect the sstables to be present (by default
we do), and we then check that not only are they all present in the live set,
but that the exact instance present is the one we made our decision to compact
against.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Assignee: Benedict
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301599#comment-14301599
]
Benedict commented on CASSANDRA-8689:
-
It is unlikely this will cause any degradation to your cluster. The problem
should just prevent summaries from being redistributed at that moment in time,
which is a harmless setback.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301596#comment-14301596
]
Jeff Liu commented on CASSANDRA-8689:
-
Surprised to know that there are so much information behind this blur java
assertion error. Thank you Benedict for the info. What is the impact of this
issue without fix? are in some kind of compacting danger ? I guess I'm really
asking for the criticality of this issue. Thanks.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299920#comment-14299920
]
Benedict commented on CASSANDRA-8689:
-
I've been meaning to file a bug about something for a while, after
investigating if it happens anywhere, and it looks to me like this may be it in
action (in one of many potential bolt holes).
In markCompacting we don't ensure the sstable we're marking compacting is
actually in the set of live sstables. So we really need to take a reference
either before or after marking compacting, or we need to ensure the
intersection with the live set is the same as the set we are marking. We can't
do this universally, though, because we sometimes deliberately markCompacting
files not in the live set.
Anyway, the upshot is that if we don't do this we can start working on a file
that has been dropped and cleaned up. The race window for grabbing such a file
is narrow, but the window over which it can cause problems once we do is
unbounded.
IndexSummaryManager.getCompactingAndNonCompactingSSTables looks to me to
exhibit this race condition.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299350#comment-14299350
]
Tyler Hobbs commented on CASSANDRA-8689:
[~jeffl] can you enable TRACE level logging on the IndexSummaryManager class to
see if we can get more details about what might be happening?
You should be able to do that by adding the following line to {{logback.xml}}:
{noformat}
logger name=org.apache.cassandra.io.sstable.IndexSummaryManager
level=TRACE/
{noformat}
The IndexSummaryManager runs a resize process once every hour (by default), so
it may take a while to see something in the logs.
Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
---
Key: CASSANDRA-8689
URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Fix For: 2.1.3
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
CassandraDaemon.java:153 - Exception in thread
Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G
memory with 2G allocated to heap. We are basically running read queries to
retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
