[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188434#comment-14188434 ] Jonathan Ellis commented on CASSANDRA-5256: --- That is a different error; please open a new ticket. > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Assignee: Jonathan Ellis >Priority: Critical > Labels: compaction > Fix For: 1.2.2 > > Attachments: 5256-v2.txt, 5256-v4.txt, 5256-v5.txt, 5256.txt > > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > --- > I've invoked the `nodetool compact` three times; this occurred after each. > The node has been up for a couple days accepting writes and has not been > restarted. > Here's the server's log since it was started a few days ago: > https://gist.github.
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187189#comment-14187189
]
Nikolai Grigoriev commented on CASSANDRA-5256:
--
I have just got this problem on multiple nodes. Cassandra 2.0.10 (DSE 4.5.2).
Should I reopen?
{code}
ERROR [CompactionExecutor:1196] 2014-10-28 17:14:50,124 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:1196,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at
org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:692)
at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:663)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:354)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:125)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:113)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:192)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> "Memory was freed" AssertionError During Major Compaction
> -
>
> Key: CASSANDRA-5256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0
> Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
> Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_30"
> Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
> Ubuntu 12.04.2 LTS
>Reporter: C. Scott Andreas
>Assignee: Jonathan Ellis
>Priority: Critical
> Labels: compaction
> Fix For: 1.2.2
>
> Attachments: 5256-v2.txt, 5256-v4.txt, 5256-v5.txt, 5256.txt
>
>
> When initiating a major compaction with `./nodetool -h localhost compact`, an
> AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
> ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
> (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
> java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
> at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
> at
> org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93)
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
> at
> or
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581905#comment-13581905 ] Sylvain Lebresne commented on CASSANDRA-5256: - +1 on v5. Nits: the comment of CompactionTask.execute is now outdated about the unmarking part (I would also probably have move the unmarking part in AbsractCompactionTask rather than CompactionTask as that felt "safer" to me in principle, but that doesn't really matter). > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Assignee: Jonathan Ellis >Priority: Critical > Labels: compaction > Fix For: 1.2.2 > > Attachments: 5256.txt, 5256-v2.txt, 5256-v4.txt, 5256-v5.txt > > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > --- > I've in
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581291#comment-13581291 ] Jonathan Ellis commented on CASSANDRA-5256: --- I think there are two problems here: # I was wrong, we're not starting with non-compacting candidates (this is the case for STCS, but not LCS, which checks for compacting-ness later on). # The assert is broken in a concurrent compactions world; it's possible that the sstable we're checking for overlaps with, gets compacted away already by a different task before that assert fires. v5 edits findDroppableSSTable to check for compacting status, and removes the getOverlappingSSTables assert. > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Assignee: Jonathan Ellis >Priority: Critical > Labels: compaction > Fix For: 1.2.2 > > Attachments: 5256.txt, 5256-v2.txt, 5256-v4.txt > > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExec
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579718#comment-13579718
]
Jonathan Ellis commented on CASSANDRA-5256:
---
bq. One problem I had was CompactionsTest was throwing AssertionError on
testSingleSSTableCompactionWithLeveledCompaction, when trying to find
overlapping sstables to determine droppable ones.
I'm worried this patch fixes a symptom instead of a deeper bug:
# We already start with {{Set candidates =
cfs.getUncompactingSSTables();}}; fixing a race by adding additional checks
seems like it will just make the race less likely rather than eliminate it
entirely
# I'm not sure why compacting status should affect getOverlappingSSTables
For the record, here's the stack trace in question:
{noformat}
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.AssertionError
[junit] at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:373)
[junit] Caused by: java.lang.AssertionError
[junit] at
org.apache.cassandra.db.ColumnFamilyStore.getOverlappingSSTables(ColumnFamilyStore.java:882)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:184)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:296)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:107)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:95)
[junit] at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:185)
{noformat}
> "Memory was freed" AssertionError During Major Compaction
> -
>
> Key: CASSANDRA-5256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0
> Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
> Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_30"
> Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
> Ubuntu 12.04.2 LTS
>Reporter: C. Scott Andreas
>Assignee: Jonathan Ellis
>Priority: Critical
> Labels: compaction
> Fix For: 1.2.2
>
> Attachments: 5256.txt, 5256-v2.txt
>
>
> When initiating a major compaction with `./nodetool -h localhost compact`, an
> AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
> ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
> (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
> java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
> at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
> at
> org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93)
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
> at
> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
> at
> org.apache.cassandra.db.compaction.CompactionIterable$R
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579565#comment-13579565
]
Sylvain Lebresne commented on CASSANDRA-5256:
-
lgtm, +1.
Nits:
* we could also move the try \{ task.execute() \} finally \{
task.unmarkSSTables(); \} inside AbstractCompactionTask to 1) avoid repeating
it 3 times and 2) making sure we don't forget about it.
* maybe the ctor of AbstractCompactionTask could assert that the sstable it
gets are 'marked compacting' to avoid that kind of mistake in the future.
> "Memory was freed" AssertionError During Major Compaction
> -
>
> Key: CASSANDRA-5256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.2.0
> Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
> Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_30"
> Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
> Ubuntu 12.04.2 LTS
>Reporter: C. Scott Andreas
>Assignee: Jonathan Ellis
>Priority: Critical
> Labels: compaction
> Fix For: 1.2.2
>
> Attachments: 5256.txt, 5256-v2.txt
>
>
> When initiating a major compaction with `./nodetool -h localhost compact`, an
> AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
> ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
> (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
> java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
> at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
> at
> org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93)
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
> at
> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
> at
> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71)
> at
> org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579534#comment-13579534 ] Yuki Morishita commented on CASSANDRA-5256: --- Overall, looks good and simplifies things. One problem I had was CompactionsTest was throwing AssertionError on testSingleSSTableCompactionWithLeveledCompaction, when trying to find overlapping sstables to determine droppable ones. Patch for that pushed to https://github.com/yukim/cassandra/commits/5256-3. > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Assignee: Jonathan Ellis >Priority: Critical > Labels: compaction > Fix For: 1.2.2 > > Attachments: 5256.txt, 5256-v2.txt > > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > --- > I'v
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578723#comment-13578723 ] Brandon Williams commented on CASSANDRA-5256: - It's STS. > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Priority: Critical > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > --- > I've invoked the `nodetool compact` three times; this occurred after each. > The node has been up for a couple days accepting writes and has not been > restarted. > Here's the server's log since it was started a few days ago: > https://gist.github.com/cscotta/4956472/raw/95e7cbc68de1aefaeca11812cbb98d5d46f534e8/cassandra.log > Here's the code being used to issue writes to the datastore: > https://gist.github.com/cscotta/20cbd36c2503c71d06e9 >
[jira] [Commented] (CASSANDRA-5256) "Memory was freed" AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578694#comment-13578694 ] Jonathan Ellis commented on CASSANDRA-5256: --- Is this LeveledCompactionStrategy or SizeTiered? > "Memory was freed" AssertionError During Major Compaction > - > > Key: CASSANDRA-5256 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.0 > Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu > Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > java version "1.6.0_30" > Java(TM) SE Runtime Environment (build 1.6.0_30-b12) > Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) > Ubuntu 12.04.2 LTS >Reporter: C. Scott Andreas >Priority: Critical > > When initiating a major compaction with `./nodetool -h localhost compact`, an > AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: > ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java > (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] > java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) > at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) > at > org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) > at > org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:93) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) > at > org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > --- > I've invoked the `nodetool compact` three times; this occurred after each. > The node has been up for a couple days accepting writes and has not been > restarted. > Here's the server's log since it was started a few days ago: > https://gist.github.com/cscotta/4956472/raw/95e7cbc68de1aefaeca11812cbb98d5d46f534e8/cassandra.log > Here's the code being used to issue writes to the datastore: > https://gist.github
