[ https://issues.apache.org/jira/browse/CASSANDRA-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302728#comment-14302728 ]
Philip Thompson edited comment on CASSANDRA-8716 at 2/3/15 5:49 PM: -------------------------------------------------------------------- I can reproduce this very easily with java 1.7.0_67. It's just a matter of creating a 2.0.12 node with ccm, writing with stress and flushing, bootstrapping a second node, then running cleanup. Could this be related to CASSANDRA-8718? EDIT: I can confirm the issue is intermittent. I am opening a PR for a dtest for this issue. was (Author: philipthompson): I can reproduce this very easily with java 1.7.0_67. It's just a matter of creating a 2.0.12 node with ccm, writing with stress and flushing, bootstrapping a second node, then running cleanup. Could this be related to CASSANDRA-8718? > "java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory > was freed" when running cleanup > ---------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-8716 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8716 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Centos 6.6, Cassandra 2.0.12, Oracle JDK 1.7.0_67 > Reporter: Imri Zvik > Assignee: Benedict > Priority: Minor > Fix For: 2.0.13 > > Attachments: system.log.gz > > > {code}Error occurred during cleanup > java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was > freed > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234) > at > org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272) > at > org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115) > at > org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177) > at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) > at sun.rmi.transport.Transport$1.run(Transport.java:177) > at sun.rmi.transport.Transport$1.run(Transport.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:173) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.AssertionError: Memory was freed > at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259) > at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211) > at > org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79) > at > org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84) > at > org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58) > at > org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602) > at > org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947) > at > org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910) > at > org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819) > at > org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088) > at > org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564) > at > org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63) > at > org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more{code} > The error continue to happen even after scrub. > If I retry a few time, it may run successfully. > I am attaching the server log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)