[jira] [Updated] (CASSANDRA-4142) OOM Exception during repair session with LeveledCompactionStrategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Deng updated CASSANDRA-4142: Labels: lcs (was: ) > OOM Exception during repair session with LeveledCompactionStrategy > -- > > Key: CASSANDRA-4142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4142 > Project: Cassandra > Issue Type: Improvement >Affects Versions: 1.0.0 > Environment: OS: Linux CentOs 6 > JDK: Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode) > Node configuration: > Quad-core > 10 GB RAM > Xmx set to 2,5 GB (as computed by default). >Reporter: Romain Hardouin >Assignee: Sylvain Lebresne > Labels: lcs > Fix For: 1.1.1 > > Attachments: 4142-v2.txt, 4142.txt > > > We encountered an OOM Exception on 2 nodes during repair session. > Our CF are set up to use LeveledCompactionStrategy and SnappyCompressor. > These two options used together maybe the key to the problem. > Despite of setting XX:+HeapDumpOnOutOfMemoryError, no dump have been > generated. > Nonetheless a memory analysis on a live node doing a repair reveals an > hotspot: an ArrayList of SSTableBoundedScanner which appears to contain as > many objects as there are SSTables on disk. > This ArrayList consumes 786 MB of the heap space for 5757 objects. Therefore > each object is about 140 KB. > Eclipse Memory Analyzer's denominator tree shows that 99% of a > SSTableBoundedScanner object's memory is consumed by a > CompressedRandomAccessReader which contains two big byte arrays. > Cluster information: > 9 nodes > Each node handles 35 GB (RandomPartitioner) > This JIRA was created following this discussion: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Why-so-many-SSTables-td7453033.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-4142) OOM Exception during repair session with LeveledCompactionStrategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4142: -- Attachment: 4142-v2.txt v2 attached: - Renames KeyScanner to o.a.c.db.compaction.ICompactionScanner - Remove AbstractCompactionIterable.getScanners in favor of ACS.getScanners, which wires it in to normal compactions as well. (This lets ParallelCompactionIterator be more agressive about how much memory it lets each scanner use, too.) - Updated LCS.getScanners constructor to use a Multimap, which simplifies things a bit. > OOM Exception during repair session with LeveledCompactionStrategy > -- > > Key: CASSANDRA-4142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4142 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.0.0 > Environment: OS: Linux CentOs 6 > JDK: Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode) > Node configuration: > Quad-core > 10 GB RAM > Xmx set to 2,5 GB (as computed by default). >Reporter: Romain Hardouin >Assignee: Sylvain Lebresne > Fix For: 1.1.1 > > Attachments: 4142-v2.txt, 4142.txt > > > We encountered an OOM Exception on 2 nodes during repair session. > Our CF are set up to use LeveledCompactionStrategy and SnappyCompressor. > These two options used together maybe the key to the problem. > Despite of setting XX:+HeapDumpOnOutOfMemoryError, no dump have been > generated. > Nonetheless a memory analysis on a live node doing a repair reveals an > hotspot: an ArrayList of SSTableBoundedScanner which appears to contain as > many objects as there are SSTables on disk. > This ArrayList consumes 786 MB of the heap space for 5757 objects. Therefore > each object is about 140 KB. > Eclipse Memory Analyzer's denominator tree shows that 99% of a > SSTableBoundedScanner object's memory is consumed by a > CompressedRandomAccessReader which contains two big byte arrays. > Cluster information: > 9 nodes > Each node handles 35 GB (RandomPartitioner) > This JIRA was created following this discussion: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Why-so-many-SSTables-td7453033.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4142) OOM Exception during repair session with LeveledCompactionStrategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-4142: Attachment: 4142.txt Attaching patch to only open 1 sstableScanner per level during repair. It only use the trick for repair because that is the only place where we open lots of sstableScanner, but I suppose we could apply the same to normal compaction and lower slightly the memory consumption there too. Anyway, the patch includes a unit test to exercise the new code. > OOM Exception during repair session with LeveledCompactionStrategy > -- > > Key: CASSANDRA-4142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4142 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.0.0 > Environment: OS: Linux CentOs 6 > JDK: Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode) > Node configuration: > Quad-core > 10 GB RAM > Xmx set to 2,5 GB (as computed by default). >Reporter: Romain Hardouin >Assignee: Sylvain Lebresne > Fix For: 1.1.1 > > Attachments: 4142.txt > > > We encountered an OOM Exception on 2 nodes during repair session. > Our CF are set up to use LeveledCompactionStrategy and SnappyCompressor. > These two options used together maybe the key to the problem. > Despite of setting XX:+HeapDumpOnOutOfMemoryError, no dump have been > generated. > Nonetheless a memory analysis on a live node doing a repair reveals an > hotspot: an ArrayList of SSTableBoundedScanner which appears to contain as > many objects as there are SSTables on disk. > This ArrayList consumes 786 MB of the heap space for 5757 objects. Therefore > each object is about 140 KB. > Eclipse Memory Analyzer's denominator tree shows that 99% of a > SSTableBoundedScanner object's memory is consumed by a > CompressedRandomAccessReader which contains two big byte arrays. > Cluster information: > 9 nodes > Each node handles 35 GB (RandomPartitioner) > This JIRA was created following this discussion: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Why-so-many-SSTables-td7453033.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4142) OOM Exception during repair session with LeveledCompactionStrategy
[ https://issues.apache.org/jira/browse/CASSANDRA-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4142: -- Affects Version/s: (was: 1.0.6) 1.0.0 Fix Version/s: 1.1.1 Assignee: Sylvain Lebresne > OOM Exception during repair session with LeveledCompactionStrategy > -- > > Key: CASSANDRA-4142 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4142 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.0.0 > Environment: OS: Linux CentOs 6 > JDK: Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode) > Node configuration: > Quad-core > 10 GB RAM > Xmx set to 2,5 GB (as computed by default). >Reporter: Romain Hardouin >Assignee: Sylvain Lebresne > Fix For: 1.1.1 > > > We encountered an OOM Exception on 2 nodes during repair session. > Our CF are set up to use LeveledCompactionStrategy and SnappyCompressor. > These two options used together maybe the key to the problem. > Despite of setting XX:+HeapDumpOnOutOfMemoryError, no dump have been > generated. > Nonetheless a memory analysis on a live node doing a repair reveals an > hotspot: an ArrayList of SSTableBoundedScanner which appears to contain as > many objects as there are SSTables on disk. > This ArrayList consumes 786 MB of the heap space for 5757 objects. Therefore > each object is about 140 KB. > Eclipse Memory Analyzer's denominator tree shows that 99% of a > SSTableBoundedScanner object's memory is consumed by a > CompressedRandomAccessReader which contains two big byte arrays. > Cluster information: > 9 nodes > Each node handles 35 GB (RandomPartitioner) > This JIRA was created following this discussion: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Why-so-many-SSTables-td7453033.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
