[jira] [Commented] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
[ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701912#comment-13701912 ] Marcus Eriksson commented on CASSANDRA-5342: thanks, committed as 83b75754ff143d4d77b01ef76a813da47779c6f4 opted to leave the Pair in, no strong feelings either > ancestors are not cleared in SSTableMetadata after compactions are done and > old SSTables are removed > > > Key: CASSANDRA-5342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5342 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.3 >Reporter: Wei Zhu >Assignee: Marcus Eriksson > Fix For: 1.2.7 > > Attachments: 0001-CASSANDRA-5342-wip.patch, > 0001-CASSANDRA-5342-wip-v2.patch, Screen Shot 2013-03-13 at 12.05.08 PM.png > > > We are using LCS and have total of 38000 SSTables for one CF. During LCS, > there could be over a thousand SSTable involved. All those SSTable IDs are > stored in ancestors field of SSTableMetatdata for the new table. In our case, > it consumes more than 1G of heap memory for those field. Put it in > perspective, the ancestors consume 2 - 3 times more memory than bloomfilter > (fp = 0.1 by default) in LCS. > We should remove those ancestors from SSTableMetadata after the compaction is > finished and the old SSTable is removed. It might be a big deal for Sized > Compaction since there are small number of SSTable involved. But it consumes > a lot of memory for LCS. > At least, we shouldn't load those ancestors to the memory during startup if > the files are removed. > I would love to contribute and provide patch. Please let me know how to > start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
[ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701217#comment-13701217 ] Jonathan Ellis commented on CASSANDRA-5342: --- LGTM. Nit: since we only ever use either the metadata, or the anscestors, it may be cleaner to split deserializeAncestors into a separate method. OTOH that might make it easier to make mistakes in compatibility. I'm fine either way. > ancestors are not cleared in SSTableMetadata after compactions are done and > old SSTables are removed > > > Key: CASSANDRA-5342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5342 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.3 >Reporter: Wei Zhu >Assignee: Marcus Eriksson > Fix For: 1.2.7 > > Attachments: 0001-CASSANDRA-5342-wip.patch, > 0001-CASSANDRA-5342-wip-v2.patch, Screen Shot 2013-03-13 at 12.05.08 PM.png > > > We are using LCS and have total of 38000 SSTables for one CF. During LCS, > there could be over a thousand SSTable involved. All those SSTable IDs are > stored in ancestors field of SSTableMetatdata for the new table. In our case, > it consumes more than 1G of heap memory for those field. Put it in > perspective, the ancestors consume 2 - 3 times more memory than bloomfilter > (fp = 0.1 by default) in LCS. > We should remove those ancestors from SSTableMetadata after the compaction is > finished and the old SSTable is removed. It might be a big deal for Sized > Compaction since there are small number of SSTable involved. But it consumes > a lot of memory for LCS. > At least, we shouldn't load those ancestors to the memory during startup if > the files are removed. > I would love to contribute and provide patch. Please let me know how to > start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
[ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667130#comment-13667130 ] Jonathan Ellis commented on CASSANDRA-5342: --- Yeah, it sounds easy enough in principle, but it might get messy in practice since we've had this "immutable SSTableMetadata" concept. > ancestors are not cleared in SSTableMetadata after compactions are done and > old SSTables are removed > > > Key: CASSANDRA-5342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5342 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.3 >Reporter: Wei Zhu > Fix For: 1.2.6 > > Attachments: Screen Shot 2013-03-13 at 12.05.08 PM.png > > > We are using LCS and have total of 38000 SSTables for one CF. During LCS, > there could be over a thousand SSTable involved. All those SSTable IDs are > stored in ancestors field of SSTableMetatdata for the new table. In our case, > it consumes more than 1G of heap memory for those field. Put it in > perspective, the ancestors consume 2 - 3 times more memory than bloomfilter > (fp = 0.1 by default) in LCS. > We should remove those ancestors from SSTableMetadata after the compaction is > finished and the old SSTable is removed. It might be a big deal for Sized > Compaction since there are small number of SSTable involved. But it consumes > a lot of memory for LCS. > At least, we shouldn't load those ancestors to the memory during startup if > the files are removed. > I would love to contribute and provide patch. Please let me know how to > start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
[ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667085#comment-13667085 ] Marcus Eriksson commented on CASSANDRA-5342: have not really followed #5151 so these is my quite out-of-context thoughts: as far as i can see they are only read during startup, maybe we could not keep a reference to them after flushing the metadata, and then reading them on startup only to be able to remove the compaction leftovers? and then clear them out the same way we mutate sstable level? i could have a go > ancestors are not cleared in SSTableMetadata after compactions are done and > old SSTables are removed > > > Key: CASSANDRA-5342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5342 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.3 >Reporter: Wei Zhu > Fix For: 1.2.6 > > Attachments: Screen Shot 2013-03-13 at 12.05.08 PM.png > > > We are using LCS and have total of 38000 SSTables for one CF. During LCS, > there could be over a thousand SSTable involved. All those SSTable IDs are > stored in ancestors field of SSTableMetatdata for the new table. In our case, > it consumes more than 1G of heap memory for those field. Put it in > perspective, the ancestors consume 2 - 3 times more memory than bloomfilter > (fp = 0.1 by default) in LCS. > We should remove those ancestors from SSTableMetadata after the compaction is > finished and the old SSTable is removed. It might be a big deal for Sized > Compaction since there are small number of SSTable involved. But it consumes > a lot of memory for LCS. > At least, we shouldn't load those ancestors to the memory during startup if > the files are removed. > I would love to contribute and provide patch. Please let me know how to > start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5342) ancestors are not cleared in SSTableMetadata after compactions are done and old SSTables are removed
[ https://issues.apache.org/jira/browse/CASSANDRA-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601546#comment-13601546 ] Yuki Morishita commented on CASSANDRA-5342: --- Ancestors are used to track compaction leftovers. We are implementing better way to achieve that in CASSANDRA-5151 using system table. > ancestors are not cleared in SSTableMetadata after compactions are done and > old SSTables are removed > > > Key: CASSANDRA-5342 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5342 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.10, 1.2.2 >Reporter: Wei Zhu > Attachments: Screen Shot 2013-03-13 at 12.05.08 PM.png > > > We are using LCS and have total of 38000 SSTables for one CF. During LCS, > there could be over a thousand SSTable involved. All those SSTable IDs are > stored in ancestors field of SSTableMetatdata for the new table. In our case, > it consumes more than 1G of heap memory for those field. Put it in > perspective, the ancestors consume 2 - 3 times more memory than bloomfilter > (fp = 0.1 by default) in LCS. > We should remove those ancestors from SSTableMetadata after the compaction is > finished and the old SSTable is removed. It might be a big deal for Sized > Compaction since there are small number of SSTable involved. But it consumes > a lot of memory for LCS. > At least, we shouldn't load those ancestors to the memory during startup if > the files are removed. > I would love to contribute and provide patch. Please let me know how to > start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
