[
https://issues.apache.org/jira/browse/CASSANDRA-5555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-5555:
-----------------------------------
Attachment: 5555-fix-heap-and-streaming-1.2.patch
Attached patch {{5555-fix-heap-and-streaming-1.2.patch}} (and
[branch|https://github.com/thobbs/cassandra/tree/5555]) is based on Jonathan's
5555-2 branch, but fixes the estimated keys issue that Yuki brought up.
The index summary was being used just before streaming to calculate both the
estimated number of keys and the range of positions within the sstable files to
stream. Those are now calculated upfront and then the index summary reference
is dropped so that it can be GC'ed.
I did some minor stress testing, but nothing as complete as Alex's steps. (Let
me know if I should do those instead of Alex.)
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>
> Key: CASSANDRA-5555
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5555
> Project: Cassandra
> Issue Type: Improvement
> Components: Core, Tools
> Reporter: Tyler Hobbs
> Assignee: Tyler Hobbs
> Fix For: 1.2.7
>
> Attachments: 5555-01.txt, 5555-02.txt, 5555-2.txt,
> 5555-fix-heap-and-streaming-1.2.patch, cass_5555_pic_8.png,
> CASSANDRA-5555.txt, CASSANDRA-5555.txt, CASSANDRA-5555.txt
>
>
> With the default heap size, sstableloader will OOM when there are roughly 25k
> files in the directory to load. It's easy to reach this number of files in a
> single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should
> be able to increase the number of files that sstableloader can handle
> considerably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
