keith-turner commented on PR #3934:
URL: https://github.com/apache/accumulo/pull/3934#issuecomment-1800218982
Adding to @cshannon comment, we did come up with ways to work around the
issues identified. However, we came to the conclusion that making merge able
to create a single tablet with more files than can fit in memory would cause
problems for split, compaction, scan, etc. Realized it was not worthwhile
pursing these changes and decided it would be best to do the following.
* Add a property that limits the amount files that would be merged into a
single tablet. For example if the limit was 10K and if a merge tried to run on
1000 tablets each having 100 files then it would fail before making any
metadata changes. If the merge ran then it would create a single tablet with
100K files.
* Still allow merging a large number of tablets as long as there is not a
large number of files. For example merging 1 million tablets with a total of
100 file across all tablets would still work.
Going to close this PR, rework this code and open a new PR that supports the
above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
