rmahindra123 opened a new pull request #3194: URL: https://github.com/apache/hudi/pull/3194
New PR for #3117 ## What is the purpose of the pull request This pull request adds a new alternative based on RockDb for the Disk Based Map that is used within the ExternalSpillableMap. Our benchmark results shows that RockDb may improve performance significantly when the data set is large while available memory may be scarce. RockDb supports compression, efficient memory usage and native library, that may be more efficient in certain situations. By default, disk based map will be used, and a config change will be required to enable rocksDb. In this PR, the rocksDB support is only enabled for HoodieMergeHandle, and a subsequent PR will extend it to all consumers of ExternalSpillableMap (tracked here HUDI-2044) ## Brief change log - Adds a new alternative based on RockDb for the Disk Based Map that is used within the ExternalSpillableMap. - The support is currently added only for HoodieMergeHandle ## Verify this pull request This change added tests and can be verified as follows: Added the unit test in TestSpillableRocksDBBasedMap Updated the test for TestExternalSpillableMap -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org