elek opened a new pull request #665: HDDS-3160. Disable index and filter block cache for RocksDB. URL: https://github.com/apache/hadoop-ozone/pull/665 ## What changes were proposed in this pull request? During preformance tests It was noticed that the OM performance is dropped after 10-20 million of keys. (screenshot is attached to the JIRA) By default `cache_index_and_filter_blocks` is enabled for all of our RocksDB instances (see `DBProfile.java`) which is not the best option. (For example see this thread: https://github.com/facebook/rocksdb/issues/3961#) With turning on this cache the indexes and bloom filters are cached __inside the block cache__ which makes slower the cache when we have significant amount of data. Without turning it on (based on my understanding) all the indexes will remain open without any cache. With our current settings we have only a few number of sst files (even with million of keys) therefore it seems to be safe to turn this option off. With turning this option of I was able to write >200M keys with high throughput (10 k key allocation / sec + 10 k key close / sec) ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3160 ## How was this patch tested? With the help of #656 I started long-running OM load tests. (See the details there). Without the patch after 10-20M of keys the performance is dropped from 10 k key allocation / sec to a few hundreds (see the screenshot of the JIRA). With the patch It was possible to maintain the 10k / sec rate during the firstm 200M key allocation (still running...)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
