guozhangwang commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r619416799
##########
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +101,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
- /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call
Options::PrepareForBulkLoad() to your option
- */
- // (1) not in our control
- // (2) is done via bulk-loading API
- // (3) skipping because, not done in actual PrepareForBulkLoad() code
in https://github.com/facebook/rocksdb/blob/master/options/options.cc
- //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
- // (4-5) below:
- dbOptions.setMaxBackgroundFlushes(4);
- columnFamilyOptions.setDisableAutoCompactions(true);
- columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
- columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
- columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+ super.prepareForBulkLoad();
Review comment:
Maybe still valuable to leave a comment on which rocksDB version the
bulk loading behavior is updated to meet our needs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
