guozhangwang commented on a change in pull request #10568:
URL: https://github.com/apache/kafka/pull/10568#discussion_r626249438
##########
File path:
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +104,7 @@ public Env getEnv() {
@Override
public Options prepareForBulkLoad() {
- /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
- *
- * Q: What's the fastest way to load data into RocksDB?
- *
- * A: A fast way to direct insert data to the DB:
- *
- * 1. using single writer thread and insert in sorted order
- * 2. batch hundreds of keys into one write batch
- * 3. use vector memtable
- * 4. make sure options.max_background_flushes is at least 4
- * 5. before inserting the data,
- * disable automatic compaction,
- * set options.level0_file_num_compaction_trigger,
- * options.level0_slowdown_writes_trigger
- * and options.level0_stop_writes_trigger to very large.
- * After inserting all the data, issue a manual compaction.
- *
- * 3-5 will be automatically done if you call
Options::PrepareForBulkLoad() to your option
- */
- // (1) not in our control
- // (2) is done via bulk-loading API
- // (3) skipping because, not done in actual PrepareForBulkLoad() code
in https://github.com/facebook/rocksdb/blob/master/options/options.cc
- //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
- // (4-5) below:
- dbOptions.setMaxBackgroundFlushes(4);
- columnFamilyOptions.setDisableAutoCompactions(true);
- columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
- columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
- columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
Review comment:
I left a comment on the old PR as well --- I vaguely remember that in
new versions this is done inside rocksDB hence we do not need it, but maybe
@cadonna can confirm here. And if yes, we could add a one line comment for
future readers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
