[
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303428#comment-15303428
]
Henry Cai commented on KAFKA-3740:
----------------------------------
Looks like you want two sets of RocksDB settings, one for K/V store, one for
range scan store. I think for most of the small RocksDB (size < 5GB), those
settings (target file size, or bloom filter) probably won't matter.
To make it really flexible, you would need to provide per store rocksdb
settings. E.g. for A join B, if A has small-size records and B has big-size
records you would want to set the targeFileSize smaller for RocksDB A and
targetFileSize bigger for RocksDB B. It seems it would have the following
overlay config structures in StreamConfig:
rocksdb.default.target_file_size_mb=16
rocksdb.default.use_bloom_filter=true
...
rocksdb.aggregation_style.target_file_size_mb=8
...
rocksdb.join_style.target_file_size_mb=32
...
rocksdb.custom1_style.target_file_size_mb=128
System understands 'default', 'aggregation_style', 'join_style' and will choose
different style accordingly based on the context.
Application can optionally associate a RocksDB with customer style e.g.
custom1_style (need some API enhancement).
This might get more complicated, I think for the initial version, as long as we
can provide rocksdb.default.property this would be a big enhancement.
> Add configs for RocksDBStores
> -----------------------------
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Henry Cai
> Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}},
> or the default values are directly used. We need to make them configurable
> for advanced users. For example, some default values may not work perfectly
> for some scenarios:
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar
> to "StreamsConfig", which defines all related rocksDB options configs, that
> can be passed as key-value pairs to "StreamsConfig".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
