Hi Dongjin, Thanks for the KIP, and the benchmark results. It makes sense to me.
Just one question: > compression.zstd.window: enables long mode; the log of the window size that zstd uses to memorize the compressing data. (available: [10, 22], default: 0 (disables long mode.)) It said, available value is [10, 22], but default is a value out of that range, which should be wrong. Thank you. Luke On Sun, Oct 10, 2021 at 9:50 PM Dongjin Lee <dong...@apache.org> wrote: > Hi Kafka dev, > > I would like to start the discussion of KIP-780: Support fine-grained > compression options. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options > > Here is some context or history on this feature; initially, this feature > was intended to be a part of KIP-390: Support Compression Level, but when I > was working on it, I could not find the evidence that these options can > improve the performance, so it was excluded from the final proposal. Since > this (tentative) conclusion was somewhat strange, KIP-390 was passed under > the condition that a following work should be done for the > buffer/block/window-related configuration options. > > And after some repetitive prototypes and benchmarks, it seems like I > finally found the evidence. It is why I am submitting it as a separate > proposal now. The document also includes what I found during the tests in > the Benchmark section. > > All kinds of feedbacks are greatly appreciated! > > Best, > Dongjin > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* >
