Hi , Thanks for the proposal. Some parts I agree, some parts I don't and some parts are unclear
Agree : - On introducing a class that binds key, default value, provided value, and also may be a doc along with it (?). - Designing the framework to have fallback keys is good IMO. It helps us do things like https://issues.apache.org/jira/browse/HUDI-89 Disagree : - Not all configuration values are in HoodieWriteConfig, its not accurate. Configs are already split by components into HoodieIndexConfig, HoodieCompactionConfig etc.. - There are helpers for all these conveniently located in HoodieWriteConfig. I think some of the claims of usability seem subjective to me, speaking from hands-on experience writing jobs. So, if you proposing a large shake up (e.g not have a single properties file load all components), I would love to understand this at more depth. From my experience, well namespaced configs in a single properties file keeps it simple and understandable. Unclear : - What is impact on existing jobs - using RDD/WriteClient API, DataSource, DeltaStreamer level? Do you intend to change namespacing of configs? Thanks Vinoth On Tue, Dec 10, 2019 at 6:44 AM lamberken <[email protected]> wrote: > > > Hi, vino > > > Reasonable, we can refactor this step by step. The first step now is to > introduce the config framework. > When our community is familiar with the config framework mechanism, it's > easy to integrate FallbackKey in the future. > > > Best, > lamber-ken > > > > At 2019-12-10 11:51:22, "vino yang" <[email protected]> wrote: > >Hi Lamber, > > > >Thanks for the proposal. +1 from my side. > > > >When it comes to configuration, it will involve how we handle deprecated > >configuration items in the future. In my opinion, we need to take this > into > >consideration when designing. There are already some successful practices > >for our reference. For example, Flink defines some deprecated > >configurations as FallbackKey[1]. Maybe we can learn from these designs. > > > >WDYT？ > > > >[1]: > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/FallbackKey.java > > > >Best, > >Vino > > > >lamberken <[email protected]> 于2019年12月9日周一 下午11:19写道： > > > >> > >> > >> Hi, all > >> > >> > >> Currently, many configuration items and their default values are > dispersed > >> in the config file like HoodieWriteConfig. It’s very confused for > >> developers, and it's easy for developers to use them in a wrong place > >> especially when there are more and more configuration items. If we can > >> solve this, developers will benefit from it and the code structure will > be > >> more concise. > >> > >> > >> I had create a JIRA[1] and a under discuss RFC[2] to explain how to > solve > >> the problem, if you are interested in this, you can visit jira and RFC > for > >> detail. Any comments and feedback are welcome, WDYT? > >> > >> > >> Best, > >> lamber-ken > >> > >> > >> [1] https://issues.apache.org/jira/projects/HUDI/issues/HUDI-375 > >> [2] > >> > https://cwiki.apache.org/confluence/display/HUDI/RFC-11+%3A+Refactor+of+the+configuration+framework+of+hudi+project >
