Hi, @Balaji @Vinoth
I'm sorry, some places are not very clear, 1, We can see that HoodieMetricsConfig, HoodieStorageConfig, etc.. already defined in project. But we get property value by methods which defined in HoodieWriteConfig, like HoodieWriteConfig#getParquetMaxFileSize, HoodieWriteConfig#getParquetBlockSize, etc. It's means that Hoodie*Config are redundant. 2, These Hoodie*Config classes are used to set default value when call their build method, nothing else. 3, For current plan is keep the Builder pattern when configuring, when we are familiar with the config framework, We will find that Hoodie*Config class are redundant and methods prefixed with "get" in HoodieWriteConfig are also redundant. In addition, I create a pr[1] for initializing with a demo. At this demo, I create MetricsGraphiteReporterOptions which contains HOST, PORT, PREFIX, and remove getGraphiteServerHost, getGraphiteServerPort, getGraphiteMetricPrefix in HoodieMetricsConfig. https://github.com/apache/incubator-hudi/pull/1094 Best, lamber-ken At 2019-12-11 02:35:30, "Balaji Varadarajan" <[email protected]> wrote: > Hi Lamber-Ken, >Thanks for the time writing the proposal and thinking about improving Hudi >usability. >My preference would be to keep the Builder pattern when configuring. It is >something I find it natural when configuring. It is not clear to me whether >there is any external facing changes which changes this model. Would you mind >adding some more details on the RFC. It would save time to read it in one >place as opposed to checking out github repo :) >Thanks,Balaji.V > On Tuesday, December 10, 2019, 07:55:01 AM PST, Vinoth Chandar > <[email protected]> wrote: > > Hi , > >Thanks for the proposal. Some parts I agree, some parts I don't and some >parts are unclear > >Agree : >- On introducing a class that binds key, default value, provided value, and >also may be a doc along with it (?). >- Designing the framework to have fallback keys is good IMO. It helps us do >things like https://issues.apache.org/jira/browse/HUDI-89 > >Disagree : >- Not all configuration values are in HoodieWriteConfig, its not accurate. >Configs are already split by components into HoodieIndexConfig, >HoodieCompactionConfig etc.. >- There are helpers for all these conveniently located in >HoodieWriteConfig. I think some of the claims of usability seem subjective >to me, speaking from hands-on experience writing jobs. So, if you proposing >a large shake up (e.g not have a single properties file load all >components), I would love to understand this at more depth. From my >experience, well namespaced configs in a single properties file keeps it >simple and understandable. > >Unclear : >- What is impact on existing jobs - using RDD/WriteClient API, DataSource, >DeltaStreamer level? Do you intend to change namespacing of configs? > > >Thanks >Vinoth > >On Tue, Dec 10, 2019 at 6:44 AM lamberken <[email protected]> wrote: > >> >> >> Hi, vino >> >> >> Reasonable, we can refactor this step by step. The first step now is to >> introduce the config framework. >> When our community is familiar with the config framework mechanism, it's >> easy to integrate FallbackKey in the future. >> >> >> Best, >> lamber-ken >> >> >> >> At 2019-12-10 11:51:22, "vino yang" <[email protected]> wrote: >> >Hi Lamber, >> > >> >Thanks for the proposal. +1 from my side. >> > >> >When it comes to configuration, it will involve how we handle deprecated >> >configuration items in the future. In my opinion, we need to take this >> into >> >consideration when designing. There are already some successful practices >> >for our reference. For example, Flink defines some deprecated >> >configurations as FallbackKey[1]. Maybe we can learn from these designs. >> > >> >WDYT？ >> > >> >[1]: >> > >> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/FallbackKey.java >> > >> >Best, >> >Vino >> > >> >lamberken <[email protected]> 于2019年12月9日周一 下午11:19写道： >> > >> >> >> >> >> >> Hi, all >> >> >> >> >> >> Currently, many configuration items and their default values are >> dispersed >> >> in the config file like HoodieWriteConfig. It’s very confused for >> >> developers, and it's easy for developers to use them in a wrong place >> >> especially when there are more and more configuration items. If we can >> >> solve this, developers will benefit from it and the code structure will >> be >> >> more concise. >> >> >> >> >> >> I had create a JIRA[1] and a under discuss RFC[2] to explain how to >> solve >> >> the problem, if you are interested in this, you can visit jira and RFC >> for >> >> detail. Any comments and feedback are welcome, WDYT? >> >> >> >> >> >> Best, >> >> lamber-ken >> >> >> >> >> >> [1] https://issues.apache.org/jira/projects/HUDI/issues/HUDI-375 >> >> [2] >> >> >> https://cwiki.apache.org/confluence/display/HUDI/RFC-11+%3A+Refactor+of+the+configuration+framework+of+hudi+project >>
