Re:Re: Re: Re: Re: Re: Re: Re: Re: Re: [DISCUSS] Refactor of the configuration framework of hudi project

2020-01-09 Thread lamberken
hi @Vinoth Chandar, Got it, thanks. best, lamber-ken At 2020-01-09 23:52:52, "Vinoth Chandar" wrote: >Hi lamber-ken, > >A ConfigOption class would be good indeed. +1 on starting incrementally >with DataSource first and then iterating.. > >Thanks >Vinoth > >On Tue, Jan 7, 2020 at 6:58

Re: Re: Re: Re: Re: Re: Re: Re: Re: [DISCUSS] Refactor of the configuration framework of hudi project

2020-01-09 Thread Vinoth Chandar
Hi lamber-ken, A ConfigOption class would be good indeed. +1 on starting incrementally with DataSource first and then iterating.. Thanks Vinoth On Tue, Jan 7, 2020 at 6:58 PM lamberken wrote: > > > Hi @Vinoth, > > > It's time to pick up this topic. Based on the content we talked about, > here

Re:Re: Re: Re: Re: Re: Re: Re: Re: [DISCUSS] Refactor of the configuration framework of hudi project

2020-01-07 Thread lamberken
Hi @Vinoth, It's time to pick up this topic. Based on the content we talked about, here are my thoughts 1, Initial proposal aims to rework configuration framework includes(DataSource and WriteClient level), for compatibility, we can introduce a ConfigOption class and rework it on

Re: Re: Re: Re: Re: Re: Re: Re: [DISCUSS] Refactor of the configuration framework of hudi project

2019-12-18 Thread Vinoth Chandar
Sounds good.. This scoped down version per se, does not need a RFC. On Wed, Dec 18, 2019 at 3:09 PM lamberken wrote: > > > Hi @Vinoth > > > I understand what you mean, I will continue to work on this when I finish > reworking the new UI. :) > > > best, > lamber-ken > > > > > At 2019-12-18

Re: Re: Re: Re: Re: Re: Re: [DISCUSS] Refactor of the configuration framework of hudi project

2019-12-17 Thread Vinoth Chandar
Expect most users to use inputDF.write() approach... Uber uses the lower level RDD apis, like the DeltaStreamer tool does.. If we don't rename configs and still support a builder, it should be fine. I think we can scope this down to introducing a ConfigOption class that ties, the key,value,