(apologies for the delay here) Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed into the HoodieRecordPayload methods used in ordering/precombining. So, we can clearly document each payload class with its options and it should be possible for users to configure this just like any normal configs. This is probably much simpler.
Any takers for this work? :) On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu.shiyan.raym...@gmail.com> wrote: > Thank you for the feedbacks. I'm also in favor of simplifying the delta > streamer's usage. > > @vinoth, building on what you said, would it be simpler to merge the > ordering field config with the payload class config? > i.e., to remove both --source-ordering-field and payload ordering field > config, then define the ordering field within the payload class. > People who use fields other than `ts` will need to supply a payload class > and override it. > People who already supply a payload class will just override it in the > existing custom class. > WDYT? > > > On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <pratyaks...@gmail.com> > wrote: > > > Hi > > > > We can use transformer to have a combination of multiple ordering fields. > > However custom Comparable implementation is not possible in that case. So > > overall a +1 from my side as well. > > > > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <965147...@qq.com> wrote: > > > > > +1， Currently we have encountered such scenarios and look forward > to > > > supporting > > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > > 发件人: > > > "dev" > > > < > > > danny0...@apache.org>; > > > 发送时间: 2021年1月19日(星期二) 下午4:25 > > > 收件人: "dev"<dev@hudi.apache.org>; > > > > > > 主题: Re: [DISCUSS] Support multiple ordering fields > > > > > > > > > > > > > Wondering if we should just take a bunch of payload configs and > > > deprecate > > > these flags > > > > > > I have the same feeling, there are already so many config options in > > > Hoodie, the maintain work for developers or users is hard. > > > > > > Vinoth Chandar <vin...@apache.org> 于2021年1月18日周一 下午11:40写道： > > > > > > > +1 as well. > > > > > > > > Slightly orthogonal point. > > > > > > > > Wondering if we should just take a bunch of payload configs and > > > deprecate > > > > these flags? > > > > What I mean is : --source-ordering-field ultimately is used by the > > > > HoodieAvroPayload class (or its family). > > > > Our utilities expose these as flags for convenience, but its just > > > more work > > > > to maintain. > > > > > > > > On Mon, Jan 18, 2021 at 7:09 AM vino yang <yanghua1...@gmail.com > > > > > > wrote: > > > > > > > > > +1, > > > > > > > > > > We have found that such flexibility is needed in some > > scenarios. > > > > > > > > > > Best, > > > > > Vino > > > > > > > > > > Raymond Xu <xu.shiyan.raym...@gmail.com> 于2021年1月17日周日 > > > 上午3:38写道： > > > > > > > > > > > Just want to discuss a small improvement for setting > > > ordering fields. > > > > > > For > > > > > > - property `hoodie.payload.ordering.field` and > > > > > > - deltastreamer --source-ordering-field > > > > > > I think it can be useful to support multiple fields > > > (configured via a > > > > > > comma-separated list) to determine the order in some > > cases. > > > This would > > > > > need > > > > > > another config to set the Comparable implementation, say > > > > > > hoodie.payload.ordering.comparable.class to allow custom > > > logic for > > > > doing > > > > > > comparison. > > > > > > > > > > > > Any suggestions? Thanks. > > > > > > > > > > > > > > > > > >
