No worries Vinoth. Thank you for the feedback. I have created https://issues.apache.org/jira/browse/HUDI-1588 Anyone interested please feel free to pick it up. I would be happy to do it but not having cycles at the moment.
On Sun, Jan 31, 2021 at 6:32 PM Vinoth Chandar <vin...@apache.org> wrote: > (apologies for the delay here) > > Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed into > the HoodieRecordPayload methods used in ordering/precombining. > So, we can clearly document each payload class with its options and it > should be possible for users to configure this just like any > normal configs. This is probably much simpler. > > Any takers for this work? :) > > On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu.shiyan.raym...@gmail.com> > wrote: > > > Thank you for the feedbacks. I'm also in favor of simplifying the delta > > streamer's usage. > > > > @vinoth, building on what you said, would it be simpler to merge the > > ordering field config with the payload class config? > > i.e., to remove both --source-ordering-field and payload ordering field > > config, then define the ordering field within the payload class. > > People who use fields other than `ts` will need to supply a payload class > > and override it. > > People who already supply a payload class will just override it in the > > existing custom class. > > WDYT? > > > > > > On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <pratyaks...@gmail.com > > > > wrote: > > > > > Hi > > > > > > We can use transformer to have a combination of multiple ordering > fields. > > > However custom Comparable implementation is not possible in that case. > So > > > overall a +1 from my side as well. > > > > > > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <965147...@qq.com> wrote: > > > > > > > +1， Currently we have encountered such scenarios and look > forward > > to > > > > supporting > > > > > > > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > > > 发件人: > > > > "dev" > > > > < > > > > danny0...@apache.org>; > > > > 发送时间: 2021年1月19日(星期二) 下午4:25 > > > > 收件人: "dev"<dev@hudi.apache.org>; > > > > > > > > 主题: Re: [DISCUSS] Support multiple ordering fields > > > > > > > > > > > > > > > > > Wondering if we should just take a bunch of payload configs and > > > > deprecate > > > > these flags > > > > > > > > I have the same feeling, there are already so many config options in > > > > Hoodie, the maintain work for developers or users is hard. > > > > > > > > Vinoth Chandar <vin...@apache.org> 于2021年1月18日周一 下午11:40写道： > > > > > > > > > +1 as well. > > > > > > > > > > Slightly orthogonal point. > > > > > > > > > > Wondering if we should just take a bunch of payload configs and > > > > deprecate > > > > > these flags? > > > > > What I mean is : --source-ordering-field ultimately is used by > the > > > > > HoodieAvroPayload class (or its family). > > > > > Our utilities expose these as flags for convenience, but its > just > > > > more work > > > > > to maintain. > > > > > > > > > > On Mon, Jan 18, 2021 at 7:09 AM vino yang < > yanghua1...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > +1, > > > > > > > > > > > > We have found that such flexibility is needed in some > > > scenarios. > > > > > > > > > > > > Best, > > > > > > Vino > > > > > > > > > > > > Raymond Xu <xu.shiyan.raym...@gmail.com> 于2021年1月17日周日 > > > > 上午3:38写道： > > > > > > > > > > > > > Just want to discuss a small improvement for setting > > > > ordering fields. > > > > > > > For > > > > > > > - property `hoodie.payload.ordering.field` and > > > > > > > - deltastreamer --source-ordering-field > > > > > > > I think it can be useful to support multiple fields > > > > (configured via a > > > > > > > comma-separated list) to determine the order in some > > > cases. > > > > This would > > > > > > need > > > > > > > another config to set the Comparable implementation, > say > > > > > > > hoodie.payload.ordering.comparable.class to allow > custom > > > > logic for > > > > > doing > > > > > > > comparison. > > > > > > > > > > > > > > Any suggestions? Thanks. > > > > > > > > > > > > > > > > > > > > > > > >
