It is reasonable to support multiple ordering fields. The ordering fields semantics is very similar with the SQL PRIMARY KEY, the PRIMARY KEY did support multiple fields.
A little different with PRIMARY KEY is that, current ordering field also supports nested fields, E.G. "a.b.c", which the PRIMARY KEY definition can not cover. Best, Danny Chan Raymond Xu <xu.shiyan.raym...@gmail.com> 于2021年2月5日周五 下午6:48写道： > No worries Vinoth. Thank you for the feedback. I have created > https://issues.apache.org/jira/browse/HUDI-1588 > Anyone interested please feel free to pick it up. I would be happy to do it > but not having cycles at the moment. > > On Sun, Jan 31, 2021 at 6:32 PM Vinoth Chandar <vin...@apache.org> wrote: > > > (apologies for the delay here) > > > > Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed > into > > the HoodieRecordPayload methods used in ordering/precombining. > > So, we can clearly document each payload class with its options and it > > should be possible for users to configure this just like any > > normal configs. This is probably much simpler. > > > > Any takers for this work? :) > > > > On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu.shiyan.raym...@gmail.com> > > wrote: > > > > > Thank you for the feedbacks. I'm also in favor of simplifying the delta > > > streamer's usage. > > > > > > @vinoth, building on what you said, would it be simpler to merge the > > > ordering field config with the payload class config? > > > i.e., to remove both --source-ordering-field and payload ordering field > > > config, then define the ordering field within the payload class. > > > People who use fields other than `ts` will need to supply a payload > class > > > and override it. > > > People who already supply a payload class will just override it in the > > > existing custom class. > > > WDYT? > > > > > > > > > On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma < > pratyaks...@gmail.com > > > > > > wrote: > > > > > > > Hi > > > > > > > > We can use transformer to have a combination of multiple ordering > > fields. > > > > However custom Comparable implementation is not possible in that > case. > > So > > > > overall a +1 from my side as well. > > > > > > > > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <965147...@qq.com> wrote: > > > > > > > > > +1， Currently we have encountered such scenarios and look > > forward > > > to > > > > > supporting > > > > > > > > > > > > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > > > > 发件人: > > > > > "dev" > > > > > < > > > > > danny0...@apache.org>; > > > > > 发送时间: 2021年1月19日(星期二) 下午4:25 > > > > > 收件人: "dev"<dev@hudi.apache.org>; > > > > > > > > > > 主题: Re: [DISCUSS] Support multiple ordering fields > > > > > > > > > > > > > > > > > > > > > Wondering if we should just take a bunch of payload configs > and > > > > > deprecate > > > > > these flags > > > > > > > > > > I have the same feeling, there are already so many config options > in > > > > > Hoodie, the maintain work for developers or users is hard. > > > > > > > > > > Vinoth Chandar <vin...@apache.org> 于2021年1月18日周一 下午11:40写道： > > > > > > > > > > > +1 as well. > > > > > > > > > > > > Slightly orthogonal point. > > > > > > > > > > > > Wondering if we should just take a bunch of payload configs > and > > > > > deprecate > > > > > > these flags? > > > > > > What I mean is : --source-ordering-field ultimately is used by > > the > > > > > > HoodieAvroPayload class (or its family). > > > > > > Our utilities expose these as flags for convenience, but its > > just > > > > > more work > > > > > > to maintain. > > > > > > > > > > > > On Mon, Jan 18, 2021 at 7:09 AM vino yang < > > yanghua1...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > +1, > > > > > > > > > > > > > > We have found that such flexibility is needed in some > > > > scenarios. > > > > > > > > > > > > > > Best, > > > > > > > Vino > > > > > > > > > > > > > > Raymond Xu <xu.shiyan.raym...@gmail.com> > 于2021年1月17日周日 > > > > > 上午3:38写道： > > > > > > > > > > > > > > > Just want to discuss a small improvement for setting > > > > > ordering fields. > > > > > > > > For > > > > > > > > - property `hoodie.payload.ordering.field` and > > > > > > > > - deltastreamer --source-ordering-field > > > > > > > > I think it can be useful to support multiple fields > > > > > (configured via a > > > > > > > > comma-separated list) to determine the order in some > > > > cases. > > > > > This would > > > > > > > need > > > > > > > > another config to set the Comparable implementation, > > say > > > > > > > > hoodie.payload.ordering.comparable.class to allow > > custom > > > > > logic for > > > > > > doing > > > > > > > > comparison. > > > > > > > > > > > > > > > > Any suggestions? Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
