Great! On Mon, May 4, 2020 at 5:43 PM Allen Underwood <allen.underw...@broadcom.com.invalid> wrote:
> Hi Vinoth, > > Yes I was going to set some things up in the morning. I’ll let you know > how it turns out and if it’s worth a PR I’ll get one together. > > Thanks again for your help! > > Allen > > On Mon, May 4, 2020 at 8:40 PM Vinoth Chandar <vin...@apache.org> wrote: > >> Thanks both! >> >> @allen heard this many times :) hear you. You could write a small class >> yourself with your custom logic and throw it in there? >> >> If you think there is a way to fix the key generator in Hudi to be more >> resilient to these (e.g taking in a list of supported patterns vs just the >> one), let us know. >> >> On Mon, May 4, 2020 at 3:08 PM Allen Underwood >> <allen.underw...@broadcom.com.invalid> wrote: >> >> > Hi Vinoth - that was extremely helpful...I almost had it working, >> HOWEVER, >> > it appears I have dates that some have the ms on the end and others >> > don't....so if I pick adding a time format with them, then the ones >> without >> > the fail and vice versa....Good times. >> > >> > After I figure this out I'll see if I can put this information somewhere >> > easy to find. >> > >> > On Mon, May 4, 2020 at 12:23 PM Vinoth Chandar <vin...@apache.org> >> wrote: >> > >> >> Hi Allen, >> >> >> >> You are able to configure the key generator for deltastreamer using >> this >> >> property (either via a file or --config ) >> >> hoodie.datasource.write.keygenerator.class >> >> >> >> You might be interested in this built-in generator. >> >> >> >> >> https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/keygen/TimestampBasedKeyGenerator.java#L64 >> >> It takes let you configure a field as a recordKey, and if you can parse >> >> your timestamp using Java SimpleDateFormat, you can specify the >> datetime >> >> field and a pattern to parse it into.. >> >> >> >> Happy to make this work for you. >> >> >> >> community, any volunteers to faq/document this? :) >> >> >> >> >> >> On Mon, May 4, 2020 at 9:11 AM Allen Underwood >> >> <allen.underw...@broadcom.com.invalid> wrote: >> >> >> >> > I’ve tried to do my due diligence by googling / searching this slack >> and >> >> > I’ve come up empty.Is there a way through configuration / >> deltastreamer >> >> > to extract a custom partition key?Basically I have a datetime field >> in a >> >> > Kafka Source that has an ISO8601 datetime….is there a way to extract >> a >> >> > partition value out of that?I found this after some Googling, but >> this >> >> > seems like it’d only be useful if I wanted to write my own writer >> >> > application: >> >> > >> >> > >> >> >> https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.javaAny >> >> > way to do what I need through configuration of the spark job / hudi >> >> > configuration? >> >> > >> hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java >> >> > < >> >> >> https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java >> >> > >> >> > >> >> > -- >> >> > *Allen Underwood* >> >> > >> >> >> > >> > >> > -- >> > *Allen Underwood* >> > Principal Software Engineer >> > Broadcom | Symantec Enterprise Division >> > *Mobile*: 404.808.5926 >> > >> > -- > *Allen Underwood* > Principal Software Engineer > Broadcom | Symantec Enterprise Division > *Mobile*: 404.808.5926 >
