Thanks both! @allen heard this many times :) hear you. You could write a small class yourself with your custom logic and throw it in there?
If you think there is a way to fix the key generator in Hudi to be more resilient to these (e.g taking in a list of supported patterns vs just the one), let us know. On Mon, May 4, 2020 at 3:08 PM Allen Underwood <[email protected]> wrote: > Hi Vinoth - that was extremely helpful...I almost had it working, HOWEVER, > it appears I have dates that some have the ms on the end and others > don't....so if I pick adding a time format with them, then the ones without > the fail and vice versa....Good times. > > After I figure this out I'll see if I can put this information somewhere > easy to find. > > On Mon, May 4, 2020 at 12:23 PM Vinoth Chandar <[email protected]> wrote: > >> Hi Allen, >> >> You are able to configure the key generator for deltastreamer using this >> property (either via a file or --config ) >> hoodie.datasource.write.keygenerator.class >> >> You might be interested in this built-in generator. >> >> https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/keygen/TimestampBasedKeyGenerator.java#L64 >> It takes let you configure a field as a recordKey, and if you can parse >> your timestamp using Java SimpleDateFormat, you can specify the datetime >> field and a pattern to parse it into.. >> >> Happy to make this work for you. >> >> community, any volunteers to faq/document this? :) >> >> >> On Mon, May 4, 2020 at 9:11 AM Allen Underwood >> <[email protected]> wrote: >> >> > I’ve tried to do my due diligence by googling / searching this slack and >> > I’ve come up empty.Is there a way through configuration / deltastreamer >> > to extract a custom partition key?Basically I have a datetime field in a >> > Kafka Source that has an ISO8601 datetime….is there a way to extract a >> > partition value out of that?I found this after some Googling, but this >> > seems like it’d only be useful if I wanted to write my own writer >> > application: >> > >> > >> https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.javaAny >> > way to do what I need through configuration of the spark job / hudi >> > configuration? >> > hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java >> > < >> https://github.com/apache/incubator-hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java >> > >> > >> > -- >> > *Allen Underwood* >> > >> > > > -- > *Allen Underwood* > Principal Software Engineer > Broadcom | Symantec Enterprise Division > *Mobile*: 404.808.5926 >
