Thanks for ths +1 Filed : https://issues.apache.org/jira/projects/HUDI/issues/HUDI-1122 <https://issues.apache.org/jira/projects/HUDI/issues/HUDI-1122>
> 2020年7月29日 上午8:10，Vinoth Chandar <[email protected]> 写道： > > +1 we could support a built in Kafka based notification mechanism.. > Could we keep that in hudi-utilities instead of hudi-client? > > On Thu, Jul 23, 2020 at 11:18 PM wangxianghu <[email protected]> wrote: > >> Hi Gray >> Thanks for reply. It is the latter. We can use the callback to publish >> write commit message to external system. >> HTTP is the way currently implemented, Kafka is another implementation I’d >> like to do next. >> >> Thanks, >> Mathieu >> >>> 2020年7月24日 上午10:54，Gary Li <[email protected]> 写道： >>> >>> Hi xianghu, >>> >>> Thanks for implementing the callback service. That’s very helpful! >>> >>> I am curious what is the difference between the asynchronous mode you are >>> referring to and the existing timeline service. Like, if I am >> spontaneously >>> looking for the new commit, I can go to the timeline to check if there is >>> any. >>> >>> Or, do you mean something like use the callback to publish messages to >>> external pub/sub services? So the user can consume from there? >>> >>> Thanks, >>> Gary >>> >>> >>> On Thu, Jul 23, 2020 at 7:23 PM vino yang <[email protected]> wrote: >>> >>>> +1 as well >>>> >>>> We can introduce an async mode to provide a pub-sub model so that we can >>>> integrate with other systems. >>>> >>>> Best, >>>> Vino >>>> >>>> [email protected] <[email protected]> 于2020年7月23日周四 下午9:05写道： >>>> >>>>> +1. >>>>> it's great. >>>>> >>>>> >>>>> >>>>> [email protected] >>>>> >>>>> From: wangxianghu >>>>> Date: 2020-07-23 21:02 >>>>> To: [email protected] >>>>> Subject: Implement an asynchronous write commit callback >>>>> Hi all, >>>>> Currently, a write callback service implemented in HTTP has been >> merged, >>>>> it will issue an HTTP request to inform downstream that a write commit >>>> has >>>>> been executed successfully！User can leverage this notification to do >> some >>>>> cascaded incremental processes. >>>>> >>>>> But, this implementation is synchronous, user must consume this >>>>> notification immediately. This might not be what user wants. >>>>> >>>>> So, I'd like to implement an asynchronous one，implemented by Kafka for >>>>> example. user can consume the callback message as long as they want, no >>>>> need to act immediately. >>>>> >>>>> WDYT? any feedback are appreciated :) >>>>> >>>>> Best >>>>> Mathieu >>>>> >>>>> >>>> >> >>
