+1 we could support a built in Kafka based notification mechanism.. Could we keep that in hudi-utilities instead of hudi-client?
On Thu, Jul 23, 2020 at 11:18 PM wangxianghu <[email protected]> wrote: > Hi Gray > Thanks for reply. It is the latter. We can use the callback to publish > write commit message to external system. > HTTP is the way currently implemented, Kafka is another implementation I’d > like to do next. > > Thanks, > Mathieu > > > 2020年7月24日 上午10:54，Gary Li <[email protected]> 写道： > > > > Hi xianghu, > > > > Thanks for implementing the callback service. That’s very helpful! > > > > I am curious what is the difference between the asynchronous mode you are > > referring to and the existing timeline service. Like, if I am > spontaneously > > looking for the new commit, I can go to the timeline to check if there is > > any. > > > > Or, do you mean something like use the callback to publish messages to > > external pub/sub services? So the user can consume from there? > > > > Thanks, > > Gary > > > > > > On Thu, Jul 23, 2020 at 7:23 PM vino yang <[email protected]> wrote: > > > >> +1 as well > >> > >> We can introduce an async mode to provide a pub-sub model so that we can > >> integrate with other systems. > >> > >> Best, > >> Vino > >> > >> [email protected] <[email protected]> 于2020年7月23日周四 下午9:05写道： > >> > >>> +1. > >>> it's great. > >>> > >>> > >>> > >>> [email protected] > >>> > >>> From: wangxianghu > >>> Date: 2020-07-23 21:02 > >>> To: [email protected] > >>> Subject: Implement an asynchronous write commit callback > >>> Hi all, > >>> Currently, a write callback service implemented in HTTP has been > merged, > >>> it will issue an HTTP request to inform downstream that a write commit > >> has > >>> been executed successfully！User can leverage this notification to do > some > >>> cascaded incremental processes. > >>> > >>> But, this implementation is synchronous, user must consume this > >>> notification immediately. This might not be what user wants. > >>> > >>> So, I'd like to implement an asynchronous one，implemented by Kafka for > >>> example. user can consume the callback message as long as they want, no > >>> need to act immediately. > >>> > >>> WDYT? any feedback are appreciated :) > >>> > >>> Best > >>> Mathieu > >>> > >>> > >> > >
