Great. We also need it for use cases of loading data into warehouses, and would love to help.
On Wed, Jun 14, 2023 at 9:06 AM Pratyaksh Sharma <pratyaks...@gmail.com> wrote: > Hi, > > I missed this email earlier. Sure let me start an RFC this week and we can > take it from there. > > On Wed, Jun 14, 2023 at 9:20 PM Nicolas Paris <nicolas.pa...@riseup.net> > wrote: > > > Hi any rfc/ongoing efforts on the reverse delta streamer ? We have a use > > case to do hudi => Kafka and would enjoy building a more general tool. > > > > However we need a rfc basis to start some effort in the right way > > > > On April 12, 2023 3:08:22 AM UTC, Vinoth Chandar < > > mail.vinoth.chan...@gmail.com> wrote: > > >Cool. lets draw up a RFC for this? @pratyaksh - do you want to start > one, > > >given you expressed interest? > > > > > >On Mon, Apr 10, 2023 at 7:32 PM Léo Biscassi <leo.bisca...@gmail.com> > > wrote: > > > > > >> +1 > > >> This would be great! > > >> > > >> Cheers, > > >> > > >> On Mon, Apr 3, 2023 at 3:00 PM Pratyaksh Sharma < > pratyaks...@gmail.com> > > >> wrote: > > >> > > >> > Hi Vinoth, > > >> > > > >> > I am aligned with the first reason that you mentioned. Better to > have > > a > > >> > separate tool to take care of this. > > >> > > > >> > On Mon, Apr 3, 2023 at 9:01 PM Vinoth Chandar < > > >> > mail.vinoth.chan...@gmail.com> > > >> > wrote: > > >> > > > >> > > +1 > > >> > > > > >> > > I was thinking that we add a new utility and NOT extend > > DeltaStreamer > > >> by > > >> > > adding a Sink interface, for the following reasons > > >> > > > > >> > > - It will make it look like a generic Source => Sink ETL tool, > > which is > > >> > > actually not our intention to support on Hudi. There are plenty of > > good > > >> > > tools for that out there. > > >> > > - the config management can get bit hard to understand, since we > > >> overload > > >> > > ingest and reverse ETL into a single tool. So break it off at > > use-case > > >> > > level? > > >> > > > > >> > > Thoughts? > > >> > > > > >> > > David: PMC does not have control over that. Please see > unsubscribe > > >> > > instructions here. https://hudi.apache.org/community/get-involved > > >> > > Love to keep this thread about reverse streamer discussion. So > > kindly > > >> > fork > > >> > > another thread if you want to discuss unsubscribing. > > >> > > > > >> > > On Fri, Mar 31, 2023 at 1:47 AM Davidiam <david.rosa...@gmail.com > > > > >> > wrote: > > >> > > > > >> > > > Hello Vinoth, > > >> > > > > > >> > > > Can you please unsubscribe me? I have been trying to > unsubscribe > > for > > >> > > > months without success. > > >> > > > > > >> > > > Kind Regards, > > >> > > > David > > >> > > > > > >> > > > Sent from Outlook for Android<https://aka.ms/AAb9ysg> > > >> > > > ________________________________ > > >> > > > From: Vinoth Chandar <vin...@apache.org> > > >> > > > Sent: Friday, March 31, 2023 5:09:52 AM > > >> > > > To: dev <dev@hudi.apache.org> > > >> > > > Subject: [DISCUSS] Hudi Reverse Streamer > > >> > > > > > >> > > > Hi all, > > >> > > > > > >> > > > Any interest in building a reverse streaming tool, that does the > > >> > reverse > > >> > > of > > >> > > > what the DeltaStreamer tool does? It will read Hudi table > > >> incrementally > > >> > > > (only source) and write out the data to a variety of sinks - > > Kafka, > > >> > JDBC > > >> > > > Databases, DFS. > > >> > > > > > >> > > > This has come up many times with data warehouse users. Often > > times, > > >> > they > > >> > > > want to use Hudi to speed up or reduce costs on their data > > ingestion > > >> > and > > >> > > > ETL (using Spark/Flink), but want to move the derived data back > > into > > >> a > > >> > > data > > >> > > > warehouse or an operational database for serving. > > >> > > > > > >> > > > What do you all think? > > >> > > > > > >> > > > Thanks > > >> > > > Vinoth > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> -- > > >> *Léo Biscassi* > > >> Blog - https://leobiscassi.com > > >> > > >> - > > >> > > > -- Take Care, Rajesh Mahindra
