Re: Are there pipeline API's for ETL?
Hi Vino, Another use case would be I want to build a dag of batch sources, sinks and transforms and I want to schedule the jobs periodically. One can say similar to airflow but a Flink api would be lot better! Sent from my iPhone > On Jan 10, 2020, at 6:42 PM, vino yang wrote: > > > Hi kant, > > Can you provide more context about your question? What do you mean about > "pipeline API"? > > IMO, you can build an ETL pipeline via composing several Flink transform > APIs. About choosing which transform APIs, it depends on your business logic. > > Here are the generic APIs list.[1] > > Best, > Vino > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html > > kant kodali 于2020年1月11日周六 上午9:06写道： >> Hi All, >> >> I am wondering if there are pipeline API's for ETL? >> >> Thanks! >> >>
Re: Are there pipeline API's for ETL?
Hi Vino, I am new to Flink. I was thinking more like a dag builder api where I can build a dag of source,sink and transforms and hopefully fink take cares of the entire life cycle of the dag. An example would be CDAP pipeline api. Sent from my iPhone > On Jan 10, 2020, at 6:42 PM, vino yang wrote: > > > Hi kant, > > Can you provide more context about your question? What do you mean about > "pipeline API"? > > IMO, you can build an ETL pipeline via composing several Flink transform > APIs. About choosing which transform APIs, it depends on your business logic. > > Here are the generic APIs list.[1] > > Best, > Vino > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html > > kant kodali 于2020年1月11日周六 上午9:06写道： >> Hi All, >> >> I am wondering if there are pipeline API's for ETL? >> >> Thanks! >> >>
Re: Are there pipeline API's for ETL?
Hi kant, Can you provide more context about your question? What do you mean about "pipeline API"? IMO, you can build an ETL pipeline via composing several Flink transform APIs. About choosing which transform APIs, it depends on your business logic. Here are the generic APIs list.[1] Best, Vino [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html kant kodali 于2020年1月11日周六 上午9:06写道： > Hi All, > > I am wondering if there are pipeline API's for ETL? > > Thanks! > > >
