Airflow would be good but you will probably have to modify it to support stream processing. Any DAG based manager would be useful in your case. Luigi works too, but airflow has a sleeker UI. You could also try streamsets. GCP provides composer which uses airflow and dataflow for beam. AWS has Glue although might not fully cover your case.
On Sat, 9 Feb 2019 at 00:45, Rui Wang <ruw...@google.com> wrote: > Apache Airflow is a scheduling system that can help manage data pipelines. > I have seen Airflow is used to manage a few thousand hive/spark/presto > pipelines. > > > -Rui > > On Fri, Feb 8, 2019 at 4:08 PM Sridevi Nookala < > snook...@parallelwireless.com> wrote: > >> Hi, >> >> >> Our analytics app has many data pipelines , some in python /java (using >> beam) etc, >> >> Any suggestions for a pipeline manager/scheduler framework that >> manages/orchestrates these different pipelines. >> >> >> thanks >> >> Sri >> > -- Thanks, Adeel Ahmad m: (+44) 7721724715 e: aahmad1...@gmail.com
