Hi All,
Thanks for responding! We are pretty excited about using Airflow with Elyra. Regarding a time to meet and chat this week, let's shoot for sometime between Wednesday and Friday. We can set something more concrete tomorrow once we figure out everyone's timezones and availability.
Look forward to hearing from all of you!
----- Original message -----
From: Jarek Potiuk <[email protected]>
To: [email protected]
Cc: [email protected], [email protected]
Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
Date: Mon, Nov 2, 2020 1:06 PM
Cool!. I also think it's an interesting one:). But it would be great to have such integration possible from Elyra :). Let us know what comes out of it :).J.On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <[email protected]> wrote:Hi Luciano -Elyra looks like an interesting project — we'd love to connect and talk through the opportunity.You can compare your cal to mine and grab a slot here: https://calendly.com/ryw/60min — and I'll be sure to get a few of the Airflow PMC members to join as well.-RyRy WalkerFounder/CTO of Astronomer + Airflow CommitterOn Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <[email protected]> wrote:Hi All,
As mentioned in the user list [1] we are working on a visual editor
for pipelines and adding Airflow as one of the supported backends.
https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
As you are the Airflow devs, we would invite you to help us implement
the best integration possible, in two steps:
1) Getting a solid integration for building and running pipelines with
python scripts and jupyter notebooks
2) Expand the available list component types and enable more generic operators
One of the questions raised in the original e-mail is related to how
to best submit the pipeline dag to be executed by the Airflow runtime,
we have tried a few different options, starting from the experimental
REST API, S3 bucket syncs and these seem to not be the ideal solution,
will be looking into git-sync next, but would really appreciate some
suggestions on the best options, particularly if someone has already
done some external integration similar to this.
Feel free to create issues for discussion and or more details
https://github.com/elyra-ai/elyra/issues
Or use this thread for suggestions
[1] https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/--
Jarek Potiuk
Polidea | Principal Software Engineer
