Would be interested to also know possible ways to do what Luciano described. Hopefully w the serialized DAG and the new API we can start just pushing the DAG to the DB (wishful thinking)?
Gerard Casas Saez Twitter | Cortex | @casassaez <http://twitter.com/casassaez> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <[email protected]> wrote: > Cool!. I also think it's an interesting one:). But it would be great to > have such integration possible from Elyra :). Let us know what comes out of > it :). > > J. > > > On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <[email protected]> wrote: > >> Hi Luciano - >> >> Elyra looks like an interesting project — we'd love to connect and talk >> through the opportunity. >> >> You can compare your cal to mine and grab a slot here: >> https://calendly.com/ryw/60min — and I'll be sure to get a few of the >> Airflow PMC members to join as well. >> >> -Ry >> >> Ry Walker >> Founder/CTO of Astronomer + Airflow Committer >> >> >> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <[email protected]> >> wrote: >> >>> Hi All, >>> >>> As mentioned in the user list [1] we are working on a visual editor >>> for pipelines and adding Airflow as one of the supported backends. >>> >>> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html >>> >>> As you are the Airflow devs, we would invite you to help us implement >>> the best integration possible, in two steps: >>> >>> 1) Getting a solid integration for building and running pipelines with >>> python scripts and jupyter notebooks >>> >>> 2) Expand the available list component types and enable more generic >>> operators >>> >>> One of the questions raised in the original e-mail is related to how >>> to best submit the pipeline dag to be executed by the Airflow runtime, >>> we have tried a few different options, starting from the experimental >>> REST API, S3 bucket syncs and these seem to not be the ideal solution, >>> will be looking into git-sync next, but would really appreciate some >>> suggestions on the best options, particularly if someone has already >>> done some external integration similar to this. >>> >>> Feel free to create issues for discussion and or more details >>> >>> https://github.com/elyra-ai/elyra/issues >>> >>> Or use this thread for suggestions >>> >>> [1] >>> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E >>> >>> >>> -- >>> Luciano Resende >>> http://twitter.com/lresende1975 >>> http://lresende.blogspot.com/ >>> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > >
