Wishfull thinking at the moment Gerard -- the task execution still needs files on disk to run the tasks.
This was always in my long term plan for DAG serialization, but we aren't there yet. And Custom operators makes this a non-straight forward problem to solve. -ash On Nov 3 2020, at 12:18 am, Gerard Casas Saez <[email protected]> wrote: > Would be interested to also know possible ways to do what Luciano described. > Hopefully w the serialized DAG and the new API we can start just pushing the > DAG to the DB (wishful thinking)? > > Gerard Casas Saez > Twitter | Cortex | @casassaez (http://twitter.com/casassaez) > > > > > > On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <[email protected] > (mailto:[email protected])> wrote: > > Cool!. I also think it's an interesting one:). But it would be great to > > have such integration possible from Elyra :). Let us know what comes out of > > it :). > > > > J. > > > > > > On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <[email protected] > > (mailto:[email protected])> wrote: > > > Hi Luciano - > > > > > > Elyra looks like an interesting project — we'd love to connect and talk > > > through the opportunity. > > > > > > You can compare your cal to mine and grab a slot here: > > > https://calendly.com/ryw/60min — and I'll be sure to get a few of the > > > Airflow PMC members to join as well. > > > > > > -Ry > > > > > > Ry Walker > > > Founder/CTO of Astronomer + Airflow Committer > > > > > > > > > On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <[email protected] > > > (mailto:[email protected])> wrote: > > > > Hi All, > > > > > > > > As mentioned in the user list [1] we are working on a visual editor > > > > for pipelines and adding Airflow as one of the supported backends. > > > > > > > > https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html > > > > As you are the Airflow devs, we would invite you to help us implement > > > > the best integration possible, in two steps: > > > > > > > > 1) Getting a solid integration for building and running pipelines with > > > > python scripts and jupyter notebooks > > > > > > > > 2) Expand the available list component types and enable more generic > > > > operators > > > > One of the questions raised in the original e-mail is related to how > > > > to best submit the pipeline dag to be executed by the Airflow runtime, > > > > we have tried a few different options, starting from the experimental > > > > REST API, S3 bucket syncs and these seem to not be the ideal solution, > > > > will be looking into git-sync next, but would really appreciate some > > > > suggestions on the best options, particularly if someone has already > > > > done some external integration similar to this. > > > > > > > > Feel free to create issues for discussion and or more details > > > > https://github.com/elyra-ai/elyra/issues > > > > Or use this thread for suggestions > > > > [1] > > > > https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E > > > > > > > > -- > > > > Luciano Resende > > > > http://twitter.com/lresende1975 > > > > http://lresende.blogspot.com/ > > > > > > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > Polidea (https://www.polidea.com/) | Principal Software Engineer > > > > > > > > > > > > > > > > M: +48 660 796 129 (tel:+48660796129) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
