Agree - maybe 2.1 or 2.2 :). After some experiences with big customers deployments, I personally think GitSync at least for now is the best approach out there. It requires git repo + authorization, but this has all the added benefits of code change tracking, it is a very standard interface, most of the git repos provide some ways of manual review if needed and most have some kind of integration with CI/automated code analysis.
I personally think it should be the default, for any serious deployment as it provides so many benefits with very limited extra. You just need an extra "box" - git repo (which is pretty much a given in any organization). It uses a standard interface that is highly customizable (branches/folder structures, whatnot) and we already have git-sync container support in the helm chart. J. On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor <[email protected]> wrote: > Wishfull thinking at the moment Gerard -- the task execution still needs > files on disk to run the tasks. > > This was always in my long term plan for DAG serialization, but we aren't > there yet. And Custom operators makes this a non-straight forward problem > to solve. > > -ash > > On Nov 3 2020, at 12:18 am, Gerard Casas Saez > <[email protected]> wrote: > > Would be interested to also know possible ways to do what Luciano > described. Hopefully w the serialized DAG and the new API we can start just > pushing the DAG to the DB (wishful thinking)? > > Gerard Casas Saez > Twitter | Cortex | @casassaez <http://twitter.com/casassaez> > > > On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <[email protected]> > wrote: > > Cool!. I also think it's an interesting one:). But it would be great to > have such integration possible from Elyra :). Let us know what comes out of > it :). > > J. > > > On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <[email protected]> wrote: > > Hi Luciano - > > Elyra looks like an interesting project — we'd love to connect and talk > through the opportunity. > > You can compare your cal to mine and grab a slot here: > https://calendly.com/ryw/60min — and I'll be sure to get a few of the > Airflow PMC members to join as well. > > -Ry > > Ry Walker > Founder/CTO of Astronomer + Airflow Committer > > > On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <[email protected]> > wrote: > > Hi All, > > As mentioned in the user list [1] we are working on a visual editor > for pipelines and adding Airflow as one of the supported backends. > > https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html > > As you are the Airflow devs, we would invite you to help us implement > the best integration possible, in two steps: > > 1) Getting a solid integration for building and running pipelines with > python scripts and jupyter notebooks > > 2) Expand the available list component types and enable more generic > operators > > One of the questions raised in the original e-mail is related to how > to best submit the pipeline dag to be executed by the Airflow runtime, > we have tried a few different options, starting from the experimental > REST API, S3 bucket syncs and these seem to not be the ideal solution, > will be looking into git-sync next, but would really appreciate some > suggestions on the best options, particularly if someone has already > done some external integration similar to this. > > Feel free to create issues for discussion and or more details > > https://github.com/elyra-ai/elyra/issues > > Or use this thread for suggestions > > [1] > https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
