Re: Elyra 2.1 brings a visual editor for Airflow pipelines

On Thu, Mar 18, 2021 at 1:09 PM Jarek Potiuk wrote: > Hey Luciano , > > That's cool - I believe you would like to add a PR to > http://airflow.apache.org/ecosystem/ page to add it ? > > J, > > > Thanks for the suggestion, I have created a PR adding Elyra details to the "Ecosystem" page. https:/

Re: Elyra 2.1 brings a visual editor for Airflow pipelines

Hey Luciano , That's cool - I believe you would like to add a PR to http://airflow.apache.org/ecosystem/ page to add it ? J, On Thu, Mar 18, 2021 at 4:05 PM Luciano Resende wrote: > As we mentioned before [1] we have been working hard to add support > for running AI/ML pipelines on the Apac

Re: A Visual Editor for Airflow pipelines

Hey Max, I just saw https://github.com/elyra-ai/elyra . May worth check out for your use case. On Wed, Mar 17, 2021 at 10:50 AM Max Payton wrote: > We at Lyft are starting to think about developing pipelines using Jupyter, > and was wondering if there were any notes from this meeting, or follow

Elyra 2.1 brings a visual editor for Airflow pipelines

As we mentioned before [1] we have been working hard to add support for running AI/ML pipelines on the Apache Airflow runtime. With the Elyra 2.1 release, this is now available and you can take it for a spin and provide feedback on the project GitHub repo [2]. Learn more about Elyra and how to aut

Re: A Visual Editor for Airflow pipelines

We at Lyft are starting to think about developing pipelines using Jupyter, and was wondering if there were any notes from this meeting, or follow up discussions that might be relevant. We're interested in prior art and seeing what's already been done *Max Payton* He/Him/His Software Engineer 202.44

Re: A Visual Editor for Airflow pipelines

On Tue, Nov 3, 2020 at 12:05 PM Alan K Chin wrote: > @Jarek - It sounds like git-sync is or rather should be the default way users > add/modify DAGs. With that said, have you had any experience with customers > syncing their dags to other forms of dag storage (S3 etc.) and what the > outcomes w

RE: A Visual Editor for Airflow pipelines

Hello All,    I've scheduled a 60 minute call on Friday, Nov 6th @ 1PM PST to chat about this integration and have some Q and A.  The meeting link is : https://astronomer.zoom.us/j/5134172163 (thank you Ry for zoom link and usage)   --Alan ChinCODAIT, San FranciscoEmail - akc...@us.ibm.com - Or

Re: A Visual Editor for Airflow pipelines

sz Urbaszek >> To: dev@airflow.apache.org >> Cc: "luckbr1...@gmail.com" , "akc...@us.ibm.com" < >> akc...@us.ibm.com> >> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines >> Date: Tue, Nov 3, 2020 3:29 AM >> >> I think

Re: A Visual Editor for Airflow pipelines

flow.apache.org > Cc: "luckbr1...@gmail.com" , "akc...@us.ibm.com" < > akc...@us.ibm.com> > Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines > Date: Tue, Nov 3, 2020 3:29 AM > > I think the visual DAG editor would be a thing! > > Not sure i

RE: A Visual Editor for Airflow pipelines

@Tomek - Thanks for the link to the Airflow talk ! Checking it out now.    @Jarek - It sounds like git-sync is or rather should be the default way users add/modify DAGs. With that said, have you had any experience with customers syncing their dags to other forms of dag storage (S3 etc.) and what th

Re: A Visual Editor for Airflow pipelines

I think the visual DAG editor would be a thing! Not sure if you are aware of this Airflow Summit talk about visual DAG editor and the integration between Airflow and CWL: https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19 Cheers, Tomek On Tue, Nov 3, 202

Re: A Visual Editor for Airflow pipelines

Agree - maybe 2.1 or 2.2 :). After some experiences with big customers deployments, I personally think GitSync at least for now is the best approach out there. It requires git repo + authorization, but this has all the added benefits of code change tracking, it is a very standard interface, most o

Re: A Visual Editor for Airflow pipelines

Wishfull thinking at the moment Gerard -- the task execution still needs files on disk to run the tasks. This was always in my long term plan for DAG serialization, but we aren't there yet. And Custom operators makes this a non-straight forward problem to solve. -ash On Nov 3 2020, at 12:18 am,

Re: A Visual Editor for Airflow pipelines

x27;s timezones and availability. > > Look forward to hearing from all of you! > > -- > Alan Chin > CODAIT, San Francisco > Email - akc...@us.ibm.com > > > - Original message - > From: Jarek Potiuk > To: dev@airflow.apache.org > Cc: luckbr1...@gmail.com,

RE: A Visual Editor for Airflow pipelines

Hi All,    Thanks for responding! We are pretty excited about using Airflow with Elyra. Regarding a time to meet and chat this week, let's shoot for sometime between Wednesday and Friday. We can set something more concrete tomorrow once we figure out everyone's timezones and availability.     Look

Re: A Visual Editor for Airflow pipelines

Would be interested to also know possible ways to do what Luciano described. Hopefully w the serialized DAG and the new API we can start just pushing the DAG to the DB (wishful thinking)? Gerard Casas Saez Twitter | Cortex | @casassaez On Mon, Nov 2, 2020 at 2:06 P

Re: A Visual Editor for Airflow pipelines

Cool!. I also think it's an interesting one:). But it would be great to have such integration possible from Elyra :). Let us know what comes out of it :). J. On Mon, Nov 2, 2020 at 10:02 PM Ry Walker wrote: > Hi Luciano - > > Elyra looks like an interesting project — we'd love to connect and t

Re: A Visual Editor for Airflow pipelines

Hi Luciano - Elyra looks like an interesting project — we'd love to connect and talk through the opportunity. You can compare your cal to mine and grab a slot here: https://calendly.com/ryw/60min — and I'll be sure to get a few of the Airflow PMC members to join as well. -Ry Ry Walker Founder/C

A Visual Editor for Airflow pipelines

Hi All, As mentioned in the user list [1] we are working on a visual editor for pipelines and adding Airflow as one of the supported backends. https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html As you are the Airflow devs, we would invite you to help us implement the best integrati