Re: How to add hooks for strong deployment consistency?

Thanks for the reply Joy, let me walk you though things as they are today 1) we don't stop airflow or disable DAGs while deploying updates to logic, this is done live once its released 2) the python script in the DAG folder doesn't actually have DAGs in it but is a shim layer to allow us to deploy

Re: Airflow at SREcon?

Also, don't forget to update the Readme (Who uses Airflow? : https://github.com/apache/incubator-airflow) with your companies once you get to a point where you have Airflow in prod :-) -s On Fri, Feb 23, 2018 at 12:50 PM, Chris Palmer wrote: > Not directly on topic to your email, but Fitbit has

Re: How to add hooks for strong deployment consistency?

Hi David, Do you mind providing a concrete example of the scenario in which scheduler/workers see different states (I'm not 100% sure if I understood the issue at hand). And by same dag generation, are you referring to the dag version? (DAG version is currently not supported at all, but I can see

Re: How to add hooks for strong deployment consistency?

My current thinking is to add a field to the dag table that is optional and provided by the dag. We currently intercept the load path do could use this field to make sure we load the same generation. My concern here is the interaction with the scheduler, not as familiar with that logic to predict

Re: Airflow at SREcon?

Not directly on topic to your email, but Fitbit has started using Airflow for some things. In particular the Data Engineering team, which I'm a member of, and is based in Boston, is starting to use it for much of or ETL processes. Chris On Fri, Feb 23, 2018 at 3:23 PM, James Meickle wrote: > Qu

Re: Airflow at SREcon?

You could also try organizing a meet-up in the Boston area for Airflow and see who signs up. That way, you'd get sense of the community size there. We have meet-ups in Tokyo, Amsterdam, NYC, and the SF Bay Area currently... Boston sounds like a reasonable next step. -s On Fri, Feb 23, 2018 at 12:

Airflow at SREcon?

Quantopian is aiming to switch over to Airflow over the course of this year, replacing our existing systems for ingesting financial data. I'll be at SREcon Americas this year, potentially with some of my coworkers; we're located in Boston, which doesn't seem to have many users yet, so we'd love to

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Congrats Ash! > On Feb 23, 2018, at 12:19 PM, Taylor Edmiston wrote: > > Congrats, Ash! Very exciting. > > *Taylor Edmiston* > TEdmiston.com | Blog > > Stack Overflow CV | LinkedIn >

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Congrats, Ash! Very exciting. *Taylor Edmiston* TEdmiston.com | Blog Stack Overflow CV | LinkedIn | AngelList On Fri, Feb 23, 2018

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Congrats and welcome! :D On Fri, Feb 23, 2018 at 8:27 AM, Sid Anand wrote: > Folks! > Please join the Airflow PPMC in welcoming Ash Berlin-Taylor to its ranks as > both committer and PPMC member! Congrats Ash! > > Announcement : > https://cwiki.apache.org/confluence/display/AIRFLOW/ > Announcem

Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Folks! Please join the Airflow PPMC in welcoming Ash Berlin-Taylor to its ranks as both committer and PPMC member! Congrats Ash! Announcement : https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Feb15,2018 Tweet : https://twitter.com/ApacheAirflow/status/96707208341812