Re: Pinning dependencies for Apache Airflow

2018-10-09 Thread Matt Davis
flow is not installed? > > > Kind regards, > > Erik > > ________ > From: Matt Davis > Sent: Monday, October 8, 2018 10:13:34 PM > To: dev@airflow.incubator.apache.org > Subject: Re: Pinning dependencies for Apache Airflow > > It soun

Re: Pinning dependencies for Apache Airflow

2018-10-08 Thread Matt Davis
It sounds like we can get the best of both worlds with the original proposals to have minimal requirements in setup.py and "guaranteed to work" complete requirements in a separate file. That way we have flexibility for teams that run airflow and tasks in the same environment and guidance on a worki

Re: Airflow 1.10 Migration Duration

2018-09-25 Thread Matt Davis
ps://stackoverflow.com/story/taylor> > > > > > > On Tue, Sep 25, 2018 at 8:30 PM, Sid Anand wrote: > > > > > I checked with our Ops guy and he mentioned that when he upgraded from > > > 1.8.x to 1.9.x, it took a few seconds. We had 3M rows in the > >

Airflow 1.10 Migration Duration

2018-09-25 Thread Matt Davis
e table. That got us wondering what kind of maintenance people do on their Airflow metadata databases. Do folks mostly put up with long migrations and generally longer queries, or are y'all doing periodic cleanups of your metadata DB to keep it fairly light? Thanks, Matt Davis

Re: Airflow profiling

2017-06-28 Thread Matt Davis
It's not useful for online profiling, but to get a profile of specific things using the builtin cProfile module I've leveraged the airflow test CLI command, there's an example here: https://gist.github.com/jiffyclub/a3204998c8190abd8ea50090c30347fa The context here was profiling why a specific DAG