Re: Where to in codebase to add external IP to Airflow emails? TaskInstance? Some global place?

Hi Russell, The least intrusive method is to write your own "on_failure_callback" mechanism, not use the failure emails and then replicate the email handling code there with the ipgetter code. I'd then turn this into some common utility code and look into using "default_args" to have this set as

Re: Role Based Access Control for Airflow UI

Hey everyone, I recently transferred to Data Infra team here at WePay to focus on Airflow-related initiatives. Given the RBAC design is mostly hashed out, I'm happy to get this feature off the ground for Q3, starting with converting Airflow to Fab, if there are no objections. Cheers, Joy On Thu

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

FYI, can anyone pictorially describe the release process (and post it on the apache airflow wiki)? I think that would eliminate a lot of confusion in the future and avoid a rehash of this email thread on the next release. -s On Wed, Jul 19, 2017 at 10:48 AM, Hitesh Shah wrote: > To add, the mai

Re: Where to in codebase to add external IP to Airflow emails? TaskInstance? Some global place?

bump? Russell Jurney @rjurney russell.jur...@gmail.com LI FB datasyndrome.com On Tue, Jul 18, 2017 at 4:36 PM, Russell Jurney wrote: > I created a JIRA to track this issue: https://issues.apache. >

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

To add, the main source tarball should have instructions to generate the sdist and bdist versions. Additionally, as part of the release process if the plan is to publish to pypi (after the IPMC vote succeeds), then the appropriate bits also need to be verified/voted upon. There are not exactly coun

Re: How to add a module as in to requirements.txt to Airflow source?

Thanks! Russell Jurney @rjurney russell.jur...@gmail.com LI FB datasyndrome.com On Tue, Jul 18, 2017 at 8:29 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > `setup.py` > https://pythonho

[AIRFLOW-1428] Skipped tasks deadlock dag runs

I've run into issues in dag runs where skipped tasks cause the dag to deadlock. I think the deeper issue here is an airflow community issue: there has been an unresolved debate around which trigger rule should propagate skipped tasks. If you're interested in the problem, check out AIRFLOW-1428

Re: Airflow webserver forgets dags on refresh

Follow-up: I found that this issue happens when the DAG file uses a common utility method to create the Dag object, by passing some parameters to it. By looking into the airflow DB, the fileloc column contains the path to the utility file (e.g. $DAG_FOLDER/utils/dagfactory.py) instead of the path o

[AIRFLOW-1428] Skipped tasks deadlock dag runs

I've run into issues in dag runs where skipped tasks cause the dag to deadlock. I think the deeper issue here is an airflow community issue: there has been an unresolved debate around which trigger rule should propagate skipped tasks. If you're interested in the problem, check out AIRFLOW-1428