Re: Airflow 2.0

Totally agree on all your points Sid. My feeling is that at the moment the most critical thing for the project is to get a release out and get to a steady pace of high quality releases. Somehow breaking down the package seem to me like it would really help with the release process. Maybe an idea

Re: Airflow 2.0

David https://issues.apache.org/jira/browse/AIRFLOW-558 (i.e. http s://github.com/apache/incubator-airflow/pull/1830 ) Is on my plate.. have already gone through many rounds of reviews, testing, and fixes with the submitter and does not need to wait till 2.0. We should be able to merge it soon.

Re: Airflow installation error on ubuntu

Kapil, Please resubmit the email with links to the images. All embedded or attached objects are stripped from email before Apache mail servers forward them to recipients. -s On Fri, Nov 18, 2016 at 4:20 PM, Kapil Khandelwal wrote: > Hi, > > I am trying to run command pip

Airflow installation error on ubuntu

Hi, I am trying to run command pip install airflow[all] on Ubuntu box but getting below error [image: Inline image 1] Ubuntu Version [image: Inline image 2] Please let me know how to resolve the error. Thanks

Re: Airflow 2.0

> RIP out the charting application and the data profiler Yes please! +1 On Fri, Nov 18, 2016 at 2:41 PM, Maxime Beauchemin wrote: > Another point that may be controversial for Airflow 2.0: RIP out the > charting application and the data profiler. Even though it's

Re: Airflow 2.0

Another point that may be controversial for Airflow 2.0: RIP out the charting application and the data profiler. Even though it's nice to have it there, it's just out of scope and has major security issues/implications. I'm not sure how popular it actually is. We may need to run a survey at some

Re: Airflow 2.0

Using FAB's Model, we get pretty much all of that (REST API, auth/perms, CRUD) for free: http://flask-appbuilder.readthedocs.io/en/latest/quickhowto.html?highlight=rest#exposed-methods I'm pretty intimate with FAB since I use it (and contributed to it) for Superset/Caravel. All that's needed is

Re: Airflow 2.0

> It may be doable to run this as a different package `airflow-webserver`, an > alternate UI at first, and to eventually rip out the old UI off of the main > package. This is the same strategy that I was thinking of for AIRFLOW-85. You can build the new UI in parallel, and then delete the old one

Re: Airflow 2.0

A multi-tenant UI with composable roles on top of granular permissions. Migrating from Flask-Admin to Flask App Builder would be an easy-ish win (since they're both Flask). FAB Provides a good authentication and permission model that ships out-of-the-box with a REST api. Suffice to define FAB

Re: Airflow 2.0

(+1 on AIRFLOW-85.. but that's my ticket from a while ago :) Also, killing off the deprecated __init__.py magic in hooks/operators, and going with standard imports. On Fri, Nov 18, 2016 at 1:32 PM, Chris Riccomini wrote: > Full-fledged REST API (that the UI also uses)

Re: external task dependency across 2 different servers

You can simply use SqlSensor poking at the Airflow metadata database of your other cluster. https://airflow.apache.org/code.html?highlight=sqlsensor#airflow.operators.SqlSensor Max On Fri, Nov 18, 2016 at 7:26 AM, Michael Gong wrote: > Hey Rob, > > > Thanks for the valuable

Re: external task dependency across 2 different servers

Hey Rob, Thanks for the valuable information. Currently, our airflow configuration does use Celery executor and Redis backend. But the worker processes are all created locally. I assume you mean that Celery support distributed worker process to different servers/nodes. If that's the case,

Re: Airflow 2.0

Hi All, We have been using Airflow heavily for the last couple months and it’s been great so far. Here are a few things we’d like to see prioritized in 2.0. 1) Role based access to DAGs: We would like to see better role based access through the UI. There’s a related ticket out there but it

Re: external task dependency across 2 different servers

Looking ahead forward to a not too distant future: I’m currently finishing up the foundations for a Rest API. It would be trivial to add a call that returns the state of a task in this API. Then having a federated or even fully decoupled HA becomes a possibility. Obviously it is experimental at