latest engineering blog post on Airflow from Lyft

Hi All, We just published an engineering blog post( https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff) on how we run and monitor airflow at Lyft. Please take a look when you have time. And feel free to ping us if you want to know more of the details. Thanks, -Tao

Re: Requesting Wiki edit permission to create AIPs

> > > > > > On Thu, Dec 20, 2018 at 11:24 AM Sid Anand wrote: > > > >> one sec. let me grant access to all those that need it.. stay tuned. > >> -s > >> > >> On Thu, Dec 20, 2018 at 8:41 AM Kaxil Naik wrote: > >> > >>>

Re: Requesting Wiki edit permission to create AIPs

I am not an Airflow wiki admin. Could any of the admin for wiki help this request? On Wed, Dec 19, 2018 at 5:58 PM Tim Swast wrote: > Wiki ID: tswast (https://cwiki.apache.org/confluence/display/~tswast) > > Per > >

Re: Configuring/sharing Airflow github repo security alerts

Thanks Feng for the suggestion. Just file https://issues.apache.org/jira/browse/INFRA-17470. On Tue, Dec 18, 2018 at 6:25 PM Feng Lu wrote: > Cool, thank you Ash. Kindly let us know when you have opened the INFRA jira > ticket. > > On Tue, Dec 18, 2018 at 2:21 AM Ash Berlin-Taylor > wrote: > >

Re: UI not working on Master

I assume so. How do you run your test? I could try it on my side and see if I could reproduce the problem. On Tue, Dec 18, 2018 at 2:17 PM Daniel Imberman wrote: > Do we need to install NPM? Should I add npm to the docker image/bootstrap > script? > > On Tue, Dec 18, 2018 at 1:20

Re: UI not working on Master

ll see certain admin panels, but the primary DAG one doesn't show > any DAGs even when I explicitly put one in the folder. > > On Tue, Dec 18, 2018, 12:26 PM Tao Feng > > Are you using the Old UI or the new UI? Do you login as Admin user for > new > > UI? > > > &g

Re: UI not working on Master

Are you using the Old UI or the new UI? Do you login as Admin user for new UI? On Tue, Dec 18, 2018 at 11:56 AM Daniel Imberman wrote: > Hello all, > > I've been testing the K8sExecutor and I notice that when I open the UI, I'm > unable to see any of the DAGs even though they are in the

Re: Refactor models.py

We should merge the existing prs before doing this refactors. Otherwise, there will be so many rebase issues. On Mon, Dec 17, 2018 at 12:37 AM Andreas Költringer < andreas.koeltrin...@n-fuse.co> wrote: > Hi, > > a suggestion to make this process easier: > > a folder could be created called

Re: [VOTE] Graduate the Apache Airflow as a TLP

uziel > * Alex Van Boxel > * Arthur Wiedmer > * Ash Berlin-Taylor > * Bolke de Bruin > * Chris Riccomini > * Dan Davydov > * Fokko Driesprong > * Hitesh Shah > * Jakob Homan > * Jeremiah Lowin > * Joy Gao > * Kaxil Naik > * Maxime Beauchemin >

Re: [DISCUSS] Apache Airflow graduation from the incubator

Sorry, just saw Kaxil's latest email. Kaxil, is there anything else I could help with? Thanks, -Tao On Wed, Nov 28, 2018 at 2:40 PM Tao Feng wrote: > I would like to help on the documentation. Let me take a look at it. I > will work Kaxil on that. > > On Tue, Nov 27, 2018 at 12:39

Re: [DISCUSS] Apache Airflow graduation from the incubator

I would like to help on the documentation. Let me take a look at it. I will work Kaxil on that. On Tue, Nov 27, 2018 at 12:39 PM Bolke de Bruin wrote: > Hi Folks, > > Thanks all for your responses and particularly Stefan for his suggestion > to use the generic Apache way to handle security

Re: grant edit permission for airflow wiki

Thanks Max. On Mon, Nov 26, 2018 at 10:32 PM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Granted! > > Max > > On Mon, Nov 26, 2018 at 10:21 PM Tao Feng wrote: > > > Could any Airflow wiki admin grant me the edit permisssion? My wiki user > > name is tfeng. > > > > Thanks, > > -Tao > > >

grant edit permission for airflow wiki

Could any Airflow wiki admin grant me the edit permisssion? My wiki user name is tfeng. Thanks, -Tao

Re: [DISCUSS] Apache Airflow graduation from the incubator

And happy to help out any items for the TLP push. On Mon, Nov 26, 2018 at 9:53 PM Tao Feng wrote: > +1 as well. (do we need to start another vote thread per Jakob's > suggestion?) > > Great to see the thriving community! > > On Sat, Nov 24, 2018 at 3:57 AM Bolke de Bruin

Re: [DISCUSS] Apache Airflow graduation from the incubator

+1 as well. (do we need to start another vote thread per Jakob's suggestion?) Great to see the thriving community! On Sat, Nov 24, 2018 at 3:57 AM Bolke de Bruin wrote: > Hi All, > > With the Apache Airflow community healthy and growing, I think now would > be a good time to > discuss where we

Re: Airflow 1.10.1 is released

Thanks Ash for running the release! On Wed, Nov 21, 2018 at 2:20 PM Ash Berlin-Taylor wrote: > Dear Airflow community, > > I'm happy to announce that Airflow 1.10.1 was just released. > > The source release as well as the binary "sdist" release are available > here: > > >

Re: [RESULT][VOTE] Release Airflow 1.10.1 (incubating)

Thanks Ash for running the release! On Wed, Nov 21, 2018 at 2:17 PM Ash Berlin-Taylor wrote: > The vote to release Airflow 1.10.1-incubating, having been open for 3 > days is now closed. > > There were three binding +1s and no -1 votes. > > +1 (binding): > Hitesh Shah > Jakob Homan > Justin

Re: Airflow variables and data profiling hidden

Data Profiling is depreciated for new rbac UI for security reason. You could check out what has been changed in ( https://github.com/apache/incubator-airflow/blob/master/UPDATING.md). On Tue, Aug 28, 2018 at 11:15 PM, Shubham Gupta wrote: > Hi, > > Airflow variables and data profiling tabs are

Re: Python 3.6 Support for Airflow 1.10.0

+1 for keeping 2.7 as well. Lyft runs airflow on 2.7 internally. On Tue, Aug 28, 2018 at 1:12 PM, Feng Lu wrote: > +1 for keeping 2.7 as long as we can so people have time to plan and > migrate away from it. > > On Tue, Aug 28, 2018, 10:35 Arthur Wiedmer > wrote: > > > Given that Python 2.7

Re: ExternalTaskSensor alternatives

FYI, there is an existing pr and proposal for improving sensor efficiency( https://issues.apache.org/jira/browse/AIRFLOW-2747 and https://github.com/apache/incubator-airflow/pull/3596/files) by the community. For your idea, I am not sure if it is a good idea to add this dag dependent

Re: Replacement ShortCircuitOperator

Hey Kevin, It looks like a good one for an AIF(https://cwiki.apache.org/ confluence/display/AIRFLOW/Airflow+Improvements+Proposals) proposal. Thanks, -Tao On Fri, Aug 10, 2018 at 8:54 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hey, > > I agree, I always thought this would be

Re: Airflow Multi-Tenancy to filter dags by team not users

Airflow 1.9.0 doesn't ship with the RBAC( Flask-appbuilder) UI. Not sure if you could implement what you mention on top of 1.9. We recently implement Airflow DAG level access in master branch which allows to filter dag view based on role. So in this case, you will just need to create a role for

Re: Plan to change type of dag_id from String to Number?

+1 on Bolke. I don't think we have such plan. And I believe dag id has been indexed already in many tables. On Wed, Aug 8, 2018 at 11:22 PM, Bolke de Bruin wrote: > No we don’t have such plan. Dag ids are used to have a readable > identifier. If you think it makes such a big difference in speed

Re: SubdagOperator and Pools

Hi Andreas, The default executor for SubdagOperator is SequentialExecutor which makes sure all the tasks within subdag are executed in sequential order. But if you have too many subdags within single DAG and want to control with pooling(https://airflow.apache.org/concepts.html#pools),

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

Thanks everyone! On Fri, Aug 3, 2018 at 3:02 PM, Grant Nicholas < grantnicholas2...@u.northwestern.edu> wrote: > Congrats Feng! > > On Fri, Aug 3, 2018 at 12:35 PM Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > Well deserved, welcome aboard! > > > > On Fri, Aug 3, 2018 at 9:07 AM

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

wrote: > +1 > > On Tue, Jul 17, 2018 at 3:49 PM, Jin Chang wrote: > > > +1 > > > > On Tue, Jul 17, 2018 at 3:26 PM, Tao Feng wrote: > > > > > +1, should we put them in airflow wiki space(like kafka(kip)) or google > > doc > > > or

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

+1, should we put them in airflow wiki space(like kafka(kip)) or google doc or email? On Sun, Jul 15, 2018 at 11:24 AM, Arthur Wiedmer wrote: > +1 > > On Sun, Jul 15, 2018, 20:12 Maxime Beauchemin > wrote: > > > +1 > > > > On Tue, Jul 10, 2018 at 1:09 PM Sid Anand wrote: > > > > > +1 > > > >

Re: DAG Level permissions (was Re: RBAC Update)

. Thanks a lot, -Tao On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng wrote: > Hi Max, Joy and Everyone, > > Based on the discussion, I put up a work in progress pr ( > https://github.com/apache/incubator-airflow/pull/3197/) with a doc( > https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0g

Re: Is `airflow backfill` disfunctional?

t; bdbr...@gmail.com> > > > I > > > >> think you may have some context on why this may have changed at some > > > point. > > > >> I'm assuming that when DagRun handling was added to the backfill > > logic, > > > the > > &

Re: Is `airflow backfill` disfunctional?

After discussing with Max, we think it would be great if `airflow backfill` could be able to auto pick up and rerun those failed tasks. Currently, it will throw exceptions( https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L2489) without rerunning the failed tasks. But since

Re: kwargs usage in BaseOperator

Ping. Anyone has any idea on the question(whether airflow will remove kwargs in BaseOperator)? On Mon, May 21, 2018 at 11:26 AM, Tao Feng <fengta...@gmail.com> wrote: > Hi, > > I have a question regarding kwargs usage in BaseOperator(https://github. > com/apache/incubator-a

kwargs usage in BaseOperator

Hi, I have a question regarding kwargs usage in BaseOperator( https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2306-L2315). Since this pr(https://github.com/apache/incubator-airflow/pull/1285) checked in, airflow turns on deprecation warning by default. And per the

Re: Airflow tree view display issue

I would suggest you create a jira ticket and attach the image onto the ticket. I think your image is filtered by apache email and can't be viewed by others. On Tue, May 8, 2018 at 5:53 PM, Wang, Larry wrote: > Are there anybody get chance to check this thread? > > > >

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

Welcome! On Tue, May 8, 2018 at 1:47 PM, Driesprong, Fokko wrote: > Hi Airflow'ers, > > Please join the Apache Airflow PMC in welcoming its newest member and > co-committer, Naik Kaxil (a.k.a. kaxil ). > Welcome > Kaxil, great to have you on

Re: About the project support in Airflow

Hi Song, Just noted that we are also working on dag-level access on top of RBAC(AIRFLOW-2267) which should provide dag-level acl functionality. The WIP pr could be found at https://github.com/apache/incubator-airflow/pull/3197 On Wed, Apr 25, 2018 at 7:42 PM, 刘松(Cycle++开发组)

Re: Give it up for Fokko!

Thanks Fokko. Really appreciate your great feedback and your hard work on helping airflow project thrive:) On Fri, Apr 13, 2018 at 9:55 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hey all, > > I wanted to point out the amazing work that Fokko is doing, > reviewing/merging PRs

Re: Slides / Presentations for Airflow

Maybe upload the slides in slideshare and update those in https://github.com/jghoman/awesome-apache-airflow ? On Thu, Apr 12, 2018 at 2:09 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > I have some slides but they're very Airbnb-styled, I need to make a new > deck... > > Max > > On

Re: DAG Level permissions (was Re: RBAC Update)

feedbacks and see if I miss anything or am in the wrong direction as it touches the core part. In the meantime, I will still continue improving the pr for couples of todos. Looking forward to your feedback. Thanks, -Tao On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fengta...@gmail.com> wrote:

Re: DAG Level permissions (was Re: RBAC Update)

ld help shave a little bit > from > >>> that table. > >>> > >>> Best, > >>> Arthur > >>> > >>> > >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin < > >>> maximebeauche...@gmail.com> wrote: > >>> &

Re: RBAC Update

Great work Joy. This is awesome! I am interested in helping out the per dag level access. Just created a ticket to check(AIRFLOW-2267). Let me know if you have any suggestions. I will share my proposal once I am ready. On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao wrote: > Hey guys!

Re: Awesome list of resources around Apache Airflow

Thanks Max for sharing the link. This is great :) -Tao On Wed, Mar 28, 2018 at 9:21 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > I was pleasantly surprised to stumble upon this recently: > https://github.com/jghoman/awesome-apache-airflow > > Please contribute anything that you

Re: Custom DagBag collection based on dags_folder prefix

+1. I think having a design doc is good. And it would be great if you could create couples of small jiras for the related task. I am interested in helping out if possible. Thanks, -Tao On Fri, Mar 16, 2018 at 3:52 AM, Diogo Franco wrote: > Created this JIRA

Re: Very looong py files

+1 on splitting the long file. On Wed, Mar 14, 2018 at 5:50 PM, Marc Bollinger wrote: > +1 > > This was one of the first things that struck us when integrating Airflow, > but as we aren't (weren't?) pythonist[a]s at the time, didn't necessarily > feel comfortable chiming in