Re: Discuss: should we allow HTML emails on dev@ list

> > about top posting :D > > >> > > > >> > > > > > >> > > > >> > > > -ash > > >> > > > >> > > > > > >> > > > >> > > > On Sep 14 2020, at 12:42 pm, Ash Berlin-Taylor > > >> > wrote: > > >> > > > >> > > > > > >> > > > >> > > > > Hi all, > > >> > > > >> > > > > > > >> > > > >> > > > > As I'm sure you are aware, right now the dev@ list does not > > allow > > >> > > > >> > HTML > > >> > > > >> > > > > emails. I'm finding this more annoying as time goes on. > > >> > > > >> > > > > > > >> > > > >> > > > > It turns out this is a setting that we could ask the Apache > > Infra > > >> > > > >> > team > > >> > > > >> > > > > to change for us. (For example > > >> > > > >> > > > https://issues.apache.org/jira/browse/INFRA-19311) > > >> > > > >> > > > > > > >> > > > >> > > > > This email is a vote/discussion about it, and lazy consensus > > that > > >> > if > > >> > > > >> > no > > >> > > > >> > > > > one disagrees by Friday noon BST I will open a ticket to > change > > >> this > > >> > > > >> > > > setting. > > >> > > > >> > > > > > > >> > > > >> > > > > +1 from me, clearly :) > > >> > > > >> > > > > > > >> > > > >> > > > > -ash > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > > > >> > > > >> > Jarek Potiuk > > >> > > > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > > >> > > > >> > > > >> > M: +48 660 796 129 <+48660796129> > > >> > > > >> > [image: Polidea] <https://www.polidea.com/> > > >> > -- Chao-Han Tsai

Re: [VOTE] AIP-31: Airflow functional DAG definition

b00f5b246d7f18b7c796edde8c807a8d86a%40%3Cdev.airflow.apache.org%3E > > >> https://apache-airflow.slack.com/archives/CCY359SCV/p1582303386007900 > > >> > > >> This vote will last for 72 hours until 2020-03-28T20:40Z, and until at > > >> least 3 votes have been cast. > > >> > > >> Consider this my +1 (binding). > > >> > > >> Big thanks to Gerard for this AIP! > > >> > > >> Best wishes, > > >> Tomek > > >> > > > > > > -- Chao-Han Tsai

Re: [AirFlow]: Use of SubDags

periences in internet, strongly recommend to > > avoid them, what do you think? > > > > Thanks in advance. > > > > Kind regards. > > > > David. > -- Chao-Han Tsai

Re: [DISCUSS] Packaging DAG/operator dependencies in wheels

gt; > > with wheel format (which is also .zip BTW). Maybe it was > already > > > > >> > mentioned > > > > >> > > in some discussions before but I have not found any. > > > > >> > > > > > > >> > > *Context:* > > > > >> > > > > > > >> > > We are well on the way of implementing "AIP-21 Changing import > > > > >paths" > > > > >> and > > > > >> > > will provide backport packages for Airflow 1.10. As a next > step > > > > >we want > > > > >> > to > > > > >> > > target AIP-8. > > > > >> > > One of the problems to implement AIP-8 (split hooks/operators > > > > >into > > > > >> > separate > > > > >> > > packages) is the problem of dependencies. Different > > > > >operators/hooks > > > > >> might > > > > >> > > have different dependencies if maintained separately. > Currently > > > > >we > > > > >> have a > > > > >> > > common set of dependencies as we have only one setup.py, but > if > > > > >we > > > > >> split > > > > >> > to > > > > >> > > separate packages, this might change. > > > > >> > > > > > > >> > > *Proposal:* > > > > >> > > > > > > >> > > Our users - who love the .zip DAG distribution - proposed that > > we > > > > >> package > > > > >> > > the DAGs and all related packages in a wheel package instead > of > > > > >pure > > > > >> > .zip. > > > > >> > > This would allow the users to install extra dependencies > needed > > > > >by the > > > > >> > DAG. > > > > >> > > And it struck me that we could indeed do that for DAGs but > also > > > > >> mitigate > > > > >> > > most of the dependency problems for separately-packaged > > > > >operators. > > > > >> > > > > > > >> > > The proposal from our users was to package the extra > > dependencies > > > > >> > together > > > > >> > > with the DAG in a wheel file. This is quite cool on it's own, > > but > > > > >I > > > > >> > thought > > > > >> > > we might actually use the same approach to solve dependency > > > > >problem > > > > >> with > > > > >> > > AIP-8. > > > > >> > > > > > > >> > > I think we could implement "operator group" -> extra -> "pip > > > > >packages" > > > > >> > > dependencies (we need them anyway for AIP-21) and then we > could > > > > >have > > > > >> > wheel > > > > >> > > packages with all the "extra" dependencies for each group of > > > > >operators. > > > > >> > > > > > > >> > > Worker executing an operator could have the "core" > dependencies > > > > >> installed > > > > >> > > initially but then when it is supposed to run an operator it > > > > >could > > > > >> > create a > > > > >> > > virtualenv, install the required "extra" from wheels and run > the > > > > >task > > > > >> for > > > > >> > > this operator in this virtualenv (and remove virtualenv). We > > > > >could have > > > > >> > > such package-wheels prepared (one wheel package per operator > > > > >group) and > > > > >> > > distributed either same way as DAGs or using some shared > binary > > > > >> > repository > > > > >> > > (and cached in the worker). > > > > >> > > > > > > >> > > Having such dynamically created virtualenv has also the > > advantage > > > > >that > > > > >> if > > > > >> > > someone has a DAG with specific dependencies - they could be > > > > >embedded > > > > >> in > > > > >> > > the DAG wheel, installed from it to this virtualenv, and the > > > > >virtualenv > > > > >> > > would be removed after the task is finished. > > > > >> > > > > > > >> > > The advantage of this approach is that each DAG's extra > > > > >dependencies > > > > >> are > > > > >> > > isolated and you could have even different versions of the > same > > > > >> > dependency > > > > >> > > used by different DAGs. I think that could save a lot of > > > > >headaches for > > > > >> > many > > > > >> > > users. > > > > >> > > > > > > >> > > For me that whole idea sounds pretty cool. > > > > >> > > > > > > >> > > Let me know what you think. > > > > >> > > > > > > >> > > J. > > > > >> > > > > > > >> > > > > > > >> > > -- > > > > >> > > > > > > >> > > Jarek Potiuk > > > > >> > > Polidea <https://www.polidea.com/> | Principal Software > > Engineer > > > > >> > > > > > > >> > > M: +48 660 796 129 <+48660796129> > > > > >> > > [image: Polidea] <https://www.polidea.com/> > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Chao-Han Tsai

Re: [VOTE] Accept donation of Airflow k8s Operator to Apache Airflow

t; > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/6623 > > > > > > > > > > > > > > > > > > > > > > This vote is to determine if the Airflow PMC is in > favor > > of > > > > > > accepting > > > > > > > > > > this > > > > > > > > > > > donation, subject to the fulfillment of the ASF IP > > > Clearance > > > > > > process. > > > > > > > > > > > > > > > > > > > > > > [ ] +1 : Accept contribution of Airflow k8s operator > > > > > > > > > > > [ ] 0 : No opinion > > > > > > > > > > > [ ] -1 : Reject contribution because... > > > > > > > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. > > > > > > > > > > > > > > > > > > > > > > The specifics about the location of the code are not > > > relevant > > > > > at > > > > > > the > > > > > > > > > > > moment. For now, I’m just looking to finalize the > > donation, > > > > and > > > > > > we > > > > > > > > can > > > > > > > > > > > later deal with the practical considerations regarding > > > where > > > > to > > > > > > keep > > > > > > > > > the > > > > > > > > > > > code, and setting up CI/CD for it. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Aizhamal > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/bfbb57da417a01169efd0ec459ebf0419a1258d7302c5e37ef22db12@%3Cdev.airflow.apache.org%3E > > > > > > > > > > > [2] > > > https://github.com/GoogleCloudPlatform/airflow-operator > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > > > Engineer > > > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > -- > > > > > > > > > > Tomasz Urbaszek > > > > > Polidea <https://www.polidea.com/> | Junior Software Engineer > > > > > > > > > > M: +48 505 628 493 <+48505628493> > > > > > E: tomasz.urbas...@polidea.com > > > > > > > > > > Unique Tech > > > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > > > > > > > > > > -- Chao-Han Tsai

Re: Drop Python 3.5 support?

gt;>>>>> Airflow: > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Formatted string literals : > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.python.org/3/whatsnew/3.6.html#pep-498-formatted-string-literals > >>>>>>>>>>>>> (my > >>>>>>>>>>>>> favourite one) > >>>>>>>>>>>>> - Type annotations for variables : > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.python.org/3/whatsnew/3.6.html#pep-526-syntax-for-variable-annotations > >>>>>>>>>>>>> - > >>>>>>>>>>>>> now especially that we use MyPy more and more this one > >> is > >>>>>> the > >>>>>>>>> only > >>>>>>>>>>>>> place we > >>>>>>>>>>>>> have to leave ugly comments rather than annotations. > >>>>>>>>>>>>> - Path-like types: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.python.org/3/whatsnew/3.6.html#pep-519-adding-a-file-system-path-protocol > >>>>>>>>>>>>> - > >>>>>>>>>>>>> nicer handling of file access > >>>>>>>>>>>>> - Local Time disambiguation: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.python.org/3/whatsnew/3.6.html#pep-495-local-time-disambiguation > >>>>>>>>>>>>> - > >>>>>>>>>>>>> this had already caused problems in the past (I fixed a > >>>>>> problem > >>>>>>>>>>> where > >>>>>>>>>>>>> pendulum and datetime objects were mixed and caused > >> wrong > >>>>>>>>>> behaviour > >>>>>>>>>>> on > >>>>>>>>>>>>> 3.5) > >>>>>>>>>>>>> - Json loads supports binary format - > >>>>>>>>>>>>> https://docs.python.org/3/whatsnew/3.6.html#json - > >> this > >>>>>> has > >>>>>>>>>> already > >>>>>>>>>>>>> bitten us as well. there was code working fine in py2.7 > >>>>>> and 3.6 > >>>>>>>>>> but > >>>>>>>>>>>> not > >>>>>>>>>>>>> working with 3.5(!). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Last but not least - it might free some resources on > >>>>>> Travis (I hope > >>>>>>>>>>>> GitLab > >>>>>>>>>>>>> fix will be out in 10 days or so and we will be able to > >>>>>> start > >>>>>>>>> testing > >>>>>>>>>>>>> migration to it). > >>>>>>>>>>>>> > >>>>>>>>>>>>> J.. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Nov 12, 2019 at 10:00 PM Bolke de Bruin > >>>>>> >>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Can we drop python 3.5 support and switch to 3.6 as a > >>>>>> minimum? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers > >>>>>>>>>>>>>> Bolke > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> Software > >>>>>> Engineer > >>>>>>>>>>>>> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> Jarek Potiuk > >>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>> Engineer > >>>>>>>>>> > >>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>> > >>>>>>>>> > >>>>> > >>>> > >>> > >>> > >>> -- > >>> > >>> Jarek Potiuk > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > >>> M: +48 660 796 129 <+48660796129> > >>> [image: Polidea] <https://www.polidea.com/> > >>> > >> > -- Chao-Han Tsai

Re: [PROPOSAL] Migrate to Pytest

ek.pot...@polidea.com> > >>> > >>>>>> wrote: > >>> > >>>>>>>>> > >>> > >>>>>>>>> Fantastic! Pytest is sooo much better than nosetests! > >>> > >>>>>>>>> > >>> > >>>>>>>>> On Mon, Oct 14, 2019 at 1:44 PM Ash Berlin-Taylor < > >>> > >>> a...@apache.org > >>> > >>>>> > >>> > >>>>>>>> wrote: > >>> > >>>>>>>>> > >>> > >>>>>>>>>> Awesome, thanks! > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> I tried this at the start of the year but we were a few > 100 > >>> > >>>> failing > >>> > >>>>>>>> tests > >>> > >>>>>>>>>> away. > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> -ash > >>> > >>>>>>>>>> > >>> > >>>>>>>>>>> On 14 Oct 2019, at 11:44, Tomasz Urbaszek < > >>> > >>>>>> tomasz.urbas...@polidea.com > >>> > >>>>>>>>> > >>> > >>>>>>>>>> wrote: > >>> > >>>>>>>>>>> > >>> > >>>>>>>>>>> Hi all, > >>> > >>>>>>>>>>> > >>> > >>>>>>>>>>> I've just created AIP-27 Migrate to pytest > >>> > >>>>>>>>>>> < > >>> > >>>>>>>>>> > >>> > >>>>>>>> > >>> > >>>>>> > >>> > >>>>> > >>> > >>>> > >>> > >>> > >>> > >> > >>> > > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-27+Migrate+to+pytest > >>> > >>>>>>>>>>> . > >>> > >>>>>>>>>>> I think this proposal will improve our test suite > especially > >>> > >>>> that > >>> > >>>>> it > >>> > >>>>>>>>>> seems > >>> > >>>>>>>>>>> that we only need to fix 16 tests to obtain green CI > build. > >>> > >>>>>>>>>>> > >>> > >>>>>>>>>>> Your considerations and feedback are highly welcomed! > >>> > >>>>>>>>>>> > >>> > >>>>>>>>>>> Bests, > >>> > >>>>>>>>>>> Tomek > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>> > >>> > >>>>>>>>> -- > >>> > >>>>>>>>> > >>> > >>>>>>>>> Jarek Potiuk > >>> > >>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>> > >>> Engineer > >>> > >>>>>>>>> > >>> > >>>>>>>>> M: +48 660 796 129 <+48660796129> > >>> > >>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>> > >>>>>>>> > >>> > >>>>>>>> > >>> > >>>>> > >>> > >>>> > >>> > >>> > >>> > >> > >>> > >> > >>> > >> -- > >>> > >> > >>> > >> Tomasz Urbaszek > >>> > >> Polidea <https://www.polidea.com/> | Junior Software Engineer > >>> > >> > >>> > >> M: +48 505 628 493 <+48505628493> > >>> > >> E: tomasz.urbas...@polidea.com > >>> > >> > >>> > >> Unique Tech > >>> > >> Check out our projects! <https://www.polidea.com/our-work> > >>> > >> > >>> > > > >>> > > > >>> > > -- > >>> > > > >>> > > Jarek Potiuk > >>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > > > >>> > > M: +48 660 796 129 <+48660796129> > >>> > > [image: Polidea] <https://www.polidea.com/> > >>> > > >>> > > >>> > >> -- Chao-Han Tsai

Re: Proposed roadmap for Airflow 2.0

gt; > >> >>> > >> > > >> >>> I'd like to start working on a concrete plan to get > >Airflow 2.0 > >> > out, > >> > > >> and > >> > > >> >> as a result I've started updating > >> > > >> >> > >https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 > >> > > >> >>> > >> > > >> >>> In addition to all the tidy up work ("spring cleaning", > >finish > >> > tidy > >> > > up > >> > > >> >> after dropping Py2 etc) I'd propose the following 6 high > >level > >> > items: > >> > > >> >>> > >> > > >> >>> Knative Executor (AIP-25, currently draft. Being worked on > >by > >> > Daniel > >> > > >> >> Imberman ) > >> > > >> >>> Improve Webserver performance (AIP-24, currently draft. > >Being > >> > worked > >> > > >> on > >> > > >> >> by myself, Kaxil Naik and Zhou Fang) > >> > > >> >>> Enhanced real-time UI > >> > > >> >>> Improve Scheduler performance > >> > > >> >>> Extend/finish the API (AIP-13 is part of this, but not > >all) > >> > > >> >>> Production Docker image + Helm chart > >> > > >> >>> We at Astronomer are committing to work on these in > >roughly this > >> > > order > >> > > >> >> if no one else gets to them first. I also propose that we > >create > >> > SIGs > >> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly > >(every > >> > 14 > >> > > >> days) > >> > > >> >> "calls"/update sessions. We already have #sig-ui and > >> > > >> #sig-dag-serialisation. > >> > > >> >>> > >> > > >> >>> This roadmap is also not a promise that all of these will > >be > >> done > >> > > >> before > >> > > >> >> Airflow 2.0 - we may decide later to push something back to > >v2.1 > >> > etc. > >> > > >> >>> > >> > > >> >>> Does anyone disagree strongly with these priorities, or > >have > >> > > anything > >> > > >> >> they want to add that you are willing to work on? > >> > > >> >>> > >> > > >> >>> Thanks, > >> > > >> >>> Ash > >> > > >> >> > >> > > >> >> > >> > > >> > > >> > > >> > -- > >> > > >> > > >> > > >> > Tomasz Urbaszek > >> > > >> > Polidea <https://www.polidea.com/> | Junior Software > >Engineer > >> > > >> > > >> > > >> > M: +48 505 628 493 <+48505628493> > >> > > >> > E: tomasz.urbas...@polidea.com > > > >> > > >> > > >> > > >> > Unique Tech > >> > > >> > Check out our projects! <https://www.polidea.com/our-work> > >> > > >> > >> > > >> > >> > > > >> > > >> > > >> > -- > >> > > >> > Jarek Potiuk > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > >> > M: +48 660 796 129 <+48660796129> > >> > [image: Polidea] <https://www.polidea.com/> > >> > > >> > -- Chao-Han Tsai

Re: [ANNOUNCE] Please welcome new Airflow committer Felix Uellendall

ipating in the mailing list as well as AIP discussion. > > > > Please join in me congratulating Felix for becoming an Airflow > > committer. > > > > > > > > Welcome Felix! > > > > > > > > Cheer, > > > > > > > > Ash > > > > on behalf of the Airflow PMC > > > > > > -- Chao-Han Tsai

Re: "Breeze" dev environment

ls to help to solve them. Feel free to ping me in case > > you try to use Breeze and have problems - slack is probably best in such > > case. > > > > J > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Chao-Han Tsai

Re: Airflow maintenance blog post

.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Chao-Han Tsai

Re: [IE] Re: [VOTE] Change the Airflow logo

isting logo" I have not put that > >>>>>>>>> as > >>>>>>>>> an > >>>>>>>>> option. Shout very loudly now if you disagree) > >>>>>>>>> > >>>>>>>>> This vote will last for 72 hours, until Saturday 24th Agust at > >>>>>>>>> 10:42 > >>>>>>>>> UTC, > >>>>>>>>> and all votes count this time (no binding or non-binding on this > >>>>>>>>> one). > >>>>>>>>> The > >>>>>>>>> options are: > >>>>>>>>> > >>>>>>>>> Option 1. (Chris' pinwheel one) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Proposed+Logo+Redesign?preview=/125308376/125309631/airflowOption1.png > >>>>>>>>> Option 2. (Chris' shuriken one) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Proposed+Logo+Redesign?preview=/125308376/125309632/airflowOption2.png > >>>>>>>>> Option 3. (Daniel's one) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Proposed+Logo+Redesign?preview=/125308376/125309633/airflowOption3.png > >>>>>>>>> > >>>>>>>>> All those images in one place https://imgur.com/a/WVpQJGd (I've > >>>>>>>>> attached > >>>>>>>>> them as images too, but I don't think the mailing list allows > >>>>>>>>> attachments) > >>>>>>>>> > >>>>>>>>> Please cast your vote clearly for one of the three options. In > >>>> the > >>>>>>>>> case > >>>>>>>>> of a tie I will make the casting deciding vote from the tied > >>>>>>>>> options. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> My vote is for Option 2. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Ash > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> *Mohannad Albanayosy* > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> *Sunil Varma Chiluvuri* > >>>>>> Data & Analytics > >>>>>> Equifax Inc. > >>>>>> > >>>>>> 352-215-3391 > >>>>>> sunilvarma.chiluv...@equifax.com > >>>>>> > >>>>>> -- > >>>>>> This message contains proprietary information from Equifax which may > >> be > >>>>>> confidential. If you are not an intended recipient, please refrain > >> from > >>>>>> any > >>>>>> disclosure, copying, distribution or use of this information and > note > >>>>> that > >>>>>> such actions are prohibited. If you have received this transmission > in > >>>>>> error, please notify by e-mail postmas...@equifax.com > >>>>>> <mailto:postmas...@equifax.com>. > >>>>>> > >>>>>> > >>>>>> Equifax® is a registered trademark of > >>>>>> Equifax Inc. All rights reserved. > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> Leah Cole | Developer Programs Engineer | colel...@google.com | > (925) > >>>>> 257-2112 > >>>>> > >>>> > >> > -- Chao-Han Tsai

Re: [DISCUSS] Tweaks to the Airflow logo

08/13 10:28:48, Ash Berlin-Taylor > > > wrote: > > > > > > > > > Hi everyone,> > > > > > > > > > > > > > > > > > > Here at Astronomer we've just had our Lead Designer, Chris > > > > > Hendrixson, > > > > > > > > join us recently and he's proposed a few subtle tweaks to the > > > Airflow > > > > > > > logo > > > > > > > > that I think keep it still recognisably the same logo, but > just > > > a bit > > > > > > > more > > > > > > > > refined.> > > > > > > > > > > > > > > > > > > In his words:> > > > > > > > > > > > > > > > > > > > The new Airflow icon was designed by me, Chris > Hendrixson. > > > I’m > > > > > the > > > > > > > > Lead Designer at Astronomer. I’ve been designing logos and > > > > > interfaces and > > > > > > > > many other things for over a decade. This rebrand is free, of > > > > > course. I’m > > > > > > > > excited about contributing to the open source community.> > > > > > > > > > > > > > > > > > > I don't think the mailing list allows attachments, so you > can > > > view > > > > > it > > > > > > > on > > > > > > > > the wiki < > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Proposed+Logo+Redesign > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do people think of this?> > > > > > > > > > > > > > > > > > > -ash> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Kaxil Naik* > > > > > > > *Big Data Consultant | DevOps Data Engineer* > > > > > > > *Certified *Google Cloud Data Engineer | *Certified* Apache > Spark & > > > > > Neo4j > > > > > > > Developer > > > > > > > *LinkedIn*: https://www.linkedin.com/in/kaxil > > > > > > > > > > > > > > > > > > > > > > -- Chao-Han Tsai

Re: [VOTE] Release Airflow 1.10.4 from RC5

-[4338](tel:4338)] Change k8s pod_request_factory to use > yaml safe_load > >> > (#[5120](tel:5120)) > >> > - [AIRFLOW-[4869](tel:4869)] Reorganize sql to gcs operators. > (#[5504](tel:5504)) > >> > - [AIRFLOW-[5021](tel:5021)] move gitpython into setup_requires > (#[5640](tel:5640)) > >> > - [AIRFLOW-[4583](tel:4583)] Fixes type error in GKEPodOperator > (#[5612](tel:5612)) > >> > - [AIRFLOW-[4116](tel:4116)] Dockerfile now supports CI image build > on DockerHub > >> > (#[4937](tel:4937)) > >> > - [AIRFLOW-[4115](tel:4115)] Multi-staging Aiflow Docker image > (#[4936](tel:4936)) > >> > - [AIRFLOW-[4963](tel:4963)] Avoid recreating task context > (#[5596](tel:5596)) > >> > - [AIRFLOW-[4865](tel:4865)] Add context manager to set temporary > config values in > >> > tests. (#[5569](tel:5569)) > >> > - [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with version > below [4.17.13](tel:41713) > >> > (#[5572](tel:5572)) (used only in build-pipeline, not runtime) > >> > - [AIRFLOW-[4868](tel:4868)] Fix typo in kubernetes/docker/build.sh > (#[5505](tel:5505)) > >> > - [AIRFLOW-[4211](tel:4211)] Add tests for WebHDFSHook > (#[5015](tel:5015)) > >> > - [AIRFLOW-[4320](tel:4320)] Add tests for SegmentTrackEventOperator > (#[5104](tel:5104)) > >> > - [AIRFLOW-[4319](tel:4319)] Add tests for Bigquery related Operators > (#[5101](tel:5101)) > >> > - [AIRFLOW-[4014](tel:4014)] Change DatastoreHook and add tests > (#[4842](tel:4842)) > >> > - [AIRFLOW-[4322](tel:4322)] Add test for VerticaOperator > (#[5107](tel:5107)) > >> > - [AIRFLOW-[4323](tel:4323)] Add 2 tests for WinRMOperator > (#[5108](tel:5108)) > >> > - [AIRFLOW-[3677](tel:3677)] Improve CheckOperator test coverage > (#[4756](tel:4756)) > >> > - [AIRFLOW-[4659](tel:4659)] Fix pylint problems for api module > (#[5398](tel:5398)) > >> > - [AIRFLOW-[4358](tel:4358)] Speed up test_jobs by not running tasks > (#[5162](tel:5162)) > >> > - [AIRFLOW-[4394](tel:4394)] Don't test behaviour of BackfillJob from > CLI tests > >> (#[5160](tel:5160)) > >> > - [AIRFLOW-[3471](tel:3471)] Move XCom out of models.py > (#[4629](tel:4629)) > >> > - [AIRFLOW-[4379](tel:4379)] Remove duplicate code & Add validation > in gcs_to_gcs.py > >> > (#[5145](tel:5145)) > >> > - [AIRFLOW-[4259](tel:4259)] Move models out of models.py > (#[5056](tel:5056)) > >> > - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis > (#[5233](tel:5233)) > >> > - [AIRFLOW-[4535](tel:4535)] Break jobs.py into multiple files > (#[5303](tel:5303)) > >> > - [AIRFLOW-[1464](tel:1464)] Batch update task_instance state > (#[5323](tel:5323)) > >> > - [AIRFLOW-[4554](tel:4554)] Test for sudo command, add some other > test docs (#[5310](tel:5310)) > >> > - [AIRFLOW-[4419](tel:4419)] Refine concurrency check in scheduler > (#[5194](tel:5194)) > >> > - [AIRFLOW-[4269](tel:4269)] Minor acceleration of > jobs._process_task_instances() > >> > (#[5076](tel:5076)) > >> > - [AIRFLOW-[4341](tel:4341)] Remove ``View.render()`` already exists > in fab.BaseView > >> > (#[5125](tel:5125)) > >> > - [AIRFLOW-[4342](tel:4342)] Use @cached_property instead of > re-implementing it each > >> > time (#[5126](tel:5126)) > >> > - [AIRFLOW-[4256](tel:4256)] Remove noqa from migrations > (#[5055](tel:5055)) > >> > - [AIRFLOW-[4034](tel:4034)] Remove unnecessary string formatting > with ``**locals()`` > >> > (#[4861](tel:4861)) > >> > - [AIRFLOW-[3944](tel:3944)] Remove code smells (#[4762](tel:4762)) > >> > > >> > Doc-only changes > >> > """""""""""""""" > >> > - [AIRFLOW-XXX] Add missing doc for annotations param of > >> > KubernetesPodOperator (#[5666](tel:5666)) > >> > - [AIRFLOW-XXX] Fix typos in CONTRIBUTING.md (#[5626](tel:5626)) > >> > - [AIRFLOW-XXX] Correct BaseSensorOperator docs (#[5562](tel:5562)) > >> > - [AIRFLOW-[4926](tel:4926)] Fix example dags where its start_date is > >> > datetime.utcnow() (#[5553](tel:5553)) > >> > - [AIRFLOW-[4860](tel:4860)] Remove Redundant Information in Example > Dags (#[5497](tel:5497)) > >> > - [AIRFLOW-[4767](tel:4767)] Fix errors in the documentation of > Dataproc Operator > >> > (#[5487](tel:5487)) > >> > - [AIRFLOW-[1684](tel:1684)] Branching based on XCom variable (Docs) > (#[4365](tel:4365)) > >> > - [AIRFLOW-[3341](tel:3341)] FAQ return DAG object example > (#[4605](tel:4605)) > >> > - [AIRFLOW-[4433](tel:4433)] Add missing type in DockerOperator doc > string (#[5205](tel:5205)) > >> > - [AIRFLOW-[4321](tel:4321)] Replace incorrect info of Max Size limit > of GCS Object > >> > Size (#[5106](tel:5106)) > >> > - [AIRFLOW-XXX] Add information about user list (#[5341](tel:5341)) > >> > - [AIRFLOW-XXX] Clarify documentation related to autodetect parameter > in > >> > GCS_to_BQ Op (#[5294](tel:5294)) > >> > - [AIRFLOW-XXX] Remove mention of pytz compatibility from timezone > >> > documentation (#[5316](tel:5316)) > >> > - [AIRFLOW-XXX] Add missing docs for GoogleCloudStorageDeleteOperator > >> > (#[5274](tel:5274)) > >> > - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP connection > >> > (#[5242](tel:5242)) > >> > - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators > (#[5192](tel:5192)) > >> > - [AIRFLOW-XXX] Link to correct class for timedelta in macros.rst > (#[5226](tel:5226)) > >> > - [AIRFLOW-XXX] Add Kamil as committer (#[5216](tel:5216)) > >> > - [AIRFLOW-XXX] Add Joshua and Kevin as committer (#[5207](tel:5207)) > >> > - [AIRFLOW-XXX] Reduce log spam in tests (#[5174](tel:5174)) > >> > - [AIRFLOW-XXX] Speed up tests for PythonSensor (#[5158](tel:5158)) > >> > - [AIRFLOW-XXX] Add Bas Harenslak to committer list > (#[5157](tel:5157)) > >> > - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#[5132](tel:5132)) > >> > - [AIRFLOW-XXX] Update docstring for SchedulerJob (#[5105](tel:5105)) > >> > - [AIRFLOW-XXX] Fix docstrings for > CassandraToGoogleCloudStorageOperator > >> > (#[5103](tel:5103)) > >> > - [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator > >> docstring > >> > (#[5074](tel:5074)) > >> > - [AIRFLOW-XXX] Ignore python files under node_modules in docs > (#[5063](tel:5063)) > >> > - [AIRFLOW-XXX] Build a universal wheel with LICNESE files > (#[5052](tel:5052)) > >> > - [AIRFLOW-XXX] Fix docstrings of SQSHook (#[5099](tel:5099)) > >> > - [AIRFLOW-XXX] Use Py3.7 on readthedocs > >> > - [AIRFLOW-[4446](tel:4446)] Fix typos (#[5217](tel:5217)) > >> > >> > >> > >> -- > >> *Kaxil Naik* > >> *Big Data Consultant | DevOps Data Engineer* > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > Neo4j > >> Developer > >> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> -- Chao-Han Tsai

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

lasy consensus as this is > > > something we should have rather quickly) > > > > > > Looking forward to your comments! > > > > > > J. > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Chao-Han Tsai

Re: [2.0 spring cleaning] Deprecate subdags

e some feedback from you guys. Chao-Han On 2019/05/16 05:40:53, Chao-Han Tsai wrote: > Hi all,> > > I have a WIP PR <https://github.com/apache/airflow/pull/5279/files> that> > aims to make SubDagOperator to use normal scheduler instead of the backfill> > scheduler a

Re: I have joined Astronomer full-time !!

tribute to > the > project. This will now change as I will be able to work on Airflow > during > my normal working hours. > > Look forward to working with you all in the community. > > PS: I will be OOO between during the first half of July as I am > getting > married :) but would be active before and after that period. > > Regards, > Kaxil > > > > > > -- Chao-Han Tsai

Re: [DISCUSS] Pull Request Improvements

> >> > >> So I just noticed that GitHub acquired Pull Panda: > >> https://github.blog/2019-06-17-github-acquires-pull-panda/ > >> > >> > > > > > > > > > > > > > > >> > >> > >> It would help us to remind pr's ready for being reviewed, balance the > >> workload for committers to review them and overall being able to get > Pr's > >> more quickly merged and released. > >> > >> > > > > > > > > > > > > > > >> > >> > >> I think this would be a huge benefit for us all. What do you think? > >> > >> > > > > > > > > > > > > > > >> > >> > >> Kind regards, > >> feluelle > >> > >> > > > > > > > > > > > > > > >> > >> > >> Sent from ProtonMail mobile > >> > >> > > > > > > > > > > > > > > > > > > > > >> > >> > >> -- > >> > >> > > > > > > > > > > > > > > >> > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> > > > > > > > > > > > > > > >> > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > >> > > > > > > > > > > > > > > > > > > -- Chao-Han Tsai

Re: [VOTE] AIP-16: CLI: Use nested commands instead of flags

;optional arguments: > > > -h, --help show this help message and exit > > > > > > This is my (binding) +1 vote. > > > > > > A reminder about binding/non-binding votes: PMC members and Committer's > > > votes are binding for AIP; other members of the community are > encouraged > > to > > > vote with a "(non-binding)". > > > > > > Thanks, > > > Ash > > > > > > > > > > > > > > > > > -- > Best regards, > > Andrii Soldatenko > Python Developer > > skype: andrii.soldatenko > andrii.soldate...@gmail.com > http://asoldatenko.com > -- Chao-Han Tsai

Re: Call for fixes for 1.10.4

> > > > > > > Hello all, > > > > > > > > Please reply to this email wiith PR, commit hash or Jira issues that > > you > > > > want it to be included in *1.10.4* > > > > > > > > Regards, > > > > *Kaxil Naik* > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > E: jarek.pot...@polidea.com > -- Chao-Han Tsai

Re: [2.0 spring cleaning] Deprecate subdags

ation? If a new and better implementation is created, > > >>> would > > >>>> it > > >>>>>>> even be similar enough to subDAGs that we'd want to continue to > > >>> call > > >>>>> the > > >>>>>>> feature that? > > >>>>>>> > > >>>>>>> They feel like a "new Airflow user trap" right now - I have had > > >>> to > > >>>> tell > > >>>>>> my > > >>>>>>> team never to use them, because they seem appealing and are in > > >>> the > > >>>>>> official > > >>>>>>> docs. > > >>>>>>> > > >>>>>>> On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor > > >>> > > >>>>>> wrote: > > >>>>>>>> I'd like to find time to fix subdags as they do provide a > > >>> useful > > >>>>>>>> abstraction - but I agree right now they aren't great (I > > >>> avoid > > >>> them > > >>>>>>> because > > >>>>>>>> of this) > > >>>>>>>> > > >>>>>>>> I have half thoughts of how to it should work, I just need to > > >>> look > > >>>> at > > >>>>>> the > > >>>>>>>> code in depth to see if that makes sense. Now 1.10.3 is out I > > >>> might > > >>>>>> have > > >>>>>>> a > > >>>>>>>> bit more time to do this. > > >>>>>>>> > > >>>>>>>> -ash > > >>>>>>>> > > >>>>>>>>> On 12 Apr 2019, at 15:48, James Meickle > > >>> > >>>>>>> .INVALID> > > >>>>>>>> wrote: > > >>>>>>>>> I think we should deprecate SubDAGs given the complexity > > >>> they > > >>> add > > >>>>> and > > >>>>>>> the > > >>>>>>>>> limited usage and use cases. Or, we should invest effort in > > >>>>>> redesigning > > >>>>>>>>> their API and implementation. I think that having to > > >>> account > > >>> for > > >>>>>>>>> subdag-introduced complexity makes Airflow's code much > > >>> harder > > >>> to > > >>>>>>> maintain > > >>>>>>>>> and buggier, looking at how many open issues there are that > > >>>>> reference > > >>>>>>>>> subdags (and how unrelated in topic they are): > > >>>>>>>>> > > >> > > > https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22 > > >>>>>>>> > > > > > > -- Chao-Han Tsai

Re: [DISCUSS] AIP-5: Remote DAG Fetcher

versions of > DAGs to Airflow to support versioned DagRuns. > > Cheers, > > Chao-Han > > > > On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai > wrote: > >> Hi Airflowers, >> >> We had an email thread >> <https://lists.apache.org/thread.h

Re: [DISCUSS] AIP-5: Remote DAG Fetcher

-Han On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai wrote: > Hi Airflowers, > > We had an email thread > <https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E> > and PR <https://github.com/apache/

Re: [ANNOUNCE] Please welcome new Airflow committer Kevin Yang

r > > some > > > > > time > > > > > > now, contributing patches and participating in the mailing list. > > > > > > Implementing optimizations to the core of Apache Airflow to make > it > > > > > scale. > > > > > > Please join in me congratulating Kevin for becoming an Airflow > > > > committer. > > > > > > > > > > > > Welcome Kevin! > > > > > > > > > > > > Cheers, Fokko > > > > > > on behalf of the Airflow PMC > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > E: jarek.pot...@polidea.com > > > > > > > > > > > > > > > -- Chao-Han Tsai

Re: [ANNOUNCE] Please welcome new Airflow committer Kamil Breguła

> as > > AIP discussion. > > He spends lots of efforts improving the documentation of Airflow. > > Please join in me congratulating Kamil for becoming an Airflow committer. > > > > Welcome Kamil! > > > > Best, > > > > Tao > > on behalf of the Airflow PMC > > > -- Chao-Han Tsai

Re: [ANNOUNCE] Please welcome new Airflow committer Joshua Carp

>>>>>>> > >>>>>>>> On Mon, Apr 29, 2019 at 11:21 PM Jakob Homan > >>>>>> wrote: > >>>>>>>> > >>>>>>>>> Howdy all- > >>>>>>>>> I'm pleased to announce that the Airflow PMC has voted Joshua > >>> Carp > >>>>>>>>> to be a committer. Joshua's been actively involved in the > >> project > >>>>>> for > >>>>>>>>> quite a while, turning out great patches that have improved the > >>> code > >>>>>>>>> base. > >>>>>>>>> > >>>>>>>>> Welcome aboard, Joshua! > >>>>>>>>> > >>>>>>>>> -Jakob > >>>>>>>>> on behalf of the Airflow PMC > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>>> Jarek Potiuk > >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>>>> > >>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>> E: jarek.pot...@polidea.com > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >>> > >> > > -- Chao-Han Tsai

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

;> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> E: jarek.pot...@polidea.com > >> > > > > > > -- > > *Kaxil Naik* > > *Big Data Consultant *@ *Data Reply UK* > > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j > > Developer > > *LinkedIn*: https://www.linkedin.com/in/kaxil > > -- Chao-Han Tsai

Re: Proposal: Automatically mark stale PRs in github

; > As part of our effort to reduce the PR backlog I wanted to proposed > that > > > we set the github stale action https://github.com/apps/stale. This > will > > > allow us to temporarily close PRs/tickets that are not actively being > > > worked on. > > > > > > (note that this will not remove PRs, it will simply mark PRs as stale > to > > > make it easier for committers) > > > > > > > > > > > > > > > -- Chao-Han Tsai

Re: schedule_interval question

; Setting schedule_interval as > > schedule_interval=timedelta(hours=24), > > correct me if I am wrong but Airflow seems to schedule DAG 24 hours in the > past from the time DAG was deployed. > > Thanks, > Pawel > -- Chao-Han Tsai

Re: [ANNOUNCE] Please welcome new Airflow committer Jarek Potiuk

https://github.com/Polidea> [image: Facebook] > <https://www.facebook.com/Polidea.Software> [image: Twitter] > <https://twitter.com/polidea> [image: Linkedin] > <https://www.linkedin.com/company/polidea> [image: Instagram] > <https://instagram.com/polidea> [image: Behance] > <https://www.behance.net/polidea> > -- Chao-Han Tsai

Re: Is `airflow backfill` disfunctional?

;>>>>>> > > > >>>>>>>> First I start getting a ton of `logging.info` of each tasks > > that > > > >>>> cannot > > > >>>>>>> be > > > >>>>>>>> started just yet at every tick flooding my terminal with the > > > >> keyword > > > >>>>>>>> `FAILED` in it, looking like a million of lines like this one: > > > >>>>>>>> > > > >>>>>>>> [2018-05-24 14:33:07,852] {models.py:1123} INFO - Dependencies > > not > > > >>> met > > > >>>>>>> for > > > >>>>>>>> > > >>> [scheduled]>, > > > >>>>>>>> dependency 'Trigger Rule' FAILED: Task's trigger rule > > > >> 'all_success' > > > >>> re > > > >>>>>>>> quires all upstream tasks to have succeeded, but found 1 > > > >>>> non-success(es). > > > >>>>>>>> upstream_tasks_state={'successes': 0L, 'failed': 0L, > > > >>>> 'upstream_failed': > > > >>>>>>>> 0L, > > > >>>>>>>> 'skipped': 0L, 'done': 0L}, upstream_task_ids=['some_other > > > >> _task_id'] > > > >>>>>>>> > > > >>>>>>>> Good thing I triggered 1 month and not 2 years like I actually > > > >> need, > > > >>>> just > > > >>>>>>>> the logs here would be "big data". Now I'm unclear whether > > there's > > > >>>>>>> anything > > > >>>>>>>> actually running or if I did something wrong, so I decide to > > kill > > > >>> the > > > >>>>>>>> process so I can set a smaller date range and get a better > > picture > > > >>> of > > > >>>>>>>> what's up. > > > >>>>>>>> > > > >>>>>>>> I check my logging level, am I in DEBUG? Nope. Just INFO. So I > > > >> take > > > >>> a > > > >>>>>>> note > > > >>>>>>>> that I'll need to find that log-flooding line and demote it to > > > >> DEBUG > > > >>>> in a > > > >>>>>>>> quick PR, no biggy. > > > >>>>>>>> > > > >>>>>>>> Now I restart with just a single schedule, and get an error > `Dag > > > >>>>>>> {some_dag} > > > >>>>>>>> has reached maximum amount of 3 dag runs`. Hmmm, I wish > backfill > > > >>> could > > > >>>>>>> just > > > >>>>>>>> pickup where it left off. Maybe I need to run an `airflow > clear` > > > >>>> command > > > >>>>>>>> and restart? Ok, ran my clear command, same error is showing > up. > > > >>> Dead > > > >>>>>>> end. > > > >>>>>>>> > > > >>>>>>>> Maybe there is some new `airflow clear --reset-dagruns` > option? > > > >>>> Doesn't > > > >>>>>>>> look like it... Maybe `airflow backfill` has some new switches > > to > > > >>>> pick up > > > >>>>>>>> where it left off? Can't find it. Am I supposed to clear the > DAG > > > >>> Runs > > > >>>>>>>> manually in the UI? This is a pre-production, in-development > > DAG, > > > >> so > > > >>>>>>> it's > > > >>>>>>>> not on the production web server. Am I supposed to fire up my > > own > > > >>> web > > > >>>>>>>> server to go and manually handle the backfill-related DAG > Runs? > > > >>>> Cannot to > > > >>>>>>>> my staging MySQL and do manually clear some DAG runs? > > > >>>>>>>> > > > >>>>>>>> So. Fire up a web server, navigate to my dag_id, delete the > DAG > > > >>> runs, > > > >>>> it > > > >>>>>>>> appears I can finally start over. > > > >>>>>>>> > > > >>>>>>>> Next thought was: "Alright looks like I need to go Linus on > the > > > >>>> mailing > > > >>>>>>>> list". > > > >>>>>>>> > > > >>>>>>>> What am I missing? I'm really hoping these issues specific to > > > >> 1.8.2! > > > >>>>>>>> > > > >>>>>>>> Backfilling is core to Airflow and should work very well. I > want > > > >> to > > > >>>>>>> restate > > > >>>>>>>> some reqs for Airflow backfill: > > > >>>>>>>> * when failing / interrupted, it should seamlessly be able to > > > >> pickup > > > >>>>>>> where > > > >>>>>>>> it left off > > > >>>>>>>> * terminal logging at the INFO level should be a clear, human > > > >>>> consumable, > > > >>>>>>>> indicator of progress > > > >>>>>>>> * backfill-related operations (including restarts) should be > > > >> doable > > > >>>>>>> through > > > >>>>>>>> CLI interactions, and not require web server interactions as > the > > > >>>> typical > > > >>>>>>>> sandbox (dev environment) shouldn't assume the existence of a > > web > > > >>>> server > > > >>>>>>>> > > > >>>>>>>> Let's fix this. > > > >>>>>>>> > > > >>>>>>>> Max > > > >>>>>>>> > > > >>>>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > -- Chao-Han Tsai

Re: [VOTE] AIP-6 Apply Pylint to Airflow

>>>> be > > >>>>>>>>> ignored. This should benefit the Airflow code base with > > consistent, > > >>>>>>>>> documented code and less errors. > > >>>>>>>>> > > >>>>>>>>> Note that Pylint 2.0 works with Python 3 only, so if the vote > is > > >>>>>>>>> successful, it should be introduced after dropping support for > > >>>>>> Python 2 > > >>>>>>>>> (AIRFLOW-4196< > https://issues.apache.org/jira/browse/AIRFLOW-4196 > > >>>> ). > > >>>>>>>>> Note2: to keep the scope as small as possible, I’ll create a > > >>> separate > > >>>>>>>> vote > > >>>>>>>>> for Black formatting. > > >>>>>>>>> > > >>>>>>>>> Cheers, > > >>>>>>>>> Bas > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -- > > >>>>>>>> > > >>>>>>>> Jarek Potiuk > > >>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > Engineer > > >>>>>>>> > > >>>>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>>>> E: jarek.pot...@polidea.com > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> *Kaxil Naik* > > >>>>>>> *Big Data Consultant *@ *Data Reply UK* > > >>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > Spark & > > >>>> Neo4j > > >>>>>>> Developer > > >>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >>>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> > > >>>>> Kamil Breguła > > >>>>> Polidea <https://www.polidea.com/> | Software Engineer > > >>>>> > > >>>>> M: +48 505 458 451 <+48505458451> > > >>>>> E: kamil.breg...@polidea.com > > >>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>> > > >>>>> We create human & business stories through technology. > > >>>>> Check out our projects! <https://www.polidea.com/our-work> > > >>>>> [image: Github] <https://github.com/Polidea> [image: Facebook] > > >>>>> <https://www.facebook.com/Polidea.Software> [image: Twitter] > > >>>>> <https://twitter.com/polidea> [image: Linkedin] > > >>>>> <https://www.linkedin.com/company/polidea> [image: Instagram] > > >>>>> <https://instagram.com/polidea> [image: Behance] > > >>>>> <https://www.behance.net/polidea> > > >>>> > > >>> > > > > > > > > -- Chao-Han Tsai

Re: [2.0 spring cleaning] Deprecate subdags

; > of this) > >> > > > > > >> > > > > I have half thoughts of how to it should work, I just need to > >look > >> at > >> > > the > >> > > > > code in depth to see if that makes sense. Now 1.10.3 is out I > >might > >> > > have > >> > > > a > >> > > > > bit more time to do this. > >> > > > > > >> > > > > -ash > >> > > > > > >> > > > > > On 12 Apr 2019, at 15:48, James Meickle > > >> > > > .INVALID> > >> > > > > wrote: > >> > > > > > > >> > > > > > I think we should deprecate SubDAGs given the complexity > >they > >add > >> > and > >> > > > the > >> > > > > > limited usage and use cases. Or, we should invest effort in > >> > > redesigning > >> > > > > > their API and implementation. I think that having to > >account > >for > >> > > > > > subdag-introduced complexity makes Airflow's code much > >harder > >to > >> > > > maintain > >> > > > > > and buggier, looking at how many open issues there are that > >> > reference > >> > > > > > subdags (and how unrelated in topic they are): > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22 > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > -- Chao-Han Tsai

Re: Sync DAGs folder with Git

gt; # i.e. {AIRFLOW_HOME}/dags > git_dags_folder_mount_point = > > So I was wondering what is the best way to keep in sync Airflow with a > remote repository that contains DAGs and if this functionality is already > integrated natively in Airflow! > > Thank you so much! > > Best, > > > Gabriele > -- Chao-Han Tsai

Re: Airflow 1.10.3 has been released!

t; > > > > > > We also made this version available on PyPi for convenience (`pip > > > install > > > > > apache-airflow`): > > > > > > > > > > https://pypi.python.org/pypi/apache-airflow > > > > > > > > > > The documentation is available on: > > > > > https://airflow.apache.org/1.10.3/ > > > > > https://airflow.apache.org/ > > > > > > > > > > Find the CHANGELOG here for more details: > > > > > > > > > > > > > > > > > > > > https://airflow.apache.org/1.10.3/changelog.html#airflow-1-10-3-2019-04-09 > > > > > > > > > > Cheers, > > > > > Ash > > > > > > > > > > -- Chao-Han Tsai

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

discussing the proposal in depth, please > > > refer to the Wiki to keep all the communication in a single place: > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+image > > > > > > This is my +1 (binding) vote. > > > > > > Thanks, > > > Fokko > > > > > > -- Chao-Han Tsai

Re: [DISCUSS] Utility DAGs

to Airflow, or could/should > > they be just handled internally by the scheduler? > > > > -a > > > > On 31 March 2019 03:57:08 BST, Chao-Han Tsai > wrote: > > >Hi all, > > > > > >I have been thinking about adding some DAGs that are for the

[DISCUSS] Utility DAGs

to let user decide whether to load the utility_dags (just like what we did for example_dags). What do you think? -- Chao-Han Tsai

Re: Hangout: Learn how to hack on Airflow (CHANGE OF DAY)

> 17:00 GMT/1pm EST/10am PDT (note, times are only 4/7 hours apart next > > week, > > > not the usual 5/8) where I'll run through how I hack on Airflow, > tricks I > > > use for running tests locally (Sneak preview: I never run them all > > locally, > > > it just takes too long!) and will try to answer any questions. > > > >>> > > > >>> I'll try and record it too for people who can't make it. > > > >>> > > > >>> I'll follow up with a URL once I work out which platform to use (it > > > may be a Google Hangout, or something else) > > > >>> > > > >>> If you could reply to me OFF LIST so I can get an idea of interest, > > > that would be useful, but not required. > > > >>> > > > >>> Thanks, > > > >>> Ash > > > >> > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > E: jarek.pot...@polidea.com > -- Chao-Han Tsai

[DISCUSS] AIP-5: Remote DAG Fetcher

your feedback and suggestions for improvement! Chao-Han Tsai