Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Maxime Beauchemin
+1 (binding) Also check out black to auto-pep8! https://github.com/ambv/black On Thu, Apr 11, 2019 at 5:12 PM Tao Feng wrote: > +1 > > On Thu, Apr 11, 2019 at 4:27 PM Beau Barker > wrote: > > > +1 non binding > > > > Pylint is extremely strict so your may want to be selective about the > >

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

2019-04-11 Thread Maxime Beauchemin
`end_date` is useful, some DAGs / tasks may be scheduled to expire, say an A/B test in an A/B testing framework with an experiment duration, a backfill framework, or a framework/UI that allows users to schedule to run a task to run for say 30 days. I'd keep this one for sure, not all pipelines /

Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Tao Feng
+1 On Thu, Apr 11, 2019 at 4:27 PM Beau Barker wrote: > +1 non binding > > Pylint is extremely strict so your may want to be selective about the > rules it applies. > > > > On 12 Apr 2019, at 7:15 am, Kamil Breguła > wrote: > > > > +! (non-binding) > > > > On Thu, Apr 11, 2019 at 11:05 PM

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Beau Barker
Thanks Ash and contributors. > On 12 Apr 2019, at 3:38 am, Gabriel Silk wrote: > > Nice work! > >> On Thu, Apr 11, 2019 at 9:21 AM Sid Anand wrote: >> >> Woot! Piling on with the kudos to Ash! >> >> -s >> >> On Thu, Apr 11, 2019 at 6:55 AM Jarek Potiuk >> wrote: >> >>> Great work Ash!

Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Beau Barker
+1 non binding Pylint is extremely strict so your may want to be selective about the rules it applies. > On 12 Apr 2019, at 7:15 am, Kamil Breguła wrote: > > +! (non-binding) > > On Thu, Apr 11, 2019 at 11:05 PM Driesprong, Fokko > wrote: > >> +1 (binding) >> >> For more information

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

2019-04-11 Thread Driesprong, Fokko
I agree with Max here, we should be careful. Regarding the yesterday_ds, yesterday_ds_nodash, tomorrow_ds, tomorrow_ds_nodash. I'm not against having better readable shorthands, but more about the fact that having a tomorrow_ds doesn't make sense when we have an hourly (or weekly) job. I'm

Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Kamil Breguła
+! (non-binding) On Thu, Apr 11, 2019 at 11:05 PM Driesprong, Fokko wrote: > +1 (binding) > > For more information about AIP-6: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=99844429 > > Op do 11 apr. 2019 om 23:03 schreef Kaxil Naik : > > > +1 (binding) > > > > On Thu, Apr

Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Kaxil Naik
+1 (binding) On Thu, Apr 11, 2019 at 9:51 PM Jarek Potiuk wrote: > +1 (non-binding). > > On Thu, Apr 11, 2019 at 10:50 PM Bas Harenslak < > basharens...@godatadriven.com> wrote: > > > Hello Airflow community, > > > > This email calls for a vote to introduce Pylint in the Airflow project. > >

Re: [VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Jarek Potiuk
+1 (non-binding). On Thu, Apr 11, 2019 at 10:50 PM Bas Harenslak < basharens...@godatadriven.com> wrote: > Hello Airflow community, > > This email calls for a vote to introduce Pylint in the Airflow project. > The vote will last for at least 1 week (April 18th 23:00 CET), and at least > three +1

[VOTE] AIP-6 Apply Pylint to Airflow

2019-04-11 Thread Bas Harenslak
Hello Airflow community, This email calls for a vote to introduce Pylint in the Airflow project. The vote will last for at least 1 week (April 18th 23:00 CET), and at least three +1 (binding) votes have been cast. After feedback on AIP-6 and discussion on

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Gabriel Silk
Nice work! On Thu, Apr 11, 2019 at 9:21 AM Sid Anand wrote: > Woot! Piling on with the kudos to Ash! > > -s > > On Thu, Apr 11, 2019 at 6:55 AM Jarek Potiuk > wrote: > > > Great work Ash! > > > > On Thu, Apr 11, 2019 at 10:19 AM Sumit Maheshwari < > sumeet.ma...@gmail.com> > > wrote: > > > > >

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Sid Anand
Woot! Piling on with the kudos to Ash! -s On Thu, Apr 11, 2019 at 6:55 AM Jarek Potiuk wrote: > Great work Ash! > > On Thu, Apr 11, 2019 at 10:19 AM Sumit Maheshwari > wrote: > > > Kudos to Ash!! > > > > On Thu, Apr 11, 2019 at 12:34 PM Chao-Han Tsai > > wrote: > > > > > Thanks Ash! > > > >

Sync DAGs folder with Git

2019-04-11 Thread Gabriele Di Bernardo
Hello everyone, I am relatively new to Airflow and I am not sure if this is the right place to ask questions. We set up a simple Kubernetes pod running a Docker image that contains Airflow. We want the DAGs to be synced from a Git repository. The first idea we had was to define an extra dag

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Jarek Potiuk
Great work Ash! On Thu, Apr 11, 2019 at 10:19 AM Sumit Maheshwari wrote: > Kudos to Ash!! > > On Thu, Apr 11, 2019 at 12:34 PM Chao-Han Tsai > wrote: > > > Thanks Ash! > > > > On Wed, Apr 10, 2019 at 6:07 PM Jiajie Zhong > > > wrote: > > > > > Thanks Ash! > > > > > > Best wish. > > > --

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Sumit Maheshwari
Kudos to Ash!! On Thu, Apr 11, 2019 at 12:34 PM Chao-Han Tsai wrote: > Thanks Ash! > > On Wed, Apr 10, 2019 at 6:07 PM Jiajie Zhong > wrote: > > > Thanks Ash! > > > > Best wish. > > -- Jiajie > > > > From: Deng Xiaodong > > Sent: Thursday, April 11, 2019 7:13

Re: Airflow 1.10.3 has been released!

2019-04-11 Thread Chao-Han Tsai
Thanks Ash! On Wed, Apr 10, 2019 at 6:07 PM Jiajie Zhong wrote: > Thanks Ash! > > Best wish. > -- Jiajie > > From: Deng Xiaodong > Sent: Thursday, April 11, 2019 7:13 > To: dev@airflow.apache.org > Subject: Re: Airflow 1.10.3 has been released! > > Thanks Ash!

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

2019-04-11 Thread Bas Harenslak
Great discussion, let’s stay on track. If I can summarise: * yesterday_ds, yesterday_ds_nodash, tomorrow_ds, tomorrow_ds_nodash * Arthur: some users use these for convenience * Bas/Fokko: these are values that can be easily derived in a one-liner * tables * nobody?

Re: Longer term Airflow planning

2019-04-11 Thread Deng Xiaodong
Some personal thoughts about the PR processing speed specifically. I'm trying to benchmark Airflow with other Apache projects (like Spark, Kafka), in terms of PR reviewing/merging speed: as at this moment, there are 400+ open PRs in Spark and 500+ open PRs in Kafka. On the other hand, there are