Re: [VOTE] AIP-31: Airflow functional DAG definition

2020-03-25 Thread Chao-Han Tsai
+1 binding On Wed, Mar 25, 2020 at 1:56 PM Kaxil Naik wrote: > +1 binding > > On Wed, Mar 25, 2020 at 8:55 PM Dan Davydov > wrote: > > > Sorry +1 (binding) :). > > > > On Wed, Mar 25, 2020 at 4:53 PM Dan Davydov > wrote: > > > > > +1 > > > > > > On Wed, Mar 25, 2020 at 4:40 PM Tomasz Urbaszek

Re: [VOTE] AIP-31: Airflow functional DAG definition

2020-03-25 Thread Kaxil Naik
+1 binding On Wed, Mar 25, 2020 at 8:55 PM Dan Davydov wrote: > Sorry +1 (binding) :). > > On Wed, Mar 25, 2020 at 4:53 PM Dan Davydov wrote: > > > +1 > > > > On Wed, Mar 25, 2020 at 4:40 PM Tomasz Urbaszek < > > tomasz.urbas...@polidea.com> wrote: > > > >> Hello everyone! > >> > >> This email

Re: [VOTE] AIP-31: Airflow functional DAG definition

2020-03-25 Thread Dan Davydov
Sorry +1 (binding) :). On Wed, Mar 25, 2020 at 4:53 PM Dan Davydov wrote: > +1 > > On Wed, Mar 25, 2020 at 4:40 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> wrote: > >> Hello everyone! >> >> This email calls for a vote on the design proposed in AIP-31: >> >>

Re: [VOTE] AIP-31: Airflow functional DAG definition

2020-03-25 Thread Dan Davydov
+1 On Wed, Mar 25, 2020 at 4:40 PM Tomasz Urbaszek wrote: > Hello everyone! > > This email calls for a vote on the design proposed in AIP-31: > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-31:+Airflow+functional+DAG+definition > > Discussion threads: > >

Re: Tasks scheduled but never run

2020-03-25 Thread kumar pavan
Hi , We have faced the similar issue and debugged the scheduler code and found that back fill don't have much options given( Back fill logic is different in more scheduling point) .In this case once the DAG is stuck it is stuck . Instead of using backfill you can use trigger_dag which has more

Re: Requesting to give details on FacebookAdsHook project

2020-03-25 Thread Daniel Imberman
Hi Chetan, Please see our documentation here https://airflow.apache.org/. Please feel free to reach out if you have further questions. On Mar 25, 2020, 10:29 AM -0700, Chetan Agarwal , wrote: > Dear Sir, > > I am Chetan Agarwal, third-year Computer Science and Engineering > Student at National

Re: SIG-Autoscaling meeting

2020-03-25 Thread Jarek Potiuk
+1 On Wed, Mar 25, 2020 at 6:20 PM Tomasz Urbaszek wrote: > +1 > > Tomek > > > On Wed, Mar 25, 2020 at 5:14 PM Kaxil Naik wrote: > > > > Sounds good, thanks Daniel. > > > > > > > > On Wed, Mar 25, 2020 at 3:57 PM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > > > Hello all! >

Requesting to give details on FacebookAdsHook project

2020-03-25 Thread Chetan Agarwal
Dear Sir, I am Chetan Agarwal, third-year Computer Science and Engineering Student at National Institute of Technology Karnataka, Surathkal. I would like to know more about this project. Can you please share some resources or details of this project. I have some prior experience with

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-03-25 Thread Tomasz Urbaszek
Hi all, Should we proceed with the vote? T. On Fri, Mar 20, 2020 at 1:37 AM Gerard Casas Saez wrote: > Hi everyone, > > Sorry for leaving this unfinished. I was commenting an extension that > Tomasz suggested in the AIP doc, to add fields in the operator that serve > as templated args. This

Re: SIG-Autoscaling meeting

2020-03-25 Thread Tomasz Urbaszek
+1 Tomek On Wed, Mar 25, 2020 at 5:14 PM Kaxil Naik wrote: > > Sounds good, thanks Daniel. > > > > On Wed, Mar 25, 2020 at 3:57 PM Daniel Imberman > wrote: > > > Hello all! > > > > Since the KEDA autoscaler has been released, for the moment there’s not > > too much going on with

Re: SIG-Autoscaling meeting

2020-03-25 Thread Kaxil Naik
Sounds good, thanks Daniel. On Wed, Mar 25, 2020 at 3:57 PM Daniel Imberman wrote: > Hello all! > > Since the KEDA autoscaler has been released, for the moment there’s not > too much going on with SIG-Autoscaling. For this reason I think we should > suspend the monthly meetings and take them

SIG-Autoscaling meeting

2020-03-25 Thread Daniel Imberman
Hello all! Since the KEDA autoscaler has been released, for the moment there’s not too much going on with SIG-Autoscaling. For this reason I think we should suspend the monthly meetings and take them on a more ad-hoc basis (e.g. if there is a call for feedback on the FargateExecutor). If new

Re: Helm Chart for Airflow

2020-03-25 Thread Jarek Potiuk
One thing for the donation. Did you you want to have separate repository Greg ? I think we should simply create a folder in Airflow repo and keep it there (similarly as we keep Dockerfile). I am going to switch our Kubernetes Tests to the production image (will make the tests much faster) and I

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-25 Thread Jarek Potiuk
OK. I think that one does not need voting then. I will proceed with my PR for that :). J. On Wed, Mar 25, 2020 at 3:38 PM Daniel Imberman wrote: > Agreed. Kind of a “best we can do” considering the current nature of > python. > On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko , > wrote: > >

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-25 Thread Daniel Imberman
Agreed. Kind of a “best we can do” considering the current nature of python. On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko , wrote: > Yes, I'd be in favor of not having two packages, and just pinning the > versions then. In this case, all the versions will be pinned, so if a user > wants to

Tasks scheduled but never run

2020-03-25 Thread heng gu
I was doing backfill running a one task dag. There were over 1000 dag_runs ready to run, I set concurrency=16 and max_active_run=16, I use Celery executor and have 12 workers. Task instances were quickly scheduled, but they stuck there and never get run. I did a search online and got this:

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

2020-03-25 Thread Karolina Rosół
Hi guys, > I think the problem here is not the tool like Jira but the way we are using > it. I fully agree with Michał that there should be some rules / guidelines agreed among community to follow. > Automation can help here. Auto-labelling and good templates might be > a good start. We should

Gsoc 2020 - Proposal Overview

2020-03-25 Thread Akbar B
Hi Team, Currently, I am in the process of drafting a proposal for Gsoc. The project title was *Enhancing the Airflow UI*, in this I want to resolve the existing UI issues in the codebase. I have planned to add the following issues into my proposal, 1. AIRFLOW-5647