Re: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs are missing in airflow.cfg

2018-09-21 Thread Holden Karau
Thanks for the heads up :) So it should automatically restore the assignee, which it looks like it did, and it's set up to only do this for new PRs (no back-fills). I can turn it all the way off if it's annoying to folks but otherwise it will automatically switch JIRAs to in-progress as folks make

Fwd: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs are missing in airflow.cfg

2018-09-21 Thread Kaxil Naik
Hi Holden Karau, Not sure on what is happening, but seems like the dashboard you are working on is setting assignee. Pinging you just in-case if you aren't aware :) -- Forwarded message - From: Holden Karau's magical unicorn (JIRA) Date: Fri, Sep 21, 2018 at 11:31 PM Subject: [ji

Re: Creating dynamic pool from task

2018-09-21 Thread Chris Palmer
I see, so for a given DagRun you want to limit the compute tasks that are running. But I'm guessing you want multiple DagRuns to be able to run concurrently to operate on their own clusters independently. >From what I could tell in the code, the pool gets checked before execution (which is when te

Re: Airflow: Apache Graduation

2018-09-21 Thread Bolke de Bruin
I love champions šŸ˜ Op vr 21 sep. 2018 14:01 schreef Jakob Homan : > The big list that *has* to be filled out and correct is here: > http://incubator.apache.org/projects/airflow.html > > This is usually done by the Champion, Chris Riccomini. Chris - are > you still interested to do this? > > Aft

Available tickets for the Airflow meetup

2018-09-21 Thread Jossie Haines
Hi there! Manish just told us about the meetup on Monday and some of of us from Tile wanted to come watch but noticed that RSVPs are closed. Ā Are there any spaces available? Thanks, Ā Jossie

Re: Airflow: Apache Graduation

2018-09-21 Thread Jakob Homan
The big list that *has* to be filled out and correct is here: http://incubator.apache.org/projects/airflow.html This is usually done by the Champion, Chris Riccomini. Chris - are you still interested to do this? After that and the maturity questionnaire are in good shape, it's: 1) Graduation DI

Re: Airflow: Apache Graduation

2018-09-21 Thread Bolke de Bruin
There is a maturity check that we can do. Don't have the link ready now. But can look it up later. To me two things need to be more mature 1. Handling of security issues 2. Licensing. 2a having ppl aware that with new dependencies updates to dependencies the respective licenses need to be update

Re: Creating dynamic pool from task

2018-09-21 Thread David Szakallas
Chris, the tasks are independent of each other so they can run concurrently. I have to limit the concurrency though, so they donā€™t starve. As the cluster is created dynamically with a task, a shared pool with other DAGs or other runs of the same DAG is not preferable. I imagined something like

Re: Creating dynamic pool from task

2018-09-21 Thread Chris Palmer
What would cause multiple computation tasks to run on the cluster at the same time? Are you worried about concurrent DagRuns? Does setting dag concurrency and/or task concurrency appropriately solve your problem? Chris On Thu, Sep 20, 2018 at 8:31 PM David Szakallas wrote: > What I am doing is

Re: Airflow: Apache Graduation

2018-09-21 Thread Ash Berlin-Taylor
Your guess is as good as mine on what is involved in graduation. I think that we sorted the Licensing issues in 1.10.0 (even if the way we sorted it was a little annoying - having to specify the environment variable at `pip install` time is a littlle bit un-pythonic, but I wasn't thinking of fix