Re: Airflow cli to remote host

A quick side note to say that it's common to deploy one or many Airflow sandboxes which are effectively the same configuration as a worker without an actual worker instance working on it. It's similar to the concept of a "gateway node" in Hadoop. Users typically work in user space with a modified

Re: Airflow cli to remote host

Fabric looks perfect for this. From: Kyle Hamlin Sent: Thursday, May 24, 2018 6:22 AM To: dev@airflow.incubator.apache.org Subject: Re: Airflow cli to remote host I'd suggest using something like Fabric for

Re: Airflow cli to remote host

Just use SSH to exec the job on the remote server. Luke. Sent from my iPhone > On 24/05/2018, at 6:19 AM, Frank Maritato wrote: > > Hi All, > > I need to be able to run backfill for my jobs against our production airflow > server. Is there a way to run > > airflow

Re: Airflow cli to remote host

I'd suggest using something like Fabric for this. This is is how I accomplish the same task. On Wed, May 23, 2018 at 2:19 PM Frank Maritato wrote: > Hi All, > > I need to be able to run backfill for my jobs against our production > airflow

Airflow cli to remote host

Hi All, I need to be able to run backfill for my jobs against our production airflow server. Is there a way to run airflow backfill job_name -s 2018-05-01 against a remote server? I didn’t see a -h option to specify a hostname. If not, is there a way through the ui to do this? I'd rather not

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

Welcome back Ash! Firstly, it was great meeting you in person in London in March, right before you went on Paternity Leave. Hope you are getting some sleep :-) -s On Wed, May 23, 2018 at 1:44 AM, Ash Berlin-Taylor wrote: > Clearing :) > > I've gotten as far as having the

Re: Improving Airflow SLAs

Hi folks, I've created a branch off of v1-10-test; the diff can be found here: https://github.com/apache/incubator-airflow/compare/v1-10-test...Eronarn:sla_improvements As a recap, this work is expected to do the following: - split the "sla" parameter into three independent SLAs: expected

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

Clearing :) I've gotten as far as having the tab with this Jira issue open again. Which is progress :) I've been back at work for a week, so things are getting less hectic both at home and at work, so I should be able to find time for this this week or next. If someone picks it up before I

Scheduler gets slow with increasing number of Dags

Hi All, We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local Dag folder. We are observing that scheduler is taking lot of time(around 20 minutes) in picking

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

Clearing :) I've gotten as far as having the tab with this Jira issue open again. Which is progress :) I've been back at work for a week, so things are getting less hectic both at home and at work, so I should be able to find time for this this week or next. If someone picks it up before I

Re: SubDagOperator clear improvement

Ping On Thu, May 17, 2018 at 9:04 AM, Milan van der Meer < milan.vanderm...@riaktr.com> wrote: > Hi all, > > I have tried tackling https://issues.apache.org/jira/browse/AIRFLOW-1195 > but without any success as the logic of clearing tasks and subdags is > spread all over the place. > > Could