Re: What is the best way to retry an entire DAG instead of just a single task?

2016-10-26 Thread Rob Froetscher
Sure, I'd be happy to. On Wed, Oct 26, 2016 at 1:13 AM, siddharth anand wrote: > Rob, > Would you mind writing up a blog about this and sharing it with the list? > To be honest, many of the committers themselves have little to no > experience with subdags. It's an area few people are familiar wi

Re: What is the best way to retry an entire DAG instead of just a single task?

2016-10-26 Thread siddharth anand
Rob, Would you mind writing up a blog about this and sharing it with the list? To be honest, many of the committers themselves have little to no experience with subdags. It's an area few people are familiar with. -s On Mon, Oct 24, 2016 at 11:08 AM, Rob Froetscher wrote: > Just following up her

Re: What is the best way to retry an entire DAG instead of just a single task?

2016-10-24 Thread Rob Froetscher
Just following up here. But we have found that SubDags can be another way to do this. We have had a few problems due to SubDags using the backfill execution style (if there aren't workers available immediately to pick up these tasks then it counts as "deadlocked"). However, for the most part, Sub

Re: What is the best way to retry an entire DAG instead of just a single task?

2016-08-02 Thread siddharth anand
Hi Arthur, It's not a dumb question. We don't have the ability to retry a DAG based on a task failure in a programatic way to the best of my knowledge. Also, we don't allow cyclic dependencies.. hence the Acyclic part of DAG. TriggerDagRunOperator won't work because the execution is async. The TDR

Re: What is the best way to retry an entire DAG instead of just a single task?

2016-08-02 Thread Wang Yajun
Arthur Purvis u can try: 1. airflow backfill -s start_date -e end_date your_DAG 2. airflow trigger_dag your_DAG u can see the detail information by the official document hope to help u Arthur Purvis 于2016年8月2日 周二下午10:55写道: > Apologies if this is a dumb question, but I'm looking for a way to re

What is the best way to retry an entire DAG instead of just a single task?

2016-08-02 Thread Arthur Purvis
Apologies if this is a dumb question, but I'm looking for a way to retry an entire DAG if a single task fails, rather than retry just that task. The context is that of a job starter + sensor, and if the sensor fails it means the job needs to be restarted, not just re-sensored. >From reading the d