Yep, thanks exactly what I understand as well!
On Wed, Mar 22, 2017 at 4:02 PM, Sumit Maheshwari
wrote:
> Vincent,
>
> I think "dag.catchup = False" affects the whole DAG, means skipping all
> tasks in it. While "LatestOnlyOperator" can be used to skip only some of
> the tasks in a DAG as well.
Vincent,
I think "dag.catchup = False" affects the whole DAG, means skipping all
tasks in it. While "LatestOnlyOperator" can be used to skip only some of
the tasks in a DAG as well.
On Wed, Mar 22, 2017 at 7:05 PM, Vincent Poulain <
vincent.poul...@tinyclues.com> wrote:
> I did not see the cle
I did not see the clear explanation there :
http://airflow.incubator.apache.org/concepts.html?highlight=provide_context#latest-run-only
All good!
On Wed, Mar 22, 2017 at 2:22 PM, Vincent Poulain <
vincent.poul...@tinyclues.com> wrote:
> Sid, in your example what is the difference between using t
Sid, in your example what is the difference between using the
LatestOnlyOperator & set catch_up feature to False ? "[The catch up
feature] kick off a DAG Run for any interval that has not been run"
I am still learning Airflow concepts too..
Thanks!
On Tue, Mar 21, 2017 at 10:31 PM, Ruslan Dautkha
Thank you for the detailed explanation Boris.
Best regards,
Ruslan Dautkhanov
On Mon, Mar 20, 2017 at 12:12 PM, Boris Tyukin
wrote:
> depends_on_past is looking at previous task instance which sounds the same
> as "latestonly" but the difference becomes apparent if you look at this
> example.
depends_on_past is looking at previous task instance which sounds the same
as "latestonly" but the difference becomes apparent if you look at this
example.
Let's say you have a dag, scheduled to run every day and it has been
failing for the past 3 days. The whole purpose of that dag is to populate
Thanks Boris. It does make sense.
Although how it's different from depends_on_past task-level parameter?
In both cases, a task will be skipped if there is another TI of this task
is still running (from a previous dagrun), right?
Thanks,
Ruslan
On Sat, Mar 18, 2017 at 7:11 PM, Boris Tyukin wrot
you would just chain them - there is an example that came with airflow 1.8
https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_latest_only.py
so in your case, instead of dummy operator, you would use your Oracle
operator.
Does it make sense?
On Sat, Mar 18, 2017
Is there is a way to combine scheduling behavior operators (like this
LatestOnlyOperator)
with a functional operator (like Oracle_Operator)? I was thinking multiple
inheritance would do,like
> class Oracle_LatestOnly_Operator (Oracle_Operator, LatestOnlyOperator):
> ...
I might be overthinking t
Thanks George for that feature!
sure, just created a jira on this
https://issues.apache.org/jira/browse/AIRFLOW-1008
On Sat, Mar 18, 2017 at 12:05 PM, siddharth anand wrote:
> Thx Boris . Credit goes to George (gwax) for the implementation of the
> LatestOnlyOperator.
>
> Boris,
> Can you desc
Thx Boris . Credit goes to George (gwax) for the implementation of the
LatestOnlyOperator.
Boris,
Can you describe what you mean in a Jira?
-s
On Fri, Mar 17, 2017 at 6:02 PM, Boris Tyukin wrote:
> this is nice indeed along with the new catchup option
> https://airflow.incubator.apache.org/sche
this is nice indeed along with the new catchup option
https://airflow.incubator.apache.org/scheduler.html#backfill-and-catchup
Thanks Sid and Ben for adding these new options!
for a complete picture, it would be nice to force only one dag run at the
time.
On Fri, Mar 17, 2017 at 7:33 PM, siddhar
With the Apache Airflow 1.8 release imminent, you may want to try out the
*LatestOnlyOperator.*
If you want your DAG to only run on the most recent scheduled slot,
regardless of backlog, this operator will skip running downstream tasks for
all DAG Runs prior to the current time slot.
For example
13 matches
Mail list logo
