A DAG Run is a run for a particular dag at a particular time. For example,
if your DAG were called FOO and if its schedule interval were @hourly, then
you would have DAG runs : Foo @ noon, Foo @ 1p, Foo @ 2p, etc...
If you need to seed a table prior to each run, just add a task at the start
of you
Hey all,
Just a gentle reminder that the next Airflow meetup is happening on
November 16, 2016 at WePay in Redwood City. Details are here:
http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/234778571/
Looking forward to seeing you all there!
Cheers,
Chris
I believe you could also set downstream tasks to success to avoid it being
picked up why the scheduler again
> On Nov 1, 2016, at 12:12 PM, Ben Tallman wrote:
>
> So to kill a running DAG (and keep it killed), we need to clear the state
> of each task instance? Do we then pause the DAG? Or do t
Yeah, clearing will reschedule I'd have thought
We had a use case for this feature and got there by writing a basic plugin that
calls
https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L106
-Original Message-
From: Vishal Doshi [mailto:vis...@celect.com]
Sent: Tue
Our dags are triggered manually. Clearing the state alone will end up with the
cleared tasks being rescheduled, I think?
Thanks,
Vishal
On 11/1/16, 12:11, "Bolke de Bruin" wrote:
Clearing the state of the task, kills it. So the feature is already in, but
maybe not so clearly.
> O
So to kill a running DAG (and keep it killed), we need to clear the state
of each task instance? Do we then pause the DAG? Or do that in advance?
Thanks,
Ben
--
Ben Tallman - 503.680.5709
On Tue, Nov 1, 2016 at 9:11 AM, Bolke de Bruin wrote:
> Clearing the state of the task, kills it. So the f
Clearing the state of the task, kills it. So the feature is already in, but
maybe not so clearly.
> Op 1 nov. 2016, om 17:09 heeft Ben Tallman het volgende
> geschreven:
>
> I vote for this feature! Preferably a polite and a NOW option.
>
> Thanks,
> Ben
>
> --
> Ben Tallman - 503.680.5709
>
I vote for this feature! Preferably a polite and a NOW option.
Thanks,
Ben
--
Ben Tallman - 503.680.5709
On Tue, Nov 1, 2016 at 9:08 AM, Vishal Doshi wrote:
> I haven’t been able to find anything on this in the code / docs. Is there
> a supported way to kill a DAG (and its still running tasks)
Hey Gerard,
For our use case we have 1 pool per dag and each dag has between 50 and 100
tasks. We have the pool size set between 3 and 5 in most cases and have seen
up to 50 running tasks within the pool
Best,
David
> On Nov 1, 2016, at 11:17 AM, Gerard Toonstra wrote:
>
> David,
>
> When
I haven’t been able to find anything on this in the code / docs. Is there a
supported way to kill a DAG (and its still running tasks)?
David,
When you say "massive" oversubscribing, are you running a lot of dags in
parallel that
use your configured pools? Access to pools is not atomic at the moment.
Can you also quantify "massive" ? Not that it matters, but to get a better
idea.
Rgds,
Gerard
On Tue, Nov 1, 2016 at 4:09 PM,
Yes.. we have seen the over-subscription of pools. We do need a fix for it
--- I don't believe there is one. We need someone to own and fix it.. happy
to review a PR.
We use pools at Agari for all of our needs. We are okay with mild
oversubscription, so we do see numbers slightly higher, but our p
Hi,
I have a MySQL table, which will be stored some static information. The
information could be different for different airflow runs, so I hope to use
python code to initialize it whenever airflow starts.
Where is the best place to put such code ?
Is the class DagBag's __init__() a good can
I've been seeing some weird behavior when using Airflow's execution pool
feature. Pools have been massively over-filled leading to failed tasks. I
created a bug for this issue, but in the mean time, has anyone else experienced
this behavior?
https://issues.apache.org/jira/browse/AIRFLOW-584
B
14 matches
Mail list logo
