A few related thoughts about the scheduler.
The scheduler is growing to take on much more than just scheduling, so much
so that "supervisor" would be a better name for it. It includes:
* parsing DAGs (eventually it may serialize their metadata to the database
to help make the web server stateless)
The only thing is that this is a change in semantics and changing semantics
(breaking some DAGs) and then changing them back (and breaking things
again) isn't great.
On Wed, Mar 15, 2017 at 7:02 PM, Bolke de Bruin wrote:
> Indeed that could be the case. Let's get 1.8.0 out the door so we can
> f
Upon further investigation this was caused by a change in the semantics of
ALL_SUCCESS, which I have these feelings about:
Intuitively you would expect to skip any task that has dependencies that
weren't run by default, i.e. the trigger rule is called ALL_SUCCESS and
skipped tasks are not successfu
Indeed that could be the case. Let's get 1.8.0 out the door so we can focus on
these bug fixes for 1.8.1.
Bolke
Sent from my iPhone
> On 15 Mar 2017, at 18:25, Dan Davydov wrote:
>
> Another issue we are seeing is
> https://issues.apache.org/jira/browse/AIRFLOW-992 - tasks that have both
> s
Another issue we are seeing is
https://issues.apache.org/jira/browse/AIRFLOW-992 - tasks that have both
skipped children and successful children are run instead of skipped. Not
blocking the release on this just letting you guys know for the release bug
notes. We will be cherrypicking a fix for this
Confirmed that Bolke's PR above fixes the issue.
Also, I agree this is not a blocker for the current airflow release, so my
+1 (binding) stands.
-s
On Wed, Mar 15, 2017 at 3:11 PM, Bolke de Bruin wrote:
> PR is available: https://github.com/apache/incubator-airflow/pull/2154
>
> But marked for
PR is available: https://github.com/apache/incubator-airflow/pull/2154
But marked for 1.8.1.
- Bolke
> On 15 Mar 2017, at 14:37, Bolke de Bruin wrote:
>
> On second thought I do consider it a bug and can have a fix out pretty
> quickly, but I don’t consider it a blocker.
>
> - B.
>
>> On 15
On second thought I do consider it a bug and can have a fix out pretty quickly,
but I don’t consider it a blocker.
- B.
> On 15 Mar 2017, at 14:21, Bolke de Bruin wrote:
>
> Just to be clear: Also in 1.7.1 the DagRun was marked successful, but its
> tasks continued to be scheduled. So one cou
Just to be clear: Also in 1.7.1 the DagRun was marked successful, but its tasks
continued to be scheduled. So one could also consider 1.7.1 behaviour a bug. I
am not sure here, but I think it kind of makes sense to consider the behaviour
of 1.7.1 a bug. It has been present throughout all the 1.8
Hi Rui,
We have been discussing this during the hackathon at Airbnb as well. Besides
the reservations Gerard is documenting, I am also not enthusiastic about this
design. Currently, the scheduler is our main issue in scaling. Scheduler runs
will take longer and longer with more DAGs and more co
Here's the JIRA :
https://issues.apache.org/jira/browse/AIRFLOW-989
I confirmed it is a regression from 1.7.1.3, which I installed via pip and
tested against the same DAG in the JIRA.
The issue occurs if a leaf / last / terminal downstream task is not
cleared. You won't see this issue if you clea
FYI: When all root tasks (i.e. the last tasks to run) have succeeded the DagRun
is considered successful and the scheduler will not consider any other tasks in
the dag run. The code is here:
https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L4095
for version 1.8, and has
+1 (binding)
On Wed, Mar 15, 2017 at 9:19 AM, Bolke de Bruin wrote:
> I have asked Sid to create and Jira and to make it reproducible.
> Nevertheless, I do not consider it a blocker as a workaround exists and it
> is relatively small in scope (while slightly annoying I understand that).
>
> Let’
@Sid, does this happen if you clear downstream as well?
On Wed, Mar 15, 2017 at 9:04 AM, Chris Riccomini
wrote:
> Has anyone been able to reproduce Sid's issue?
>
> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin
> wrote:
>
>> That is not an airflow error, but a Kerberos error. Try executing t
I have asked Sid to create and Jira and to make it reproducible. Nevertheless,
I do not consider it a blocker as a workaround exists and it is relatively
small in scope (while slightly annoying I understand that).
Let’s get 1.8 out and do bug fixes in 1.8.1. More bugs will inevitably pop up
:)
Has anyone been able to reproduce Sid's issue?
On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin wrote:
> That is not an airflow error, but a Kerberos error. Try executing the
> kinit command on the command line by yourself.
>
> Bolke
>
> Sent from my iPhone
>
> > On 14 Mar 2017, at 23:11, Ruslan
Hi Rui,
I worked a bit on the scheduler and added some of my comments below.
On Tue, Mar 14, 2017 at 11:08 PM, Rui Wang
wrote:
> Hi,
> The design doc below I created is trying to make airflow scheduler more
> centralized. Briefly speaking, I propose moving state change of
> TaskInstance to sch
17 matches
Mail list logo
