[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906857#comment-15906857 ] ASF subversion and git services commented on AIRFLOW-932: - Commit a8f2c27ed9e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906856#comment-15906856 ] ASF subversion and git services commented on AIRFLOW-932: - Commit a8f2c27ed9e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906799#comment-15906799 ] ASF subversion and git services commented on AIRFLOW-932: - Commit a8f2c27ed9e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906798#comment-15906798 ] ASF subversion and git services commented on AIRFLOW-932: - Commit a8f2c27ed9e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906284#comment-15906284 ] ASF subversion and git services commented on AIRFLOW-932: - Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906285#comment-15906285 ] ASF subversion and git services commented on AIRFLOW-932: - Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ] AIRFLOW-932][AIRFLOW-932][AIRFLOW-921][AIRFLOW-910] Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/AIRFLOW-921 > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894640#comment-15894640 ] Bolke de Bruin commented on AIRFLOW-932: PR is available with unit tests [~aoen]. Please review > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893157#comment-15893157 ] Dan Davydov commented on AIRFLOW-932: - If the goal is to eventually move all the logic to the scheduler I think it's OK (but maybe there are some edge cases I'm not considering). It will also simplify the logic/reasoning around the REMOVED state and what could have created it. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893158#comment-15893158 ] Dan Davydov commented on AIRFLOW-932: - Thanks for fixing! > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893151#comment-15893151 ] Bolke de Bruin commented on AIRFLOW-932: It would avoid the issue, but from a dag run integrity perspective that seems off. Nevertheless I will create a PR that avoids the issue, and mark it "to do' for 1.8.1+ . > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893135#comment-15893135 ] Dan Davydov commented on AIRFLOW-932: - I think tasks should only be verified/removed in the scheduler dagrun loop. That would avoid this issue if I understand correctly. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[
https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893124#comment-15893124
]
Bolke de Bruin commented on AIRFLOW-932:
Issue resides in cli.py and will only happen when a specific task is used:
{code}
if args.task_regex:
dag = dag.sub_dag(
task_regex=args.task_regex,
include_upstream=not args.ignore_dependencies)
{code}
This creates a subset of the tasks from a dag_run, with the same name as the
original. Hence it will set a task to removed if you verify the integrity of a
dag run.
get_task_instances picks up all instances, including "removed' ones from the
original (whole) dag and this is not filtered in the backfill. Hence the lists
mismatch and an AirflowException is thrown.
The quick and dirty fix is to not mark removed if run from backfill and filter
the list of "get_task_instances".
However the functionality if sub_dag is awkward imho and might need a real fix.
What do you think?
> Backfills delete existing task instances and mark them as removed
> -
>
> Key: AIRFLOW-932
> URL: https://issues.apache.org/jira/browse/AIRFLOW-932
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: backfill
>Reporter: Dan Davydov
>Priority: Blocker
>
> I'm still investigating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893083#comment-15893083 ] Bolke de Bruin commented on AIRFLOW-932: Ok. Confirmed now - really strange. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892823#comment-15892823 ] Dan Davydov commented on AIRFLOW-932: - Ah I think I know why it's not reproducing for you. Can you try letting the scheduler run a dagrun, and then backfill the dagrun once it completes? > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892818#comment-15892818 ] Dan Davydov commented on AIRFLOW-932: - This is off of master. I just reproduced again (you are right about the start date though, should have been 2016). > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[
https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891767#comment-15891767
]
Bolke de Bruin commented on AIRFLOW-932:
I just ran this on my AIRFLOW-910 branch and I am not seeing this behaviour. I
did change your execution date to 2016 as 2017 won't run.
{code}
MariaDB [airflow]> select * from task_instance;
++--++++--+-++--+--++--+-+-++-+---+
| task_id| dag_id |
execution_date | start_date | end_date
| duration | state | try_number | hostname | unixname |
job_id | pool | queue | priority_weight | operator | queued_dttm | pid
|
++--++++--+-++--+--++--+-+-++-+---+
| create_and_transfer_roster | teztg| 2016-06-14
00:00:00.00 | 2017-03-02 08:58:35.867219 | 2017-03-02 08:58:46.868470 |
11.0013 | success | 1 | bolkes-macbook-pro.local | bolke| 2 |
NULL | default | 1 | SubDagOperator | NULL| 46856 |
| create_roster | teztg.create_and_transfer_roster | 2016-06-14
00:00:00.00 | 2017-03-02 08:58:42.681159 | 2017-03-02 08:58:42.702009 |
0.02085 | success | 1 | bolkes-macbook-pro.local | bolke| 4 |
NULL | default | 1 | PythonOperator | NULL| 46862 |
++--++++--+-++--+--++--+-+-++-+---+
{code}
{code}
[2017-03-02 08:58:29,039] {jobs.py:1756} INFO - Checking run
[2017-03-02 08:58:29,044] {models.py:1125} INFO - Dependencies all met for
[2017-03-02 08:58:29,047] {base_executor.py:50} INFO - Adding to queue: airflow
run teztg create_and_transfer_roster 2016-06-14T00:00:00 --local -sd
DAGS_FOLDER/task_removed_bug.py
[2017-03-02 08:58:33,985] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:33,987] {local_executor.py:45} INFO - LocalWorker running
airflow run teztg create_and_transfer_roster 2016-06-14T00:00:00 --local -sd
DAGS_FOLDER/task_removed_bug.py
[2017-03-02 08:58:33,989] {models.py:1125} INFO - Dependencies all met for
/Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/configuration.py:548:
DeprecationWarning: This method will be removed in future versions. Use
'parser.read_file()' instead.
self.readfp(StringIO.StringIO(string))
[2017-03-02 08:58:34,561] {__init__.py:56} INFO - Using executor LocalExecutor
Logging into:
/Users/bolke/airflow/logs/teztg/create_and_transfer_roster/2016-06-14T00:00:00
[2017-03-02 08:58:38,987] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:38,990] {models.py:1119} INFO - Dependencies not met for
,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2017-03-02 08:58:38,990] {models.py:1119} INFO - Dependencies not met for
,
dependency 'Task Instance Not Already Running' FAILED: Task is already running,
it started on 2017-03-02 08:58:35.867219.
[2017-03-02 08:58:43,994] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:43,997] {models.py:1119} INFO - Dependencies not met for
,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2017-03-02 08:58:43,997] {models.py:1119} INFO - Dependencies not met for
,
dependency 'Task Instance Not Already Running' FAILED: Task is already running,
it started on 2017-03-02 08:58:35.867219.
[2017-03-02 08:58:49,001] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:54,007] {jo
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891763#comment-15891763 ] Bolke de Bruin commented on AIRFLOW-932: I'll have a look and see if I can reproduce it. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[
https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891251#comment-15891251
]
Dan Davydov commented on AIRFLOW-932:
-
Command:
{quote}airflow backfill teztg -t create_and_transfer_roster -s 2017-06-14 -e
2017-06-14{quote}
DAG:
{quote}
import airflow
from airflow.executors import SequentialExecutor
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.operators.subdag_operator import SubDagOperator
from airflow.models import DAG
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'start_date': datetime(2016, 3, 15),
}
dag = DAG('teztg', default_args=default_args)
subdag = DAG('teztg.create_and_transfer_roster', default_args=default_args)
subdag_operator = SubDagOperator(task_id='create_and_transfer_roster',
subdag=subdag,
executor=SequentialExecutor(),
dag=dag)
def create_roster(ds, macros, **kwargs):
pass
PythonOperator(
task_id='create_roster',
python_callable=create_roster,
dag=subdag,
provide_context=True)
first_task = DummyOperator(task_id="first_task", dag=dag)
{quote}
Expected:
first_task TI does not change (stays as none)
create_and_transfer_roster TI is run
Actual:
first_task TI gets its state set to removed
create_and_transfer_roster TI is not run, backfill errors with
airflow.exceptions.AirflowException: Task first_task not found
[~bolke] if you think these are related to your changes (addition of
task.REMOVED state/backfill changes) do you mind taking a look? Otherwise let
me know and I can do so.
> Backfills delete existing task instances and mark them as removed
> -
>
> Key: AIRFLOW-932
> URL: https://issues.apache.org/jira/browse/AIRFLOW-932
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: backfill
>Reporter: Dan Davydov
>Priority: Blocker
>
> I'm still investigating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890822#comment-15890822 ] Dan Davydov commented on AIRFLOW-932: - I'm still investigating and will update the ticket with more details as soon as I am able to. My first guess is that the regex field of backfills makes Airflow think that a task doesn't exist anymore if it's not included in the regex or something like that. This is a blocker because it overwrites existing task instances that were e.g. successful with the removed state. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890810#comment-15890810 ] Bolke de Bruin commented on AIRFLOW-932: verify_integrity in models.py is where you would like to look at. As mentioned it sets a state to REMOVED if a task instance does not have a corresponding task in a DAG. It is there for auditing and versioning reasons. Does it affect running anything? A changed DAG cannot have a dependency on a non existent task. > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
