[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578536#comment-16578536
]
Brian Hann commented on AIRFLOW-184:
{quote}Is this issue still relevant?
{quote}
Yes. Clearing failed tasks through the CLI does not clear downstream tasks that
are in "upstream_failed" state.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>Assignee: Joy Gao
>Priority: Major
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386953#comment-16386953 ] Ash Berlin-Taylor commented on AIRFLOW-184: --- Is this issue still relevant? > Add clear/mark success to CLI > - > > Key: AIRFLOW-184 > URL: https://issues.apache.org/jira/browse/AIRFLOW-184 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Reporter: Chris Riccomini >Assignee: Joy Gao >Priority: Major > > AIRFLOW-177 pointed out that the current CLI does not allow us to clear or > mark success a task (including upstream, downstream, past, future, and > recursive) the way that the UI widget does. Given a goal of keeping parity > between the UI and CLI, it seems like we should support this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317223#comment-15317223
]
Chris Riccomini commented on AIRFLOW-184:
-
{quote}
Ideally, this should need to be queued indeed
{quote}
Agreed, but would rather first get the CLI and UI in sync. Once they are the
same, we can think about how we make things more async/scalable.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>Assignee: Joy Gao
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317196#comment-15317196 ] Arthur Wiedmer commented on AIRFLOW-184: Sounds good to me. Ideally, this should need to be queued indeed. Should the commands mark_success just be a wrapper around a more general set_state? Marking large swath of tasks as success is a pain in the ui, and the backfill with regex matching was useful for this. But I agree that it does not make sense anymore and should be refactored into something more useful + that does not go through the scheduler as it is a waste of slots. > Add clear/mark success to CLI > - > > Key: AIRFLOW-184 > URL: https://issues.apache.org/jira/browse/AIRFLOW-184 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Reporter: Chris Riccomini >Assignee: Joy Gao > > AIRFLOW-177 pointed out that the current CLI does not allow us to clear or > mark success a task (including upstream, downstream, past, future, and > recursive) the way that the UI widget does. Given a goal of keeping parity > between the UI and CLI, it seems like we should support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317184#comment-15317184
]
Chris Riccomini commented on AIRFLOW-184:
-
{quote}
I don't think the command even needs to interact with workers at all like the
current way backfill + mark_success works
{quote}
+1
{quote}
I think it is ok to break backwards incompatibility here without bumping to a
major release risk-wise but I think 1-2 other committers should +1 this since
it's a bit of a grey area.
{quote}
Sounds good. So far we've got your +1 and mine. Could you do a quick ping from
[~artwr] or [~maxime.beauche...@apache.org]? And we can get one of [~bolke] or
[~jlowin] to +1.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>Assignee: Joy Gao
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317128#comment-15317128 ] Dan Davydov commented on AIRFLOW-184: - I agree that mark_success should be moved into a separate command. I don't think the command even needs to interact with workers at all like the current way backfill + mark_success works, and we could also simplify a lot of the code. I think it is ok to break backwards incompatibility here without bumping to a major release risk-wise but I think 1-2 other committers should +1 this since it's a bit of a grey area. Otherwise we can add the new --mark-success command, deprecate backfill --mark-success and break compatibility in 2.0. > Add clear/mark success to CLI > - > > Key: AIRFLOW-184 > URL: https://issues.apache.org/jira/browse/AIRFLOW-184 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Reporter: Chris Riccomini >Assignee: Joy Gao > > AIRFLOW-177 pointed out that the current CLI does not allow us to clear or > mark success a task (including upstream, downstream, past, future, and > recursive) the way that the UI widget does. Given a goal of keeping parity > between the UI and CLI, it seems like we should support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316887#comment-15316887
]
Chris Riccomini commented on AIRFLOW-184:
-
{quote}
As for the --only_failed flag, personally it is more intuitive to treat
'upstream_failed' the same as 'failed' and have those tasks cleared as well
(right now it is ignored).
{quote}
IMO, if we're shooting for parity with the UI, we should handle the state
change regardless of what the upstream/downstream states are. I believe this is
how the UI works. If I set "downstream", then it will clear all downstream
states of the task(s), regardless of whether they're upstream_failed, success,
failed, up_for_retry, or whatever.
{quote}
Just noticed that 'airflow backfill' and 'airflow run' both have a
--mark_success flag which can be used to set Task Instances success just like
the UI widget.
{quote}
I'm not a huge fan of the way that backfill works. I'd prefer to remove the
ability for backfill to `mark success`, and just have a single place in the CLI
to do this. [~jlowin]/[~bolke] given that the UI currently fiddles with task
state independent of the scheduler, is it OK to do this via the UI as well (vs
going through the scheduler, and skipping actual execution to mark state
changes).
My current vote is to:
# Remove the ability to mark success via backfill.
# Make `airflow clear` have parity with the web UI, which means supporting
upstream/downstream/recursive the way that the UI does.
I'm not so concerned about past/present now, given that you can specify date
ranges via
Note: (1) is backwards incompatible. I suspect the only people who might care
about this are the AirBNB folks. [~maxime.beauche...@apache.org], [~pauly],
[~aoen].
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>Assignee: Joy Gao
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316177#comment-15316177 ] Joy Gao commented on AIRFLOW-184: - Just noticed that 'airflow backfill' and 'airflow run' both have a --mark_success flag which can be used to set Task Instances success just like the UI widget. > Add clear/mark success to CLI > - > > Key: AIRFLOW-184 > URL: https://issues.apache.org/jira/browse/AIRFLOW-184 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Reporter: Chris Riccomini >Assignee: Joy Gao > > AIRFLOW-177 pointed out that the current CLI does not allow us to clear or > mark success a task (including upstream, downstream, past, future, and > recursive) the way that the UI widget does. Given a goal of keeping parity > between the UI and CLI, it seems like we should support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315728#comment-15315728 ] Joy Gao commented on AIRFLOW-184: - Agree with point above. The Airflow UI provides the visual granularity that makes selecting a specific Task Instance at a point in time easy, and consequently ‘future’ and ‘past’ are relative to the TI selected. The CLI for 'clear' does offer past/future functionalities (recursive is easy to add). Omitting start_date/end_date is the equivalent of selecting past/future. This is not very intuitive, but there doesn't seem to be a good way to select a specific Task Instance other than pinning it down with start_date and end_date (which may be confused with the started/ended timestamp of the job being run, when in reality it is referring to the execution_date). AFAIK, the only unique identifier for a task instance is the job_id, but that is incrementally assigned by the job scheduler from the job queue and is global to all jobs in Airflow. As for the --only_failed flag, personally it is more intuitive to treat 'upstream_failed' the same as 'failed' and have those tasks cleared as well (right now it is ignored). > Add clear/mark success to CLI > - > > Key: AIRFLOW-184 > URL: https://issues.apache.org/jira/browse/AIRFLOW-184 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Reporter: Chris Riccomini >Assignee: Joy Gao > > AIRFLOW-177 pointed out that the current CLI does not allow us to clear or > mark success a task (including upstream, downstream, past, future, and > recursive) the way that the UI widget does. Given a goal of keeping parity > between the UI and CLI, it seems like we should support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307235#comment-15307235
]
Sumit Maheshwari commented on AIRFLOW-184:
--
Was checking {{ airflow clear }} and it seems like its doable by saying {{
airflow -f -s start_date -e end_date dag_id }} and cleverly choosing start_date
and end_date at the boundaries of desire execution date, which is tricky and
risky for automated scripts.
Also seems that it clears task instances with {{ failed }} state only, not sure
how does it affects instances with state {{ upstream_failed }}.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
