Antalares commented on issue #72386:
URL: https://github.com/apache/airflow/issues/72386#issuecomment-5814153772
My main use case is when the DAG run is still in progress and is blocked
waiting for an input that we already know will never arrive.
For example, imagine a DAG that processes a set of input files:
file_A → ingestion_A ┐
file_B → ingestion_B ├→ transformations → ...
file_C → ingestion_C ┘
The DAG is currently waiting for "file_B" to arrive.
Now imagine a production incident has occurred and we know that "file_B"
will not be delivered for this particular run.
As an operator, I would like to be able to manually mark "ingestion_B" as
"SKIPPED" so that the DAG can continue:
ingestion_A → SUCCESS
ingestion_B → SKIPPED ← manually bypassed
ingestion_C → SUCCESS
↓
transformations
The important point is that I don't want to mark the task as "SUCCESS",
because that would incorrectly indicate that the ingestion actually took place.
I also don't want to simply clear/remove the task instance, because I want
the execution history to retain the information that this particular input was
intentionally bypassed as part of the incident recovery.
So the use case is essentially:
1. DAG run is active and waiting for an input.
2. A production incident makes it known that the input will never arrive.
3. The operator manually marks the corresponding task as "SKIPPED".
4. The DAG can proceed according to its normal dependency/trigger-rule
semantics.
5. The Airflow UI/history clearly shows that this task was intentionally
skipped rather than successfully executed.
I agree that marking an already completed task as "SKIPPED" after the fact
would be a different use case, and I don't think that is particularly useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
