Hi Shivam, Elad, Thanks for the proposal and for the demo. I read the AIP and the comments from Niko and Dheeraj on the wiki. My feedback is below; I can copy the detailed points to the wiki as inline comments if that is easier for you.
Summary: I am in favour of the task instance hold, built the way you propose (it only blocks the start, it is enforced in the scheduler, and there is no new state). I would leave the task hold across all runs out of the first version, and I would rebuild the motivation. 1. The Motivation The three examples can each be solved today when the Dag is designed for it: - Bad data: validate the data, emit an Asset only when the check passes, and let a second Dag do the publishing. When the check fails there is no Asset event, the publishing Dag never starts and there is nothing to hold. For bad data that the check missed, pausing the publishing Dag does the same job. - Vendor outage: give the external system its own pool and set it to 0 slots. The task stays in scheduled, its state is untouched and everything else runs. That is the behaviour the AIP describes for a task hold. - Wrong run: put a human-in-the-loop task (AIP-90) before the irreversible step. So these examples read more as an argument for better Dag design than for a new feature, which is close to what Niko asks in his first comments. What is left is the unplanned case, and I think it is a real one: - A Dag you operate but did not write. A platform team on call for the Dags of other teams cannot split a Dag or add a pool during an incident, because both need a deploy by the owner. - One run out of many. Pausing a Dag and emptying a pool act on every run. Only a task instance hold stops the 09:00 run while the 10:00 run and a backfill continue. I would build the Motivation on these two. 2. Scope: task instance hold only To your question whether the two scopes are right: I agree with Niko and Dheeraj to start with the task instance hold. My extra reason is that the task hold behaves like a 0-slot pool with a reason and an end time. The AIP sets pools aside in one line; it would help to say what a task hold adds beyond that. 3. A hold cannot be placed before clearing a finished run The AIP says a task instance hold ends when its run finishes and stays ended if the run is cleared. Problems in a run are often noticed after it has finished. Then I have to clear first and place the hold second, while the scheduler is already picking up the cleared tasks. Could clearing take a "hold these tasks" option, or could a hold placed on a finished run stay dormant until the task is cleared? 4. Attempts that are already queued To your other question, whether a hold should only block the start: yes, but the cost is higher than it looks. Holds are placed during incidents, which is when the executor backlog is deepest, so most attempts get through exactly then. Did you consider a third check in the Execution API, when the worker asks to move the task from queued to running? It does not depend on the executor, and that endpoint already answers 409 when the task is not in the state it expects. If you considered and dropped it, the reason would fit under "Rejected alternatives". 5. Mapped tasks and task loops The design sections do not mention mapped tasks. Does a hold on a mapped task stop the expansion, or does the task expand and the expanded task instances wait? Does a hold also cover map indexes created after it was placed? The same question decides what a hold does to a task loop (AIP-111, in 3.4). The AIP lists loops under "Related features" as something that keeps working, but a loop is the case where it matters most: every iteration is its own task instance, created through the mapped-task expansion and started in order by the scheduler. So: - A hold on a task inside a loop covers all its iterations. The loop then stops before the next pass and the passes that already ran are kept. Is that the intended behaviour? Holding one given iteration would need the hold per map index that is listed as future work. - Can the synthesized gate task and completion node be held? Holding the gate is a way to stop a loop after the current pass. - Clearing an iteration re-opens the iterations after it. Does a hold that was placed before the clear still apply to them? I think this is a good use case for the task instance hold: stopping a refinement loop between two passes to look at the result, without a human-in-the-loop step planned in the Dag. It would fit in the Motivation. Task iteration (AIP-104) and task steps (AIP-112) are the simple cases and one line each is enough: the items and the steps run inside one task instance on the worker, so a hold blocks the start of the whole task and never acts between items or steps. 6. Expiry A hold can expire at its end time, and the expiry gets an audit log entry. Which component ends the hold and writes that entry, and how is it written once when several schedulers run? Best regards, David ________________________________ From: [email protected] <[email protected]> Sent: Thursday, November 05, 2026 01:54 To: [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: [DISCUSS] AIP-128: Task/TaskInstance Hold EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto:[email protected]>. Hi all, Elad and I would like to start the discussion on AIP-128, which proposes introducing task holds in Airflow. Users sometimes need one task to wait while unrelated work continues—for example, while correcting bad data or waiting for a vendor API to recover. A hold lets them prevent that task from starting without pausing the whole Dag. AIP: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FAIRFLOW%2Fpages%2F462815942%2FAIP-128%2BTask%2BTaskInstance%2BHold&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069508792%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9PmV937Pams2fBP9CxuX5xgcfhU6Byag45pVYWwjkY8%3D&reserved=0<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462815942/AIP-128+Task+TaskInstance+Hold> Demo (1.5 min): https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdownload%2Fattachments%2F462815942%2Ftask-holds-demo-short.mp4%3Fapi%3Dv2&data=05%7C02%7Cdavid.blain%40infrabel.be%7C30a3ee14a3d544ddd58808df3b130743%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639294625069537794%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mWnu%2FCu9NSqPQ2OQdAmM8L8dzUdELbKxIjqPcqPmZ0s%3D&reserved=0<https://cwiki.apache.org/confluence/download/attachments/462815942/task-holds-demo-short.mp4?api=v2> The proposal supports two kinds of hold: - Task instance hold: one task in one run, for example publish_report in today's run while bad data is fixed. - Task hold: one task in every run, for example while a vendor API is unavailable. The scheduler checks holds before picking tasks to start. An attempt it has already picked may still run. A hold doesn't change the task's state; other state changes, such as upstream failure, happen as usual. Once all holds end, normal scheduling can continue. Every hold has a reason, an optional end time and an audit log entry. We have a working prototype, shown in the video. The AIP explains the key design decisions and their trade-offs. We'd particularly welcome feedback on whether a hold should only block a task's start, and whether the two proposed scopes are sufficient for a first version. Please comment on the AIP doc, or reply here. Thanks, Shivam --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] General (Internal Property)
