Suggestion: Maybe we should add agentic suggestions / SKILLS to do it and even run some agentic checks and create an issue pointing to some of those? This might then kick in when either author or reviewer use agents to do what they are doing. I think those are far more effective nowadays than relying on people a) reading it b) remembering that it's there, or c) implementing it :)
J. On Fri, Jul 31, 2026 at 8:30 AM Amogh Desai <[email protected]> wrote: > Hi All, > > Through this email, I am trying to direct you to a better pattern to reduce > your operational overhead of having to > maintain some custom piece of code that you own across airflow upgrades. > > I came across a few users on r/apache_airflow > < > https://www.reddit.com/r/apache_airflow/comments/1vatr6h/looking_for_feedback_is_anyone_using_task_state/ > > > who > are running custom submit + sensor task groups for > external jobs such as dbt cloud / databricks / snowflake jobs: one task > submits, a separate task polls for completion. > > It got me curious and I built the same pattern for Databricks and want to > share why task state store (AIP-103, shipped in 3.3) > is the better fit. > > My setup looked like this: > > submit = DatabricksSubmitRunOperator( > task_id="submit", > wait_for_termination=False, > tasks=[...], > ) > # custom sensor I wrote > sense = DatabricksRunSensor( > task_id="sense", > run_id=submit.output["run_id"], > poke_interval=15, > mode="reschedule", > ) > submit >> sense > > It works, but it has two real problems. > > a) (Semantic) The submit task tells you nothing about whether the job > succeeded, only that it started, you have to look at sense > for the actual outcome. > b) The sensor's own timeout runs on a clock separate from the job itself, > so it can fail the task even after the job completed successfully > underneath it. > > Task state store collapses this into one task: > > submit = DatabricksSubmitRunOperator( > task_id="submit_and_wait", > tasks=[...], > durable=True, > retries=3, > ) > > durable=True writes the run id to task state store the moment the job > exists, before the task itself completes. If the worker dies right after > submission, a retry reads that stored id back and reconnects to the same > run instead of resubmitting. And because the whole submit-and-wait > sequence lives in one task, its final status is the job's real outcome. > Retries are ordinary retries, each one re-checks the actual job status from > the > source of truth rather than tracking a separate poke/timeout state. > > Net result: > - one task instead of two, > - no custom sensor to maintain, > - and the crash-safety guarantee. > > If you are running a submit+sensor setup like mine for any provider, I > would like to know whether this covers your case or whether something is > still missing. (feel free to ping me on Airflow Slack and we can chat) > > Thanks & Regards, > Amogh Desai >
