Hi All, Through this email, I am trying to direct you to a better pattern to reduce your operational overhead of having to maintain some custom piece of code that you own across airflow upgrades.
I came across a few users on r/apache_airflow <https://www.reddit.com/r/apache_airflow/comments/1vatr6h/looking_for_feedback_is_anyone_using_task_state/> who are running custom submit + sensor task groups for external jobs such as dbt cloud / databricks / snowflake jobs: one task submits, a separate task polls for completion. It got me curious and I built the same pattern for Databricks and want to share why task state store (AIP-103, shipped in 3.3) is the better fit. My setup looked like this: submit = DatabricksSubmitRunOperator( task_id="submit", wait_for_termination=False, tasks=[...], ) # custom sensor I wrote sense = DatabricksRunSensor( task_id="sense", run_id=submit.output["run_id"], poke_interval=15, mode="reschedule", ) submit >> sense It works, but it has two real problems. a) (Semantic) The submit task tells you nothing about whether the job succeeded, only that it started, you have to look at sense for the actual outcome. b) The sensor's own timeout runs on a clock separate from the job itself, so it can fail the task even after the job completed successfully underneath it. Task state store collapses this into one task: submit = DatabricksSubmitRunOperator( task_id="submit_and_wait", tasks=[...], durable=True, retries=3, ) durable=True writes the run id to task state store the moment the job exists, before the task itself completes. If the worker dies right after submission, a retry reads that stored id back and reconnects to the same run instead of resubmitting. And because the whole submit-and-wait sequence lives in one task, its final status is the job's real outcome. Retries are ordinary retries, each one re-checks the actual job status from the source of truth rather than tracking a separate poke/timeout state. Net result: - one task instead of two, - no custom sensor to maintain, - and the crash-safety guarantee. If you are running a submit+sensor setup like mine for any provider, I would like to know whether this covers your case or whether something is still missing. (feel free to ping me on Airflow Slack and we can chat) Thanks & Regards, Amogh Desai
