Cool... and maybe an issue to make an inventory of those :) ? On Fri, Jul 31, 2026 at 4:08 PM Kaxil Naik <[email protected]> wrote:
> Yup, Amogh already did that 3 weeks ago :) > https://github.com/astronomer/agents/pull/222 > > On Fri, 31 Jul 2026 at 15:56, Jarek Potiuk <[email protected]> wrote: > > > Suggestion: > > > > Maybe we should add agentic suggestions / SKILLS to do it and even run > some > > agentic checks and create an issue pointing to some of those? This might > > then kick in when either author or reviewer use agents to do what they > are > > doing. > > I think those are far more effective nowadays than relying on people a) > > reading it b) remembering that it's there, or c) implementing it :) > > > > J. > > > > On Fri, Jul 31, 2026 at 8:30 AM Amogh Desai <[email protected]> > wrote: > > > > > Hi All, > > > > > > Through this email, I am trying to direct you to a better pattern to > > reduce > > > your operational overhead of having to > > > maintain some custom piece of code that you own across airflow > upgrades. > > > > > > I came across a few users on r/apache_airflow > > > < > > > > > > https://www.reddit.com/r/apache_airflow/comments/1vatr6h/looking_for_feedback_is_anyone_using_task_state/ > > > > > > > who > > > are running custom submit + sensor task groups for > > > external jobs such as dbt cloud / databricks / snowflake jobs: one task > > > submits, a separate task polls for completion. > > > > > > It got me curious and I built the same pattern for Databricks and want > to > > > share why task state store (AIP-103, shipped in 3.3) > > > is the better fit. > > > > > > My setup looked like this: > > > > > > submit = DatabricksSubmitRunOperator( > > > task_id="submit", > > > wait_for_termination=False, > > > tasks=[...], > > > ) > > > # custom sensor I wrote > > > sense = DatabricksRunSensor( > > > task_id="sense", > > > run_id=submit.output["run_id"], > > > poke_interval=15, > > > mode="reschedule", > > > ) > > > submit >> sense > > > > > > It works, but it has two real problems. > > > > > > a) (Semantic) The submit task tells you nothing about whether the job > > > succeeded, only that it started, you have to look at sense > > > for the actual outcome. > > > b) The sensor's own timeout runs on a clock separate from the job > itself, > > > so it can fail the task even after the job completed successfully > > > underneath it. > > > > > > Task state store collapses this into one task: > > > > > > submit = DatabricksSubmitRunOperator( > > > task_id="submit_and_wait", > > > tasks=[...], > > > durable=True, > > > retries=3, > > > ) > > > > > > durable=True writes the run id to task state store the moment the job > > > exists, before the task itself completes. If the worker dies right > after > > > submission, a retry reads that stored id back and reconnects to the > same > > > run instead of resubmitting. And because the whole submit-and-wait > > > sequence lives in one task, its final status is the job's real outcome. > > > Retries are ordinary retries, each one re-checks the actual job status > > from > > > the > > > source of truth rather than tracking a separate poke/timeout state. > > > > > > Net result: > > > - one task instead of two, > > > - no custom sensor to maintain, > > > - and the crash-safety guarantee. > > > > > > If you are running a submit+sensor setup like mine for any provider, I > > > would like to know whether this covers your case or whether something > is > > > still missing. (feel free to ping me on Airflow Slack and we can chat) > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > >
