amoghrajesh commented on code in PR #76295: URL: https://github.com/apache/airflow/pull/76295#discussion_r4546215952
########## providers/databricks/docs/operators/notebook.rst: ########## @@ -80,3 +82,57 @@ attempt is observed, so downstream failure handling is not delayed by long-runni Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt under the same ``task_key`` until then. Sibling tasks in the run continue independently. + + +Durable execution +----------------- + +Outside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +this operator submits a run and then waits for it, on the worker or in the Triggerer when +``deferrable=True``. By default it runs in a *durable* mode: the Databricks run id is persisted to +:doc:`task state store <apache-airflow:core-concepts/task-state-store>` before the wait begins, so if +the worker crashes or is preempted and the task is retried, the operator reconnects to the run that +is already executing on Databricks instead of submitting a duplicate. + +On retry the operator checks the prior run's state: + +* if it is still running, the operator reconnects and continues polling +* if it already succeeded, the operator returns immediately without resubmitting +* if it failed terminally, the operator submits a fresh run + +Pass ``durable=False`` to always submit a fresh run on retry. + +Unlike :class:`~airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator` and +:class:`~airflow.providers.databricks.operators.databricks.DatabricksRunNowOperator`, where +``durable`` has no effect in deferrable mode, durable execution here also covers +``deferrable=True``: the run id is stored before the task defers, so a try lost while the Triggerer +was waiting still reconnects on the next one. + +Durable execution requires Airflow 3.3 or newer, since it relies on the task state store. On +earlier Airflow versions the flag is a no-op and the operator always submits a fresh run on retry. +If the task state store is unavailable at runtime, the operator logs that crash recovery is +disabled and behaves the same way. + +Inside a ``DatabricksWorkflowTaskGroup`` nothing changes: the run is submitted by the workflow +launch task and this operator only monitors its own task within that shared run, so a retry already +reconnects rather than resubmitting. + +The stored run id is not deleted automatically; that only happens when someone runs +``airflow state-store clean``. If a task's ``retry_delay`` is longer than +``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run +id is gone by the next retry and the operator submits a fresh run instead of reconnecting. Avoid +running cleanup on a schedule shorter than your longest ``retry_delay``. See +:doc:`apache-airflow:administration-and-deployment/task-and-asset-state-store-cleanup`. + +Which clear you use decides whether the stored run id survives. Clearing a single task from the Review Comment: Handled in [kaxil review](https://github.com/apache/airflow/pull/76295/commits/db20dcae1f6eb55b451ef26a33f4861641d0793f) ########## providers/databricks/src/airflow/providers/databricks/operators/databricks.py: ########## @@ -82,6 +82,62 @@ XCOM_RUN_PAGE_URL_KEY = "run_page_url" XCOM_STATEMENT_ID_KEY = "statement_id" +# Databricks splits run state across life_cycle_state and result_state, while the ResumableJobMixin +# status interface is a single string. The durable operators encode both as "LIFE_CYCLE:RESULT" and +# decode them with the predicates below, so all three share one view of which runs can be rejoined. +RUN_NOT_FOUND_STATUS = "NOT_FOUND:" +TERMINATING_SETTLE_TIMEOUT_SECONDS = 60 +ACTIVE_RUN_LIFE_CYCLE_STATES = ( + "PENDING", + "QUEUED", + "RUNNING", + "TERMINATING", + "BLOCKED", + "WAITING_FOR_RETRY", +) + + +def fetch_run_status( + hook: DatabricksHook, + run_id: JsonValue, + *, + poll_interval: float = 5, + settle_timeout: float = TERMINATING_SETTLE_TIMEOUT_SECONDS, +) -> str: + """ + Return the encoded status of ``run_id``, or a sentinel if its run history is gone. + + ``TERMINATING`` is waited out because it does not say whether the run is finishing on its own or + being cancelled, and ``on_kill`` cancels asynchronously without clearing the stored id. On + timeout it is reported as-is, which counts as active: rejoining a slow run beats duplicating it. + """ + deadline = time.monotonic() + settle_timeout + while True: + try: + # The stored external id is always a Databricks run id; the mixin types it as JsonValue. + run_info = hook.get_run(cast("int", run_id)) + except DatabricksApiError as e: + # A stored run whose history expired (or was deleted) returns 404. The sentinel is neither + # active nor succeeded, so the mixin resubmits fresh instead of failing the task. + if e.http_status_code == 404: + return RUN_NOT_FOUND_STATUS + raise + state = RunState(**run_info["state"]) + if state.life_cycle_state != "TERMINATING" or time.monotonic() >= deadline: + return f"{state.life_cycle_state}:{state.result_state}" + time.sleep(poll_interval) Review Comment: Handled in [kaxil review](https://github.com/apache/airflow/pull/76295/commits/db20dcae1f6eb55b451ef26a33f4861641d0793f) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
