amoghrajesh commented on code in PR #76295:
URL: https://github.com/apache/airflow/pull/76295#discussion_r4509221042
##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -2020,15 +2029,24 @@ def _get_run_json(self) -> dict[str, Any]:
def _launch_job(self, context: Context | None = None) -> int | None:
"""Launch the job on Databricks."""
run_json = self._get_run_json()
+ # Return as soon as the run exists. A page URL lookup or an xcom push
here would run
+ # before the mixin can persist the run id, so a failure would strand
the run and let the
+ # retry submit a duplicate.
self.databricks_run_id = self._hook.submit_run(run_json)
Review Comment:
Agreed, following this up separately. Confirmed `_get_run_json` never sets a
token and `DatabricksTaskBaseOperator` doesn't expose the parameter, and
`_retryable_error` does
POST on `Timeout` / `ConnectionError` / 5xx, so the duplicate run window is
real as mentioned.
I also think `map_index` has to be in the token too, or every instance of a
mapped task shares one run; and the 64 character limit means the whole key has
to be hashed rather than concatenated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]