rjgoyln opened a new pull request, #72685: URL: https://github.com/apache/airflow/pull/72685
## Summary `DataprocSubmitJobDirectTrigger`, the `start_from_trigger` path of `DataprocSubmitJobOperator`, learns the job id only from the submit response and cannot write it back into the Trigger row it is inflated from. Two ways to lose a job follow: - a triggerer that restarts while the job is running re-enters `run()` and submits a second Dataproc job, leaving the first untracked - a kill that lands while the submit call is in flight leaves `on_kill()` with no id, so the job it created keeps running ## Change - assign the job id before submitting instead of reading it from the response - default `request_id` to that same id - resume polling on `AlreadyExists`, but only for a job this trigger named - tolerate `NotFound` when cancelling a job that may never have reached Dataproc The id has to be derivable rather than remembered: the Trigger row's kwargs come from the operator's `start_trigger_args`, so nothing the trigger learns at runtime survives its re-creation. The trigger row id identifies one deferral and differs between deployments, so two Airflow instances sharing a Dataproc project do not converge on the same id. A job id or `request_id` the caller supplied is left alone, as in `WorkflowsCreateWorkflowOperator` — idempotency is then theirs to manage. ## Behavior change Jobs submitted through `start_from_trigger` now carry an Airflow-assigned id (`airflow-<hash>`) rather than one generated by Dataproc. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
