seanghaeli commented on PR #73702: URL: https://github.com/apache/airflow/pull/73702#issuecomment-6226541701
On my machine I'm seeing a problem arise with this PR when a deferred task is cleared. `clear` instructs the task to run again, and on 3.3+ the re-run looks for an existing Glue job from the same task. It finds the old one still running, and decides to just wait on that one. But clearing also triggers this PR's "on kill" step, which stops that same old Glue job. So the re-run ends up waiting on a stopped job and fails with `STOPPED`. On main, the re-run reattached and succeeded. Could the re-run start a fresh Glue job when `stop_job_run_on_kill=True`, instead of reattaching? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
