SEPURI-SAI-KRISHNA commented on issue #59120:
URL: https://github.com/apache/airflow/issues/59120#issuecomment-5929969532
`_create_dag_runs` runs inside `prohibit_commit`: `_do_scheduling` opens the
guard and calls `_create_dagruns_for_dags(guard, session)`, which calls
`_create_dag_runs`. On main today that is `scheduler_job_runner.py` 2107, 2109
and 2583. `CommitProhibitorGuard` listens on `before_commit`, and SQLAlchemy
dispatches that for nested transactions too:
```python
if self._parent is None or self.nested:
self.session.dispatch.before_commit(self.session)
```
so releasing the savepoint raises `RuntimeError: UNEXPECTED COMMIT - THIS
WILL BREAK HA LOCKS!`. Re-checked on SQLAlchemy 2.0.52.
`session.connection().begin_nested()` sidesteps the Session event layer, but
mixing connection-level savepoints with ORM flush leaves the Session in
`PendingRollbackError` after the first failure, so that is also a dead end.
What works is returning early from `_validate_commit` when
`session.in_nested_transaction()`. Releasing a savepoint neither ends the outer
transaction nor drops row locks, so it cannot break the HA locks the guard
protects, and a real commit inside the guard is still rejected. Full writeup,
including the loop pattern and a test result:
https://github.com/apache/airflow/issues/59120#issuecomment-5077987321
Two things are unsettled. It changes `CommitProhibitorGuard`, a core
primitive rather than anything local to this loop, and whether that is
acceptable is still unanswered on this issue. And verification was on SQLite,
which does not reproduce the Postgres aborted-transaction state, so the loop
mechanics are established but the failure mode in the description is not
confirmed fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
