kjh0623 opened a new pull request, #70002:
URL: https://github.com/apache/airflow/pull/70002
With multiple dag processors running in HA (e.g. `replicas: 2`), two
processors can parse the same file concurrently: both read the (empty or stale)
`DagModel.tags` relationship, then both INSERT the same `dag_tag` row. The
plain ORM INSERT raises a unique-constraint violation
(`psycopg2.errors.UniqueViolation: duplicate key value violates unique
constraint "dag_tag_pkey"`), which poisons the session, surfaces as
`PendingRollbackError` on the retry, and crashes the dag processor.
We hit this in production while running two dag processors over ~2,900 DAGs
and have been running this fix as an internal patch since; this upstreams it.
**Fix**
- `_update_dag_tags` now inserts tags with a dialect-aware conflict-ignoring
statement — `ON CONFLICT DO NOTHING` on PostgreSQL/SQLite, self-assign `ON
DUPLICATE KEY UPDATE` on MySQL — following the existing pattern used for
`AssetActive` in the same module (`activate_assets_if_possible`).
- Pending tag DELETEs are flushed before the INSERT. This also guarantees
deletes hit the database before inserts for case-only renames on MySQL
(case-insensitive `(name, dag_id)` primary key), so the MySQL special-case
block from #57113 is folded into the new flow, and it ensures a not-yet-flushed
`DagModel` row exists to satisfy the `dag_tag` FK.
- The relationship is expired after the insert so the next access reloads it
from the database.
**Tests**
- Regression test simulating the HA race deterministically: the tags
relationship is loaded, then the same tag is inserted behind the ORM's back
before `_update_dag_tags` runs — previously `IntegrityError`, now tolerated
with the remaining tags still applied.
- Test that a not-yet-flushed `DagModel` is flushed before the tag insert
(FK safety).
- The existing parametrized `test_update_dag_tags` cases (including MySQL
case-only renames) cover the folded-in #57113 behaviour across all backend
matrices.
closes: #68263
related: #50815
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
