vatsrahul1001 opened a new pull request, #66016:
URL: https://github.com/apache/airflow/pull/66016
## What does this PR do?
Fixes both the `upgrade` and `downgrade` paths of migration
`0080_3_1_0_modify_deadline_callback_schema.py` (`808787349f22`), which added
`NOT NULL` columns to the `deadline` table without migrating existing row data,
causing `IntegrityError` on non-empty databases.
### Root cause
**Upgrade** (`3bda03debd04 → 808787349f22`): drops the old `callback
VARCHAR(500)` / `callback_kwargs JSON` columns and then does `ADD COLUMN
callback JSON NOT NULL`. PostgreSQL raises `NotNullViolation` immediately
because existing rows have no value for the new column.
**Downgrade** (`808787349f22 → 3bda03debd04`): drops the new JSON `callback`
column and then does `ADD COLUMN callback VARCHAR(500) NOT NULL`. Same
`NotNullViolation` on PostgreSQL; MySQL silently writes `NULL`, which then
crashes migration `0094`'s upgrade step with `json.loads(None)`.
### Fix
Both paths now follow the same safe-migration pattern already used by
migration `0094`:
1. **Read** the existing column data *before* the schema change.
2. **Add** the new column(s) as `nullable=True` so the DDL succeeds on all
supported databases.
3. **Back-fill** every row using a typed SQLAlchemy `table()` clause
(dialect-aware JSON serialisation).
4. **Enforce** `NOT NULL` only after all rows have valid values.
The upgrade serialises the old `(callback VARCHAR, callback_kwargs JSON)`
pair into the `{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0}` envelope that migration `0094`'s upgrade expects.
The downgrade extracts `path` and `kwargs` from that same envelope before
restoring the old VARCHAR column.
## How to reproduce
1. Start Airflow on any version that already ran migration `0080` (i.e. >=
3.1.0) with at least one `deadline` row in the database.
2. Run `airflow db downgrade` past `808787349f22` (or `airflow db upgrade`
from a pre-3.1.0 snapshot) – observe `IntegrityError: NotNullViolation` on
PostgreSQL.
## Checklist
- [x] My PR is targeted at the `main` branch
- [x] I have added a meaningful commit message that explains *why* the
change is necessary
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
