seanmuth opened a new pull request, #69064:
URL: https://github.com/apache/airflow/pull/69064
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
-->
## Why
Migrations `0049_3_0_0_remove_pickled_data_from_xcom_table` and
`0055_3_0_0_remove_pickled_data_from_dagrun_table` convert the legacy pickled
`bytea` columns (`xcom.value`, `dag_run.conf`) to JSON/JSONB during the 2.x ->
3.x upgrade. They already sanitize the non-finite float tokens
`NaN`/`Infinity`/`-Infinity` (#57893), but **not** the `U+0000` (NUL) escape
that `json.dumps` emits for embedded null bytes.
PostgreSQL JSON/JSONB cannot represent `U+0000`:
```
sqlalchemy.exc.DataError: (psycopg2.errors.UntranslatableCharacter)
unsupported Unicode escape sequence
DETAIL: <NUL> cannot be converted to text.
CONTEXT: JSON data, line 1: ..."col": "F<NUL>...
```
So a single XCom/conf value carrying a NUL (common with data sourced from
fixed-width exports, C buffers, Mongo, or bad UTF-16->UTF-8 conversions) breaks
the upgrade:
- **0049 (xcom):** the bulk `ALTER COLUMN value TYPE JSONB USING
CAST(CONVERT_FROM(value,'UTF8') AS JSONB)` has no per-row guard, so it **aborts
the entire migration**.
- **0055 (dag_run.conf):** the per-row `try/except` swallows the failure and
increments `err_count`, so the conf is **silently dropped** (data loss).
This is the same class of "legal in pickle, illegal in strict JSON/JSONB"
value already handled for the non-finite floats. Fixes #69063.
## What
Strip the `U+0000` escape before the JSON(B) conversion. Unlike
`NaN`/`Infinity`, a NUL cannot be quoted/kept, so removal is the only option
(the alternative would be substituting `U+FFFD`; stripping is the least
surprising for metadata).
- **0049** - add a NUL strip to all three dialect branches:
- PostgreSQL: `replace(convert_from(value,'UTF8'), '<NUL-escape>', '')`
wrapping the existing `regexp_replace`.
- MySQL: `REPLACE(CONVERT(value USING utf8mb4), '<NUL-escape>', '')` (same
file-vs-SQL backslash escaping as the existing `NaN` regex in that branch).
- SQLite: an extra inner `REPLACE(... , '<NUL-escape>', '')` (SQLite has
no `REGEXP_REPLACE`, consistent with the existing approach).
- **0055** - strip the escape from the serialized JSON:
`json.dumps(original_data).replace('<NUL-escape>', '')`, so the conf is
preserved instead of dropped.
`json.dumps()` (with the default `ensure_ascii=True`) emits a literal NUL as
the 6-char escape, never a raw `0x00` byte, so stripping the escape covers
values produced by normal XCom/conf serialization. These migration files are
amended in place per the project convention for migration bugfixes (an
already-applied migration can't be re-run).
## Minimal repro
```sql
-- '\x7b226b223a2022465c75303030306f6f227d' decodes to: {"k": "F<NUL>oo"}
SELECT CAST(CONVERT_FROM('\x7b226b223a2022465c75303030306f6f227d'::bytea,
'UTF8') AS JSONB);
-- ERROR: unsupported Unicode escape sequence / DETAIL: <NUL> cannot be
converted to text.
```
## TODO before un-drafting
- [ ] Add migration unit tests covering a NUL-bearing value for all dialects.
- [ ] Confirm whether a newsfragment is desired for an in-place migration
bugfix.
Related: #57893 (the `NaN`/`Infinity` sanitization for these same
conversions).
---
*Authored with [Claude Code](https://claude.com/claude-code) (Claude Opus
4.8).*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
