waterWang opened a new pull request, #71927:
URL: https://github.com/apache/airflow/pull/71927
## Summary
The 0049 migration's NaN sanitizer (``regexp_replace``) incorrectly matches
``NaN``/``Infinity`` tokens inside JSON string values, injecting an unescaped
quote that breaks the outer JSON structure. This causes the migration to abort
with ``INVALID_TEXT_REPRESENTATION`` on PostgreSQL and equivalent errors on
MySQL/SQLite, making the upgrade 2.x → 3.x fail for any XCom value that is a
JSON string wrapping a JSON document containing NaN floats.
This happens when:
- A task pushes already-serialized JSON (e.g.
``json.dumps(json.dumps({"amount": float("nan")}))``)
- A dict value is itself a JSON string (``{"report": "{\"amount\": NaN}"}``)
Both shapes are **already valid JSON** — the NaN token lives inside a JSON
string, not at the JSON syntax level. The sanitizer should not touch them.
## Fix
Guard the per-dialect sanitize SQL with the native JSON validity check for
each engine:
| Dialect | Function | Available since |
|:---|:---|:---|
| PostgreSQL | ``json_valid()`` | PG 14 (Airflow 3 requires PG 14+) |
| MySQL | ``JSON_VALID()`` | MySQL 5.7+ |
| SQLite | ``json_valid()`` | JSON1 extension (enabled in all modern builds)
|
Values that are already valid JSON pass through the sanitizer untouched;
only values containing bare NaN/Infinity tokens (which are illegal in strict
JSON/JSONB) are quoted.
## Tests
- ``test_sqlite_sanitize_skips_already_valid_json`` — regression test for
both shapes using the SQLite branch (backend-independent).
- The existing PG/MySQL backend tests in ``TestPostgresSanitize`` /
``TestMysqlSanitize`` continue to pass (bare NaN/Infinity are still quoted
correctly).
Closes #71921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
