shantanutheone opened a new pull request, #26500:
URL: https://github.com/apache/datafusion/pull/26500
## Which issue does this PR close?
- Closes #26493.
## Rationale for this change
`COPY ... STORED AS JSON` and `INSERT INTO` a JSON table drop every MAP
entry whose value is NULL. `map(['color','size'], ['red', NULL])` is written as
`{"color":"red"}`, so reading the file back gives a different map: the `size`
key is gone and `cardinality` drops from 2 to 1. A map with only NULL values,
such as `map(['flag'], [NULL])`, comes back as `{}`.
The JSON sink uses arrow-json's writer with its default `explicit_nulls =
false`, and arrow-json's `MapEncoder` skips NULL-valued entries in that mode.
For a struct field a missing key still reads back as NULL when the table has a
schema, but for a map the entry is the data, so leaving it out loses it.
## What changes are included in this PR?
- `datafusion/datasource-json/src/file_format.rs`: a small `ndjson_writer`
helper that builds the line-delimited writer with `with_explicit_nulls(true)`
when the schema contains a Map at any depth (inside List, LargeList,
FixedSizeList, ListView, LargeListView, Struct, Dictionary or RunEndEncoded).
`JsonSerializer` uses it, which covers both `COPY` and `INSERT INTO`.
- `datafusion/datasource-json/src/source.rs`: `plan_to_json` uses the same
helper, so both write paths behave the same.
- An upgrade guide note in `56.0.0.md` for the output change.
Schemas without a Map are written exactly as before.
## What is the testing strategy for this PR?
New round-trip cases at the end of `copy.slt`: write tables with NULL-valued
MAP entries to JSON, read them back with a declared schema, and check the
values and `cardinality`. One case has the map at the top level, the other has
it inside a struct and inside a list, so the nested check is covered. Both
cases fail without the source change (the maps read back as `{}` or lose their
NULL keys) and pass with it.
Two small unit tests in `file_format.rs` cover what SQL round trips cannot
see: `contains_map` finds a Map inside List/Struct, Dictionary and
RunEndEncoded, and `JsonSerializer` writes `{"id":null,"m":{"k":null}}` when a
Map is present but still omits the NULL (`{}`) when there is none.
## Are there any user-facing changes?
Yes, for JSON files whose schema contains a Map. Explicit nulls are a
writer-wide setting in arrow-json, so in those files every other NULL
(top-level columns and struct fields) is now written as `"field":null` instead
of being omitted. Reading back with a declared schema gives the same values.
With schema inference, a column or struct field that is NULL in every sampled
row is now inferred as type `Null` instead of being missing. Files without a
Map are unchanged. No public API changes.
Not covered here: unbounded JSON stream tables
(`datafusion/catalog/src/stream.rs`) build their own `LineDelimitedWriter`, so
they still drop these entries, and the CLI's `--format json` / `ndjson` output
is unchanged. `datafusion-catalog` does not depend on `datasource-json`, so
sharing the helper means moving it (probably to `datafusion-datasource`). I can
do that in a follow-up.
Open question: whether maintainers would prefer a `format.explicit_nulls`
option on `JsonOptions` instead of tying this to Map columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
