CaroFernando opened a new pull request, #8740:
URL: https://github.com/apache/texera/pull/8740
### What changes were proposed in this PR?
When a JSONL file has a JSON null in a row, the JSONL File Scan reads it as
the literal text "null". That happens because JSONToMap writes every value
node into the map using asText(), and for NullNode that returns "null". One
null in a column then breaks schema inference: it can't parse "null" as a
number, so the column falls back to STRING and the cell shows the word null.
This PR makes JSONToMap skip NullNode in both places it can show up, object
fields and array elements in flatten mode. A null now simply has no entry in
the map, which is exactly how a missing key already behaves. The JSONL scan
operator handles missing keys fine already: at runtime the cell becomes an
empty value, and during schema inference the other rows decide the type.
So for the example in the issue:
{"a":1} {"a":null} used to give column a as STRING with values "1" and "null"
it now gives INTEGER with values 1 and an empty cell
The two tests in JSONUtilsSpec that expected the string "null" were written
on purpose to pin the old behavior until a fix came along, so they are
updated here. I also added two cases: a row where every field is null gives
an empty map, and a null inside a flattened array skips only its own indexed
key.
### Any related issues, documentation, discussions?
Closes #8674
### How was this PR tested?
The new tests fail before the fix and pass after it.
sbt "WorkflowCore/testOnly org.apache.texera.amber.util.JSONUtilsSpec"
(25/25)
sbt WorkflowCore/test (820/820)
sbt WorkflowOperator/test (2605 passed, 11 canceled which also fail on main)
sbt scalafmtAll
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
