andygrove opened a new issue, #244:
URL: https://github.com/apache/datafusion-spark/issues/244

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `from_json(jsonStr, schema[, options])` parses a JSON string into a value of 
the given type. `schema` must be a
   foldable string: a DDL string (`'a INT, b DOUBLE'`, `'array<int>'`, 
`'map<string, int>'`), a JSON-format schema, or
   `schema_of_json(...)`. A non-literal schema fails with 
`INVALID_SCHEMA.NON_STRING_LITERAL`. The top-level type may be
   `STRUCT`, `ARRAY` or `MAP`, plus `VARIANT` from 4.0. Map keys anywhere in 
the schema must be `STRING`, otherwise
   analysis fails with `DATATYPE_MISMATCH.INVALID_JSON_MAP_KEY_TYPE`. `options` 
must be a `map(...)` of string literals,
   otherwise it fails with `INVALID_OPTIONS.NON_MAP_FUNCTION` or 
`INVALID_OPTIONS.NON_STRING_TYPE`.
   
   The return type is the schema with nullability forced on everywhere 
(`schema.asNullable`), and the result is always
   nullable. The input must be a string (collation-aware from 4.0). The 
function is time-zone aware
   (`spark.sql.session.timeZone`, or a `timeZone` option). Available since 
Spark 2.2.0.
   
   Parsing uses Spark's `JacksonParser` with `JSONOptions`. The defaults are: 
`allowSingleQuotes=true`,
   `allowNonNumericNumbers=true`, and `allowComments`, 
`allowUnquotedFieldNames`, `allowNumericLeadingZeros`,
   `allowBackslashEscapingAnyCharacter` and `allowUnquotedControlChars` all 
false. Records go through
   `FailureSafeParser`. Only the `PERMISSIVE` (default) and `FAILFAST` modes 
are allowed; `DROPMALFORMED` fails with
   `PARSE_MODE_UNSUPPORTED`.
   
   ### Examples
   
   ```sql
   SELECT from_json('{"a":1, "b":0.8}', 'a INT, b DOUBLE');
   -- {"a":1,"b":0.8}
   SELECT from_json('{"time":"26/08/2015"}', 'time Timestamp', 
map('timestampFormat', 'dd/MM/yyyy'));
   -- {"time":2015-08-26 00:00:00}
   SELECT from_json('{"a":1, "b":2}', 'map<string, int>');
   -- {"a":1,"b":2}
   SELECT from_json('[1, 2, 3]', 'array<int>');
   -- [1,2,3]
   SELECT from_json('[1, "2", 3]', 'array<int>');
   -- NULL
   SELECT from_json('[1, 2, null]', 'array<int>');
   -- [1,2,null]
   SELECT from_json('{"a": 1}', 'array<struct<a:int>>');
   -- [{"a":1}]
   SELECT from_json('[null, {"a":2}]', 'array<struct<a:int>>');
   -- [null,{"a":2}]
   SELECT from_json('[{"a": 1}, 2]', 'array<map<string,int>>');
   -- NULL
   SELECT from_json('{"d": "02-29"}', 'd date', map('dateFormat', 'MM-dd'));
   -- {"d":null}          (4.0+; 3.4/3.5 raise 
INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER)
   SELECT from_json('{"c1":[1, 2, 3]}', schema_of_json('{"c1":[0]}'));
   -- {"c1":[1,2,3]}
   ```
   
   These come from the `ExpressionDescription` and `json-functions.sql.out`.
   
   ### Edge cases to test
   
   - NULL input gives NULL. An empty or whitespace-only string has no first 
token, so the parser returns no rows and the
     result is NULL.
   - Malformed input in `PERMISSIVE` mode. A struct schema gives a struct whose 
fields are all NULL, not a NULL struct
     (`from_json('{"a" 1}', 'a INT')` gives `{"a":null}`). Array and map 
schemas give NULL. If the schema contains the
     corrupt-record column (`spark.sql.columnNameOfCorruptRecord`, default 
`_corrupt_record`, or the
     `columnNameOfCorruptRecord` option), that column gets the raw input, and 
it can sit anywhere in the struct.
   - Partial results (`spark.sql.json.enablePartialResults`, internal, default 
true since 3.4). A field that fails to
     convert becomes NULL while its siblings are kept: `{"a": "1", "b": 11}` 
with `a INT, b INT` gives `{null, 11}`.
     With the config false, nested failures null out the whole enclosing value. 
See the `SPARK-33134` and `SPARK-40646`
     cases in `JsonFunctionsSuite`.
   - `FAILFAST` mode fails with 
`MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION`, where
     `badRecord = [null,null,{"a" 1, "b": 11}]`. The cause is, for example,
     `CANNOT_PARSE_JSON_FIELD(fieldName, fieldValue, jsonType, dataType)`. In 
3.4 the error has no subclass.
   - Type conversion is strict. A JSON string is not converted to INT, BIGINT 
or BOOLEAN. A float token is not converted
     to INT. Values out of range for INT or BIGINT, and DECIMAL values whose 
precision overflows after HALF_UP rounding
     to the scale (`NUMERIC_VALUE_OUT_OF_RANGE`), make the field NULL. FLOAT 
and DOUBLE accept integer tokens and the
     strings `NaN`, `Infinity`, `+Infinity`, `+INF`, `-Infinity` and `-INF` 
when `allowNonNumericNumbers` is true.
   - An empty string for a non-string field fails 
(`EmptyJsonFieldValueException`). With
     `spark.sql.legacy.json.allowEmptyString.enabled` it becomes NULL, except 
for FLOAT, DOUBLE, DATE and TIMESTAMP,
     which still fail.
   - A `STRING` field receiving an object, array, number or boolean gets the 
Jackson re-serialization of that token: no
     whitespace, floats written with `Double.toString` (`1.0E10`).
   - Duplicate keys. By code reading, in a struct the last occurrence wins 
because each occurrence overwrites the field.
     In a `MAP` both entries are kept: `convertMap` builds `ArrayBasedMapData` 
directly, with no
     `spark.sql.mapKeyDedupPolicy` check.
   - Field-name matching is case-sensitive and exact. Unknown fields are 
skipped. Missing fields are NULL
     (`'b INT'` on `{"a": 1}` gives `{"b":null}`).
   - An array-of-struct schema given a single object input wraps it 
(`[{"a":1}]` above).
   - Trailing content. The parser reads one root value and never checks for 
EOF. By code reading,
     `from_json('{"a":1} x', 'a INT')` gives `{"a":1}`.
   - Options to cover: `mode`, `columnNameOfCorruptRecord`, `allowComments`, 
`allowUnquotedFieldNames`,
     `allowSingleQuotes=false`, `allowNumericLeadingZeros`, 
`allowBackslashEscapingAnyCharacter`,
     `allowUnquotedControlChars`, `allowNonNumericNumbers`, `dateFormat`, 
`timestampFormat`, `timestampNTZFormat`,
     `timeZone` and `locale`. Option keys are case-insensitive. A NULL option 
value is ignored. A NULL option key fails
     with `NULL_MAP_KEY`.
   - Dates and timestamps: the default ISO formats, custom patterns, the 
session time zone, DST transitions, and
     `spark.sql.legacy.timeParserPolicy`.
   - `to_json` round-trips (`to_json` is tracked in 
apache/datafusion-spark#85). `from_json` followed by `to_json` should
     reproduce `{"a":1,"b":2,"c":3}` for `map<string,int>`, and `[{"a":1}]` for 
an array of structs.
   - Jackson stream limits (3.5+): number length, nesting depth, and string 
length. These can be overridden with the
     `maxNumLen`, `maxNestingDepth` and `maxStringLen` options.
   - ANSI mode does not change parsing.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5: a datetime that parses under the legacy parser but not the 
new one raises
     `INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER`. From 
4.0 the value is NULL (see the `02-29`
     example).
   - `MALFORMED_RECORD_IN_PARSING` gained the `WITHOUT_SUGGESTION` subclass in 
3.5.
   - 4.0 adds `VARIANT` schemas (with `spark.sql.variant.allowDuplicateKeys`) 
and collation support for the input.
   - 4.2 adds the `TIME` type and the `timeFormat` option (the `TIME(p)` cases 
in the 4.2 `json-functions.sql.out`).
   - 3.4 uses Jackson 2.14 (no stream limits), 3.5 uses 2.15, 4.0 uses 2.18, 
and 4.1 and 4.2 use 2.21.
   
   ### Implementation notes
   
   - Spark: `JsonToStructs` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala`.
     The evaluator is `JsonToStructsEvaluator` in 
`.../expressions/json/JsonExpressionEvalUtils.scala`. The parser is
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala`,
 with options in
     `JSONOptions.scala` and modes in `catalyst/util/FailureSafeParser.scala`.
   - DataFusion core has no JSON functions. Arrow's `arrow-json` reader is 
strict JSON and does not have Spark's
     per-field failure semantics, so it can't be used as it is.
   - This crate has no Spark DDL type parser. One option is a planner-side 
parser for the schema literal that
     `return_field_from_args` can use. Another is to let callers such as Comet 
pass the Arrow type directly. Reuse
     `crates/spark/src/function/json/` for the lenient tokenizer shared with 
the other JSON functions.
   - A first version could support `PERMISSIVE` and `FAILFAST`, primitive, 
struct, array and map types, and the leniency
     options, and leave datetime patterns and `VARIANT` for follow-ups.
   - Add `crates/sqllogictest/test_files/json/from_json.slt` and document the 
function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to