andygrove opened a new issue, #242:
URL: https://github.com/apache/datafusion-spark/issues/242

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `json_array_length(jsonArray)` returns the number of elements in the 
outermost JSON array as an `INT`. It returns
   NULL for NULL input, for an empty string, for valid JSON whose top-level 
value is not an array, and for malformed
   JSON. The result is nullable.
   
   The argument must be a string (`ExpectsInputTypes`, so no implicit cast; 
`json_array_length(2)` fails analysis with
   `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`). From 4.0 the input is 
`StringTypeWithCollation(supportsTrimCollation =
   true)`. Available since Spark 3.1.0. No aliases.
   
   The implementation reads the first token with a Jackson parser from the same 
lenient `SharedFactory` used by
   `get_json_object` (`ALLOW_SINGLE_QUOTES` and `ALLOW_UNESCAPED_CONTROL_CHARS` 
enabled). If that token is
   `START_ARRAY`, it counts top-level elements with `skipChildren()` until 
`END_ARRAY`. Any `IOException`, which
   includes Jackson parse errors, gives NULL.
   
   ### Examples
   
   ```sql
   SELECT json_array_length('[1,2,3,4]');
   -- 4
   SELECT json_array_length('[1,2,3,{"f1":1,"f2":[5,6]},4]');
   -- 5
   SELECT json_array_length('[1,2');
   -- NULL
   SELECT json_array_length(null);
   -- NULL
   SELECT json_array_length('');
   -- NULL
   SELECT json_array_length('[]');
   -- 0
   SELECT json_array_length('[[1,2],[5,6,7]]');
   -- 2
   SELECT json_array_length('[{"a":123},{"b":"hello"}]');
   -- 2
   SELECT json_array_length('{"key":"not a json array"}');
   -- NULL
   SELECT json_array_length('[1,2,3,4,5');
   -- NULL
   SELECT json_array_length('Random String');
   -- NULL
   ```
   
   The first three come from the `ExpressionDescription`. The others come from 
`json-functions.sql.out` and
   `JsonExpressionsSuite`.
   
   ### Edge cases to test
   
   - NULL, empty string, and whitespace-only input give NULL, because the first 
token is `null`.
   - Top-level scalars (`1`, `"a"`, `true`, `null`) and objects give NULL.
   - Nested arrays and objects count as one element each, and their contents 
are skipped rather than counted.
   - Lenient syntax. Single-quoted strings (`['a', 'b']` gives 2) and raw 
control characters inside strings are accepted.
     A strict parser such as `serde_json` gets these wrong.
   - Trailing content. Spark stops at the outer `END_ARRAY` and never checks 
for EOF. By code reading,
     `json_array_length('[1,2] trailing')` returns `2`.
   - Malformed content before the closing bracket gives NULL: `[1,2`, `[1,,2]`, 
`[1 2]`, and a malformed nested element.
     Skipping a nested element still tokenizes it, so errors inside it are 
detected.
   - Leading-zero numbers (`[01]`), `NaN`, and comments are not enabled in 
`SharedFactory`, so these are malformed and
     give NULL.
   - Jackson stream limits (3.5+). A number token longer than 1000 digits, or 
nesting deeper than 1000, gives NULL.
     Spark 3.4 has no such limit.
   - Very long arrays: the count is a Java `int`.
   - Multi-byte UTF-8 and `\uXXXX` escapes inside elements.
   - Scalar and column input.
   - The function is not affected by ANSI mode.
   
   ### Differences across Spark versions
   
   - The behaviour is the same from 3.4 through 4.2, apart from the Jackson 
upgrade. 3.4 uses Jackson 2.14.2 with no
     default number-length or nesting limits, and 3.5 and later have those 
limits.
   - 4.0 turns the expression into a `RuntimeReplaceable` that calls 
`JsonExpressionUtils.lengthOfJsonArray` (no
     behaviour change) and adds collation support for the input.
   - The error parameter for the type mismatch changed from `"paramIndex" : 
"1"` (3.x) to `"first"` (4.0+).
   
   ### Implementation notes
   
   - Spark: `LengthOfJsonArray` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala`.
     From 4.0 the logic is in
     
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionUtils.java`
     (`lengthOfJsonArray`).
   - DataFusion core has no JSON functions.
   - In datafusion-spark, `crates/spark/src/function/json/json_tuple.rs` uses 
`serde_json::from_str`. That is strict and
     also rejects trailing content, so it cannot be used as it is. A small 
streaming tokenizer that emulates Jackson's
     lenient features would be enough here and could be shared with 
`get_json_object`, `json_object_keys` and
     `json_tuple`.
   - Put it in `crates/spark/src/function/json/`, add 
`crates/sqllogictest/test_files/json/json_array_length.slt`, and
     document it with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to