SreeramaYeshwanthGowd opened a new pull request, #57479:
URL: https://github.com/apache/spark/pull/57479
### What changes were proposed in this pull request?
Add a built in `json_typeof(json)` scalar function that returns the type of
the outermost value in a JSON string, as one of `object`, `array`, `string`,
`number`, `boolean`, or `null`. It returns SQL `NULL` when the input is not a
valid JSON string or is an empty string.
API surface added:
- SQL: `json_typeof(json)`
- Scala DataFrame: `functions.json_typeof(col)`
- PySpark, classic and Spark Connect:
`pyspark.sql.functions.json_typeof(col)`
Implementation notes:
- Modeled on the existing `json_object_keys` function: a
`RuntimeReplaceable` expression whose replacement is a `StaticInvoke` into
`JsonExpressionUtils`, using the same Jackson parser (`CreateJacksonParser` /
`SharedFactory`) that already backs `json_array_length` and `json_object_keys`.
No new dependency.
- Like those functions, malformed input returns `NULL`: `json_typeof` reads
the outermost value and consumes it, so a structurally invalid JSON value
returns `NULL` (consistent with `json_object_keys`).
- Both integer and floating point JSON numbers map to `number`, matching
PostgreSQL and BigQuery.
### Why are the changes needed?
Spark can already ask a JSON string for its array length
(`json_array_length`) and its object keys (`json_object_keys`), but not for the
type of its outermost value. `json_typeof` completes that family. It is a
lightweight top level type probe: unlike `schema_of_json`, which does full
recursive schema inference and returns a DDL string such as `ARRAY<BIGINT>`,
`json_typeof` returns a single simple type token. It is provided by PostgreSQL
(`json_typeof` and `jsonb_typeof`) and BigQuery (`JSON_TYPE`), so it also
improves parity with the engines Spark users migrate from.
### Does this PR introduce _any_ user-facing change?
Yes. It adds a new built in SQL function `json_typeof` and the corresponding
Scala and PySpark DataFrame API entries. No existing behavior changes.
Example:
```
spark-sql> SELECT json_typeof('{"a": 1}');
object
spark-sql> SELECT json_typeof('[1, 2, 3]');
array
spark-sql> SELECT json_typeof('123');
number
spark-sql> SELECT json_typeof('not json');
NULL
```
### How was this patch tested?
Added catalyst unit tests in `JsonExpressionsSuite` covering each JSON type
(object, array, string, number, boolean, null), invalid and empty inputs
returning null, and null propagation, and a DataFrame API test in
`JsonFunctionsSuite`. Added a PySpark doctest and regenerated
`sql-expression-schema.md`.
### Was this patch authored or co-authored using generative AI tooling? No
<!-- Complete this section before submitting the PR. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
