andygrove opened a new issue, #224:
URL: https://github.com/apache/datafusion-spark/issues/224
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`named_struct(name1, val1, name2, val2, ...)` builds a STRUCT whose fields
are named `name1`, `name2`, ... and
hold the matching values. The struct itself is never NULL (`nullable =
false`). Each field's nullability is the
nullability of its value expression, and a NULL value becomes a NULL field.
Type checks (`checkInputDataTypes`):
- An odd number of arguments fails with `WRONG_NUM_ARGS.WITHOUT_SUGGESTION`
(expectedNum `2n (n > 0)`).
- Each name must be a foldable STRING expression. A literal or a constant
expression such as
`concat('a', 'b')` is accepted, but a column or a non-string literal fails
with
`DATATYPE_MISMATCH.CREATE_NAMED_STRUCT_WITHOUT_FOLDABLE_STRING`: "Only
foldable `STRING` expressions are
allowed to appear at odd position, but they are <inputExprs>."
- A name that evaluates to NULL fails with
`DATATYPE_MISMATCH.UNEXPECTED_NULL`.
- Duplicate field names are not checked and are accepted. Empty-string names
are also accepted.
Values may have any type. The function exists since Spark 1.5.0.
`struct(...)` is lowered to the same
`CreateNamedStruct` expression with generated names (`col1`, `col2`, ... or
the input column/alias names), so an
implementation could serve both.
### Examples
```sql
SELECT named_struct("a", 1, "b", 2, "c", 3);
-- {"a":1,"b":2,"c":3}
SELECT struct(1, 2, 3);
-- {"col1":1,"col2":2,"col3":3}
SELECT typeof(named_struct('C', 'gamma', 'D', 'delta'));
-- struct<C:string,D:string>
```
From `ComplexTypeSuite`:
- `named_struct('a', 'x', 'b', 2.0)` gives `{"a":"x","b":2.0}`.
- `named_struct('a', CAST(NULL AS INT))` gives `{"a":null}`, a non-NULL
struct with a NULL field.
- `named_struct(1, 2, 3)` fails with `WRONG_NUM_ARGS.WITHOUT_SUGGESTION`,
and actualNum is `3`.
### Edge cases to test
- A NULL value in each position. The struct stays non-NULL and only that
field is NULL.
- Field nullability in the result schema: a non-nullable value gives a
non-nullable field.
- Duplicate names, as in `named_struct('a', 1, 'a', 2)`:
- Spark accepts them and produces a struct with two fields named `a`.
- The Arrow `Fields` must keep both. This is the case Comet currently
falls back on.
- Accessing such a field by name is a separate concern.
- Empty-string name `''`: Spark accepts it.
- Case-sensitive names: `'A'` and `'a'` are distinct fields.
- Names that are foldable expressions rather than literals (`concat('a',
'b')`, `CAST(1 AS STRING)`).
- Rejected names:
- a column used as a name,
- a non-string literal (`named_struct(1, 2)`),
- `CAST(NULL AS STRING)` as a name,
- an odd argument count.
- Zero arguments: according to the code, `named_struct()` passes
`checkInputDataTypes`, since `0 % 2 == 0`, and
yields an empty `STRUCT<>`. Arrow needs
`StructArray::new_empty_fields(len, None)` for this.
- Value types: nested struct, array, map, binary, decimal, timestamp, and
`NULL` (VOID). A mix of scalar and
column values must be broadcast to the batch length.
- A wide struct with many fields (Spark splits codegen for these, so they
are worth one test).
### Differences across Spark versions
- 3.4, 3.5 and 4.0: `CreateNamedStruct` is identical in all three. Names
must be exactly `StringType`, so a
collated-string name is rejected.
- 4.1 (checked in 4.1.3) and 4.2:
- Names may be any `StringType`, including collated strings
(`dataType.isInstanceOf[StringType]`).
- `contextIndependentFoldable` is added.
- Field metadata is also carried over from a `GetStructField` value.
- `dataType` no longer throws for NULL names (SPARK-57736). This is
analysis-only.
- Runtime semantics are unchanged.
### Implementation notes
- Spark implements this in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala`
(`CreateNamedStruct`, plus `CreateStruct` for `struct`).
- DataFusion core's `named_struct` is in
`datafusion/functions/src/core/named_struct.rs`. It differs from Spark
as follows:
- It marks every field `nullable = true`, where Spark uses each value's
nullability.
- It rejects an empty-string name ("non-empty constant string") and zero
arguments, both of which Spark
accepts.
- It requires each name to be a scalar string at planning time
(`scalar_arguments`), so a foldable but not yet
simplified expression such as `concat('a','b')` may be rejected.
- Its error messages differ from Spark's error conditions.
- It does not check for duplicate names, which matches Spark. Confirm that
the `struct_field_mapping`
optimizer hook, which maps names to argument indices, behaves correctly
with duplicates.
- A datafusion-spark version can mostly reuse the core kernel. It needs
`return_field_from_args` with Spark's
field nullability, acceptance of `''` and zero arguments, and Spark-style
errors.
- There are placeholder files
`crates/sqllogictest/test_files/struct/named_struct.slt` and `struct.slt`,
ported
from Sail with their queries commented out. Enable them, put the UDF in
`crates/spark/src/function/struct/`
(currently empty), and document it with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]