andygrove opened a new issue, #191:
URL: https://github.com/apache/datafusion-spark/issues/191
Part of #173 and apache/datafusion-comet#7300.
### What the function does
`inline(expr)` is a generator that explodes an `ARRAY<STRUCT<...>>` into one
output row per array element,
with one output column per struct field. The output column names and types
are the struct's field names
and types (`col1`, `col2`, ... for `struct(...)` with unnamed fields). If
the array's `containsNull` is
true, all output columns become nullable (`st.asNullable`). Since Spark
2.0.0 (table-valued function form
`SELECT * FROM inline(...)` since 3.4.0).
`inline_outer(expr)` is the same generator wrapped in `GeneratorOuter`: when
the array is NULL or empty
it emits one row with every column NULL instead of no rows.
Only `ARRAY<STRUCT>` is accepted; there are no implicit casts. Anything else
(including `NULL`
literals, `array()` which is `ARRAY<VOID>`, and a bare struct) fails
analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (`requiredType`
`"ARRAY<STRUCT>"`). From 4.0 the argument can
be passed by name (`input => ...`).
### Examples
```sql
SELECT inline(array(struct(1, 'a'), struct(2, 'b')));
-- 1 a
-- 2 b
SELECT * FROM inline(array(struct(1, 'a'), struct(2, 'b'))) t(x, y);
-- schema struct<x:int,y:string>
-- 1 a
-- 2 b
SELECT inline(array(struct(null, 'a'), struct(null, 'b')));
-- schema struct<col1:void,col2:string>
-- NULL a
-- NULL b
-- empty array: inline gives no rows, inline_outer gives one NULL row
SELECT * FROM inline(array_remove(array(struct(1, 'a')), struct(1, 'a')));
-- (no rows)
SELECT * FROM inline_outer(array_remove(array(struct(1, 'a')), struct(1,
'a')));
-- NULL NULL
-- NULL struct elements become a row of NULLs (GeneratorFunctionSuite,
SPARK-39061)
SELECT inline(b) FROM VALUES (1, array(named_struct('c1', 0, 'c2', 1), null,
named_struct('c1', 2, 'c2', 3),
null)) AS tbl(a, b);
-- 0 1
-- NULL NULL
-- 2 3
-- NULL NULL
-- LATERAL VIEW OUTER over arrays, an empty array and NULL
(GeneratorFunctionSuite)
CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES
array(struct(0, 1), struct(3, 4)), array(struct(6, 7)), array(), null AS
tbl(arr);
SELECT f1, f2 FROM t1 LATERAL VIEW inline_outer(arr) AS f1, f2;
-- 0 1
-- 3 4
-- 6 7
-- NULL NULL
-- NULL NULL
SELECT inline(1);
-- DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (inputType "INT", requiredType
"ARRAY<STRUCT>")
```
### Edge cases to test
- NULL array: `inline` emits no rows; `inline_outer` emits one all-NULL row.
- Empty array: same as NULL (no rows vs one all-NULL row).
- NULL struct element inside the array: emits a row whose fields are all
NULL (both forms), and it must not
be confused with the outer padding row.
- Struct with NULL fields, and with nested types (arrays, maps, structs) as
fields.
- Struct with a single field, and with many fields; field names with dots or
duplicates (Spark keeps the
struct's names, e.g. `a` from `struct(a, 'a')`).
- Output nullability: non-nullable struct fields become nullable when the
array may contain NULLs.
- Other columns of the input row are repeated for each generated row
(`SELECT a, inline(b)`).
- Literal array vs column input; many input rows with arrays of varying
length, including length 0.
- Invalid input types: `inline(NULL)` (`VOID`), `inline(array())`
(`ARRAY<VOID>`), `inline(array(1, 'a'))`,
`inline(struct(1, 'a'))` all raise
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` at analysis.
- Deterministic: output order follows array order within each input row.
### Differences across Spark versions
- `Inline` itself (type check, null-struct handling, nullability) is the
same in 3.4.3 through 4.2.0.
- 3.4/3.5 register `inline` via `expression[Inline]` and `inline_outer` via
`expressionGeneratorOuter`;
4.0+ use `InlineExpressionBuilder` / `InlineGeneratorBuilder` /
`InlineOuterGeneratorBuilder`, which add
the named argument `input =>`. Results are unchanged.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala`,
classes
`Inline`, `InlineExpressionBuilder`, `InlineGeneratorBuilderBase`,
`InlineGeneratorBuilder`,
`InlineOuterGeneratorBuilder`; outer handling is `GeneratorOuter` plus
`GenerateExec` (`outer = true`).
- DataFusion core has no `inline`. The closest is `unnest` on a
`List<Struct>` followed by struct
expansion. Its default `NullHandling::Preserve` keeps a NULL list as one
NULL row, which matches neither
form: `inline` needs `NullHandling::Drop` and `inline_outer` needs
`NullHandling::PreserveAndExpandEmpty`
(`datafusion/common/src/unnest.rs`). A NULL struct element must expand to
all-NULL fields.
- In datafusion-spark this is a row-generating operation, like the existing
`stack`
(`crates/spark/src/function/generator/stack.rs`), which returns a list of
structs for the caller to
expand. Since `inline`'s input already is a list of structs, the useful
pieces here are likely planning
helpers (output schema with Spark's names and nullability, and the
outer/non-outer null policy) rather
than a scalar kernel; agree on the shape before writing code. Place it in
the `generator` module.
- Tests: add `crates/sqllogictest/test_files/generator/inline.slt` (cover
both functions), and document
with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]