andygrove opened a new issue, #191:
URL: https://github.com/apache/datafusion-spark/issues/191

   Part of #173 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `inline(expr)` is a generator that explodes an `ARRAY<STRUCT<...>>` into one 
output row per array element,
   with one output column per struct field. The output column names and types 
are the struct's field names
   and types (`col1`, `col2`, ... for `struct(...)` with unnamed fields). If 
the array's `containsNull` is
   true, all output columns become nullable (`st.asNullable`). Since Spark 
2.0.0 (table-valued function form
   `SELECT * FROM inline(...)` since 3.4.0).
   
   `inline_outer(expr)` is the same generator wrapped in `GeneratorOuter`: when 
the array is NULL or empty
   it emits one row with every column NULL instead of no rows.
   
   Only `ARRAY<STRUCT>` is accepted; there are no implicit casts. Anything else 
(including `NULL`
   literals, `array()` which is `ARRAY<VOID>`, and a bare struct) fails 
analysis with
   `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (`requiredType` 
`"ARRAY<STRUCT>"`). From 4.0 the argument can
   be passed by name (`input => ...`).
   
   ### Examples
   
   ```sql
   SELECT inline(array(struct(1, 'a'), struct(2, 'b')));
   -- 1 a
   -- 2 b
   
   SELECT * FROM inline(array(struct(1, 'a'), struct(2, 'b'))) t(x, y);
   -- schema struct<x:int,y:string>
   -- 1 a
   -- 2 b
   
   SELECT inline(array(struct(null, 'a'), struct(null, 'b')));
   -- schema struct<col1:void,col2:string>
   -- NULL      a
   -- NULL      b
   
   -- empty array: inline gives no rows, inline_outer gives one NULL row
   SELECT * FROM inline(array_remove(array(struct(1, 'a')), struct(1, 'a')));
   -- (no rows)
   SELECT * FROM inline_outer(array_remove(array(struct(1, 'a')), struct(1, 
'a')));
   -- NULL      NULL
   
   -- NULL struct elements become a row of NULLs (GeneratorFunctionSuite, 
SPARK-39061)
   SELECT inline(b) FROM VALUES (1, array(named_struct('c1', 0, 'c2', 1), null,
                                           named_struct('c1', 2, 'c2', 3), 
null)) AS tbl(a, b);
   -- 0 1
   -- NULL      NULL
   -- 2 3
   -- NULL      NULL
   
   -- LATERAL VIEW OUTER over arrays, an empty array and NULL 
(GeneratorFunctionSuite)
   CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES
     array(struct(0, 1), struct(3, 4)), array(struct(6, 7)), array(), null AS 
tbl(arr);
   SELECT f1, f2 FROM t1 LATERAL VIEW inline_outer(arr) AS f1, f2;
   -- 0 1
   -- 3 4
   -- 6 7
   -- NULL      NULL
   -- NULL      NULL
   
   SELECT inline(1);
   -- DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (inputType "INT", requiredType 
"ARRAY<STRUCT>")
   ```
   
   ### Edge cases to test
   
   - NULL array: `inline` emits no rows; `inline_outer` emits one all-NULL row.
   - Empty array: same as NULL (no rows vs one all-NULL row).
   - NULL struct element inside the array: emits a row whose fields are all 
NULL (both forms), and it must not
     be confused with the outer padding row.
   - Struct with NULL fields, and with nested types (arrays, maps, structs) as 
fields.
   - Struct with a single field, and with many fields; field names with dots or 
duplicates (Spark keeps the
     struct's names, e.g. `a` from `struct(a, 'a')`).
   - Output nullability: non-nullable struct fields become nullable when the 
array may contain NULLs.
   - Other columns of the input row are repeated for each generated row 
(`SELECT a, inline(b)`).
   - Literal array vs column input; many input rows with arrays of varying 
length, including length 0.
   - Invalid input types: `inline(NULL)` (`VOID`), `inline(array())` 
(`ARRAY<VOID>`), `inline(array(1, 'a'))`,
     `inline(struct(1, 'a'))` all raise 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` at analysis.
   - Deterministic: output order follows array order within each input row.
   
   ### Differences across Spark versions
   
   - `Inline` itself (type check, null-struct handling, nullability) is the 
same in 3.4.3 through 4.2.0.
   - 3.4/3.5 register `inline` via `expression[Inline]` and `inline_outer` via 
`expressionGeneratorOuter`;
     4.0+ use `InlineExpressionBuilder` / `InlineGeneratorBuilder` / 
`InlineOuterGeneratorBuilder`, which add
     the named argument `input =>`. Results are unchanged.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala`,
 classes
     `Inline`, `InlineExpressionBuilder`, `InlineGeneratorBuilderBase`, 
`InlineGeneratorBuilder`,
     `InlineOuterGeneratorBuilder`; outer handling is `GeneratorOuter` plus 
`GenerateExec` (`outer = true`).
   - DataFusion core has no `inline`. The closest is `unnest` on a 
`List<Struct>` followed by struct
     expansion. Its default `NullHandling::Preserve` keeps a NULL list as one 
NULL row, which matches neither
     form: `inline` needs `NullHandling::Drop` and `inline_outer` needs 
`NullHandling::PreserveAndExpandEmpty`
     (`datafusion/common/src/unnest.rs`). A NULL struct element must expand to 
all-NULL fields.
   - In datafusion-spark this is a row-generating operation, like the existing 
`stack`
     (`crates/spark/src/function/generator/stack.rs`), which returns a list of 
structs for the caller to
     expand. Since `inline`'s input already is a list of structs, the useful 
pieces here are likely planning
     helpers (output schema with Spark's names and nullability, and the 
outer/non-outer null policy) rather
     than a scalar kernel; agree on the shape before writing code. Place it in 
the `generator` module.
   - Tests: add `crates/sqllogictest/test_files/generator/inline.slt` (cover 
both functions), and document
     with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to