neilconway opened a new issue, #22095:
URL: https://github.com/apache/datafusion/issues/22095
### Is your feature request related to a problem or challenge?
DataFusion’s Substrait consumer currently normalizes some nested literal
fields to nullable when converting Substrait literals into ScalarValues.
This is intentional today because `VirtualTable` / `Values` consumption
converts the schema and literal rows separately. DataFusion later materializes
the values into Arrow arrays and builds a `RecordBatch` using the schema. Arrow
requires nested field types to match exactly, including nullability, so a
mismatch like this can fail even
when the value itself is valid:
```
schema: List(Field { name: "item", data_type: Int32, nullable: false })
literal: List(Field { name: "item", data_type: Int32, nullable: true })
```
### Describe the solution you'd like
Add an expected-type-aware Substrait literal conversion path.
For `VirtualTable` / `Values`, the consumer has access to the `NamedStruct`
schema. When converting each literal row, pass the expected top-level Field
into literal conversion, and recursively use the expected nested fields for
Struct, List, and Map literals.
The existing schema-less literal conversion path should remain available
for ordinary expression literals where no expected field/type is available.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
