Jefffrey opened a new issue, #17670:
URL: https://github.com/apache/datafusion/issues/17670
### Is your feature request related to a problem or challenge?
Currently cannot flatten a `List` (or `FixedSizeList`) that contains a
`LargeList` as it's inner element.
We should be able to support this (at least for queries we expect to
succeed). For example, expect something like this to succeed in `array.slt`:
```sql
query ???
select flatten(arrow_cast(make_array([1], [2, 3], [null], make_array(4,
null, 5)), 'FixedSizeList(4, LargeList(Int64))')),
flatten(arrow_cast(make_array([[1.1], [2.2]], [[3.3], [4.4]]),
'List(LargeList(FixedSizeList(1, Float64)))'));
----
[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [[1.1], [2.2], [3.3], [4.4]]
```
### Describe the solution you'd like
Need to consider return type, see how `LargeList` is missing here for the
inner `field` match:
https://github.com/apache/datafusion/blob/35c1cfdd151b14bbba87dbdf03d49afd9f4718a6/datafusion/functions-nested/src/flatten.rs#L107-L110
This is where current error is happening:
https://github.com/apache/datafusion/blob/35c1cfdd151b14bbba87dbdf03d49afd9f4718a6/datafusion/functions-nested/src/flatten.rs#L166-L168
- We just throw error without trying to see if it's possible
Perhaps we can try some sort of "best effort" where we try to downcast the
`LargeList` child to a `List` and if that succeeds (i.e. all offsets of
`LargeList` can fit inside a `List`) we can flatten it to the parent `List`,
otherwise error; alternatively just upcast the parent `List` to a `LargeList`,
though this might be tricky considering `return_type()` wouldn't know this
until execution and I don't think we want to blindly upcast all parent `List`s
to `LargeList`.
Open to any other suggestions.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
