duongcongtoai commented on issue #11689: URL: https://github.com/apache/datafusion/issues/11689#issuecomment-2254394579
This is maybe because of the logical plan generated as ``` logical_plan 01)Unnest: lists[unnest(unnest(make_array(make_array(Int64(1),Int64(2),Int64(3)))))] structs[] 02)--Projection: unnest(make_array(Int64(1),Int64(2),Int64(3))), unnest(make_array(make_array(Int64(1),Int64(2),Int64(3)))) AS unnest(unnest(make_array(make_array(Int64(1),Int64(2),Int64(3))))) 03)----Unnest: lists[unnest(make_array(Int64(1),Int64(2),Int64(3))), unnest(make_array(make_array(Int64(1),Int64(2),Int64(3))))] structs[] 04)------Projection: List([1, 2, 3]) AS unnest(make_array(Int64(1),Int64(2),Int64(3))), List([[1, 2, 3]]) AS unnest(make_array(make_array(Int64(1),Int64(2),Int64(3)))) ``` While the correct plan may need to look like ``` 01)--Projection: unnest(make_array(Int64(1),Int64(2),Int64(3))), unnest_depth_2(make_array(make_array(Int64(1),Int64(2),Int64(3)))) AS unnest(unnest(make_array(make_array(Int64(1),Int64(2),Int64(3))))) 02)----Unnest: lists[unnest(make_array(Int64(1),Int64(2),Int64(3))), unnest_depth_2(make_array(make_array(Int64(1),Int64(2),Int64(3))))] structs[] 03)------Projection: List([1, 2, 3]) AS unnest(make_array(Int64(1),Int64(2),Int64(3))), List([[1, 2, 3]]) AS unnest(make_array(make_array(Int64(1),Int64(2),Int64(3)))) ``` Meaning all the necessary unnest will have to happen inside one logical node This bring the needs for the following items: 1. unnest logical/physical plan has to be aware of the depth of unnest 2. try_process_unnest function has to detect consecutive unnests in the query plan, and determine the depth for each unnest operation on each columns For the second items, we have to be aware of the expr that has recursive unnest, but they are not consecutive, for example this expr: ``` select unnest(unnest(unnest(struct_arr_col)['col0'])), unnest([1,2,3]); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
