fbx31 commented on PR #18921:
URL: https://github.com/apache/datafusion/pull/18921#issuecomment-3655460423
Hello @gstvg, thanks for your last submission, some questions came to my
mind reading your proposal, I see quite well the map/filter operations that are
producing array in result but I am a bit "confused" about the support of
"reduce" like operation that could produce any type as output like another
array or scalar values with an aggegation value that has to be first intialized
by an Expr through the call to HoF and the "propagation" of this value accross
the plan architecture of datafusion.
I think about use cases like **array_reduce(input: [], init_aggregate_val:
Expr, lamba: Expr)** with **lambda(current_value_of_input, aggregate_value) ->
aggregate_value**
Is it possible with your proposal ? (I saw potential several parameters for
the lambda so I suppose your proposal is compliant ?)
Also, is it possible to "nest" the HoF like:
array_transform(arr_transform([], lamda1), lamba2) ?
For this last question, I don't see any "obstacles" but I prefer to confirm
with you... :-)
And also, pratically, how to reference "current_value" and "aggregate_value"
in the dataframe API? These are not named in the schema and these are not query
parameter as well so it's not clear for me how to reference these values inside
the expressions:
**df.array_transform(col("my_array"), max(<????> * 2, 1000)**
Sorry to question like that, but I was bumping my head on the walls on this
topic since months without success (not enough mastering datafusion
logical/physical plans logic).
Many, many thanks !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
