tustvold commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2715020448
> Arrow-rs is not very configurable unlike to DataFusion If you have a concrete proposal, feel free to raise an issue. FWIW most kernels do take various options to alter their behaviour, a non-trivial number of which specifically exist for Spark compatibility. > It is possible to do in DataFusion although it is gonna be a huge change to cover all array functions, so the first idea was if it is possible to have a solution on arrow-rs I think you misunderstand what I am saying, arrow can't be opinionated about what people are permitted to use as the list field name, as people can always do something different. Even if you made new_list_field default to using "element" rather than "item" it wouldn't resolve your problem as kernels could still construct lists with different naming. FWIW arrow-rs doesn't use new_list_field outside of tests for this reason, and in fact I actually objected to its addition for this reason - https://github.com/apache/arrow-rs/issues/4544#issuecomment-1642452355, kernels that construct lists should make this attribute configurable. If DF has such kernels, then DF should make this behaviour configurable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
