dwsmith1983 commented on PR #5867: URL: https://github.com/apache/datafusion-comet/pull/5867#issuecomment-5779025979
> Are you and @LinSimon-901101 happy for this one to carry that too? Yes. This pull request already routes `map_from_arrays` through the dispatcher and rewrites `map_from_arrays_dedup_policy.sql` the same way, so #5844's remaining pieces can rebase on top like #5875's. On Spark 3.4.3 the LAST_WIN fixture here reads its arrays from parquet with the nested vectorized reader left at its default, and the 3.4 run you did came back green, so the data avoids SPARK-48019. > Could you cite those numbers here and say plainly that the lookup-only regression is accepted in exchange for keeping the projection in Comet? Added to the description under a Performance heading, with the 65,536-row figures from #5875 and the tradeoff stated as you put it. The four inline points are addressed in 0d61cac1e, with a reply on each thread. The SQL file suite passes on Spark 3.5 with the new column-fed `arrays_zip` query, and the regenerated compatibility guide shows the reworded reasons and the new `array_append` note. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
