Yicong-Huang commented on PR #57911: URL: https://github.com/apache/spark/pull/57911#issuecomment-5292056362
> Could you try the micro benchmark with the bool change again? Yes. In the last round, I only appended the e2e results. I have now rerun the ASV microbenchmark with Arrow 25.0.1 as well. The numbers changed slightly, but the savings on the common paths remain significant, and the worst-case regression is acceptable in my view. > Also the benchmark is `spark.createDataFrame`, which means there's no UDF or other encoding involved right? Yup. It is intended to measure the conversion from Python types to Arrow types. No other computation is involved, if that’s what you’re asking. With UDF logic included, the impact becomes a matter of proportion: total time = UDF processing time + conversion time. This change will also benefit the Arrow-optimized Python UDF path, though we do not enable that path by default due to its performance trade-offs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
