JoshRosen commented on PR #46908: URL: https://github.com/apache/spark/pull/46908#issuecomment-2154434517
I'm switching this back to WIP pending some further performance profiling over a wider array of workloads. It turns out that the Scala compiler generates specialized implementations of `applyOrElse` for `PartialFunction` instances which can be faster than calling `isDefinedAt()` followed by `apply()`. In the case where functions rarely apply my approach here may be faster, but if the case conditions are complex then my change may lead to duplicate evaluations of the case conditions and could potentially harm performance. Given this, I may end up splitting this PR into smaller pieces, opening a separate PR to land only the non `applyOrElse` parts of the changes (e.g. keeping the `ColumnResolutionHelper` changes and the changes to `withOrigin` itself but deferring the other changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
