hvanhovell opened a new pull request, #44162: URL: https://github.com/apache/spark/pull/44162
### What changes were proposed in this pull request? This PR makes the following two changes: - The `withColumns` message is now stackable. This means that stacked `withColumns` calls can now be represented by a single message. This should reduce the chance of hitting protobuf recursion limits. - Resolution of `withColumns` calls is moved into the analyzer. This should make resolving stacked withColumns call significantly faster. ### Why are the changes needed? It is fairly common to stack withColumns calls on top of each other (e.g. when computing stats for all columns in a Dataframe). In connect plans with large withColumns chains tend to hit the protobuf recursion limit quickly. On top of this building a dataframe in these cases is rather small because each withColumns call creates a new server DataFrame. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests. I have added ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
