Vivek1106-04 commented on PR #59100: URL: https://github.com/apache/spark/pull/59100#issuecomment-5883949172
Thanks for flagging. I benchmarked chained `withColumn` in a pipeline flow (median of 3, local): - Chains over an internal dataset already fell back to the lazy path before this change (eager analysis fails since the dataset isn't in the catalog yet), so timings are unchanged. - Chains over external/local sources: lazy resolution adds ~20-55 ms at n=10..40 (roughly linear), paid once per flow at graph resolution. - Longer chains can't reach the server from SDP: at n=48 the Python client fails in `register_flow` with `Exceeded upb_DecodeOptions_MaxDepth` while building `WriteRelationFlowDetails`, so the long-chain case from #52253 isn't reachable here. Given that, I'd rather not add a conf, since re-enabling eager would bring the dependency bug back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
