dongjoon-hyun commented on PR #60117: URL: https://github.com/apache/spark/pull/60117#issuecomment-6663620025
Thank you for the review. 1. Projection split: I'd keep the setting to the aliases. In the sharing analysis a Python UDF still counts as expensive, so two aliases that share one are computed in one layer and do not run it twice. The split only adds layers above the child of the `Project`, so a restart does not depend on it, and 4.3.0 has no split to match. Version 0 of the regex estimate differs because it has to add the regex expressions as parts. Otherwise an alias that is expensive only because of a regex would have none. I added a comment near `expensivePartsOf` and a line to the PR description. 2. Restarts that cross no stateful operator or watermark: I'd keep the uniform default. Confining it would make a generic optimizer rule reason about streaming placement, and any placement-sensitive operator the check missed would bring the failures back. The cost for those queries is the same as in released 4.3, and the per-checkpoint pinning matches `regexPatternCostEstimateVersion` and SPARK-60885. I added to the PR description that such queries need a new checkpoint to benefit from SPARK-59693. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
