huaxingao commented on PR #57248: URL: https://github.com/apache/spark/pull/57248#issuecomment-4971869888
Thanks @dongjoon-hyun for the heads up. Agreed this is a real regression introduced in 4.2.0 by the KGP/SPJ refactor in #54330. Before I announce the RC6 result, I want to sync on release impact. I don't think this rises to a blocker, for a few reasons: - Results are never silently wrong. The reorder only affects the canonical copy of the plan, the one Spark uses to compare plans for reuse/subquery/AQE, not the copy it executes. Result-producing execution always runs the original, correctly-ordered plan. - The SPJ planning decision is unaffected. Join-key matching compares partition expressions individually (order-preserving), so the decision to apply SPJ, and the actual join, never depends on the reordered list. - Worst case is a loud, contained failure. Building the canonical copy only rearranges a column list in memory, so it can't fail on its own. Even if the reordered metadata were ever lined up against mixed-type partition values, it would surface as a visible ClassCastException confined to the SPJ / key-grouped-partitioning paths, not silent corruption. - Narrow trigger and a safe fallback. It needs multi-column key-grouped partitioning whose expressions reorder under hashCode. Anyone who hits it can disable SPJ with spark.sql.sources.v2.bucketing.enabled=false and fall back to a normal shuffle join (correct results, just without the SPJ optimization) until 4.2.1. So I'm leaning toward not failing RC6. If anyone feels it should block 4.2.0, please -1 on the vote thread. I'm happy to hold the announcement to discuss first. Also cc @sunchao @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
