ammarchalifah commented on PR #59410: URL: https://github.com/apache/spark/pull/59410#issuecomment-6163843135
Thank you for the thorough review @peter-toth ! > The test section says the shape fails in createShuffleSpec with a type mismatch on base. With the in-memory source it doesn't throw, because GenericInternalRow.get doesn't check the type. The test fails only at the arity assertion. A throw needs a connector whose key rows check types, like Iceberg's StructInternalRow. Seems like the type mismatch was raised in my production workload that actually read from Iceberg, not in-memory storage. I should change the description, but I wonder if we could also do test on Iceberg source rather than in-memory only. > AQE isn't required. An outer join over an SPJ join calls createShuffleSpec on the inner scan's projected partitioning in EnsureRequirements (lines 108 and 778). With AQE off the misaligned rows reach KeyGroupedPartitioning.apply there too. Got it! Will update the description. > 3. JIRA fields: [SPARK-60203](https://issues.apache.org/jira/browse/SPARK-60203)'s component is Spark Core, but the change is in SQL. [SPARK-55588](https://issues.apache.org/jira/browse/SPARK-55588) (open) reports the same Iceberg error on a table partitioned by identity(collected_date), bucket(32, user_id), so it's worth linking. Got it, issue updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
