[ https://issues.apache.org/jira/browse/SPARK-36918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh resolved SPARK-36918. --------------------------------- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34166 [https://github.com/apache/spark/pull/34166] > unionByName shouldn't consider types when comparing structs > ----------------------------------------------------------- > > Key: SPARK-36918 > URL: https://issues.apache.org/jira/browse/SPARK-36918 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: Adam Binford > Assignee: Adam Binford > Priority: Major > Fix For: 3.3.0 > > > Improvement/follow-on of https://issues.apache.org/jira/browse/SPARK-35290. > We use StructType.sameType to see if we need to recreate the struct, but this > can lead to false positives if the structure is the same but the types are > different, and will lead to simply creating a new struct that's exactly the > same as the original. This can cause significant overhead when unioning > multiple deeply nested nullable structs, as each time it's recreated it gets > wrapped in a If(IsNull()). Only comparing the field names can lead to more > efficient plans. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org