tgravescs opened a new pull request #30504: URL: https://github.com/apache/spark/pull/30504
### What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to insert a filter for not null and size > 0 when using inner explode/inline. This is fine in most cases but the extra filter is not needed if the explode is with a create array and not using Literals (it already handles LIterals). When this happens you know that the values aren't null and it has a size. It already handles the empty array. This PR makes it such that it doesn't insert the unneeded filter when used with createArray or CreateMap. To do this I added an extra trait which both of those inherit from. This will make it cleaner to match on that trait and if anyone is adding other expressions or extending via plugins the match will still work if they just extend this new trait. I was looking for the cleanest way to do this. I'm not sure I like the name of this trait but it works and putting it up to get feedback and see if people have better ideas. Note I also changed to match against CollectionGenerator instead of Inline and ExplodeBase explicitly, this seemed cleaner but if there is a reason not to do this let me know. ### Why are the changes needed? remove unneeded filter ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Unit tests added and manually tested various cases ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
