[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096939#comment-17096939 ]
Dongjoon Hyun commented on SPARK-29048: --------------------------------------- BTW, this was merged and reverted for `3.0.0` only. This JIRA is irrelevant to 2.4.6 release~ So, we can ignore for 2.4.6 release. > Query optimizer slow when using Column.isInCollection() with a large size > collection > ------------------------------------------------------------------------------------ > > Key: SPARK-29048 > URL: https://issues.apache.org/jira/browse/SPARK-29048 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.4, 2.4.5, 2.4.6 > Reporter: Weichen Xu > Priority: Major > > Query optimizer slow when using Column.isInCollection() with a large size > collection. > The query optimizer takes a long time to do its thing and on the UI all I see > is "Running commands". This can take from 10s of minutes to 11 hours > depending on how many values there are. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org