[ https://issues.apache.org/jira/browse/SPARK-37392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454192#comment-17454192 ]
Apache Spark commented on SPARK-37392: -------------------------------------- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/34823 > Catalyst optimizer very time-consuming and memory-intensive with some > "explode(array)" > --------------------------------------------------------------------------------------- > > Key: SPARK-37392 > URL: https://issues.apache.org/jira/browse/SPARK-37392 > Project: Spark > Issue Type: Bug > Components: Optimizer > Affects Versions: 3.1.2, 3.2.0 > Reporter: Francois MARTIN > Priority: Major > > The problem occurs with the simple code below: > {code:java} > import session.implicits._ > Seq( > (1, "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", > "x", "x", "x", "x", "x", "x") > ).toDF() > .checkpoint() // or save and reload to truncate lineage > .createOrReplaceTempView("sub") > session.sql(""" > SELECT > * > FROM > ( > SELECT > EXPLODE( ARRAY( * ) ) result > FROM > ( > SELECT > _1 a, _2 b, _3 c, _4 d, _5 e, _6 f, _7 g, _8 h, _9 i, _10 j, _11 k, > _12 l, _13 m, _14 n, _15 o, _16 p, _17 q, _18 r, _19 s, _20 t, _21 u > FROM > sub > ) > ) > WHERE > result != '' > """).show() {code} > It takes several minutes and a very high Java heap usage, when it should be > immediate. > It does not occur when replacing the unique integer value (1) with a string > value ({_}"x"{_}). > All the time is spent in the _PruneFilters_ optimization rule. > Not reproduced in Spark 2.4.1. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org