[
https://issues.apache.org/jira/browse/SPARK-37392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-37392:
------------------------------------
Assignee: (was: Apache Spark)
> Catalyst optimizer very time-consuming and memory-intensive with some
> "explode(array)"
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-37392
> URL: https://issues.apache.org/jira/browse/SPARK-37392
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
> Affects Versions: 3.1.2, 3.2.0
> Reporter: Francois MARTIN
> Priority: Major
>
> The problem occurs with the simple code below:
> {code:java}
> import session.implicits._
> Seq(
> (1, "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x", "x",
> "x", "x", "x", "x", "x", "x")
> ).toDF()
> .checkpoint() // or save and reload to truncate lineage
> .createOrReplaceTempView("sub")
> session.sql("""
> SELECT
> *
> FROM
> (
> SELECT
> EXPLODE( ARRAY( * ) ) result
> FROM
> (
> SELECT
> _1 a, _2 b, _3 c, _4 d, _5 e, _6 f, _7 g, _8 h, _9 i, _10 j, _11 k,
> _12 l, _13 m, _14 n, _15 o, _16 p, _17 q, _18 r, _19 s, _20 t, _21 u
> FROM
> sub
> )
> )
> WHERE
> result != ''
> """).show() {code}
> It takes several minutes and a very high Java heap usage, when it should be
> immediate.
> It does not occur when replacing the unique integer value (1) with a string
> value ({_}"x"{_}).
> All the time is spent in the _PruneFilters_ optimization rule.
> Not reproduced in Spark 2.4.1.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
