[
https://issues.apache.org/jira/browse/SPARK-49950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xie shuiahu updated SPARK-49950:
--------------------------------
Description:
```scala
import spark.implicits._
val data = (0 until 10000000).toArray
// spend more than 10s
val ds = spark.createDataset(data)
// spend more than 5s
ds.selectExpr("value+2 as value").selectExpr("value+3 as
value").selectExpr("value+4 as value").selectExpr("value+5 as value")
```
This is caused by `LocalRelation`, because `mapExpressions` will go into the
data and spend a lot of time to tranversal it. Any idea to fix this issue?
was:
```scala
import spark.implicits._
val data = (0 until 10000000).toArray
// spend more than 10s
val ds = spark.createDataset(data)
// spend more than 5s
ds.selectExpr("value+2 as value").selectExpr("value+3 as
value").selectExpr("value+4 as value").selectExpr("value+5 as value")
```
This is caused by `LocalRelation`, because `mapExpressions` will go into the
data and spend a lot of time to tranversal it. Any idea to fix this issue?
> spark planing is too slow when LocalRelation has a lot of data
> --------------------------------------------------------------
>
> Key: SPARK-49950
> URL: https://issues.apache.org/jira/browse/SPARK-49950
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: xie shuiahu
> Priority: Trivial
>
> ```scala
> import spark.implicits._
> val data = (0 until 10000000).toArray
>
> // spend more than 10s
> val ds = spark.createDataset(data)
> // spend more than 5s
> ds.selectExpr("value+2 as value").selectExpr("value+3 as
> value").selectExpr("value+4 as value").selectExpr("value+5 as value")
> ```
>
> This is caused by `LocalRelation`, because `mapExpressions` will go into the
> data and spend a lot of time to tranversal it. Any idea to fix this issue?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
