Kousuke Saruta created SPARK-28548:
--------------------------------------
Summary: explain() shows wrong result for persisted DataFrames
after some operations
Key: SPARK-28548
URL: https://issues.apache.org/jira/browse/SPARK-28548
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta
After some operations against Datasets and then persist them, Dataset.explain
shows wrong result.
One of those operations is explain() itself.
An example here.
{code}
val df = spark.range(10)
df.explain
df.persist
df.explain
{code}
Expected result is like as follows.
{code}
== Physical Plan ==
*(1) ColumnarToRow
+- InMemoryTableScan [id#7L]
+- InMemoryRelation [id#7L], StorageLevel(disk, memory, deserialized, 1
replicas)
+- *(1) Range (0, 10, step=1, splits=12)
{code}
But I got this.
{code}
== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
