Xiao Li created SPARK-24867:
-------------------------------
Summary: Add AnalysisBarrier to DataFrameWriter
Key: SPARK-24867
URL: https://issues.apache.org/jira/browse/SPARK-24867
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.1, 2.3.0
Reporter: Xiao Li
Assignee: Xiao Li
{code}
val udf1 = udf({(x: Int, y: Int) => x + y})
val df = spark.range(0, 3).toDF("a")
.withColumn("b", udf1($"a", udf1($"a", lit(10))))
df.cache()
df.write.saveAsTable("t")
df.write.saveAsTable("t1")
{code}
Cache is not being used because the plans do not match with the cached plan.
This is a regression caused by the changes we made in AnalysisBarrier, since
not all the Analyzer rules are idempotent. We need to fix it to Spark 2.3.2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
