Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20521#discussion_r166409729
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -493,9 +510,23 @@ case class DataSource(
dataSource.createRelation(
sparkSession.sqlContext, mode, caseInsensitiveOptions,
Dataset.ofRows(sparkSession, data))
case format: FileFormat =>
-
sparkSession.sessionState.executePlan(planForWritingFileFormat(format, mode,
data)).toRdd
+ val cmd = planForWritingFileFormat(format, mode, data)
+ val resolvedPartCols = cmd.partitionColumns.map { col =>
+ // The partition columns created in `planForWritingFileFormat`
should always be
+ // `UnresolvedAttribute` with a single name part.
+ assert(col.isInstanceOf[UnresolvedAttribute])
+ val unresolved = col.asInstanceOf[UnresolvedAttribute]
+ assert(unresolved.nameParts.length == 1)
+ val name = unresolved.nameParts.head
+ outputColumns.find(a => equality(a.name, name)).getOrElse {
+ throw new AnalysisException(
+ s"Unable to resolve $name given
[${data.output.map(_.name).mkString(", ")}]")
+ }
+ }
+ val resolved = cmd.copy(partitionColumns = resolvedPartCols,
outputColumns = outputColumns)
--- End diff --
The given physical plan has been registered to the UI, and we can collect
its metrics if we execute it. However if we run
`sparkSession.sessionState.executePlan`, we get a new physical plan which
semantically equals to the given physical plan but not the same object. This
new physical plan is not registered to UI so we can't show metrics correctly in
the UI.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
