Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r100231056
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -218,7 +247,14 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
bucketSpec = getBucketSpec,
options = extraOptions.toMap)
- dataSource.write(mode, df)
+ val destination = source match {
+ case "jdbc" => extraOptions.get(JDBCOptions.JDBC_TABLE_NAME)
+ case _ => extraOptions.get("path")
--- End diff --
> is like metadata
It is metadata, but that doesn't mean it doesn't have meaning and thus
doesn't need structure. Some of the metadata currently models the "where" the
data is being written. Internally it doesn't really matter much how much it's
handled (it's an "implementation detail"), but, for someone building an
application that uses this information, knowing that a particular key means
"where the data will end up" *is* very important, and a structured type with
proper, documented fields helps that.
We just happen to want that information, and we could use it either way,
but that's beside the point. I'm arguing that there's value in exposing this
data in a more structured manner than just an opaque map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
