GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/15580
[SPARK-18042][SQL] OutputWriter should expose file path written
## What changes were proposed in this pull request?
This patch adds a new "path" method on OutputWriter that returns the path
of the file to be written (or written) by the OutputWriter. This is part of the
necessary work to consolidate structured streaming and batch write paths.
The batch write path has a nice feature that each data source can define
the extension of the files, and allow Spark to specify the staging directory
and the prefix for the files. However, in the streaming path we need to collect
the list of files written, and there is no interface right now to do that.
## How was this patch tested?
N/A - there is no behavior change and this should be covered by existing
tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-18042
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15580
----
commit 5f3f7288e09fbfe906ebd85310f5efcd6cd58a36
Author: Reynold Xin <r...@databricks.com>
Date: 2016-10-21T07:30:44Z
[SPARK-18042][SQL] OutputWriter should expose file path written
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
