GitHub user lw-lin reopened a pull request: https://github.com/apache/spark/pull/13518
[WIP][SPARK-15472][SQL] Add support for writing in `csv`, `json`, `text` formats in Structured Streaming ## What changes were proposed in this pull request? This patch adds support for writing in `csv`, `json`, `text` formats in Structured Streaming: **1. at a high level, this patch forms the following hierarchy**(`text` as an example): ``` <OutputWriter> â TextOutputWriterBase â â BatchTextOutputWriter StreamingTextOutputWriter ``` ``` <OutputWriterFactory> â â BatchTextOutputWriterFactory StreamingOutputWriterFactory â StreamingTextOutputWriterFactory ``` The `StreamingTextOutputWriter` and other 'streaming' output writers would write data **without** using an `OutputCommitter`. This was the same approach taken by [SPARK-14716](https://github.com/apache/spark/pull/12409). **2. to support compression, this patch attaches an extension to the path assigned by `FileStreamSink`**, which is slightly different from [SPARK-14716](https://github.com/apache/spark/pull/12409). For example, if we are writing out using the `gzip` compression and `FileStreamSink` assigns path `${uuid}` to a text writer, then in the end the file written out will be `${uuid}.txt.gz` -- so that when we read the file back, we'll correctly interpret it as `gzip` compressed. ## How was this patch tested? `FileStreamSinkSuite` is expanded much more to cover the added `csv`, `json`, `text` formats: ```scala test(" csv - unpartitioned data - codecs: none/gzip") test("json - unpartitioned data - codecs: none/gzip") test("text - unpartitioned data - codecs: none/gzip") test(" csv - partitioned data - codecs: none/gzip") test("json - partitioned data - codecs: none/gzip") test("text - partitioned data - codecs: none/gzip") test(" csv - unpartitioned writing and batch reading - codecs: none/gzip") test("json - unpartitioned writing and batch reading - codecs: none/gzip") test("text - unpartitioned writing and batch reading - codecs: none/gzip") test(" csv - partitioned writing and batch reading - codecs: none/gzip") test("json - partitioned writing and batch reading - codecs: none/gzip") test("text - partitioned writing and batch reading - codecs: none/gzip") ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark add-csv-json-text-for-ss Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13518.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13518 ---- commit 97034f9aeb092b10e1606e60a8e6b4878ebd54cf Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T09:03:04Z Add csv, json, text commit 2035b597b44aa519d8da3b155036446f88b3050e Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T09:03:15Z Fix parquet extension commit 4737361489fd680405b291ec498ab91374685ffe Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T11:52:14Z Fix style commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T08:02:32Z Fix tests commit daec480bd16ed52137d32f332debe3806953f4d2 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T09:03:08Z Revert "Fix tests" This reverts commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b. commit 43b68d426e9b64061095eca7a1db0e762843adef Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T09:09:10Z Fix tests commit 56dbb9b4f0f7e2bf76935e0d1d2fc6c6cdf141ff Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T12:34:07Z Investigate test commit 91e51aed5caf663d5068057e5cf28f21eb768310 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T12:43:39Z Investigate test commit eb2090ce9fa04efbd23370f7d3e6cb98fd0b4c74 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T13:00:26Z Update run-tests commit 1e9af1cef892706de8b07728c192dd8ca5e5851e Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T04:10:34Z Investigate test ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org