GitHub user lw-lin reopened a pull request:
https://github.com/apache/spark/pull/13518
[WIP][SPARK-15472][SQL] Add support for writing in `csv`, `json`, `text`
formats in Structured Streaming
## What changes were proposed in this pull request?
This patch adds support for writing in `csv`, `json`, `text` formats in
Structured Streaming:
**1. at a high level, this patch forms the following hierarchy**(`text` as
an example):
```
<OutputWriter>
â
TextOutputWriterBase
â â
BatchTextOutputWriter StreamingTextOutputWriter
```
```
<OutputWriterFactory>
â â
BatchTextOutputWriterFactory StreamingOutputWriterFactory
â
StreamingTextOutputWriterFactory
```
The `StreamingTextOutputWriter` and other 'streaming' output writers would
write data **without** using an `OutputCommitter`. This was the same approach
taken by [SPARK-14716](https://github.com/apache/spark/pull/12409).
**2. to support compression, this patch attaches an extension to the path
assigned by `FileStreamSink`**, which is slightly different from
[SPARK-14716](https://github.com/apache/spark/pull/12409). For example, if we
are writing out using the `gzip` compression and `FileStreamSink` assigns path
`${uuid}` to a text writer, then in the end the file written out will be
`${uuid}.txt.gz` -- so that when we read the file back, we'll correctly
interpret it as `gzip` compressed.
## How was this patch tested?
`FileStreamSinkSuite` is expanded much more to cover the added `csv`,
`json`, `text` formats:
```scala
test(" csv - unpartitioned data - codecs: none/gzip")
test("json - unpartitioned data - codecs: none/gzip")
test("text - unpartitioned data - codecs: none/gzip")
test(" csv - partitioned data - codecs: none/gzip")
test("json - partitioned data - codecs: none/gzip")
test("text - partitioned data - codecs: none/gzip")
test(" csv - unpartitioned writing and batch reading - codecs: none/gzip")
test("json - unpartitioned writing and batch reading - codecs: none/gzip")
test("text - unpartitioned writing and batch reading - codecs: none/gzip")
test(" csv - partitioned writing and batch reading - codecs: none/gzip")
test("json - partitioned writing and batch reading - codecs: none/gzip")
test("text - partitioned writing and batch reading - codecs: none/gzip")
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lw-lin/spark add-csv-json-text-for-ss
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13518.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13518
----
commit 97034f9aeb092b10e1606e60a8e6b4878ebd54cf
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-05T09:03:04Z
Add csv, json, text
commit 2035b597b44aa519d8da3b155036446f88b3050e
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-05T09:03:15Z
Fix parquet extension
commit 4737361489fd680405b291ec498ab91374685ffe
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-05T11:52:14Z
Fix style
commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T08:02:32Z
Fix tests
commit daec480bd16ed52137d32f332debe3806953f4d2
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T09:03:08Z
Revert "Fix tests"
This reverts commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b.
commit 43b68d426e9b64061095eca7a1db0e762843adef
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T09:09:10Z
Fix tests
commit 56dbb9b4f0f7e2bf76935e0d1d2fc6c6cdf141ff
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T12:34:07Z
Investigate test
commit 91e51aed5caf663d5068057e5cf28f21eb768310
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T12:43:39Z
Investigate test
commit eb2090ce9fa04efbd23370f7d3e6cb98fd0b4c74
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-06T13:00:26Z
Update run-tests
commit 1e9af1cef892706de8b07728c192dd8ca5e5851e
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T04:10:34Z
Investigate test
commit f76b1d9fe5e4970ad66dfb6d4b2fed939b68728c
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T04:49:12Z
Fix tests
commit 7fca579d9fec2589f13403b0eb0f2d5f5e6bd52a
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T04:50:24Z
Fix tests
commit 2ead307d01d8f908951fbc059b8e49bbc77947b1
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T05:02:55Z
Fix tests
commit b64afc64d3121479eb5c3f8c8b5663b6e05349b7
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T05:35:34Z
Fix tests
commit 2c52c2c6ac961d0b71c2dd3802cd53d51ba1926a
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T05:58:58Z
Fix style
commit aa61f1dab4fa367296fa83948c173985a289a629
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T06:39:22Z
Fix tests
commit c5ac63e139c37e9e29c071ef17bfd0c45b40676b
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T06:53:44Z
Fix tests
commit cc842733d7ad2644ed8ddb7ebd9dcc4def301130
Author: Liwei Lin <lwl...@gmail.com>
Date: 2016-06-07T07:12:46Z
Investigate tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
