[
https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15519970#comment-15519970
]
zhangxinyu commented on SPARK-17631:
------------------------------------
h4. A short design for this feature
h5. Goal
Build an http sink for structured streaming. Streaming query results can be
written out to http servers.
h5. Usage
# The streaming query results should have a single string column.
# We should configure ".format("http").option("url", yourHttpUrl)" in our
programs to create http sinks.
e.g.
val query = counts.writeStream
.outputMode("complete")
.format("http")
.option("url", "yourHttpUrl")
.start()
h5. Design
# Add a class "HttpSink" that extends trait "Sink", and override function
"addBatch".
Override "addBatch": echo Row in dataFrame will be written out through
sending an http post request.
# Add a class "HttpStreamSink" that extends both trait "StreamSinkProvider" and
trait "DataSourceRegister". It overrides two functions:
- shortname: return an string "http"
- createSink: return an HttpSink instance
h5. Other features to debate
# should we support https too?
# do we need to set any headers (i.e. maybe the batch id?)
> Structured Streaming - Add Http Stream Sink
> -------------------------------------------
>
> Key: SPARK-17631
> URL: https://issues.apache.org/jira/browse/SPARK-17631
> Project: Spark
> Issue Type: New Feature
> Components: SQL, Streaming
> Affects Versions: 2.0.0
> Reporter: zhangxinyu
> Priority: Minor
>
> Streaming query results can be sinked to http server through http post request
> github: https://github.com/apache/spark/pull/15194
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
