[ https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528403#comment-15528403 ]
Frederick Reiss commented on SPARK-17631: ----------------------------------------- I can think of two use cases where one might want to have the option to send the output of a Structured Streaming application out over HTTP. The first use case involves using HTTP as a lightweight protocol for transferring bulk data out of a Structured Streaming query to a remote system. In this case, I would think the best approach would be to use a {{ForeachSink}}. That way, the HTTP connections would originate from the executors instead of having all the data pulled into the driver. I'm not sure that there's really a need to add a built-in sink type for this use case, as the existing {{ForeachSink}} provides most of what one would need. The second use case involves tunneling data out of the Spark cluster to an existing legacy system that speaks HTTP or HTTPS. For example, one might want a Structured Streaming application to send ML model updates to an application server via a secured connection. In this use case, I would think that it would be appropriate for the data to be pulled to the driver as in the PR. However, you would want a lot more configurability in any facility built into Spark; otherwise, there isn't much benefit over what a user could throw together in a few minutes over the (as yet unopened, but hopefully soon available) Sink API. > Structured Streaming - Do we need to output results through http API? > --------------------------------------------------------------------- > > Key: SPARK-17631 > URL: https://issues.apache.org/jira/browse/SPARK-17631 > Project: Spark > Issue Type: New Feature > Components: SQL, Streaming > Affects Versions: 2.0.0 > Reporter: zhangxinyu > Priority: Minor > > Streaming query results can be sinked to http server through http post request -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org