[ https://issues.apache.org/jira/browse/SPARK-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15785953#comment-15785953 ]
Shixiong Zhu commented on SPARK-18942: -------------------------------------- Thanks for your prototype. Actually, you can just implement an RDD action, or DataFrame DataSource and put them as a Spark package like [spark-redshift|https://github.com/databricks/spark-redshift]. [Spark Packages|https://spark-packages.org/] is a better place for such third-party data sources. > Support output operations for kinesis > ------------------------------------- > > Key: SPARK-18942 > URL: https://issues.apache.org/jira/browse/SPARK-18942 > Project: Spark > Issue Type: New Feature > Components: DStreams > Affects Versions: 2.0.2 > Reporter: Takeshi Yamamuro > Priority: Trivial > > Spark does not support output operations (e.g. DStream#saveAsTextFile) for > Kinesis. So, officially supporting this is useful for some AWS users, I > think. An usage of the output operations is assumed as follows; > {code} > // Import a class that includes an output function > scala> import org.apache.spark.streaming.kinesis.KinesisDStreamFunctions._ > // Create a DStream > scala> val stream: DStream[String] = ... > // Define a handler to convert the DStream type for output > scala> val msgHandler = (s: String) => s.getBytes("UTF-8") > // Define the output operation > scala> kinesisStream.count().saveAsKinesisStream(streamName, endpointUrl, > msgHandler) > {code} > A prototype I made is here: > https://github.com/apache/spark/compare/master...maropu:OutputOpForKinesis -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org