[
https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044850#comment-15044850
]
Rodrigo Boavida commented on SPARK-12178:
-----------------------------------------
This is probably the first step onto the effort of making the concept of Direct
Streams generic and reusable for different technologies (not just Kafka).
Reactive streams concept is an example of a further step.
I'd like to tag here Prakash Chockalingam from Databricks with who I had this
conversation about on the latest Spark Summit, but can't find his user name.
CCing Iulian as well
[~dragos]
Tnks,
Rod
> Expose reporting of StreamInputInfo for custom made streams
> -----------------------------------------------------------
>
> Key: SPARK-12178
> URL: https://issues.apache.org/jira/browse/SPARK-12178
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Rodrigo Boavida
> Priority: Minor
>
> For custom made direct streams, the Spark Streaming context needs to be
> informed of the RDD count per batch execution. This is not exposed by the
> InputDStream abstract class.
> The suggestion is to create a method in the InputDStream class that reports
> to the streaming context and make that available to child classes of
> InputDStream.
> Signature example:
> def reportInfo(validTime : org.apache.spark.streaming.Time, inputInfo :
> org.apache.spark.streaming.scheduler.StreamInputInfo)
> I have already done this on my own private branch. I can merge that change in
> if approval is given.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
