[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265508#comment-14265508 ]
Saisai Shao commented on SPARK-4960: ------------------------------------ Hi TD, thanks a lot for your suggestions. What I concerned about changing the interceptor pattern to T => Iterator\[T\] is that it will narrow the usage scenario of this functionality, at least it cannot meet the requirement [~c...@koeninger.org] mentioned for KafkaInputDStream. Is it necessary to add just T => Iterator\[T\] pattern in receiver rather than moving into DStream transformation? For the problem you mentioned, I think maybe we can create a interceptor receiver to ship the data from actual receiver and do some transformation before storing, so each actual receiver will have a shadow interceptor receiver to intercept the data, this might fix the problem but will increase the implementation complexity. I will re-think about my design and try to figure out a better way if possible. Thanks a lot. > Interceptor pattern in receivers > -------------------------------- > > Key: SPARK-4960 > URL: https://issues.apache.org/jira/browse/SPARK-4960 > Project: Spark > Issue Type: New Feature > Components: Streaming > Reporter: Tathagata Das > > Sometimes it is good to intercept a message received through a receiver and > modify / do something with the message before it is stored into Spark. This > is often referred to as the interceptor pattern. There should be general way > to specify an interceptor function that gets applied to all receivers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org