[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258994#comment-14258994 ]
Saisai Shao commented on SPARK-4960: ------------------------------------ What confused me a lot is the type inference, as the interceptor add, say M => Iterable\[T\], so the type of DStream\[M\] need to be changed into DStream\[T\], while actually the type of input DStream is M, like: {code} DStream[T] => InputDStream[T] => ReceiverInputDStream[T] => SocketInputDStream[M] {code} Assuming I add the interceptor before the ReceiverInputDStream. I'm not sure there's way to do it? A possible way is changing the interceptor to Any => Iterable\[T\], but how to reflect the received data type is also a problem, still function like bytesToObjects: InputStream => Iterator\[T\] is hard to work. Another way is to change the signature of SocketInputDStream, like: {code} DStream[T] => InputDStream[T] => ReceiverInputDStream[T] => SocketInputDStream[M, T] {code} But this will require related socketStream API change to add another ClassTag. To keep without changing the current API, while get the correct type information seems a little hard to design. Any suggestion would be greatly appreciated. > Interceptor pattern in receivers > -------------------------------- > > Key: SPARK-4960 > URL: https://issues.apache.org/jira/browse/SPARK-4960 > Project: Spark > Issue Type: New Feature > Components: Streaming > Reporter: Tathagata Das > > Sometimes it is good to intercept a message received through a receiver and > modify / do something with the message before it is stored into Spark. This > is often referred to as the interceptor pattern. There should be general way > to specify an interceptor function that gets applied to all receivers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org