RE: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

smime.p7m Description: S/MIME encrypted message

Re: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

This sounds like something you can solve by a stateful operator. check out mapWithState. If both the message can be keyed with a common key, then you can define a keyed-state. the state will have a field for the first message.When you see the first message for a key, fill the first field with

Re: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

any valuable feedback is appreciated! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-How-to-do-join-two-messages-in-spark-streaming-Probabaly-messasges-are-in-differnet--tp28161p28163.html Sent from the Apache Spark User List mailing list

RE: [Spark Streaming] How to do join two messages in spark streaming(Probabaly messasges are in differnet RDD) ?

smime.p7m Description: S/MIME encrypted message