If i understand it correctly, then in your transform function make it return DStream[OldMsg, TranformedMsg] and then in the try...catch you can access ._1 for oldMsg and then ._2 for transformedMsg.
Thanks Best Regards On Wed, Sep 16, 2015 at 2:46 PM, Samya MAITI <[email protected]> wrote: > Hi Akhil, > > > > I suppose this will give me the transformed msg & not the original msg. > > > > I need the data corresponding to *msgStream* & not *wordCountPair*. > > > > As per my understanding, we need to keep a copy of incoming stream (not > sure how), so as to refer to that in catch block. > > > > Regards, > > Sam > > > > *From:* Akhil Das [mailto:[email protected]] > *Sent:* Wednesday, September 16, 2015 12:24 PM > *To:* Samya MAITI <[email protected]> > *Cc:* [email protected] > *Subject:* Re: Getting parent RDD > > > > How many RDDs are you having in that stream? If its a single RDD then > you could do a .foreach and log the message, something like: > > > > > > val ssc = .... > val msgStream = ..... //SparkKafkaDirectAPI > val wordCountPair = TransformStream.transform(msgStream) > /wordCountPair.foreach( > > msg > > => > try{ > //Some action that causes exception > }catch { > case ex1 : Exception => { > // *How to get hold of the msgStream, so that I can log the > actual message that caused the exception.* > > Log.error("Whoops! This message :=>" + msg) > > > } > > )/ > > > > > Thanks > > Best Regards > > > > On Tue, Sep 15, 2015 at 9:13 PM, Samya <[email protected]> wrote: > > Hi Team > > I have the below situation. > > val ssc = .... > val msgStream = ..... //SparkKafkaDirectAPI > val wordCountPair = TransformStream.transform(msgStream) > /wordCountPair.foreachRDD(rdd => > try{ > //Some action that causes exception > }catch { > case ex1 : Exception => { > // *How to get hold of the msgStream, so that I can log the > actual message that caused the exception.* > } > )/ > > > Regards, > Sam > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Getting-parent-RDD-tp24701.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > >
