Sorry guys may bad,
Here is a high level code sample,
val unionStreams = ssc.union(kinesisStreams)
unionStreams.foreachRDD(rdd => {
rdd.foreach(tweet =>
val strTweet = new String(tweet, "UTF-8")
val interaction = InteractionParser.parser(strTweet)
interactionDAL.insert(interaction)
)
})
Here I have to close the connection for interactionDAL other wise the JVM gives
me error that the connection is open. I tried with sticky connection as well
with keep_alive true. So my guess was that at the point of
“unionStreams.foreachRDD” or at “rdd.foreach” the code is marshaled and send to
workers and workers un-marshals and execute the process, which is why the
connection is alway opened for each RDD. I might be completely wrong. I would
love to know what is going on underneath.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
