Hi, I have a scenario with spark streaming, where I need to write to a database from within updateStateByKey[1].
That means that inside my update function I need a connection. I have so far understood that I should create a new (lazy) connection for every partition. But since I am not working in foreachRDD I wonder where I can iterate over the partitions. Should I use mapPartitions() somewhere up the chain? Jan [1] The use case being saving ‘done' sessions during web tracking. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
