Re: Kafka not shutting down cleanly; Actor serializtion?

2013-12-09 Thread Michael Malak
UTION. From: Michael Malak To: "dev@spark.incubator.apache.org" Sent: Thursday, September 26, 2013 12:27 PM Subject: Kafka not shutting down cleanly; Actor serializtion? Tathagata: I don't believe Kafka streams are being shut down cleanly, which im

Kafka not shutting down cleanly; Actor serializtion?

2013-09-26 Thread Michael Malak
Tathagata: I don't believe Kafka streams are being shut down cleanly, which implies that the most recent Kafka offsets are not being committed back to Zookeeper, which implies starting/restarting a Spark Streaming process would result in duplicate events. The simple Spark Streaming code (runn