Spark Streaming Listener to Kill Stages?

Hello, I was wondering if there's a way to kill stages in spark streaming via a StreamingListener. Sometimes stages will hang, and simply killing it via the UI: http://apache-spark-user-list.1001560.n3.nabble.com/file/n24478/Screen_Shot_2015-08-27_at_11.png is enough to let the streaming job

Spark Streaming Checkpointing Restarts with 0 Event Batches

Hello, I'm using direct spark streaming (from kafka) with checkpointing, and everything works well until a restart. When I shut down (^C) the first streaming job, wait 1 minute, then re-submit, there is somehow a series of 0 event batches that get queued (corresponding to the 1 minute when the

Re: Spark Direct Streaming With ZK Updates

Forgot to include the PR I was referencing: https://github.com/apache/spark/pull/4805/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Direct-Streaming-With-ZK-Updates-tp24423p24424.html Sent from the Apache Spark User List mailing list archive at

Re: Spark Direct Streaming With ZK Updates

When updating the ZK offset in the driver (within foreachRDD), there is somehow a serialization exception getting thrown: 15/08/24 15:45:40 ERROR JobScheduler: Error in job generator java.io.NotSerializableException: org.I0Itec.zkclient.ZkClient at