Saisai Shao created SPARK-6051: ---------------------------------- Summary: Add an option for DirectKafkaInputDStream to commit the offsets into ZK Key: SPARK-6051 URL: https://issues.apache.org/jira/browse/SPARK-6051 Project: Spark Issue Type: Improvement Components: Streaming Affects Versions: 1.3.0 Reporter: Saisai Shao
Currently in DirectKafkaInputDStream, offset is managed by Spark Streaming itself without ZK or Kafka involved, which will make several third-party offset monitoring tools fail to monitor the status of Kafka consumer. So here as a option to commit the offset to ZK when each job is finished, the process is implemented as a asynchronized way, so the main processing flow will not be blocked, already tested with KafkaOffsetMonitor tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org