If you’re using WAL with Kafka, Spark Streaming will ignore this
configuration(autocommit.enable) and explicitly call commitOffset to update
offset to Kafka AFTER WAL is done.
No matter what you’re setting with autocommit.enable, internally Spark
Streaming will set it to false to turn off
Please see the inline comments.
From: Shushant Arora [mailto:shushantaror...@gmail.com]
Sent: Monday, July 6, 2015 8:51 PM
To: Shao, Saisai
Cc: user
Subject: Re: kafka offset commit in spark streaming 1.2
So If WAL is disabled, how developer can commit offset explicitly in spark
streaming app
commitment mechanism is actually a timer way, so it is
asynchronized with replication.
From: Shushant Arora [mailto:shushantaror...@gmail.com]
Sent: Monday, July 6, 2015 8:30 PM
To: Shao, Saisai
Cc: user
Subject: Re: kafka offset commit in spark streaming 1.2
And what if I disable WAL and use
:30 PM
*To:* Shao, Saisai
*Cc:* user
*Subject:* Re: kafka offset commit in spark streaming 1.2
And what if I disable WAL and use replication of receiver data using
StorageLevel.MEMORY_ONLY2().
Will it commit offset after replicating the message or will it use
autocommit.enable
value
