In spark streaming 1.2 , Is offset of kafka message consumed are updated in
zookeeper only after writing in WAL if WAL and checkpointig are enabled or
is it depends upon kafkaparams while initialing the kafkaDstream.
MapString,String kafkaParams = new HashMapString, String();
autocommit mechanism.
Thanks
Jerry
From: Shushant Arora [mailto:shushantaror...@gmail.com]
Sent: Monday, July 6, 2015 8:11 PM
To: user
Subject: kafka offset commit in spark streaming 1.2
In spark streaming 1.2 , Is offset of kafka message consumed are updated in
zookeeper only after writing in WAL
Please see the inline comments.
From: Shushant Arora [mailto:shushantaror...@gmail.com]
Sent: Monday, July 6, 2015 8:51 PM
To: Shao, Saisai
Cc: user
Subject: Re: kafka offset commit in spark streaming 1.2
So If WAL is disabled, how developer can commit offset explicitly in spark
streaming app
commitment mechanism is actually a timer way, so it is
asynchronized with replication.
From: Shushant Arora [mailto:shushantaror...@gmail.com]
Sent: Monday, July 6, 2015 8:30 PM
To: Shao, Saisai
Cc: user
Subject: Re: kafka offset commit in spark streaming 1.2
And what if I disable WAL and use
:30 PM
*To:* Shao, Saisai
*Cc:* user
*Subject:* Re: kafka offset commit in spark streaming 1.2
And what if I disable WAL and use replication of receiver data using
StorageLevel.MEMORY_ONLY2().
Will it commit offset after replicating the message or will it use
autocommit.enable
value
