Hi, Joong,
Please check the following two links:
-
https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement
-
https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API
They might help you understand the problem.
Cheers,
Xiao Li
If you want 0 data loss you should also look into the min.insync.repica
setting in 0.8.2.1 as it guarantees data in multiple racks.
If you don't have that set then you have this scenario as possible.
lets say 1 topic, 1 partition, replication 3. You are producing with ACK=-1
b1, b2, b3 (where b=
It is based on our understanding from reading the documents.
We aren't concerned of data duplication as that is going to be handled by
elasticsearch.
> On May 1, 2015, at 12:15 AM, Daniel Compton
> wrote:
>
> When we evaluated MirrorMaker last year we didn't find any risk of data
> loss, on
0.8.2.1
> On Apr 30, 2015, at 11:28 PM, Jiangjie Qin wrote:
>
> Which mirror maker version did you look at? The MirrorMaker in trunk
> should not have data loss if you just use the default setting.
>
>> On 4/30/15, 7:53 PM, "Joong Lee" wrote:
>>
>> Hi,
>> We are exploring Kafka to keep two da
When we evaluated MirrorMaker last year we didn't find any risk of data
loss, only duplicate messages in the case of a network partition.
Did you discover data loss in your tests, or were you just looking at the
docs?
On Fri, 1 May 2015 at 4:31 pm Jiangjie Qin
wrote:
> Which mirror maker version
Which mirror maker version did you look at? The MirrorMaker in trunk
should not have data loss if you just use the default setting.
On 4/30/15, 7:53 PM, "Joong Lee" wrote:
>Hi,
>We are exploring Kafka to keep two data centers (primary and DR) running
>hosts of elastic search nodes in sync. One k
Hi,
We are exploring Kafka to keep two data centers (primary and DR) running hosts
of elastic search nodes in sync. One key requirement is that we can't lose any
data. We POC'd use of MirrorMaker and felt it may not meet out data loss
requirement.
I would like ask the community if we should lo