Bump up for another round of discussion.
To follow up last question raised about running Source (MirrorCheckpoint)
Connector and Sink (MirrorSinkConnector) Connector simultaneously in MM2. From
my testing, they all are functioning well and there seems no significant
degradation or complexity
Hi Mickael,
Definitely we can include Sink connector for checkpoints and even heartbeats,
but I am thinking if the existing 3 connectors, MirrorSourceConnector,
MirrorCheckpointConnector and MirrorHeartbeatConnector are managed separately,
so that we could reduce the footprint of introducing
Hi Ning,
Thanks for the updates.
1. If you have to run a Sink (the new MirrorSinkConnector) and Source
(MirrorCheckpoint) connector for MM2 you will need 2 Connect runtimes.
So this does not work well for users of Connect. I've not really
looked into it yet but I wonder if we should include a
bump for another potential more discussion
On 2020/08/27 23:31:38, Ning Zhang wrote:
> Hello Mickael,
>
> > 1. How does offset translation work with this new sink connector?
> > Should we also include a CheckpointSinkConnector?
>
> CheckpointSourceConnector will be re-used as the same as
Hello Mickael,
> 1. How does offset translation work with this new sink connector?
> Should we also include a CheckpointSinkConnector?
CheckpointSourceConnector will be re-used as the same as current. When EOS is
enabled, we will run 3 connectors:
MirrorSinkConnector (based on SinkConnector)
Thanks Ning for the KIP. Having stronger guarantees when mirroring
data would be a nice improvement!
A few comments:
1. How does offset translation work with this new sink connector?
Should we also include a CheckpointSinkConnector?
2. Migrating to this new connector could be tricky as
Awesome, this will be a huge advancement. I also want to point out that
this KIP implements MirrorSinkConnector as well, finally, which is a very
often requested missing feature in my experience.
Ryanne
On Fri, Aug 21, 2020, 9:45 AM Ning Zhang wrote:
> Hello, I wrote a KIP about MirrorMaker2
Hello, I wrote a KIP about MirrorMaker2 Exactly-once Semantics (EOS)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-656%3A+MirrorMaker2+Exactly-once+Semantics
At the high-level, it resembles the idea of how HDFS Sink Connector achieves
EOS across clusters by managing and storing the
