Good morning,
I would like to have more information about the acquisition of the licence
Mirror Maker 2.0 and the commercial support that is offered
Thank you very much !
[cid:image001.png@01D87B03.2452BCB0]
Josée Brodeur
Conseillère Approvisionnement stratégique, Technologies d'aff
I was exploring the Mirror maker 2.0. I read through this
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
documentation and I have a few questions.
1) For running mirror maker as a dedicated mirror maker cluster, the
documentation specifies a config file and a
Hi
I was exploring the Mirror maker 2.0. I read through this
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
documentation
and I have a few questions.
1. For running mirror maker as a dedicated mirror maker cluster, the
documentation specifies a config file
Ananya, yes the driver is distributed, but each worker only communicates
via kafka. They do not listen on any ports.
Ryanne
On Sat, Jul 11, 2020, 11:28 AM Ananya Sen wrote:
> Hi
>
> I was exploring the Mirror maker 2.0. I read through this
>
> https://cwiki.apache.org/confluenc
Thank you Ryanne for the quick response.
I further want to clarify a few points.
The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka connect
we have multiple workers and each worker has some assigned task. To map this to
Mirror Maker 2.0, A mirror Maker will driver have some
Any help here would be greatly appreciated.
On Sat, Aug 8, 2020, 12:13 PM Ananya Sen wrote:
> Thank you Ryanne for the quick response.
> I further want to clarify a few points.
>
> The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka
> connect we have multiple w
.max' applies.
> How can I scale up the mirror maker instance so that I can have very
little lag?
Tweak 'tasks.max' and spin up more driver instances.
Ryanne
On Sat, Aug 8, 2020 at 1:43 AM Ananya Sen wrote:
> Thank you Ryanne for the quick response.
> I further want t
Thanks, Ryanne. That answers my questions. I was actually missing this
"tasks.max" property. Thanks for pointing that out.
Furthermore, as per the KIP of Mirror Maker 2.0, there are 3 types of
connectors in a Mirror Maker Cluster:
1. KafkaSourceConnector - focus on replica
pointing that out.
>
> Furthermore, as per the KIP of Mirror Maker 2.0, there are 3 types of
> connectors in a Mirror Maker Cluster:
>
>1. KafkaSourceConnector - focus on replicating topic partitions
>2. KafkaCheckpointConnector - focus on replicating consumer groups
>
12:30 PM Ananya Sen
> wrote:
>
> > Thanks, Ryanne. That answers my questions. I was actually missing this
> > "tasks.max" property. Thanks for pointing that out.
> >
> > Furthermore, as per the KIP of Mirror Maker 2.0, there are 3 types of
> > conn
Hi All,
I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm receiving
the following errors on startup:
ERROR Plugin class loader for connector
'org.apache.kafka.connect.mirror.MirrorSourceConnector' was n
ed
to modify the plugin.path or copy jars around.
Ryanne
On Mon, Dec 16, 2019, 6:23 AM Jamie wrote:
> Hi All,
> I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm
> receiving the following errors on startup:
> ERROR Plugin
e
>
> On Mon, Dec 16, 2019, 6:23 AM Jamie wrote:
>
> > Hi All,
> > I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm
> > receiving the following errors on startup:
> > ERROR Plugin class loader for connector
> > 'org
Connect cluster running
> > 2.4.0 as well? The jars should land in the Connect runtime without any
> need
> > to modify the plugin.path or copy jars around.
> >
> > Ryanne
> >
> > On Mon, Dec 16, 2019, 6:23 AM Jamie wrote:
> >
> > > Hi All,
>
ssing the former. Is the Connect cluster
>> running
>> > 2.4.0 as well? The jars should land in the Connect runtime without any
>> need
>> > to modify the plugin.path or copy jars around.
>> >
>> > Ryanne
>> >
>> > On Mon, Dec 16,
ven your
> question
> >> > about plugin.path I'm guessing the former. Is the Connect cluster
> >> running
> >> > 2.4.0 as well? The jars should land in the Connect runtime without any
> >> need
> >> > to modify the plugin.path or copy jars aro
nd.
Ryanne
On Mon, Dec 16, 2019, 6:23 AM Jamie
wrote:
Hi All,
I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm
receiving the following errors on startup:
ERROR Plugin class loader for connector
'org.apache.kafka.connect.mirror.MirrorSourceConne
Using https://github.com/apache/kafka/tree/trunk/connect/mirror as a guide,
I have build from source the origin/KIP-382 of
https://github.com/apache/kafka.git.
I am seeing 2 issues
* I brought up 2 processes on 2 different nodes ( they are actually pods on
k8s but that should not matter ). They s
Vishal, the first issue is easy: you must set tasks.max to something above
1 (the default) in order to achieve any parallelism. This property is
passed along to the internal Connect workers. It's unfortunate that Connect
is not smart enough to default this property to the number of workers. I
suspe
2nd/restore issue ( I think I need to solve the offsets topic issue before
I go with the scale up and down issue )
As you had indicated, I went ahead and created the offsets topic. The
status of the cluster ( destination ) is thus
opic# Partitions# BrokersBrokers Spread %Brokers Skew %Brokers L
> I do not have a single record in the offsets topic
That's definitely not normal. You are correct that without records in that
topic, MM2 will restart from EARLIEST. The offsets should be stored
periodically and whenever the connectors gracefully shutdown or restart.
Is it possible the topics do
I think this might be it.. Could you confirm. It seems to be on the path to
commit the offsets.. but not sure...
[2019-10-14 15:29:14,531] ERROR Scheduler for MirrorSourceConnector caught
exception in scheduled task: syncing topic ACLs
(org.apache.kafka.connect.mirror.Scheduler:102)
java.util.con
Aah no.. this is more to it. Note sure if related to the above.
https://github.com/axbaretto/kafka/blob/master/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceTaskOffsetCommitter.java#L114
Is timing out based on
https://github.com/axbaretto/kafka/blob/master/connect/runtime
You can disable ACL sync to avoid that error, but it doesn't step on
anything else. That can't be it. Consider enabling TRACE or DEBUG logging
within Connect to see what it's doing. Otherwise Connect tends to swallow
exceptions and retry silently.
Ryanne
On Mon, Oct 14, 2019, 2:16 PM Vishal Santo
> timed out
while waiting for producer to flush outstanding
Yeah, that's what I'd expect to see if Connect was unable to send records
to the downstream remote topics, e.g. if min.in-sync.replicas were
misconfigured. Given some data seems to arrive, it's possible that
everything is configured corr
Hey Ryanne,
The test was on topics that had a 7 day retention. Which
generally implies that the batch size for flush is pretty high ( till the
consumption becomes current ). The offset.flush.timeout.ms defaults to 5
seconds and the code will not send in the offsets if the flush is not
Hey Vishal, glad to hear you're making progress.
> 1. It seems though that flushing [...] the producer and setting the
> offset to the compacting topic is not atomic OR do we use
> transactions here ?
You are correct. I'm working on a KIP and PoC to introduce transactions to
Connect for this e
>> You are correct. I'm working on a KIP and PoC to introduce
transactions to
>> Connect for this exact purpose :)
That is awesome. Any time frame ?
In the mean time the SLA as of now
1. It is conceivable that we flush the producer to the target cluster but
fail to offset commit. If there was
That's right. MM2 is at-least-once for now, same as legacy MirrorMaker. You
can follow https://issues.apache.org/jira/browse/KAFKA-6080 for updates on
exactly-once semantics in Connect.
Ryanne
On Tue, Oct 15, 2019 at 1:24 PM Vishal Santoshi
wrote:
> >> You are correct. I'm working on a KIP an
Thank you so much for all your help. Will keep you posted on tests I do..
I hope this is helpful to other folks too..
On Tue, Oct 15, 2019 at 2:44 PM Ryanne Dolan wrote:
> That's right. MM2 is at-least-once for now, same as legacy MirrorMaker. You
> can follow https://issues.apache.org/jira/bro
Hey Ryanne,
How do I override auto.offset.reset = latest for consumers through
mm2.properties. I have tried straight up . auto.offset.reset and consumer.
auto.offset.reset but it defaults to earliest.. I do have a query in
another thread but though you might know off hand..
I would imagine
Vishal, you should be able to override the properties passed to the
internal workers using properties like A->B.consumer.auto.offset.reset or
A.consumer.auto.offset.reset in the mm2.properties file. Certain top-level
properties like tasks.max are honored without the A->B or A prefix, but
auto.offse
Hmm ( I did both )
another->another_test.enabled = true
another->another_test.topics = act_post
another->another_test.emit.heartbeats.enabled = false
another->another_test.consumer.auto.offset.reset = latest
another->another_test.sync.topic.acls.enabled = false
another.consumer.auto.offset.r
Ah, I see you are correct. Also I misspoke saying "workers" earlier, as the
consumer is not created by the worker, but the task.
I suppose the put() could be changed to putIfAbsent() here to enable this
property to be changed. Maybe submit a PR?
Ryanne
On Thu, Oct 17, 2019 at 10:00 AM Vishal San
Oh sorry a. COUNTER... is more like it
On Fri, Oct 18, 2019, 6:58 AM Vishal Santoshi
wrote:
> Will do
> One more thing the age/latency metrics seem to be analogous as in they
> seem to be calculated using similar routines. I would think a metric
> tracking
> the number of flush failures
Will do
One more thing the age/latency metrics seem to be analogous as in they
seem to be calculated using similar routines. I would think a metric
tracking
the number of flush failures ( as a GAUGE ) given offset.flush.timeout.ms
would be highly beneficial.
Regards..
On Thu, Oct 17, 2019
Hey Ryanne,
I see a definite issue. I am doing an intense test and I bring
up 12 VMs ( they are 12 pods with 8 cpus each ), replicating about 1200
plus topics ( fairly heavy 100mbps ) ... They are acquired and are
staggered as they come up..I see a fraction of these nodes not assigned
What is tasks.max? Consider bumping to something like 48 if you're running
on a dozen nodes.
Ryanne
On Fri, Oct 18, 2019, 1:43 PM Vishal Santoshi
wrote:
> Hey Ryanne,
>
>
> I see a definite issue. I am doing an intense test and I bring
> up 12 VMs ( they are 12 pods with 8 cpus each
OK, You will have to explain :)
I had 12 VMs with 8 cpus and 8 max tasks. I thought let me give a CPU to
each task, which I presumed is a java thread ( even though I know the
thread would be mostly ip bound ). . I saw the issue I pointed up.
*I now have 8 VMs 8 cpus with 48 max tasks and it did s
I misspoke
>> I now have 8 VMs 8 cpus with 48 max tasks and it did spread to the the
8 VMs. I then upscaled to 12 VMs and the tasks *have not *migrated as I
would expect .
On Fri, Oct 18, 2019 at 8:00 PM Vishal Santoshi
wrote:
> OK, You will have to explain :)
>
> I had 12 VMs with 8 cpus a
Here is what I see
* The max tasks are a a cap on a Connector across the cluster. If have 8
VMs but 8 max tasks my assumption that there would be 8 * 8 = 72 task
threads was
wring. The logs showed that the partitions were consumed by 8 threads on
the 8 VMs ( 1 per VM ) which was highly un optim
Vishal, the number of tasks created per source->target herder is determined
by both tasks.max and the total number of topic-partitions being
replicated. In order to use all 12 worker nodes, you'd need tasks.max >= 12
and number of topic-partitions >= 12. From previous emails it sounds like
you have
Hey Ryanne,
Seeing the below ERROR in the logs and then, it seems the
process does not consume ( it does not exit with any errors ) . And this is
intermittent. As in do it enough times. that does relaunch :) Is this
something a known bug
[mm2-dev-58bf5df684-ln9k2] [2019-10-2
The above may not be an issue as in it just uses the returned class
loader to resolve the Connector I think . What is not obvious, why it does
not go ahead and consume ..
[mm2-dev-749469cf68-vpm2l] [2019-10-24 20:26:07,571] INFO refreshing known
target topics took 15 ms (org.apache.kafka.connec
I might have created a build from the trunk, rather then the 2.4 branch ,
but will confirm.
On Thu, Oct 24, 2019 at 4:44 PM Vishal Santoshi
wrote:
> The above may not be an issue as in it just uses the returned class
> loader to resolve the Connector I think . What is not obvious, why it does
Yep, I increase the number of tasks and the umber of nodes executing the
tasks, the above issues appears. It creates the topics ( so non issue
reaching the clusters ) but does not create the MirrorSourceConnector/s .
The process does launch the MirrorCheckPointConnector, one for each node. I
took
Hello All,
I was using a mirror maker 2.0. I was testing the consumer checkpointing
functionality. I found that the RemoteClusterUtils.translateOffsets do not give
checkpoints for the consumer which run in assign mode.
I am using mirror maker 2.0 of Kafka Version 2.5.0 and Scala version 2.12
on.ofMillis(5500));
Properties= Bootstraps properties of cluster B
TestTopic-123= Topic name at cluster A
Thanks
On 9/7/20, 8:43 AM, "Ananya Sen" wrote:
[External]
Hello All,
I was using a mirror maker 2.0. I was testing the consumer che
"Ananya Sen" wrote:
>
> [External]
>
>
> Hello All,
>
> I was using a mirror maker 2.0. I was testing the consumer
> checkpointing functionality. I found that the
> RemoteClusterUtils.translateOffsets do not give checkpoints for the
> consumer w
mirror maker 2.
2. Mirror Maker 1.0 : working as same consumer group can be assigned in
consumer.properties file and the messages are picked right after where
replicator was stopped.
3. Tried running and configuring source.cluster.consumer.group.id in
mirror maker 2.0 in all available
as same consumer group can be assigned in
> consumer.properties file and the messages are picked right after where
> replicator was stopped.
>
> 3. Tried running and configuring source.cluster.consumer.group.id in
> mirror maker 2.0 in all available options (in cluster mode, i
: users@kafka.apache.org
Subject: Re: Maintaining same offset while migrating from Confluent Replicator
to Apache Mirror Maker 2.0
[External]
Hi Amit, I guess you may need to override the actual consumer group config
(probably not consumer.group.id) that is used in Kafka Connect
On 2020/11/26 06:47
gt; (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
>
>
>
> [2020-12-08 05:11:07,615] INFO [Consumer clientId=onprem-aws-replicator-0,
> groupId=onprem-aws-replicator] Seeking to offset 53 for partition
> ONPREM.AWS.REPLICA.TOPIC.P3R3-2
> (org.apache.kafka.cli
December 7, 2020 3:46 AM
> To: users@kafka.apache.org
> Subject: Re: Maintaining same offset while migrating from Confluent
> Replicator to Apache Mirror Maker 2.0
>
> [External]
>
>
> Hi Amit, I guess you may need to override the actual consumer group config
> (probably not co
:07,615] INFO [Consumer clientId=onprem-aws-replicator-0,
groupId=onprem-aws-replicator] Seeking to offset 53 for partition
ONPREM.AWS.REPLICA.TOPIC.P3R3-2
(org.apache.kafka.clients.consumer.KafkaConsumer:1545)
Mirror Maker 2.0:
[2020-12-08 06:10:51,385] INFO [Consumer clientId=consumer-4
icator] Seeking to offset 49 for partition
> > ONPREM.AWS.REPLICA.TOPIC.P3R3-1
> > (org.apache.kafka.clients.consumer.KafkaConsumer:1545)>
> >
> >
> >
> > [2020-12-08 05:11:07,615] INFO [Consumer clientId=onprem-aws-replicator-0,
> > groupId=onprem-aws
oupId=onprem-aws-replicator] Seeking to offset 53 for partition
> ONPREM.AWS.REPLICA.TOPIC.P3R3-2
> (org.apache.kafka.clients.consumer.KafkaConsumer:1545)
>
>
>
> Mirror Maker 2.0:
>
> [2020-12-08 06:10:51,385] INFO [Consumer clientId=consumer
ensure no duplicate messages are
replicated.
This has been tested and found to be working as expected.
Thanks and regards,
Amit
-Original Message-
From: Ning Zhang
Sent: Thursday, December 10, 2020 10:40 PM
To: users@kafka.apache.org
Subject: Re: RE: RE: Maintaining same offset while
ate messages
> are replicated.
>
> This has been tested and found to be working as expected.
>
> Thanks and regards,
> Amit
>
> -Original Message-
> From: Ning Zhang
> Sent: Thursday, December 10, 2020 10:40 PM
> To: users@kafka.apache.org
> Subject:
59 matches
Mail list logo
