Re: [DISCUSS] KIP-1039: Disable automatic topic creation for MirrorMaker2 consumers

[aaron ai]

Hi Omnia,
Thanks for your feedback!

Yes, another approach could be to disable this option manually. IMHO, it's
hard to envision a scenario where enabling it would be necessary. BTW, I've
already added this part into the KIP.

On Fri, Apr 19, 2024 at 6:18 PM Omnia Ibrahim 
wrote:

> Hi Aaron,
> You mentioned that there is no public interface changes however changing
> the default value of a config should be considered as a public change. You
> can check other KIP where we changed the default config value for a
> reference.
>
> Can you please list what is the impact of changing the behaviour of the
> topic creation along side  as well as is there any rejected alternatives
> like can’t customer disable allow.auto.create.topics manually for example
> as a workaround?
>
> Thanks
> Omnia
>
> > On 19 Apr 2024, at 10:37, aaron ai  wrote:
> >
> > Hi all,
> >
> > Here is the KIP-1039: Disable automatic topic creation for MirrorMaker2
> > consumers
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1039%3A+Disable+automatic+topic+creation+for+MirrorMaker2+consumers
> >>
> >
> > Looking forward to your feedback!
> >
> > Best regards,
> > Aaron
>
>

[DISCUSS] KIP-1039: Disable automatic topic creation for MirrorMaker2 consumers

[aaron ai]

Hi all,

Here is the KIP-1039: Disable automatic topic creation for MirrorMaker2
consumers
<
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1039%3A+Disable+automatic+topic+creation+for+MirrorMaker2+consumers
>

Looking forward to your feedback!

Best regards,
Aaron

Re: Assistance Needed with MM2 Heartbeat Topic Configuration

[aaron ai]

Hello,

Thank you very much for your suggestions and assistance. I have tried your
second method and successfully used `producer.override.bootstrap.servers`
as the key, achieving the desired effect. I appreciate your warm response
and valuable time. Thanks again!

Best regards,

On Tue, Feb 27, 2024 at 11:55 PM Ryanne Dolan  wrote:

> Hello, the heartbeat connector is a sink connector, so it normally would
> write to the target cluster. I can think of two ways to achieve what you
> want:
>
> 1) set up a second connect cluster that sinks to the source cluster, and
> run just the heartbeat connector there.
>
> 2) override the heartbeat connector's producer configuration to point to
> the source cluster. I forget the exact spelling, but something like
> `producer.bootstrap.servers` etc might work.
>
> Ryanne
>
> On Tue, Feb 27, 2024, 8:51 AM aaron ai  wrote:
>
> > Dear Kafka Community,
> >
> > I am reaching out for some guidance regarding an issue I've encountered
> > while setting up Kafka MirrorMaker2 (MM2) for data synchronization from a
> > source cluster (A) to a target cluster (B).
> >
> > During my setup with MM2 on dedicated mode, I observed that the heartbeat
> > topic is established on the source cluster, and the messages within the
> > heartbeat are being synchronized to the target cluster as expected.
> > However, when configuring MM2 on Kafka Connect using the
> > connect-distributed.properties file, it appears that I can only set the
> > bootstrap.servers to point to the endpoints of the target cluster B.
> > Consequently, when I manually establish the MirrorHeartbeatConnector, the
> > heartbeat topic also gets created on the target cluster instead of the
> > source.
> >
> > This leads me to wonder if this behavior is expected or if I might be
> > missing a step in the configuration process. My goal is to have the
> > heartbeat topic created on the source cluster. Could you please advise on
> > whether this is possible and, if so, how it can be achieved? Are there
> > specific configurations or steps that I should follow to ensure the
> > heartbeat topic is correctly established on the source cluster?
> >
> > I appreciate any insights or recommendations you might have on this
> matter.
> >
> > Best regards,
> >
>

Assistance Needed with MM2 Heartbeat Topic Configuration

[aaron ai]

Dear Kafka Community,

I am reaching out for some guidance regarding an issue I've encountered
while setting up Kafka MirrorMaker2 (MM2) for data synchronization from a
source cluster (A) to a target cluster (B).

During my setup with MM2 on dedicated mode, I observed that the heartbeat
topic is established on the source cluster, and the messages within the
heartbeat are being synchronized to the target cluster as expected.
However, when configuring MM2 on Kafka Connect using the
connect-distributed.properties file, it appears that I can only set the
bootstrap.servers to point to the endpoints of the target cluster B.
Consequently, when I manually establish the MirrorHeartbeatConnector, the
heartbeat topic also gets created on the target cluster instead of the
source.

This leads me to wonder if this behavior is expected or if I might be
missing a step in the configuration process. My goal is to have the
heartbeat topic created on the source cluster. Could you please advise on
whether this is possible and, if so, how it can be achieved? Are there
specific configurations or steps that I should follow to ensure the
heartbeat topic is correctly established on the source cluster?

I appreciate any insights or recommendations you might have on this matter.

Best regards,

Re: Seeking Assistance with MM2 Rest API Configuration - Encountering 404 Error

[aaron ai]

Hi Greg,

Thanks for the explanation. Really appreciate your help!

Best,
Aaron

On Mon, Feb 19, 2024 at 1:39 AM Greg Harris 
wrote:

> Hi Aaron,
>
> This is the expected behavior of KIP-710. The relevant section reproduced
> here:
>
> > Enabling a single Connect REST server in the MirrorMaker 2.0 node, only
> supporting the internal Connect endpoints.
>
> The endpoints that are typically exposed in a distributed Connect for
> creating connectors, getting status, etc are not enabled in dedicated mode,
> even with the new configuration. Only the internal endpoints that workers
> use to communicate with one another are enabled.
>
> If you want the other endpoints for getting the status, reconfiguring
> connectors, etc, I would recommend switching to the connect distributed
> mode and creating the MM2 connectors via the REST API.
>
> Hope this helps!
> Greg
>
> On Sun, Feb 18, 2024 at 2:00 AM aaron ai  wrote:
>
> > Kafka Development Team:
> >
> > I have enabled the internal REST API for the dedicated mode by setting
> > *dedicated.mode.enable.internal.rest* to true in my MM2 configuration
> > since KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > 2.0 clusters
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> >.
> > However, when I attempt to make requests to the REST API on
> localhost:8083,
> > I consistently receive a 404 Not Found error. This behavior has left me
> > uncertain whether I'm facing a configuration issue or if this might be
> > indicative of a bug within the system.
> >
> > BTW, I found a similar question on Stack Overflow
> > <
> https://stackoverflow.com/questions/76780739/distributed-mode-in-dedicated-mirrormaker-2-cluster-not-balancing-tasks-evenly#:~:text=If%20I%20try%20running%20curl%20targeting%20the%20leaderUrl%20above%20(on%20the%20root%20or%20at%20/connectors)%20from%20any%20node%2C%20using%20the%20advertised%20IP%20(or%20localhost%20in%20the%20leader)%20I%20end%20up%20with%20%7B%22error_code%22%3A404%2C%22message%22%3A%22HTTP%20404%20Not%20Found%22%7D
> >.
> > My Kafka Version: 3.6.0
> >
> > Here is my configuration:
> > clusters = A, B
> > A.bootstrap.servers = 10.1.1.168:9092
> > B.bootstrap.servers = 10.1.1.168:9592
> > A->B.enabled = true
> > A->B.topics = .*
> > groups=.*
> > groups.exclude = test-.*
> > replication.factor=1
> > checkpoints.topic.replication.factor=1
> > heartbeats.topic.replication.factor=1
> > offset-syncs.topic.replication.factor=1
> > offset.storage.replication.factor=1
> > status.storage.replication.factor=1
> > config.storage.replication.factor=1
> > sync.topic.acls.enabled = true
> > emit.heartbeats.interval.seconds = 5
> > sync.group.offsets.enabled=true
> > sync.topic.configs.enabled=true
> > dedicated.mode.enable.internal.rest=true
> >
> > and the HTTP response.
> > [image: image.png]
> > [image: image.png]
> >
> > I eagerly await your response and appreciate any feedback you might have
> > on this matter.
> >
>

Seeking Assistance with MM2 Rest API Configuration - Encountering 404 Error

[aaron ai]

Kafka Development Team:

I have enabled the internal REST API for the dedicated mode by setting
*dedicated.mode.enable.internal.rest* to true in my MM2 configuration
since KIP-710:
Full support for distributed mode in dedicated MirrorMaker 2.0 clusters
.
However, when I attempt to make requests to the REST API on localhost:8083,
I consistently receive a 404 Not Found error. This behavior has left me
uncertain whether I'm facing a configuration issue or if this might be
indicative of a bug within the system.

BTW, I found a similar question on Stack Overflow
.
My Kafka Version: 3.6.0

Here is my configuration:
clusters = A, B
A.bootstrap.servers = 10.1.1.168:9092
B.bootstrap.servers = 10.1.1.168:9592
A->B.enabled = true
A->B.topics = .*
groups=.*
groups.exclude = test-.*
replication.factor=1
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1
sync.topic.acls.enabled = true
emit.heartbeats.interval.seconds = 5
sync.group.offsets.enabled=true
sync.topic.configs.enabled=true
dedicated.mode.enable.internal.rest=true

and the HTTP response.
[image: image.png]
[image: image.png]

I eagerly await your response and appreciate any feedback you might have on
this matter.

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

[aaron ai]

Dear David:

I greatly appreciate your welcome and the explanation that you offered
regarding the Jenkins system!

I will try to familiarize myself with the standards and contribution
process of the Kafka community, and look forward to making some
contributions in the coming days.

Best regards,
Aaron

On Thu, May 4, 2023 at 11:47 PM David Arthur
 wrote:

> Hello, Aaron and welcome to the project!
>
> If you look towards the bottom of a Pull Request there will be a section
> reporting the status of the Checks for the latest commit.
> There is a "Details" link that takes you to the Jenkins job for that PR.
>
> The default Jenkins view is the new UI (called Blue Ocean I think?). The
> "Classic" Jenkins view is also available by clicking the button near the
> top:
>
> [image: image.png]
>
> From the Classic view it's pretty straightforward to download the console
> log for the job to see what went wrong.
>
> Our build steps are defined in the Jenkinsfile
> <https://github.com/apache/kafka/blob/trunk/Jenkinsfile> checked into the
> repository. We do have a compile + static analysis step that happens before
> running tests.
>
> -David
>
> On Thu, May 4, 2023 at 5:34 AM aaron ai  wrote:
>
>> Hey folks,
>>
>> I hope this email finds you well. My name is Aaron Ai, and I'm a beginner
>> interested in the Kafka community. I am eager to contribute to Kafka's
>> development and be part of this amazing project.
>>
>> However, I have noticed some confusion surrounding the CI pipeline in
>> Kafka
>> GitHub pull requests. It appears that the builds often fail, and the
>> results are not very clear to analyse. How can I determine if a pull
>> request build is okay for now, given the current CI pipeline situation?
>> Furthermore, is it a good idea to use Github Action to help with some
>> static checks? If so, I would be more than happy to contribute. By the
>> way,
>> is there any plan to migrate to Github Actions in the near future?
>>
>> I eagerly await your response and appreciate any feedback you might have
>> on
>> this matter. Thank you for your time and consideration.
>>
>> Best regards,
>> Aaron
>>
>
>
> --
> -David
>

Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

[aaron ai]

Hey folks,

I hope this email finds you well. My name is Aaron Ai, and I'm a beginner
interested in the Kafka community. I am eager to contribute to Kafka's
development and be part of this amazing project.

However, I have noticed some confusion surrounding the CI pipeline in Kafka
GitHub pull requests. It appears that the builds often fail, and the
results are not very clear to analyse. How can I determine if a pull
request build is okay for now, given the current CI pipeline situation?
Furthermore, is it a good idea to use Github Action to help with some
static checks? If so, I would be more than happy to contribute. By the way,
is there any plan to migrate to Github Actions in the near future?

I eagerly await your response and appreciate any feedback you might have on
this matter. Thank you for your time and consideration.

Best regards,
Aaron

Requesting permissions to contribute to Apache Kafka

[aaron ai]

Hello,

I'm following
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-GettingStarted
and have reached step 4.

wiki ID: aaronai
Jira ID: aaronai