Re: [VOTE] 3.5.1 RC0

[Federico Valeri]

Hi Divij, thanks for running the release.

I found a couple of issues with licenses. We include
plexus-utils-3.3.1, but the LICENSE file has plexus-utils-3.3.0. The
other issue is that we have classgraph-4.8.138, but we don't include
this library for a few releases (this is non blocking but it would be
good to fix).

Br
Fede




On Wed, Jul 12, 2023 at 12:14 PM Divij Vaidya  wrote:
>
> + kafka-clie...@googlegroups.com
>
> --
> Divij Vaidya
>
>
>
> On Wed, Jul 12, 2023 at 12:03 PM Divij Vaidya 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.5.1.
> >
> > This release is a security patch release. It upgrades the dependency,
> > snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
> > can find more information about the CVE at Kafka CVE list
> > .
> >
> > Additionally, this releases fixes a regression introduced in 3.3.0, which
> > caused security.protocol configuration values to be restricted to upper
> > case only. With this release, security.protocol values are
> > case insensitive. See KAFKA-15053
> >  for details.
> >
> > Release notes for the 3.5.1 release:
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, July 18, 9am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/
> >
> > Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > Javadoc:
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/javadoc/
> >
> > Tag to be voted upon (off 3.5 branch) is the 3.5.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.5.1-rc0
> >
> > Documentation:
> > https://kafka.apache.org/35/documentation.html
> > Please note that documentation will be updated with upgrade notes (
> > https://github.com/apache/kafka/commit/4c78fd64454e25e3536e8c7ed5725d3fbe944a49)
> > after the release is complete.
> >
> > Protocol:
> > https://kafka.apache.org/35/protocol.html
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/35/ (9
> > failures). I am running another couple of runs to ensure that there are no
> > consistently failing tests. I have verified that unit/integration tests on
> > my local machine successfully pass.
> >
> > System tests:
> > Not planning to run system tests since this is a patch release.
> >
> > Thank you.
> >
> > --
> > Divij Vaidya
> > Release Manager for Apache Kafka 3.5.1
> >
> >

RE: How to migrate single node Kafka with zookeeper to Kraft without ending up with additional Kraft controller node

[Pengcheng Wang]

Thanks David! That makes sense. I will keep an eye on the kafka release notes 
for the support of this in the future if possible.

From: David Arthur 
Date: Wednesday, July 5, 2023 at 9:16 AM
To: users@kafka.apache.org 
Subject: [EXTERNAL] Re: How to migrate single node Kafka with zookeeper to 
Kraft without ending up with additional Kraft controller node
Pengcheng,

Right now, migrating to a combined broker+controller is not supported. The
main reason for this is we don't have a way to change a KRaft broker into a
combined KRaft broker + controller. It is probably possible, and may be
supported at some point, but for now we're focusing on getting the
migration feature polished for production use cases.

Thanks!
David

On Mon, Jul 3, 2023 at 5:25 PM Pengcheng Wang
 wrote:

> Hi,
>
> We have a single node kafka and single node zookeeper, which we want to
> migrate to a single node kraft (which act as both controller and broker) so
> we can drop the zookeeper.
>
> I know we have the documentation here “
> https://kafka.apache.org/documentation/#kraft_zk_migration ” for zookeeper
> to Kraft Migration. I followed it and the migration was successful, but I
> end up with an additional kraft controller node which was provisioned for
> the migration purpose. So my question is: is there a way for us to remove
> that additional kraft controller node, so we end up with a single node
> kafka as before?
>
> I know it’s not recommended for production, but given that a single kraft
> node can act as both broker and controller, I guess this is possible?
> Thanks!
>
> Best,
> Pengcheng
>


--
-David

Re: Replication factor for a topic Increase

[Gaurav Pande]

Thanks Luke,

One last thing for future topics that gets auto created is there a way to
put this auto replication.factor setting dynamically rather than modifying
server.properties and restarting all 3  Brokers?
Regards,
GP


On Thu, 13 Jul, 2023, 14:59 Luke Chen,  wrote:

> Once the topic is created, you can only increase the replication factor by:
> 1. delete the topic and re-create the same topic (note: data will be lost
> after this step)
> or
> 2. follow the doc to increase the replication factor (data will be kept)
>
> Thank you.
> Luke
>
> On Thu, Jul 13, 2023 at 5:23 PM Gaurav Pande  wrote:
>
> > Hi Luke ,
> >
> > Thanks does that mean that setting default.replication.factor to more
> than
> > 1 and doing Kafka broker restart wouldn't dynamically change the existing
> > partitions replication? We have to use cli shell script, as per
> > documentation?
> > Regards,
> > GP
> >
> > On Thu, 13 Jul, 2023, 13:54 Luke Chen,  wrote:
> >
> > > Hi Gaurav,
> > >
> > > > Why do I see topic ReplicationFactor:1 during description of a topic
> ?
> > > I think you should set this config: `default.replication.factor`
> > >
> > > > And how can we increase replication factor?  I have 3 Kafka brokers
> > > running on 2.7.0 version.
> > > Please check this doc:
> > >
> > >
> >
> https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have noticed that when I describe my Kafka topic I see
> > > > ReplicationFactor:1 but in my server.properties  file I already have
> > > > defined below values i.e a replication factor of 3 :
> > > >
> > > > num.partions=3
> > > > offsets.topic.replication.factor=3
> > > > transaction.state.log.replication.factor=3
> > > > transaction.state.log.min.isr=1
> > > >
> > > > Why do I see topic ReplicationFactor:1 during description of a topic
> ?
> > > >
> > > > And how can we increase replication factor?  I have 3 Kafka brokers
> > > running
> > > > on 2.7.0 version.
> > > >
> > > > Regards
> > > > GP
> > > >
> > >
> >
>

Re: Replication factor for a topic Increase

[Luke Chen]

Once the topic is created, you can only increase the replication factor by:
1. delete the topic and re-create the same topic (note: data will be lost
after this step)
or
2. follow the doc to increase the replication factor (data will be kept)

Thank you.
Luke

On Thu, Jul 13, 2023 at 5:23 PM Gaurav Pande  wrote:

> Hi Luke ,
>
> Thanks does that mean that setting default.replication.factor to more than
> 1 and doing Kafka broker restart wouldn't dynamically change the existing
> partitions replication? We have to use cli shell script, as per
> documentation?
> Regards,
> GP
>
> On Thu, 13 Jul, 2023, 13:54 Luke Chen,  wrote:
>
> > Hi Gaurav,
> >
> > > Why do I see topic ReplicationFactor:1 during description of a topic ?
> > I think you should set this config: `default.replication.factor`
> >
> > > And how can we increase replication factor?  I have 3 Kafka brokers
> > running on 2.7.0 version.
> > Please check this doc:
> >
> >
> https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
> >
> > Thanks.
> > Luke
> >
> > On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have noticed that when I describe my Kafka topic I see
> > > ReplicationFactor:1 but in my server.properties  file I already have
> > > defined below values i.e a replication factor of 3 :
> > >
> > > num.partions=3
> > > offsets.topic.replication.factor=3
> > > transaction.state.log.replication.factor=3
> > > transaction.state.log.min.isr=1
> > >
> > > Why do I see topic ReplicationFactor:1 during description of a topic ?
> > >
> > > And how can we increase replication factor?  I have 3 Kafka brokers
> > running
> > > on 2.7.0 version.
> > >
> > > Regards
> > > GP
> > >
> >
>

Re: How to get Kafka cluster.id priperty

[Gaurav Pande]

Thanks Nag.

On Thu, 13 Jul, 2023, 09:54 Nag Pavan Chilakam,
 wrote:

> kafka commands are available in confluent developer pages
> https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#
> For ZK commands you may have to refer to zookeeper docs.
> btw try kraft mode, so you don't need to be worried about zookeeper :)
>
> On Wed, Jul 12, 2023 at 11:18 AM Gaurav Pande 
> wrote:
>
> > Thanks on below info is there some set of these xli commands documented
> > somewhere? If so could you share the link
> > Regards,
> > GP
> >
> > On Wed, 12 Jul, 2023, 10:53 Nag Pavan Chilakam,
> >  wrote:
> >
> > > You can get the cluster id from zookeeper shell as below
> > > nchilakam@nchilakam 7.4.0-cp-demo-examples % zookeeper-shell
> > > localhost:2181
> > > get /cluster/id Connecting to localhost:2181 WATCHER:: WatchedEvent
> > > state:SyncConnected type:None path:null
> > > {"version":"1","id":"_esLic79RVSuuMOrQaxJXA"}
> > >
> > > On Wed, Jul 12, 2023 at 8:47 AM Gaurav Pande 
> > wrote:
> > >
> > > > Hi All,
> > > > Any help on this query please?
> > > >
> > > > Regards
> > > > GP
> > > >
> > > > On Tue, 11 Jul, 2023, 21:01 Gaurav Pande, 
> > wrote:
> > > >
> > > > > Hi Guys,
> > > > >
> > > > > Iam new to Kafka I was researching a bit about finding the
> > cluster.id
> > > > for
> > > > > a newly built 3 node  zookeeper ensemble so that I can switch my
> > Kafka
> > > > > brokers to these zookeeper
> > >
> > > >  Is there a way to pre-find this cluster.id value that is used in
> meta
> > > > > properties by Kafka as right now it's reflecting old zookeeper
> > cluster
> > > > I'd
> > > > > and I want to get new cluster I'd for my zk ensemble.
> > > > >
> > > > > Regards,
> > > > > GP
> > > > >
> > > >
> > >
> >
>

Re: Replication factor for a topic Increase

[Gaurav Pande]

Hi Luke ,

Thanks does that mean that setting default.replication.factor to more than
1 and doing Kafka broker restart wouldn't dynamically change the existing
partitions replication? We have to use cli shell script, as per
documentation?
Regards,
GP

On Thu, 13 Jul, 2023, 13:54 Luke Chen,  wrote:

> Hi Gaurav,
>
> > Why do I see topic ReplicationFactor:1 during description of a topic ?
> I think you should set this config: `default.replication.factor`
>
> > And how can we increase replication factor?  I have 3 Kafka brokers
> running on 2.7.0 version.
> Please check this doc:
>
> https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
>
> Thanks.
> Luke
>
> On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande 
> wrote:
>
> > Hi all,
> >
> > I have noticed that when I describe my Kafka topic I see
> > ReplicationFactor:1 but in my server.properties  file I already have
> > defined below values i.e a replication factor of 3 :
> >
> > num.partions=3
> > offsets.topic.replication.factor=3
> > transaction.state.log.replication.factor=3
> > transaction.state.log.min.isr=1
> >
> > Why do I see topic ReplicationFactor:1 during description of a topic ?
> >
> > And how can we increase replication factor?  I have 3 Kafka brokers
> running
> > on 2.7.0 version.
> >
> > Regards
> > GP
> >
>

Re: Replication factor for a topic Increase

[Luke Chen]

Hi Gaurav,

> Why do I see topic ReplicationFactor:1 during description of a topic ?
I think you should set this config: `default.replication.factor`

> And how can we increase replication factor?  I have 3 Kafka brokers
running on 2.7.0 version.
Please check this doc:
https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

Thanks.
Luke

On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande  wrote:

> Hi all,
>
> I have noticed that when I describe my Kafka topic I see
> ReplicationFactor:1 but in my server.properties  file I already have
> defined below values i.e a replication factor of 3 :
>
> num.partions=3
> offsets.topic.replication.factor=3
> transaction.state.log.replication.factor=3
> transaction.state.log.min.isr=1
>
> Why do I see topic ReplicationFactor:1 during description of a topic ?
>
> And how can we increase replication factor?  I have 3 Kafka brokers running
> on 2.7.0 version.
>
> Regards
> GP
>