[VOTE] 2.3.1 RC0

[David Arthur]

Hello Kafka users, developers and client-developers,


This is the first candidate for release of Apache Kafka 2.3.1 which
includes many bug fixes for Apache Kafka 2.3.


Release notes for the 2.3.1 release:

https://home.apache.org/~davidarthur/kafka-2.3.1-rc0/RELEASE_NOTES.html


*** Please download, test and vote by Wednesday, September 18, 9am PT


Kafka's KEYS file containing PGP keys we use to sign the release:

https://kafka.apache.org/KEYS


* Release artifacts to be voted upon (source and binary):

https://home.apache.org/~davidarthur/kafka-2.3.1-rc0/


* Maven artifacts to be voted upon:

https://repository.apache.org/content/groups/staging/org/apache/kafka/


* Javadoc:

https://home.apache.org/~davidarthur/kafka-2.3.1-rc0/javadoc/


* Tag to be voted upon (off 2.3 branch) is the 2.3.1 tag:

https://github.com/apache/kafka/releases/tag/2.3.1-rc0


* Documentation:

https://kafka.apache.org/23/documentation.html


* Protocol:

https://kafka.apache.org/23/protocol.html


* Successful Jenkins builds for the 2.3 branch:

Unit/integration tests: https://builds.apache.org/job/kafka-2.3-jdk8/

System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.3/119



We have yet to get a successful unit/integration job run due to some flaky
failures. I will send out a follow-up email once we have a passing build.


Thanks!

David

Re: Requirements

[Hans Jespersen]

Gwen Shapira published a great whitepaper with Reference Architectures for
all Kafka and Confluent components in big and small environements and for
bare metal, VMs, and all 3 major public clouds.

https://www.confluent.io/resources/apache-kafka-confluent-enterprise-reference-architecture/


On Fri, Sep 13, 2019 at 8:26 AM Peter Menapace <
peter.menap...@vanderlande.com> wrote:

> Hi all,
>
> I have a small small question. In my company we would like to use Apache
> Kafka with KSQL.
>
> And my small question is: which hardware requirements do you have to run
> Kafka and KSQL in small and big environments?
>
>
>
> Best regards,
>
>
>
> Peter
>
>
>
> Mit freundlichen Grüßen,
>
>
>
> *Peter Menapace*
>
> Senior IT Architect - ICT Projects WP-DE
>
> T +4923197942200 | M +4915112253549
>
>
>
> [image: cid:image002.jpg@01D3CDB0.7BF22000] 
>
>
>
> *Vanderlande Industries B.V.*
>
> Vanderlandelaan 2, 5466 RB  Veghel |The Netherlands
>
> T +31 413 49 49 49 | www.vanderlande.com
>
>
>
> [image: LinkedIn.png]
>   [image:
> Twitter.png]   [image:
> cid:image005.png@01D3CDB0.7BF22000]
>   [image:
> cid:image006.png@01D3CDB0.7BF22000]
>   [image: youtube.png]
> 
>
>
>
> [image: cid:image008.jpg@01D3CDB0.7BF22000] 
>
>
> --
> Vanderlande Industries GmbH
> Sitz der Gesellschaft:
> Dortmund, Deutschland
>
>
> Amtsgericht Dortmund: HRB 8539
> Geschäftsführer: Rene Veldink
> www.vanderlande.com
>
>
> Diese E-Mail enthält vertrauliche und/oder rechtlichgeschützte
> Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich
> erhalten haben,
> informieren Sie bitte den Absender und löschen Sie diese Mail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail und
> der darin enthaltenen Informationen sind nicht gestattet.
>
> This e-mail may contain confidential and/or privilegedinformation.
> If you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail.
> Any unauthorized copying, disclosure or distribution of the material in
> this e-mail is strictly forbidden.
>
> Think green: do you really need to print this e-mail?
>
>

Requirements

[Peter Menapace]

Hi all,
I have a small small question. In my company we would like to use Apache Kafka 
with KSQL.
And my small question is: which hardware requirements do you have to run Kafka 
and KSQL in small and big environments?

Best regards,

Peter

Mit freundlichen Grüßen,

Peter Menapace
Senior IT Architect - ICT Projects WP-DE
T +4923197942200 | M +4915112253549

[cid:image002.jpg@01D3CDB0.7BF22000]

Vanderlande Industries B.V.
Vanderlandelaan 2, 5466 RB  Veghel |The Netherlands
T +31 413 49 49 49 | www.vanderlande.com

[LinkedIn.png]  
[Twitter.png]    
[cid:image005.png@01D3CDB0.7BF22000] 
   
[cid:image006.png@01D3CDB0.7BF22000] 
   [youtube.png] 


[cid:image008.jpg@01D3CDB0.7BF22000]


Vanderlande Industries GmbH
Sitz der Gesellschaft:
Dortmund, Deutschland

Amtsgericht Dortmund: HRB 8539
Geschäftsführer: Rene Veldink
www.vanderlande.com


Diese E-Mail enthält vertrauliche und/oder rechtlichgeschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben,
informieren Sie bitte den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail und der 
darin enthaltenen Informationen sind nicht gestattet.


This e-mail may contain confidential and/or privilegedinformation.
If you are not the intended recipient (or have received this e-mail in error) 
please notify the sender immediately and delete this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden.


Think green: do you really need to print this e-mail?

Re: guidelines for replacing a lost Kafka Broker

[Svante Karlsson]

Just bring a new broker up and give it the id of the lost one. It will sync
itself

/svante

Den fre 13 sep. 2019 kl 13:51 skrev saurav suman :

> Hi,
>
> When the old data is lost and another broker is added to the cluster then
> it is a new fresh broker with no data. You can reassign the partitions of
> the topics using  kafka-reassign-partitions.sh script.
>
> Please check the below links for more details.
> https://blog.imaginea.com/how-to-rebalance-topics-in-kafka-cluster/
>
> https://kafka.apache.org/documentation/
>
> Regards,
> Saurav Suman
>
> On Fri, Sep 13, 2019 at 4:21 PM Alok Dwivedi 
> wrote:
>
> > Hi All
> > Can someone please advise what is the recommended practice for replacing
> a
> > lost broker in a Kafka cluster. Lets consider this sequence of events
> >
> >
> >   1.  3 node cluster say with 1 topic of 3 partitions and RF of 2.
> >   2.  That gives 2 partitions on each Broker (B1,B2,B3)
> >   3.  Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> > Assuming each broker had only 1 leader, by losing B3 we lost 1 leader
> and 1
> > replica. Out of remaining replicas another leader will be selected for
> any
> > lost leaders so we carry on with no downtime.
> >   4.  I want to know what we should do to go back to 3 broker cluster.
> Are
> > there standard guidelines around any procedure to be done after
> restarting
> > the lost broker mainly when the lost broker has been replaced with
> another
> > instance which no longer has old log.dirs i.e. data existing on this lost
> > broker is gone. Is there a way to rebuild missing replica partitions now
> > from existing partitions or what is the recommended practice in that
> > scenario? Will we need some kind of partition re-assignment here and will
> > it ensure that we now go back from 4 partitions to 6 partitions?
> >
> >
> > Can someone please advise what happens automatically in this case and
> what
> > (if anything) should be done as a standard practice to get things back to
> > normalcy.
> >
> > Thanks
> > Alok Dwivedi
> > Data Architect
> > Maersk Always On Platform
> >
> >
> >
> >
> >
> > Classification: Public
> >
> > 
> >
> > The information contained in this message is privileged and intended only
> > for the recipients named. If the reader is not a representative of the
> > intended recipient, any review, dissemination or copying of this message
> or
> > the information it contains is prohibited. If you have received this
> > message in error, please immediately notify the sender, and delete the
> > original message and attachments.
> >
> > Maersk will as part of our communication and interaction with you collect
> > and process your personal data. You can read more about Maersk's
> collection
> > and processing of your personal data and your rights as a data subject in
> > our privacy policy <
> > https://www.maersk.com/front-page-requirements/privacy-policy>
> >
> > Please consider the environment before printing this email.
> >
>
>
> --
>
> Thanks & Regards,
>
> Saurav Suman
> MOB - 8884222451
>

Re: guidelines for replacing a lost Kafka Broker

[saurav suman]

Hi,

When the old data is lost and another broker is added to the cluster then
it is a new fresh broker with no data. You can reassign the partitions of
the topics using  kafka-reassign-partitions.sh script.

Please check the below links for more details.
https://blog.imaginea.com/how-to-rebalance-topics-in-kafka-cluster/

https://kafka.apache.org/documentation/

Regards,
Saurav Suman

On Fri, Sep 13, 2019 at 4:21 PM Alok Dwivedi 
wrote:

> Hi All
> Can someone please advise what is the recommended practice for replacing a
> lost broker in a Kafka cluster. Lets consider this sequence of events
>
>
>   1.  3 node cluster say with 1 topic of 3 partitions and RF of 2.
>   2.  That gives 2 partitions on each Broker (B1,B2,B3)
>   3.  Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> Assuming each broker had only 1 leader, by losing B3 we lost 1 leader and 1
> replica. Out of remaining replicas another leader will be selected for any
> lost leaders so we carry on with no downtime.
>   4.  I want to know what we should do to go back to 3 broker cluster. Are
> there standard guidelines around any procedure to be done after restarting
> the lost broker mainly when the lost broker has been replaced with another
> instance which no longer has old log.dirs i.e. data existing on this lost
> broker is gone. Is there a way to rebuild missing replica partitions now
> from existing partitions or what is the recommended practice in that
> scenario? Will we need some kind of partition re-assignment here and will
> it ensure that we now go back from 4 partitions to 6 partitions?
>
>
> Can someone please advise what happens automatically in this case and what
> (if anything) should be done as a standard practice to get things back to
> normalcy.
>
> Thanks
> Alok Dwivedi
> Data Architect
> Maersk Always On Platform
>
>
>
>
>
> Classification: Public
>
> 
>
> The information contained in this message is privileged and intended only
> for the recipients named. If the reader is not a representative of the
> intended recipient, any review, dissemination or copying of this message or
> the information it contains is prohibited. If you have received this
> message in error, please immediately notify the sender, and delete the
> original message and attachments.
>
> Maersk will as part of our communication and interaction with you collect
> and process your personal data. You can read more about Maersk's collection
> and processing of your personal data and your rights as a data subject in
> our privacy policy <
> https://www.maersk.com/front-page-requirements/privacy-policy>
>
> Please consider the environment before printing this email.
>


-- 

Thanks & Regards,

Saurav Suman
MOB - 8884222451

Re: guidelines for replacing a lost Kafka Broker

[M. Manna]

Not sure what you mean by 1 topic with 3 partition and replication-factor =
2. You may need to revisit the idea of partitions and replication factor
from docs.

Thanks,

On Fri, 13 Sep 2019 at 11:51, Alok Dwivedi  wrote:

> Hi All
> Can someone please advise what is the recommended practice for replacing a
> lost broker in a Kafka cluster. Lets consider this sequence of events
>
>
>   1.  3 node cluster say with 1 topic of 3 partitions and RF of 2.
>   2.  That gives 2 partitions on each Broker (B1,B2,B3)
>   3.  Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> Assuming each broker had only 1 leader, by losing B3 we lost 1 leader and 1
> replica. Out of remaining replicas another leader will be selected for any
> lost leaders so we carry on with no downtime.
>   4.  I want to know what we should do to go back to 3 broker cluster. Are
> there standard guidelines around any procedure to be done after restarting
> the lost broker mainly when the lost broker has been replaced with another
> instance which no longer has old log.dirs i.e. data existing on this lost
> broker is gone. Is there a way to rebuild missing replica partitions now
> from existing partitions or what is the recommended practice in that
> scenario? Will we need some kind of partition re-assignment here and will
> it ensure that we now go back from 4 partitions to 6 partitions?
>
>
> Can someone please advise what happens automatically in this case and what
> (if anything) should be done as a standard practice to get things back to
> normalcy.
>
> Thanks
> Alok Dwivedi
> Data Architect
> Maersk Always On Platform
>
>
>
>
>
> Classification: Public
>
> 
>
> The information contained in this message is privileged and intended only
> for the recipients named. If the reader is not a representative of the
> intended recipient, any review, dissemination or copying of this message or
> the information it contains is prohibited. If you have received this
> message in error, please immediately notify the sender, and delete the
> original message and attachments.
>
> Maersk will as part of our communication and interaction with you collect
> and process your personal data. You can read more about Maersk's collection
> and processing of your personal data and your rights as a data subject in
> our privacy policy <
> https://www.maersk.com/front-page-requirements/privacy-policy>
>
> Please consider the environment before printing this email.
>

guidelines for replacing a lost Kafka Broker

[Alok Dwivedi]

Hi All
Can someone please advise what is the recommended practice for replacing a lost 
broker in a Kafka cluster. Lets consider this sequence of events


  1.  3 node cluster say with 1 topic of 3 partitions and RF of 2.
  2.  That gives 2 partitions on each Broker (B1,B2,B3)
  3.  Lets say we lost B3. We still have B1 and B2 and 4 partitions. Assuming 
each broker had only 1 leader, by losing B3 we lost 1 leader and 1 replica. Out 
of remaining replicas another leader will be selected for any lost leaders so 
we carry on with no downtime.
  4.  I want to know what we should do to go back to 3 broker cluster. Are 
there standard guidelines around any procedure to be done after restarting the 
lost broker mainly when the lost broker has been replaced with another instance 
which no longer has old log.dirs i.e. data existing on this lost broker is 
gone. Is there a way to rebuild missing replica partitions now from existing 
partitions or what is the recommended practice in that scenario? Will we need 
some kind of partition re-assignment here and will it ensure that we now go 
back from 4 partitions to 6 partitions?


Can someone please advise what happens automatically in this case and what (if 
anything) should be done as a standard practice to get things back to normalcy.

Thanks
Alok Dwivedi
Data Architect
Maersk Always On Platform





Classification: Public



The information contained in this message is privileged and intended only for 
the recipients named. If the reader is not a representative of the intended 
recipient, any review, dissemination or copying of this message or the 
information it contains is prohibited. If you have received this message in 
error, please immediately notify the sender, and delete the original message 
and attachments.

Maersk will as part of our communication and interaction with you collect and 
process your personal data. You can read more about Maersk's collection and 
processing of your personal data and your rights as a data subject in our 
privacy policy 

Please consider the environment before printing this email.

__consumer_offsets compaction settings / metadata performance

[Aminouvic]

Hi all, 

Currently with default settings 
min.cleanable.dirty.ratio/log.cleaner.min.cleanable.ratio (0.5) 
__consumer_offsets partition size grows up to 80Mb and this seems to slow down 
metadata related operations (notably commits / join group).
What are the normal/expected partition avg size for the __consumer_offsets 
topic ?
Are there any recommendations to settings 
min.cleanable.dirty.ratio/log.cleaner.min.cleanable.ratio for the 
__consumer_offsets topic ?

Kafka Cluster 
• version 0.10.1
• 6 nodes 
• __consumer_offsets #partitions : 50 
• Cleaner config :
○ min.cleanable.dirty.ratio : 0.5
○ log.cleaner.enable : True
○ log.cleaner.io.buffer.size : 524288
○ log.cleaner.dedupe.buffer.size : 1073741824
○ log.cleaner.io.max.bytes.per.second: 104857600

Regards,
Amine