Re: compatible kafka version to use when using with logstash 7.5.1-1

[Manoj.Agrawal2]

Are you getting any error at kafka broker or producing/consuming message ?
Can you please provide more detail how did you upgrade or what error you are 
getting . it all depend how did you upgraded ?



On 8/6/20, 4:13 PM, "Satish Kumar"  wrote:

[External]


Hello,

I upgraded kafka from 0.10 to 2.5.0 and also I upgraded logstash from 2.4
to 7.5

when I have kafka 1.10 and logstash 2.4 the messages used to forward
without any problems. But after the upgrade I'm getting errors in both
logstash and kafka logs so I would like to know what is the compatible
kafka version to use with logstash ( logstash is using kafka-integration
plugin 10.0.0). Please let me know what the compatible version is. I will
do upgrades/downgrades according to that.


This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.
This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.

compatible kafka version to use when using with logstash 7.5.1-1

[Satish Kumar]

Hello,

I upgraded kafka from 0.10 to 2.5.0 and also I upgraded logstash from 2.4
to 7.5

when I have kafka 1.10 and logstash 2.4 the messages used to forward
without any problems. But after the upgrade I'm getting errors in both
logstash and kafka logs so I would like to know what is the compatible
kafka version to use with logstash ( logstash is using kafka-integration
plugin 10.0.0). Please let me know what the compatible version is. I will
do upgrades/downgrades according to that.

Re: Kafka topic partition distributing evenly on disks

[Vinicius Scheidegger]

Hi Peter,

AFAIK, everything depends on:

1) How you have configured your topic
  a) number of partitions (here I understand you have 15 partitions)
  b) partition replication configuration (each partition necessarily has a
leader - primary responsible to hold the data - and for reads and writes)
you can configure the topic to have a number of replicas
2) How you publish messages to the topic
  a) The publisher is responsible to choose the partition. This can be done
consciously (by setting the partition id while sending the message to the
topic) or unconsciously (by using the DefaultPartitioner or any other
partitioner scheme).

All messages sent to a specific partition will be written first to the
leader (meaning that the disk configured for the partition leader will
receive the load) and then replicated to the replica (followers).
Kafka does not automatically distribute the data equally to the different
brokers - you need to think about your architecture having that in mind.

I hope it helps

On Thu, Aug 6, 2020 at 10:23 PM Péter Nagykátai 
wrote:

> I initially started with one data disk (mounted solely to hold Kafka data)
> and recently added a new one.
>
> On Thu, Aug 6, 2020 at 10:13 PM  wrote:
>
> > What do you mean older disk ?
> >
> > On 8/6/20, 12:05 PM, "Péter Nagykátai"  wrote:
> >
> > [External]
> >
> >
> > Yeah, but it doesn't do that. My "older" disks have ~70 partitions,
> the
> > newer ones ~5 partitions. That's why I'm asking what went wrong.
> >
> > On Thu, Aug 6, 2020 at 8:35 PM  wrote:
> >
> > > Kafka  evenly distributed number of partition on each disk so in
> > your case
> > > every disk should have 3/2 topic partitions .
> > > It is producer job to evenly produce data by partition key  to
> topic
> > > partition .
> > > How it partition key , it is auto generated or producer sending key
> > along
> > > with message .
> > >
> > >
> > > On 8/6/20, 7:29 AM, "Péter Nagykátai" 
> wrote:
> > >
> > > [External]
> > >
> > >
> > > Hello,
> > >
> > > I have a Kafka cluster with 3 brokers (v2.3.0) and each broker
> > has 2
> > > disks
> > > attached. I added a new topic (heavyweight) and was surprised
> > that
> > > even if
> > > the topic has 15 partitions, those weren't distributed evenly
> on
> > the
> > > disks.
> > > Thus I got one disk that's almost empty and the other almost
> > filled
> > > up. Is
> > > there any way to have Kafka evenly distribute data on its
> disks?
> > >
> > > Thank you!
> > >
> > >
> > > This e-mail and any files transmitted with it are for the sole use
> > of the
> > > intended recipient(s) and may contain confidential and privileged
> > > information. If you are not the intended recipient(s), please reply
> > to the
> > > sender and destroy all copies of the original message. Any
> > unauthorized
> > > review, use, disclosure, dissemination, forwarding, printing or
> > copying of
> > > this email, and/or any action taken in reliance on the contents of
> > this
> > > e-mail is strictly prohibited and may be unlawful. Where permitted
> by
> > > applicable law, this e-mail and other e-mail communications sent to
> > and
> > > from Cognizant e-mail addresses may be monitored.
> > > This e-mail and any files transmitted with it are for the sole use
> > of the
> > > intended recipient(s) and may contain confidential and privileged
> > > information. If you are not the intended recipient(s), please reply
> > to the
> > > sender and destroy all copies of the original message. Any
> > unauthorized
> > > review, use, disclosure, dissemination, forwarding, printing or
> > copying of
> > > this email, and/or any action taken in reliance on the contents of
> > this
> > > e-mail is strictly prohibited and may be unlawful. Where permitted
> by
> > > applicable law, this e-mail and other e-mail communications sent to
> > and
> > > from Cognizant e-mail addresses may be monitored.
> > >
> >
> >
> > This e-mail and any files transmitted with it are for the sole use of the
> > intended recipient(s) and may contain confidential and privileged
> > information. If you are not the intended recipient(s), please reply to
> the
> > sender and destroy all copies of the original message. Any unauthorized
> > review, use, disclosure, dissemination, forwarding, printing or copying
> of
> > this email, and/or any action taken in reliance on the contents of this
> > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > applicable law, this e-mail and other e-mail communications sent to and
> > from Cognizant e-mail addresses may be monitored.
> > This e-mail and any files transmitted with it are for the sole use of the
> > intended recipient(s) and may contain confidential and privileged
> > information. If you are not the intended 

Re: Kafka topic partition distributing evenly on disks

[Péter Nagykátai]

I initially started with one data disk (mounted solely to hold Kafka data)
and recently added a new one.

On Thu, Aug 6, 2020 at 10:13 PM  wrote:

> What do you mean older disk ?
>
> On 8/6/20, 12:05 PM, "Péter Nagykátai"  wrote:
>
> [External]
>
>
> Yeah, but it doesn't do that. My "older" disks have ~70 partitions, the
> newer ones ~5 partitions. That's why I'm asking what went wrong.
>
> On Thu, Aug 6, 2020 at 8:35 PM  wrote:
>
> > Kafka  evenly distributed number of partition on each disk so in
> your case
> > every disk should have 3/2 topic partitions .
> > It is producer job to evenly produce data by partition key  to topic
> > partition .
> > How it partition key , it is auto generated or producer sending key
> along
> > with message .
> >
> >
> > On 8/6/20, 7:29 AM, "Péter Nagykátai"  wrote:
> >
> > [External]
> >
> >
> > Hello,
> >
> > I have a Kafka cluster with 3 brokers (v2.3.0) and each broker
> has 2
> > disks
> > attached. I added a new topic (heavyweight) and was surprised
> that
> > even if
> > the topic has 15 partitions, those weren't distributed evenly on
> the
> > disks.
> > Thus I got one disk that's almost empty and the other almost
> filled
> > up. Is
> > there any way to have Kafka evenly distribute data on its disks?
> >
> > Thank you!
> >
> >
> > This e-mail and any files transmitted with it are for the sole use
> of the
> > intended recipient(s) and may contain confidential and privileged
> > information. If you are not the intended recipient(s), please reply
> to the
> > sender and destroy all copies of the original message. Any
> unauthorized
> > review, use, disclosure, dissemination, forwarding, printing or
> copying of
> > this email, and/or any action taken in reliance on the contents of
> this
> > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > applicable law, this e-mail and other e-mail communications sent to
> and
> > from Cognizant e-mail addresses may be monitored.
> > This e-mail and any files transmitted with it are for the sole use
> of the
> > intended recipient(s) and may contain confidential and privileged
> > information. If you are not the intended recipient(s), please reply
> to the
> > sender and destroy all copies of the original message. Any
> unauthorized
> > review, use, disclosure, dissemination, forwarding, printing or
> copying of
> > this email, and/or any action taken in reliance on the contents of
> this
> > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > applicable law, this e-mail and other e-mail communications sent to
> and
> > from Cognizant e-mail addresses may be monitored.
> >
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>

Re: Kafka topic partition distributing evenly on disks

[Manoj.Agrawal2]

What do you mean older disk ?

On 8/6/20, 12:05 PM, "Péter Nagykátai"  wrote:

[External]


Yeah, but it doesn't do that. My "older" disks have ~70 partitions, the
newer ones ~5 partitions. That's why I'm asking what went wrong.

On Thu, Aug 6, 2020 at 8:35 PM  wrote:

> Kafka  evenly distributed number of partition on each disk so in your case
> every disk should have 3/2 topic partitions .
> It is producer job to evenly produce data by partition key  to topic
> partition .
> How it partition key , it is auto generated or producer sending key along
> with message .
>
>
> On 8/6/20, 7:29 AM, "Péter Nagykátai"  wrote:
>
> [External]
>
>
> Hello,
>
> I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2
> disks
> attached. I added a new topic (heavyweight) and was surprised that
> even if
> the topic has 15 partitions, those weren't distributed evenly on the
> disks.
> Thus I got one disk that's almost empty and the other almost filled
> up. Is
> there any way to have Kafka evenly distribute data on its disks?
>
> Thank you!
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>


This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.
This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.

Re: Kafka topic partition distributing evenly on disks

[Péter Nagykátai]

Yeah, but it doesn't do that. My "older" disks have ~70 partitions, the
newer ones ~5 partitions. That's why I'm asking what went wrong.

On Thu, Aug 6, 2020 at 8:35 PM  wrote:

> Kafka  evenly distributed number of partition on each disk so in your case
> every disk should have 3/2 topic partitions .
> It is producer job to evenly produce data by partition key  to topic
> partition .
> How it partition key , it is auto generated or producer sending key along
> with message .
>
>
> On 8/6/20, 7:29 AM, "Péter Nagykátai"  wrote:
>
> [External]
>
>
> Hello,
>
> I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2
> disks
> attached. I added a new topic (heavyweight) and was surprised that
> even if
> the topic has 15 partitions, those weren't distributed evenly on the
> disks.
> Thus I got one disk that's almost empty and the other almost filled
> up. Is
> there any way to have Kafka evenly distribute data on its disks?
>
> Thank you!
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>

Re: Kafka topic partition distributing evenly on disks

[Manoj.Agrawal2]

Kafka  evenly distributed number of partition on each disk so in your case 
every disk should have 3/2 topic partitions .
It is producer job to evenly produce data by partition key  to topic partition .
How it partition key , it is auto generated or producer sending key along with 
message .


On 8/6/20, 7:29 AM, "Péter Nagykátai"  wrote:

[External]


Hello,

I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2 disks
attached. I added a new topic (heavyweight) and was surprised that even if
the topic has 15 partitions, those weren't distributed evenly on the disks.
Thus I got one disk that's almost empty and the other almost filled up. Is
there any way to have Kafka evenly distribute data on its disks?

Thank you!


This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.
This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. Where permitted by applicable law, this e-mail 
and other e-mail communications sent to and from Cognizant e-mail addresses may 
be monitored.

Re: New Website Layout

[Ben Weintraub]

Plus one to Tom's request - the ability to easily generate links to
specific config options is extremely valuable.

On Thu, Aug 6, 2020 at 10:09 AM Tom Bentley  wrote:

> Hi Ben,
>
> The documentation for the configs (broker, producer etc) used to function
> as links as well as anchors, which made the url fragments more
> discoverable, because you could click on the link and then copy+paste the
> browser URL:
>
> 
>href="#batch.size">batch.size
> 
>
> What seems to have happened with the new layout is the  tags are empty,
> and no longer enclose the config name,
>
> 
>   
>   batch.size
> 
>
> meaning you can't click on the link to copy and paste the URL. Could the
> old behaviour be restored?
>
> Thanks,
>
> Tom
>
> On Wed, Aug 5, 2020 at 12:43 PM Luke Chen  wrote:
>
> > When entering streams doc, it'll always show:
> > *You're viewing documentation for an older version of Kafka - check out
> our
> > current documentation here.*
> >
> >
> >
> > On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:
> >
> > > Thanks for the PR and feedback Michael. Appreciated.
> > >
> > > On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> > > wrote:
> > >
> > > > Thank you, it looks great!
> > > >
> > > > I found a couple of small issues:
> > > > - It's not rendering correctly with http.
> > > > - It's printing "called" to the console. I opened a PR to remove the
> > > > console.log() call: https://github.com/apache/kafka-site/pull/278
> > > >
> > > > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford 
> wrote:
> > > > >
> > > > > The new website layout has gone live as you may have seen. There
> are
> > a
> > > > > couple of rendering issues in the streams developer guide that
> we're
> > > > > getting addressed. If anyone spots anything else could they please
> > > reply
> > > > to
> > > > > this thread.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Ben
> > > > >
> > > > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford 
> wrote:
> > > > >
> > > > > > Hey folks
> > > > > >
> > > > > > We've made some updates to the website's look and feel. There is
> a
> > > > staged
> > > > > > version in the link below.
> > > > > >
> > > > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > > > username: kafka
> > > > > > password: streaming
> > > > > >
> > > > > > Comments welcomed.
> > > > > >
> > > > > > Ben
> > > > > >
> > > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Ben Stopford
> > >
> > > Lead Technologist, Office of the CTO
> > >
> > > 
> > >
> >
>

Re: New Website Layout

[Tom Bentley]

Hi Ben,

The documentation for the configs (broker, producer etc) used to function
as links as well as anchors, which made the url fragments more
discoverable, because you could click on the link and then copy+paste the
browser URL:


  batch.size


What seems to have happened with the new layout is the  tags are empty,
and no longer enclose the config name,


  
  batch.size


meaning you can't click on the link to copy and paste the URL. Could the
old behaviour be restored?

Thanks,

Tom

On Wed, Aug 5, 2020 at 12:43 PM Luke Chen  wrote:

> When entering streams doc, it'll always show:
> *You're viewing documentation for an older version of Kafka - check out our
> current documentation here.*
>
>
>
> On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:
>
> > Thanks for the PR and feedback Michael. Appreciated.
> >
> > On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> > wrote:
> >
> > > Thank you, it looks great!
> > >
> > > I found a couple of small issues:
> > > - It's not rendering correctly with http.
> > > - It's printing "called" to the console. I opened a PR to remove the
> > > console.log() call: https://github.com/apache/kafka-site/pull/278
> > >
> > > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
> > > >
> > > > The new website layout has gone live as you may have seen. There are
> a
> > > > couple of rendering issues in the streams developer guide that we're
> > > > getting addressed. If anyone spots anything else could they please
> > reply
> > > to
> > > > this thread.
> > > >
> > > > Thanks
> > > >
> > > > Ben
> > > >
> > > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
> > > >
> > > > > Hey folks
> > > > >
> > > > > We've made some updates to the website's look and feel. There is a
> > > staged
> > > > > version in the link below.
> > > > >
> > > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > > username: kafka
> > > > > password: streaming
> > > > >
> > > > > Comments welcomed.
> > > > >
> > > > > Ben
> > > > >
> > > > >
> > >
> >
> >
> > --
> >
> > Ben Stopford
> >
> > Lead Technologist, Office of the CTO
> >
> > 
> >
>

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Stanislav Kozlovski]

Thanks for driving the release Randall!
Congratulations to everybody involved - awesome work!

On Thu, Aug 6, 2020 at 5:21 PM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>


-- 
Best,
Stanislav

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Ismael Juma]

Thanks for driving the releas, Randall. Congratulations to all the
contributors! :)

Ismael

On Thu, Aug 6, 2020, 7:21 AM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>

Kafka topic partition distributing evenly on disks

[Péter Nagykátai]

Hello,

I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2 disks
attached. I added a new topic (heavyweight) and was surprised that even if
the topic has 15 partitions, those weren't distributed evenly on the disks.
Thus I got one disk that's almost empty and the other almost filled up. Is
there any way to have Kafka evenly distribute data on its disks?

Thank you!

[ANNOUNCE] Apache Kafka 2.6.0

[Randall Hauch]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.6.0

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Significant performance improvements, especially when the broker has
large numbers of partitions
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes other features, 74 improvements, 175 bug fixes,
plus other changes.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#2.6.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 127 contributors to this release!

17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Randall Hauch

Concept and implementation questions

[hamidreza hosseini]

Hi,
I have some questions about kafka concept,
Thanks in advance for your answer:
1.I want when kafka are deleting kafka's partition (because it achieved to its 
time or max volume) it send me a notification and says that "I was deleted 
offset 22  od topic a and partition 2"
Can i do this in kafka?
2.every database save its database in a format, this is my question that  Kafka 
save its data in a text file or it has a specific format and structure?
And what is that?
3.
How can i take backup from my kafka cluster and restore it?
4.I want a dedicated dashboard to monitor and manage kafka (i dont want to use 
prometheus or something like that) i want a seprate and dedicated dashboard
Is there any opensource project for this purpose?
5.
I want to run kafka in production , is there any sample configuration that it 
set some important and tuned parameters  to download and modify it?

Best Regards

Kafka Time-based offset fetching not working when in Kafka Produce Transactional semantics are enabled

[PATEL Himaliben]

Hi,



We are implementing exactly once in our application with help of 
KafakaConsumer's offsetsForTimes API.



Out kafka producer is using transactional semantics  as below : pseudo steps



It is enabled using :  producerProps.put("transactional.id", "h3");

KafkaProducer 
producer = new KafkaProducer(producerProps);



And transactions are committed using :

1. producer.initTransactions();

2. producer.beginTransaction();

3. // push some data

4. producer.commitTransaction();

5. producer.beginTransaction();

// Repeat 3rd, 4th and 5th statements in loop

//when done

6. producer.commitTransaction();

7. producer.close();



Our observations are :

I have tried combinations for transaction semantics 
enabled/disabled + CreateTime is current / 1 day old.

1. Transaction disabled , CreateTime not provided (Will take 
system time) : Offset fetching is proper

2. Transaction enabled , CreateTime not provided (Will take 
system time) : Offset fetching is proper

3. Transaction disabled , CreateTime is 1 day old : Offset 
fetching is proper

4. Transaction enabled , CreateTime is 1 day old : Offset 
return is seems like very first batch commit offset only.

(Regarding above case #4 , we have tried to use properties 
message.timestamp.difference.max.ms and retention.ms also. But result is same. 
And our data is ranging from few hours to days but again result is same.)





Please help if we are missing something when we are using transaction semantics 
for populating kafka topic.