Re: what happened in case of single disk failure

[张祥]

Thanks Peter, it makes a lot of sense.

Peter Bukowinski  于2020年3月3日周二 上午11:56写道：

> Whether your brokers have a single data directory or multiple data
> directories on separate disks, when a disk fails, the topic partitions
> located on that disk become unavailable. What happens next depends on how
> your cluster and topics are configured.
>
> If the topics on the affected broker have replicas and the minimum ISR
> (in-sync replicas) count is met, then all topic partitions will remain
> online and leaders will move to another broker. Producers and consumers
> will continue to operate as usual.
>
> If the topics don’t have replicas or the minimum ISR count is not met,
> then the topic partitions on the failed disk will be offline. Producers can
> still send data to the affected topics — it will just go to the online
> partitions. Consumers can still consume data from the online partitions.
>
> -- Peter
>
> > On Mar 2, 2020, at 7:00 PM, 张祥  wrote:
> >
> > Hi community，
> >
> > I ran into disk failure when using Kafka, and fortunately it did not
> crash
> > the entire cluster. So I am wondering how Kafka handles multiple disks
> and
> > it manages to work in case of single disk failure. The more detailed, the
> > better. Thanks !
>

Re: Issue in retention with compact,delete cleanup policy

[Matthias J. Sax]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

If I understand the issue correctly, the problem is that some data,
even if it is not updated for a long period of time will, never be
deleted because it's always in a segment with new data?

As your segment size is already fairly small, I don't see any other
solution as to write corresponding tombstones.

Changing the behavior would for sure require a KIP. One could
basically change the compaction to leave data that is older than the
retention time in their own segments to make them available for
deletion. I am not familiar with this part of the code and thus cannot
say how complex it would be to implement.


- -Matthias


On 3/3/20 7:02 PM, Koushik Chitta wrote:
> Bubbling this up to understand if anyone else are in similar use
> case.
>
>
> -Original Message- From: Koushik Chitta
>  Sent: Sunday, February 23, 2020
> 1:35 PM To: users@kafka.apache.org; d...@kafka.apache.org Subject:
> [EXTERNAL] Issue in retention with compact,delete cleanup policy
>
> Hi,
>
> I have a Topic with following config.
>
> cleanup.policy  =  compact,delete segment.bytes = 52428800 (~52
> mb) min.compaction.lag.ms = 180 (30 min) delete.retention.ms =
> 8640 (1 day) retention.ms = 25920 (3 days)
>
> Ideally I would want the old records > 3 days to be deleted without
> producing an explicit delete(null value of a key) of the record.
> But there can be a case due to continuous compaction, the segments
> can contain a very old record(eg: > 30 days) and new recent record
> (eg: 1hr) which will make the segment ineligible for retention
> delete.
>
> Currently I don't see a work around for this. Please suggest. I
> plan to start a KIP to address this use case.
>
> Thanks, Koushik
>
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5fMAEACgkQO4miYXKq
/OgBpA//auSjqG8bnpKD44Svey2GK7cI1kA6pyEY/NMgS4PLav5q+dWiPnADBDQV
ZmgekEcXLk6TRggl8oHs0zJCf9ETesGwAUQEcQnIstK+lPIBT/fOEpVdJiZzwNt5
24t1flhO3NBgFty+XQm5J0DrNJmMaysbhptuulPpOfn2Cqj6L26Co22rWkm1w9Pa
03bmrswj5f2kVXWvKExY1kZLRxNhGu4fQaovyDF8dgTZAoHQWAwXpUOxB70tl0Ct
fDwzzYd+waGNnQ7cbsbdY6QxvhQJj2aijXCgih3tqJ2ww14BIlTHBku9FAIU77Ww
OesfAFAt8tcZmYX5pntVyaG2FblATDKSnf1WEAzSRNzuw+dGsnrkAZSwizZJWVYq
JgRo3EO55HHPDqol+T1jI/KFL7tUb549rEd6Er3sxM/+PHX7CeKoS+Y/+VOPY8Z3
uc6qVe1y+rjqikBE9dmOjBmJngopbYtAK8Wu6CbdNRqDXXZtkXsnoj1vr7SlsJZ1
yov+11gLEAtGHSrAwGw6W5Jydcwdbxug2xSgWI0W0XD6djiyS3NBw59vBfB4+wPP
YLrS5UvxB3W6/wUgivx3iPNTOtHMuqb4lElxVLgHQzKJUmQkouDGQTpUNKCEfncb
9631uJP9hAMLOl/rvPW6CDp2PlMZLAgB14vIgqyEB0vkU7D+1qs=
=YTFS
-END PGP SIGNATURE-

Re: Does the response to a OffsetFetch include non-committed transactional offsets?

[Matthias J. Sax]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

A fetch-offset request would return the latest "stable" offset, ie,
either non-transactional or transactional+committed.

If there is a pending transaction, the corresponding offset would not
be returned.

Btw: Kafka 2.5 allows you to block a fetch-offset request for this
case: ie, if there is a pending transaction, you can wait until the
transaction is either committed (and the committed offset would be
returned) or aborted (and the "old" offset would be returned).

Check out KIP-447 for more details:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+sc
alability+for+exactly+once+semantics

The broker side changes will be included in 2.5 release.


- -Matthias



On 3/3/20 6:11 AM, Reftel, Magnus wrote:
> If a consumer sends its offset for a topic-partition as part of a
> transaction, and someone sends an OffsetFetch request for that
> consumer group and topic-partition before the transaction is
> committed, is the OffsetFetch response meant to include that
> pending offset, or only the last offset sent outside of a
> non-committed transaction? I find no discussion of it in the Kafka
> protocol guide, and the code in GroupMetadata.scala seems to
> indicate that pending offsets are not included, but I'm observing
> some behavior that might suggest otherwise.
>
> Best Regards Magnus Reftel
>
>
>  Denne e-posten og eventuelle
> vedlegg er beregnet utelukkende for den institusjon eller person
> den er rettet til og kan vaere belagt med lovbestemt taushetsplikt.
> Dersom e-posten er feilsendt, vennligst slett den og kontakt
> Skatteetaten. The contents of this email message and any
> attachments are intended solely for the addressee(s) and may
> contain confidential information and may be legally protected from
> disclosure. If you are not the intended recipient of this message,
> please immediately delete the message and alert the Norwegian Tax
> Administration.
>
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5fLikACgkQO4miYXKq
/OjqWBAAxM6S8fmOhD99h4xdg9oV4ce6p2UnPficAuLhiAJT3yG7zrP3K9roz47/
Kqrw5sO0Hu1cGDGDJyjE3ODPRF1IA5Dou5/5H094biElzJIf6170hYkjvLKZUum6
5tjSAFXuotZn6CXDaD2l3/LORqufdq9qYVkfe6S89zTz4cD1v4ULe1+B6zddh8+A
VDanCB1usJo6VyJ2kU3/IGPkDPKYLoSN3+ijBIdGX7rj/d/RaaH6HO5G4fWXhAae
9RJ05BLSuTo1WqglfZs0PHAhMqurzkHlyXNHxa1W+llxh8AJ/eYz3NmwAKHBrW3M
bw/PEPIAcF5xj2xR1p+2FjYNJbeK1qxBwLRw8jbUaX+yoqn7YQmEvjAuOizr4moF
qHFrIpjIi5SCG5iXpSUJRxY0Wlt/RJG1WYqwdCOIlJtSzgL3+aEbaKdQ8EFMckhA
K7jUaF/TQrsfNszOCI9YvtwqYdDI1b85K0l6+5H5Ki69akQwWSR1nI/2M1WQ07oS
YJLENV17qEJBKdK7wBrqRMRKgBYwlQvjDhthrroCgPdQe0jySwpMwIHzKdi2yhVH
hem8Q8u6fjtfTMDLD7S/+sTATEhJsjN97b/t+wUrK2L3BjkXm83BUziftd+6+L2S
ahDwuZJciYt5U0eFkP+4co26U/BkNituTmGRCTCtmlSyahbocus=
=aSd3
-END PGP SIGNATURE-

RE: Producer-Perf-test

[Sunil CHAUDHARI]

Hi Himanshu,
Pasting screenshot for better view.
My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?


[cid:image001.png@01D5F202.89FE2BD0]
From: Sunil CHAUDHARI
Sent: Tuesday, March 3, 2020 2:43 PM
To: users@kafka.apache.org
Subject: Producer-Perf-test

Hi,
I have done performance testing of Kafka cluster using 
kafka-producer-perf-test.sh
I created diff type of topics and did perf testing. Example: MB1P1R= MB is my 
topic name with 1 Partition and 1 replica.
I have 3 nodes cluster.

My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?

Sr NO

Topic

Partitions

Replicas

Network.threads

IO-Threads

Records Sent

Batch-size-avg

Throughput

Latency (ms)

Records/sec

Size/sec

Avg

Max

1

MB1P1R

1

1

3

8

65495

16220

68653.03983

13.78 MB/sec

163.67

245

2

MB2P1R

2

1

3

8

65495

15865.719

58269.57295

11.70 MB/sec

173.81

470

3

MB2P2R

2

2

3

8

65495

16202.813

46286.21908

9.29 MB/sec

417.33

704

4

MB3P2R

3

2

3

8

65495

16184

56412.57537

11.32 MB/sec

229.82

550

5

MB3P3R

3

3

3

8

65495



46417.43444

9.32 MB/sec

411.44

705



Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

RE: Issue in retention with compact,delete cleanup policy

[Koushik Chitta]

Bubbling this up to understand if anyone else are in similar use case.


-Original Message-
From: Koushik Chitta  
Sent: Sunday, February 23, 2020 1:35 PM
To: users@kafka.apache.org; d...@kafka.apache.org
Subject: [EXTERNAL] Issue in retention with compact,delete cleanup policy

Hi,

I have a Topic with following config.

cleanup.policy  =  compact,delete
segment.bytes = 52428800 (~52 mb)
min.compaction.lag.ms = 180 (30 min) delete.retention.ms = 8640 (1 day) 
retention.ms = 25920 (3 days)

Ideally I would want the old records > 3 days to be deleted without producing 
an explicit delete(null value of a key) of the record.
But there can be a case due to continuous compaction, the segments can contain 
a very old record(eg: > 30 days) and new recent record (eg: 1hr) which will 
make the segment ineligible for retention delete.

Currently I don't see a work around for this. Please suggest.
I plan to start a KIP to address this use case.

Thanks,
Koushik

Re: Subject: [VOTE] 2.4.1 RC0

[Eric Lalonde]

Hi,

I ran:
$  https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh 
 2.4.1 
https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 


All checksums and signatures are good and all unit and integration tests that 
were executed passed successfully.

- Eric

> On Mar 2, 2020, at 6:39 PM, Bill Bejeck  wrote:
> 
> Hello Kafka users, developers and client-developers,
> 
> This is the first candidate for release of Apache Kafka 2.4.1.
> 
> This is a bug fix release and it includes fixes and improvements from 38
> JIRAs, including a few critical bugs.
> 
> Release notes for the 2.4.1 release:
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/RELEASE_NOTES.html
> 
> *Please download, test and vote by Thursday, March 5, 9 am PT*
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
> 
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/
> 
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/javadoc/
> 
> * Tag to be voted upon (off 2.4 branch) is the 2.4.1 tag:
> https://github.com/apache/kafka/releases/tag/2.4.1-rc0
> 
> * Documentation:
> https://kafka.apache.org/24/documentation.html
> 
> * Protocol:
> https://kafka.apache.org/24/protocol.html
> 
> * Successful Jenkins builds for the 2.4 branch:
> Unit/integration tests: Links to successful unit/integration test build to
> follow
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.4/152/
> 
> 
> Thanks,
> Bill Bejeck

Re: Default values for max.request.size and message.max.bytes seem wrong

[Ismael Juma]

This was fixed recently:

https://github.com/apache/kafka/commit/bd5a1c4d368b9e91398e48400965d30d3045062e

Ismael

On Tue, Mar 3, 2020 at 11:34 AM Michael Rubin 
wrote:

> (I am re-asking my SO question:
> https://stackoverflow.com/questions/60458132/default-values-for-max-request-size-and-message-max-bytes-seem-wrong
> <
> https://stackoverflow.com/questions/60458132/default-values-for-max-request-size-and-message-max-bytes-seem-wrong
> >)
>
> The default value for the Producer setting max.request.size is 1048576.
> This controls the maximum size of a request (to a broker) in bytes.
>
> The default value for the Broker setting message.max.bytes is 112.
> This controls the largest (record batch) size allowed (by the Kafka
> server/broker).
>
> 1048576 > 112
>
> It seems that the defaults are such that a Producer (with this default
> value) could happen to produce a message which is larger than a Broker
> (with this default value) will accept, resulting in MESSAGE_TOO_LARGE
> errors.
>
> Am I misunderstanding these configuration settings? Or are the Kafka
> defaults really such that it is possible to get MESSAGE_TOO_LARGE errors
> with the default values?
>
>
>

Default values for max.request.size and message.max.bytes seem wrong

[Michael Rubin]

(I am re-asking my SO question: 
https://stackoverflow.com/questions/60458132/default-values-for-max-request-size-and-message-max-bytes-seem-wrong
 
)

The default value for the Producer setting max.request.size is 1048576. This 
controls the maximum size of a request (to a broker) in bytes.

The default value for the Broker setting message.max.bytes is 112. This 
controls the largest (record batch) size allowed (by the Kafka server/broker).

1048576 > 112

It seems that the defaults are such that a Producer (with this default value) 
could happen to produce a message which is larger than a Broker (with this 
default value) will accept, resulting in MESSAGE_TOO_LARGE errors.

Am I misunderstanding these configuration settings? Or are the Kafka defaults 
really such that it is possible to get MESSAGE_TOO_LARGE errors with the 
default values?

Re: Producer-Perf-test

[Himanshu Shukla]

It's okay. You could probably take the screenshot and share it.

On Tue, 3 Mar 2020, 20:13 sunil chaudhari, 
wrote:

> Hi Himanshu,
> Sorry but I pasted from excel. Dont know how it got messed up?
>
> Will resend it.
>
> On Tue, 3 Mar 2020 at 6:48 PM, Himanshu Shukla <
> himanshushukla...@gmail.com>
> wrote:
>
> > could you please share the result in some proper way? Each field is line
> by
> > line.
> >
> > On Tue, Mar 3, 2020 at 2:43 PM Sunil CHAUDHARI
> >  wrote:
> >
> > > Hi,
> > > I have done performance testing of Kafka cluster using
> > > kafka-producer-perf-test.sh
> > > I created diff type of topics and did perf testing. Example: MB1P1R= MB
> > is
> > > my topic name with 1 Partition and 1 replica.
> > > I have 3 nodes cluster.
> > >
> > > My Observations:
> > >
> > >   *   When I increased partitions, and keep replica same, then
> Throughput
> > > increases. Compare Row  1 and 2
> > >   *   Increase number of replica  and  partitions same: Throughput
> > > decreased than before. Row 2 and 3
> > >   *   Increase number of partitions and keep replica same as before: TP
> > > increased.  Row 3 and 4
> > >   *   Increase number of Replica and keep partitions same: TP
> decreased.
> > > Row 4 and 5.
> > >
> > > Question: Does it make any impact on throughput when number of
> partitions
> > > and replicas are equal?
> > >
> > > Sr NO
> > >
> > > Topic
> > >
> > > Partitions
> > >
> > > Replicas
> > >
> > > Network.threads
> > >
> > > IO-Threads
> > >
> > > Records Sent
> > >
> > > Batch-size-avg
> > >
> > > Throughput
> > >
> > > Latency (ms)
> > >
> > > Records/sec
> > >
> > > Size/sec
> > >
> > > Avg
> > >
> > > Max
> > >
> > > 1
> > >
> > > MB1P1R
> > >
> > > 1
> > >
> > > 1
> > >
> > > 3
> > >
> > > 8
> > >
> > > 65495
> > >
> > > 16220
> > >
> > > 68653.03983
> > >
> > > 13.78 MB/sec
> > >
> > > 163.67
> > >
> > > 245
> > >
> > > 2
> > >
> > > MB2P1R
> > >
> > > 2
> > >
> > > 1
> > >
> > > 3
> > >
> > > 8
> > >
> > > 65495
> > >
> > > 15865.719
> > >
> > > 58269.57295
> > >
> > > 11.70 MB/sec
> > >
> > > 173.81
> > >
> > > 470
> > >
> > > 3
> > >
> > > MB2P2R
> > >
> > > 2
> > >
> > > 2
> > >
> > > 3
> > >
> > > 8
> > >
> > > 65495
> > >
> > > 16202.813
> > >
> > > 46286.21908
> > >
> > > 9.29 MB/sec
> > >
> > > 417.33
> > >
> > > 704
> > >
> > > 4
> > >
> > > MB3P2R
> > >
> > > 3
> > >
> > > 2
> > >
> > > 3
> > >
> > > 8
> > >
> > > 65495
> > >
> > > 16184
> > >
> > > 56412.57537
> > >
> > > 11.32 MB/sec
> > >
> > > 229.82
> > >
> > > 550
> > >
> > > 5
> > >
> > > MB3P3R
> > >
> > > 3
> > >
> > > 3
> > >
> > > 3
> > >
> > > 8
> > >
> > > 65495
> > >
> > >
> > >
> > > 46417.43444
> > >
> > > 9.32 MB/sec
> > >
> > > 411.44
> > >
> > > 705
> > >
> > >
> > >
> > > Regards,
> > > Sunil.
> > > CONFIDENTIAL NOTE:
> > > The information contained in this email is intended only for the use of
> > > the individual or entity named above and may contain information that
> is
> > > privileged, confidential and exempt from disclosure under applicable
> law.
> > > If the reader of this message is not the intended recipient, you are
> > hereby
> > > notified that any dissemination, distribution or copying of this
> > > communication is strictly prohibited. If you have received this message
> > in
> > > error, please immediately notify the sender and delete the mail. Thank
> > you.
> > >
> >
> >
> > --
> > Regards,
> > Himanshu Shukla
> >
>

Re: Producer-Perf-test

[sunil chaudhari]

Hi Himanshu,
Sorry but I pasted from excel. Dont know how it got messed up?

Will resend it.

On Tue, 3 Mar 2020 at 6:48 PM, Himanshu Shukla 
wrote:

> could you please share the result in some proper way? Each field is line by
> line.
>
> On Tue, Mar 3, 2020 at 2:43 PM Sunil CHAUDHARI
>  wrote:
>
> > Hi,
> > I have done performance testing of Kafka cluster using
> > kafka-producer-perf-test.sh
> > I created diff type of topics and did perf testing. Example: MB1P1R= MB
> is
> > my topic name with 1 Partition and 1 replica.
> > I have 3 nodes cluster.
> >
> > My Observations:
> >
> >   *   When I increased partitions, and keep replica same, then Throughput
> > increases. Compare Row  1 and 2
> >   *   Increase number of replica  and  partitions same: Throughput
> > decreased than before. Row 2 and 3
> >   *   Increase number of partitions and keep replica same as before: TP
> > increased.  Row 3 and 4
> >   *   Increase number of Replica and keep partitions same: TP decreased.
> > Row 4 and 5.
> >
> > Question: Does it make any impact on throughput when number of partitions
> > and replicas are equal?
> >
> > Sr NO
> >
> > Topic
> >
> > Partitions
> >
> > Replicas
> >
> > Network.threads
> >
> > IO-Threads
> >
> > Records Sent
> >
> > Batch-size-avg
> >
> > Throughput
> >
> > Latency (ms)
> >
> > Records/sec
> >
> > Size/sec
> >
> > Avg
> >
> > Max
> >
> > 1
> >
> > MB1P1R
> >
> > 1
> >
> > 1
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16220
> >
> > 68653.03983
> >
> > 13.78 MB/sec
> >
> > 163.67
> >
> > 245
> >
> > 2
> >
> > MB2P1R
> >
> > 2
> >
> > 1
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 15865.719
> >
> > 58269.57295
> >
> > 11.70 MB/sec
> >
> > 173.81
> >
> > 470
> >
> > 3
> >
> > MB2P2R
> >
> > 2
> >
> > 2
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16202.813
> >
> > 46286.21908
> >
> > 9.29 MB/sec
> >
> > 417.33
> >
> > 704
> >
> > 4
> >
> > MB3P2R
> >
> > 3
> >
> > 2
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16184
> >
> > 56412.57537
> >
> > 11.32 MB/sec
> >
> > 229.82
> >
> > 550
> >
> > 5
> >
> > MB3P3R
> >
> > 3
> >
> > 3
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> >
> >
> > 46417.43444
> >
> > 9.32 MB/sec
> >
> > 411.44
> >
> > 705
> >
> >
> >
> > Regards,
> > Sunil.
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use of
> > the individual or entity named above and may contain information that is
> > privileged, confidential and exempt from disclosure under applicable law.
> > If the reader of this message is not the intended recipient, you are
> hereby
> > notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have received this message
> in
> > error, please immediately notify the sender and delete the mail. Thank
> you.
> >
>
>
> --
> Regards,
> Himanshu Shukla
>

Does the response to a OffsetFetch include non-committed transactional offsets?

[Reftel, Magnus]

If a consumer sends its offset for a topic-partition as part of a transaction, 
and someone sends an OffsetFetch request for that consumer group and 
topic-partition before the transaction is committed, is the OffsetFetch 
response meant to include that pending offset, or only the last offset sent 
outside of a non-committed transaction? I find no discussion of it in the Kafka 
protocol guide, and the code in GroupMetadata.scala seems to indicate that 
pending offsets are not included, but I'm observing some behavior that might 
suggest otherwise.

Best Regards
Magnus Reftel



Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den 
institusjon eller person den er rettet til og kan vaere belagt med lovbestemt 
taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt 
Skatteetaten.
The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential information and may be legally 
protected from disclosure. If you are not the intended recipient of this 
message, please immediately delete the message and alert the Norwegian Tax 
Administration.

SV: Error handling guarantees in Kafka Streams

[Reftel, Magnus]

Thanks, the explanation makes sense. I'll dig around a bit to see if I can 
figure out why that's not what I'm seeing in my testing.

Best Regards
Magnus Reftel

-Opprinnelig melding-
Fra: Bruno Cadonna 
Sendt: onsdag 26. februar 2020 17:57
Til: Users 
Emne: Re: Error handling guarantees in Kafka Streams

Hi Magnus,

with exactly-once, the producer commits the consumer offsets. Thus, if the 
producer is not able to successfully commit a transaction, no consumer offsets 
will be successfully committed, too.

Best,
Bruno

On Wed, Feb 26, 2020 at 1:51 PM Reftel, Magnus 
 wrote:
>
> Hi,
>
> From my understanding, it is guaranteed that when a Kafka Streams application 
> running with the exactly_once processing guarantee receives a record, it will 
> either finish processing the record (including flushing any records generated 
> as a direct result of processing the message and committing the transaction), 
> invoke either the DeserializationExceptionHandler or the 
> ProductionExceptionHandler exception handler, or retry processing the 
> message. Is that correct, or are there cases where a record can be consumed 
> (and the consumption committed) without the Kafka Streams application being 
> able to either produce any output or handle an exception?
>
> Best Regards
> Magnus Reftel
>
> 
> Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den 
> institusjon eller person den er rettet til og kan vaere belagt med lovbestemt 
> taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt 
> Skatteetaten.
> The contents of this email message and any attachments are intended solely 
> for the addressee(s) and may contain confidential information and may be 
> legally protected from disclosure. If you are not the intended recipient of 
> this message, please immediately delete the message and alert the Norwegian 
> Tax Administration.


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den 
institusjon eller person den er rettet til og kan være belagt med lovbestemt 
taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt 
Skatteetaten.
The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential information and may be legally 
protected from disclosure. If you are not the intended recipient of this 
message, please immediately delete the message and alert the Norwegian Tax 
Administration.

Re: Producer-Perf-test

[Himanshu Shukla]

could you please share the result in some proper way? Each field is line by
line.

On Tue, Mar 3, 2020 at 2:43 PM Sunil CHAUDHARI
 wrote:

> Hi,
> I have done performance testing of Kafka cluster using
> kafka-producer-perf-test.sh
> I created diff type of topics and did perf testing. Example: MB1P1R= MB is
> my topic name with 1 Partition and 1 replica.
> I have 3 nodes cluster.
>
> My Observations:
>
>   *   When I increased partitions, and keep replica same, then Throughput
> increases. Compare Row  1 and 2
>   *   Increase number of replica  and  partitions same: Throughput
> decreased than before. Row 2 and 3
>   *   Increase number of partitions and keep replica same as before: TP
> increased.  Row 3 and 4
>   *   Increase number of Replica and keep partitions same: TP decreased.
> Row 4 and 5.
>
> Question: Does it make any impact on throughput when number of partitions
> and replicas are equal?
>
> Sr NO
>
> Topic
>
> Partitions
>
> Replicas
>
> Network.threads
>
> IO-Threads
>
> Records Sent
>
> Batch-size-avg
>
> Throughput
>
> Latency (ms)
>
> Records/sec
>
> Size/sec
>
> Avg
>
> Max
>
> 1
>
> MB1P1R
>
> 1
>
> 1
>
> 3
>
> 8
>
> 65495
>
> 16220
>
> 68653.03983
>
> 13.78 MB/sec
>
> 163.67
>
> 245
>
> 2
>
> MB2P1R
>
> 2
>
> 1
>
> 3
>
> 8
>
> 65495
>
> 15865.719
>
> 58269.57295
>
> 11.70 MB/sec
>
> 173.81
>
> 470
>
> 3
>
> MB2P2R
>
> 2
>
> 2
>
> 3
>
> 8
>
> 65495
>
> 16202.813
>
> 46286.21908
>
> 9.29 MB/sec
>
> 417.33
>
> 704
>
> 4
>
> MB3P2R
>
> 3
>
> 2
>
> 3
>
> 8
>
> 65495
>
> 16184
>
> 56412.57537
>
> 11.32 MB/sec
>
> 229.82
>
> 550
>
> 5
>
> MB3P3R
>
> 3
>
> 3
>
> 3
>
> 8
>
> 65495
>
>
>
> 46417.43444
>
> 9.32 MB/sec
>
> 411.44
>
> 705
>
>
>
> Regards,
> Sunil.
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use of
> the individual or entity named above and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this message in
> error, please immediately notify the sender and delete the mail. Thank you.
>


-- 
Regards,
Himanshu Shukla

Please help: How to print --reporting-interval in the perf metrics?

[Sunil CHAUDHARI]

Hi,
I want to test consumer perf using kafka-consumer-perf-test.sh
I am running below command:

./kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic MB3P3R 
--messages 65495 --num-fetch-threads  9 --print-metrics --reporting-interval 
5000 --show-detailed-stats  > MB3P3R-consumer-Perf2.log

I am getting metrics output as below:  Its not printing anything under the 
highlighted columns.
Another question: Whatevere number of partitions and replicas my topic has, it 
always giving  approx value "2159" for records-consumed-rate

time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, 
rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

Metric Name 
Value
consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-1} 
  : 2.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-rate:{client-id=consumer-1} 
  : 0.033
consumer-coordinator-metrics:commit-total:{client-id=consumer-1}
  : 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-1}  
  : 0.000
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-1} 
  : NaN
consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-1} 
  : 0.000
consumer-coordinator-metrics:join-rate:{client-id=consumer-1}   
  : 0.033

And so on...


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Producer-Perf-test

[Sunil CHAUDHARI]

Hi,
I have done performance testing of Kafka cluster using 
kafka-producer-perf-test.sh
I created diff type of topics and did perf testing. Example: MB1P1R= MB is my 
topic name with 1 Partition and 1 replica.
I have 3 nodes cluster.

My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?

Sr NO

Topic

Partitions

Replicas

Network.threads

IO-Threads

Records Sent

Batch-size-avg

Throughput

Latency (ms)

Records/sec

Size/sec

Avg

Max

1

MB1P1R

1

1

3

8

65495

16220

68653.03983

13.78 MB/sec

163.67

245

2

MB2P1R

2

1

3

8

65495

15865.719

58269.57295

11.70 MB/sec

173.81

470

3

MB2P2R

2

2

3

8

65495

16202.813

46286.21908

9.29 MB/sec

417.33

704

4

MB3P2R

3

2

3

8

65495

16184

56412.57537

11.32 MB/sec

229.82

550

5

MB3P3R

3

3

3

8

65495



46417.43444

9.32 MB/sec

411.44

705



Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Producer MESSAGE_TOO_LARGE warnning when compression is zstd

[Jiamei Xie]

Hi,
Issue:
The MESSAGE_TOO_LARGE  warning occurred when I run producer-perf-test.sh and 
the broker is arm server.  Below it’s the command I used:
bin/kafka-topics.sh --create --bootstrap-server arm-server:9092 
--replication-factor 1 --partitions 6 --topic test
bin/kafka-producer-perf-test.sh --topic test --num-records 5000 
--throughput -1 --record-size 1000 --producer-props bootstrap.servers= 
arm-server:9092 acks=1 buffer.memory=67108864 batch.size=65536 
compression.type=zstd

[2020-03-03 06:11:02,485] WARN [Producer clientId=producer-1] Got error produce 
response in correlation id 236 on topic-partition test-2, splitting and 
retrying (2147483647 attempts left). Error: MESSAGE_TOO_LARGE 
(org.apache.kafka.clients.producer.internals.Sender)
514288 records sent, 7931.4 records/sec (7.56 MB/sec), 32267.0 ms avg latency, 
60002.0 ms max latency.
[2020-03-03 06:16:15,294] ERROR Uncaught exception in thread 
'kafka-producer-network-thread | producer-1': 
(org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar 
replaced objects
After MESSAGE_TOO_LARGE occurred, the memory used in client increase 
dramatically, and the performance got worse.  Below is the dstat data in client.
dstat -cdmn
--total-cpu-usage-- -dsk/total- --memory-usage- -net/total-
usr sys idl wai stl| read  writ| used  free  buff  cach| recv  send
  0   0  99   0   0|1762k   11k|1180M 29.9G 13.3M  162M|   0 0
  5   3  92   0   0|   0 0 |1180M 29.9G 13.3M  162M|1966B 3924B
11   2  87   0   0|   016k|1238M 29.9G 13.3M  162M|5066B  379k
  9   3  87   0   0|   0 0 |1277M 29.8G 13.3M  162M|3580B  281k
  5   2  93   0   0|   0 0 |1278M 29.8G 13.3M  162M|3712B  253k
16   2  82   0   0|   0 0 |1334M 29.8G 13.3M  162M|  12k 1568k
21   4  76   0   0|   020k|2047M 29.1G 13.3M  162M|  11k   51k
  9   4  87   0   0|   0 0 |3199M 27.9G 13.3M  162M|  26k   97k
  7   5  88   0   0|   0 0 |4341M 26.8G 13.3M  162M|  25k   94k
  5   5  90   0   0|   0 0 |5543M 25.7G 13.3M  162M|  25k  100k
  5   5  90   0   0|   0 0 |6903M 24.3G 13.3M  162M|  29k  120k
12   6  82   0   0|   0 0 |8320M 22.9G 13.3M  162M|  29k  116k
  6   5  89   0   0|   028k|9568M 21.7G 13.3M  162M|  30k  124k
  7   4  89   0   0|   0 0 |10.2G 20.8G 13.3M  162M|  35k  389k
11   5  83   0   0|   0 0 |10.3G 20.7G 13.3M  162M|  50k  768k
  6   2  92   0   0|   0 0 |10.3G 20.7G 13.3M  162M|  20k  530k
  8   3  89   0   0|   0 0 |10.4G 20.7G 13.3M  162M|9183B  471k
  8   2  90   0   0|   096k|10.4G 20.7G 13.3M  162M|5552B  156k
  6   2  92   0   0|   0 0 |10.4G 20.7G 13.3M  162M|9672B  486k
13   3  84   0   0|   016k|10.5G 20.6G 13.3M  162M|5771B  310k
10   3  87   0   0|   0 0 |10.5G 20.6G 13.3M  162M|7868B  295k
14   2  85   0   0|   0 0 |10.5G 20.5G 13.3M  162M|  16k 1524k
20   3  76   0   0|   032k|11.2G 19.9G 13.3M  162M|  13k   54k
12   5  84   0   0|   0 0 |12.4G 18.7G 13.3M  162M|  27k  113k
  7   5  88   0   0|   0 0 |13.7G 17.4G 13.3M  162M|  29k  115k
11   5  84   0   0|   0 0 |14.9G 16.1G 13.3M  162M|  30k  124k
  6   5  89   0   0|   0 0 |16.3G 14.8G 13.3M  162M|  28k  118k
  6   4  89   1   0|   060k|17.5G 13.6G 13.3M  162M|  27k  107k
  7   6  87   0   0|   0 0 |18.7G 12.4G 13.3M  162M|  29k  164k
  5   1  93   0   0|   0 0 |18.7G 12.4G 13.3M  162M|  45k  713k
10   3  87   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  24k  548k
  6   2  92   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  12k  422k
  7   2  90   0   0|   044k|18.8G 12.3G 13.3M  162M|  13k  470k
16   2  82   0   0|   0 0 |18.8G 12.3G 13.3M  162M|9324B  323k
  6   2  91   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  11k  386k
  8   2  90   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  12k  403k
  8   3  89   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  12k  370k
  7   2  90   0   0|   0 0 |18.8G 12.3G 13.3M  162M|  13k  435k
  7   2  91   0   0|   020k|18.8G 12.2G 13.3M  162M|  17k  397k
  8   2  90   0   0|   0 0 |18.8G 12.2G 13.3M  162M|  16k  546k
10   2  88   0   0|   0 0 |18.9G 12.2G 13.3M  162M|  13k  436k
  5   2  93   0   0|   0 0 |18.9G 12.2G 13.3M  162M|  14k  492k
11   3  87   0   0|   0 0 |18.9G 12.2G 13.3M  162M|  13k  465k
  8   2  90   0   0|   0 0 |18.9G 12.2G 13.3M  162M|9836B  384k
10   1  88   0   0|   020k|18.9G 12.2G 13.3M  162M|9884B  392k
10   1  89   0   0|   0 0 |18.9G 12.1G 13.3M  162M|  69k  771k
  4   2  94   0   0|   0 0 |18.9G 12.1G 13.3M  162M|  71k  668k
  4   1  95   0   0|   0 0 |19.0G 12.1G 13.3M  162M|  68k  634k
  2   1  98   0   0|   016k|19.0G 12.1G 13.3M  162M|  72k  725k
  6   2  92   0   0|   0 0 |19.0G 12.1G 13.3M  162M|  73k  690k
  3   1  95   0   0| 232k0 |19.0G 12.0G 13.5M  162M|  73k  723k
  5   2  94   0   0|   0 0 |19.0G 12.0G 13.5M  162M|  79k  764k
  2   1  96   0   0|4096B   56k|19.1G 12.0G 13.5M  162M|  81k  800k