Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #944

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[Guozhang Wang]

Thanks José! For 1/2 above, just checking if we would record the
corresponding sensors only during broker bootstrap time, or whenever there
are new metadata records being committed by the controller quorum (since
there are always a short period of time, between when the records are
committed, to when the records get fetched by that broker)?

On Wed, May 18, 2022 at 7:53 PM Luke Chen  wrote:

> Hi José,
>
> Thanks for the KIP!
> +1 (binding) from me.
>
> Luke
>
> On Thu, May 19, 2022 at 2:05 AM José Armando García Sancio
>  wrote:
>
> > Hi Guozhang, thanks for the feedback.
> >
> >
> > Guozhang wrote:
> > > Could you elaborate a bit on what does "load-processing-time-us"
> > measure? I
> > > looked through the discussion thread and the KIP / JIRA but cannot find
> > its
> > > definitions.
> >
> > Yes. I updated the KIP. This is what I documented:
> > 1.
> >
> kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-avg
> > Reports the average amount of time it took for the broker to process
> > all pending records when there are pending records in the cluster
> > metadata partition. The time unit for this metric is microseconds.
> > 2.
> >
> kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-max
> > Reports the maximum amount of time it took for the broker to process
> > all pending records when there are pending records in the cluster
> > metadata partition. The time unit for this metric is microseconds.
> > 3.
> > kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
> > Reports the average byte size of the record batches in the cluster
> > metadata partition.
> > 4.
> > kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
> > Reports the maximum byte size of the record batches in the cluster
> > metadata partition.
> >
> > -José
> >
>


-- 
-- Guozhang

[jira] [Resolved] (KAFKA-13837) Return error for Fetch requests from unrecognized followers

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13837.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

> Return error for Fetch requests from unrecognized followers
> ---
>
> Key: KAFKA-13837
> URL: https://issues.apache.org/jira/browse/KAFKA-13837
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.3.0
>
>
> If the leader of a partition receives a request from a replica which is not 
> in the current replica set, we currently return an empty fetch response with 
> no error. I think the rationale for this is that the leader may not have 
> received the latest `LeaderAndIsr` update which adds the replica, so we just 
> want the follower to retry. The problem with this is that if the 
> `LeaderAndIsr` request never arrives, then there might not be an external 
> indication of a problem. This probably was the only reasonable option before 
> we added the leader epoch to the `Fetch` request API. Now that we have it, it 
> would be clearer to return an `UNKNOWN_LEADER_EPOCH` error to indicate that 
> the (replicaId, leaderEpoch) tuple is not recognized. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Luke Chen]

Hi Niket,

Thanks for the KIP!
+1 (binding) from me.

Luke

On Thu, May 19, 2022 at 8:27 AM deng ziming 
wrote:

> +1(non-binding)
>
> Thanks for this KIP.
> --
> Best,
> Ziming
>
> > On May 17, 2022, at 1:14 AM, Niket Goel 
> wrote:
> >
> > Hi all,
> >
> > I would like to start a vote for KIP-836:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Addition+of+Information+in+DescribeQuorumResponse+about+Voter+Lag
> >
> > Thanks
> > Niket Goel
>
>

Re: [VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[Luke Chen]

Hi José,

Thanks for the KIP!
+1 (binding) from me.

Luke

On Thu, May 19, 2022 at 2:05 AM José Armando García Sancio
 wrote:

> Hi Guozhang, thanks for the feedback.
>
>
> Guozhang wrote:
> > Could you elaborate a bit on what does "load-processing-time-us"
> measure? I
> > looked through the discussion thread and the KIP / JIRA but cannot find
> its
> > definitions.
>
> Yes. I updated the KIP. This is what I documented:
> 1.
> kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-avg
> Reports the average amount of time it took for the broker to process
> all pending records when there are pending records in the cluster
> metadata partition. The time unit for this metric is microseconds.
> 2.
> kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-max
> Reports the maximum amount of time it took for the broker to process
> all pending records when there are pending records in the cluster
> metadata partition. The time unit for this metric is microseconds.
> 3.
> kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
> Reports the average byte size of the record batches in the cluster
> metadata partition.
> 4.
> kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
> Reports the maximum byte size of the record batches in the cluster
> metadata partition.
>
> -José
>

Re: [VOTE] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[deng ziming]

+1(non-binding)

Thanks for this KIP.
--
Best,
Ziming

> On May 17, 2022, at 1:14 AM, Niket Goel  wrote:
> 
> Hi all,
> 
> I would like to start a vote for KIP-836:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Addition+of+Information+in+DescribeQuorumResponse+about+Voter+Lag
> 
> Thanks
> Niket Goel

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Niket Goel]

I did miss updating the KIP on Jose's comment. Have done that now, thanks
for the reminder.

For the `kafka-metadata-quorum.sh` tool, it seems that the tool's
dependence on the DescribeQuorum API is implicit given the original KIP
[1]. I will add a section in this KIP demonstrating that the tool's output
should contain the newly added fields as well.
The tool itself is tracked under this JIRA [2]

Thanks
Niket

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595%3AARaftProtocolfortheMetadataQuorum-ToolingSupport=
[2] https://issues.apache.org/jira/browse/KAFKA-13914

On Tue, May 17, 2022 at 7:31 PM deng ziming 
wrote:

> Hello Niket,
>
>
> 1. I find the DescribeQuorumResult still contains an
> DescribeQuorumResponseData, which is not allowed as Jose commented, have
> you forgot to change it?
>
> 2. You only add an Handle in AdminClient, can you also add an
> `kafka-metadata-quorum.sh` tool to help this?
>
>
> > On May 17, 2022, at 9:50 AM, Niket Goel 
> wrote:
> >
> > Thanks for the call out David. We will populate these fields for the
> > Observers as well. I will clarify this in the KIP.
> >
> > On Mon, May 16, 2022 at 1:50 PM David Arthur 
> wrote:
> >
> >> Niket, thanks for the KIP!
> >>
> >> Sorry for the late feedback on this, but I just had a quick question.
> The
> >> KIP indicates the two new fields will be set for voters only, however
> this
> >> ReplicaState struct is also used by the Observers in
> >> DescribeQuorumResponse. Will we simply fill in -1 for these values, or
> do
> >> we intend to report the last fetch and caught-up time of the observers
> as
> >> well?
> >>
> >> Thanks!
> >> David
> >>
> >>
> >> On Mon, May 16, 2022 at 1:46 PM Niket Goel 
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Thank you for the feedback on this. I have started a voting thread for
> >>> this KIP here:
> >>> https://lists.apache.org/thread/bkb7gsbxpljh5qh014ztffq7bldjrb2x
> >>>
> >>> Thanks
> >>> Niket Goel
> >>>
> >>>
> >>> From: Niket Goel 
> >>> Date: Thursday, May 12, 2022 at 5:25 PM
> >>> To: dev@kafka.apache.org 
> >>> Subject: Re: [DISCUSS] KIP-836: Addition of Information in
> >>> DescribeQuorumResponse about Voter Lag
> >>> Appreciate the careful review Jose.!
> >>>
> >>> Ack on 1 and 2. Will fix.
> >>>
> >>> For number 3 (and I am using [1] as a reference for this discussion), I
> >>> think the correct language to use would be:
> >>>
> >>> "Whenever a new fetch request
> >>> comes in the replica's last caught up time is updated to the time of
> >>> this fetch request if it requests an offset greater than or equal to
> the
> >>> leader's
> >>> current end offset"
> >>> Does that sound right now?
> >>>
> >>> Although I think I will go ahead and rewrite the explanation in a way
> >> that
> >>> is more understandable. Thanks for pointing this out.
> >>>
> >>> Thanks
> >>>
> >>> [1]
> >>>
> >>
> https://github.com/apache/kafka/blob/fa59be4e770627cd34cef85986b58ad7f606928d/core/src/main/scala/kafka/cluster/Replica.scala#L97
> >>>
> >>>
> >>>
> >>> On Thu, May 12, 2022 at 3:20 PM José Armando García Sancio
> >>>  wrote:
> >>> Thanks for the Kafka improvement Niket.
> >>>
> >>> 1. For the fields `LastFetchTime` and `LastCaughtUpTime`, Kafka tends
> >>> to use the suffix "Timestamp" when the value is an absolute wall clock
> >>> value.
> >>>
> >>> 2. The method `result()` for the type `DescribeQuorumResult` returns
> >>> the type `DescribeQuorumResponseData`. The types generated from the
> >>> RPC JSON schema are internal to Kafka and not exposed to clients. For
> >>> the admin client we should use a different type that is explicitly
> >>> public. See `org.apache.kafka.client.admin.DescribeTopicsResult` for
> >>> an example.
> >>>
> >>> 3. The proposed section has his sentence "Whenever a new fetch request
> >>> comes in the replica's last caught up time is updated to the time of
> >>> the fetch request if it requests an offset greater than the leader's
> >>> current end offset." Did you mean "previous fetch time" instead of
> >>> "last caught up time"? What do you mean by "requests an offset greater
> >>> than the leader's current end offset.?" Excluding diverging logs the
> >>> follower fetch offset should never be greater than the leader LEO.
> >>>
> >>> Thanks,
> >>> -José
> >>>
> >>>
> >>> --
> >>> - Niket
> >>>
> >>
> >
> >
> > --
> > - Niket
>
>

-- 
- Niket

[jira] [Created] (KAFKA-13914) Implement kafka-metadata-quorum.sh

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13914:
---

 Summary: Implement kafka-metadata-quorum.sh
 Key: KAFKA-13914
 URL: https://issues.apache.org/jira/browse/KAFKA-13914
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


KIP-595 documents a tool for describing quorum status 
`kafka-metadata-quorum.sh`: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-ToolingSupport.]
  We need to implement this.

Note that this depends on the Admin API for `DescribeQuorum`, which is proposed 
in KIP-836: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Addition+of+Information+in+DescribeQuorumResponse+about+Voter+Lag.]
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

[Kirk True]

Hi François,

Thanks for the KIP!

A couple of questions:

 1. Do the builders have defaults for the serializers/deserializers?
 2. Can the build method be called more than once on a given builder? 

Thanks,
Kirk

On Wed, May 18, 2022, at 10:11 AM, François Rosière wrote:
> Hi all,
> 
> KIP to create builders for
> 
>- KafkaProducer
>- KafkaConsumer
>- KafkaStreams
> 
> 
> This KIP can be seen as the continuity of the KIP-832.
> 
> KIP details:  
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884640
> Jira issue: https://issues.apache.org/jira/browse/KAFKA-13913
> 
> Kr,
> 
> F.
> 

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[José Armando García Sancio]

Hi Colin,

Thanks for the KIP.

>The rationale for deprecating ZK in the 3.4 release is so that we can remove 
>it in the 4.0 release. (In general, Kafka requires features to be deprecated 
>for at least one release before they can be removed in the following major 
>release.) During the deprecation period, ZK mode would continue to be fully 
>supported. However, we would let people know that KRaft mode is the future.

Can you also add that for existing ZK clusters the user will have to first:
1. Upgrade the Kafka cluster to a version between [3.3, 4.0).
2. Convert the ZK cluster to KRaft (pending a future KIP).
3. Upgrade the KRaft cluster to 4.x

Does this match your understanding?

-- 
-José

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Hi Jun,

Jun wrote:
> 20. For the metric type and name, we use the camel names in some cases and
> dashed lower names in some other cases. Should we make them consistent?

For the metrics group `type=KafkaController`, I am using camel names
like `MetadataLastAppliedRecordOffset` because it matches the naming
strategy for the metrics already in that group.

For the metrics group `type=broker-metadata-metrics`, I model the
naming scheme after the metrics in KIP-595 or the `raft-metrics`
group. I made the assumption that we wanted to start naming metrics
and groups using that scheme but maybe that is not correct.

What do you think?

> 21. Could you document the meaning of load-processing-time?

I updated KIP-835 to include this information. Here is a summary:

1. kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-avg
Reports the average amount of time it took for the broker to process
all pending records when there are pending records in the cluster
metadata partition. The time unit for this metric is microseconds.
2. kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-max
Reports the maximum amount of time it took for the broker to process
all pending records when there are pending records in the cluster
metadata partition. The time unit for this metric is microseconds.
3. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
Reports the average byte size of the record batches in the cluster
metadata partition.
4. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
Reports the maximum byte size of the record batches in the cluster
metadata partition.

-José

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #942

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 628464 lines...]
[2022-05-18T20:03:32.591Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentKeyValueStoreToTimestampedKeyValueStoreUsingPapi PASSED
[2022-05-18T20:03:32.591Z] 
[2022-05-18T20:03:32.591Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi STARTED
[2022-05-18T20:04:24.858Z] 
[2022-05-18T20:04:24.858Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi PASSED
[2022-05-18T20:04:24.858Z] 
[2022-05-18T20:04:24.858Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi STARTED
[2022-05-18T20:05:11.985Z] 
[2022-05-18T20:05:11.985Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi PASSED
[2022-05-18T20:05:11.985Z] 
[2022-05-18T20:05:11.985Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigrateInMemoryWindowStoreToTimestampedWindowStoreUsingPapi STARTED
[2022-05-18T20:05:58.846Z] 
[2022-05-18T20:05:58.846Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigrateInMemoryWindowStoreToTimestampedWindowStoreUsingPapi PASSED
[2022-05-18T20:05:58.846Z] 
[2022-05-18T20:05:58.846Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-05-18T20:06:04.312Z] 
[2022-05-18T20:06:04.312Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-05-18T20:06:04.312Z] 
[2022-05-18T20:06:04.312Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2022-05-18T20:06:08.870Z] 
[2022-05-18T20:06:08.870Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2022-05-18T20:06:08.870Z] 
[2022-05-18T20:06:08.870Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2022-05-18T20:06:13.125Z] 
[2022-05-18T20:06:13.125Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2022-05-18T20:06:13.125Z] 
[2022-05-18T20:06:13.125Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-05-18T20:06:16.421Z] 
[2022-05-18T20:06:16.421Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2022-05-18T20:06:16.421Z] 
[2022-05-18T20:06:16.421Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceThreads STARTED
[2022-05-18T20:06:17.464Z] 
[2022-05-18T20:06:17.464Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-3ceefbbc-d3a7-4871-9a4a-4f5e39b6b673-StreamThread-1"
[2022-05-18T20:06:20.727Z] 
[2022-05-18T20:06:20.727Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-3ceefbbc-d3a7-4871-9a4a-4f5e39b6b673-StreamThread-2"
[2022-05-18T20:06:23.996Z] 
[2022-05-18T20:06:23.996Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceThreads PASSED
[2022-05-18T20:06:23.996Z] 
[2022-05-18T20:06:23.996Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException STARTED
[2022-05-18T20:06:29.499Z] 
[2022-05-18T20:06:29.499Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException PASSED
[2022-05-18T20:06:29.499Z] 
[2022-05-18T20:06:29.499Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler STARTED
[2022-05-18T20:06:34.006Z] 
[2022-05-18T20:06:34.006Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler PASSED
[2022-05-18T20:06:34.006Z] 
[2022-05-18T20:06:34.006Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread STARTED
[2022-05-18T20:06:36.100Z] 
[2022-05-18T20:06:36.100Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-fcf5646b-df38-4abb-a771-49c8ed577140-GlobalStreamThread"
[2022-05-18T20:06:36.100Z] 
[2022-05

[DISCUSS] KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

[François Rosière]

Hi all,

KIP to create builders for

   - KafkaProducer
   - KafkaConsumer
   - KafkaStreams


This KIP can be seen as the continuity of the KIP-832.

KIP details:  
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884640
Jira issue: https://issues.apache.org/jira/browse/KAFKA-13913

Kr,

F.

[jira] [Created] (KAFKA-13913) Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

[Jira]

François Rosière created KAFKA-13913:


 Summary: Provide builders for KafkaProducer/KafkaConsumer and 
KafkaStreams
 Key: KAFKA-13913
 URL: https://issues.apache.org/jira/browse/KAFKA-13913
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: François Rosière
Assignee: François Rosière


To have more flexibility, builders should be provided for the following objects
 * KafkaProducer
 * KafkaConsumer
 * KafkaStreams 

These builders will give an easy way to construct these objects using different 
arguments/combinations without having to add a new constructor every time a new 
parameter is required.

They will also allow using already configured dependencies coming from an 
injection framework such as Spring (see 
[https://github.com/spring-projects/spring-kafka/issues/2244]).

>From a user point of view, builders would be used as follow

 
{noformat}
KafkaProducer kafkaProducer = new KafkaProducerBuilder()  .withKeySerializer()  
.withValueSerializer()  
.withInterceptors()  .withPartitioner()  
.withMetricsReporter()  .build();  KafkaConsumer consumer = 
new KafkaConsumerBuilder()  
.withKeyDeserializer()  
.withValueDeserializer()  
.withInterceptors()  
.withMetricsReporter()  .build(); KafkaStreams kafkaStreams = 
new KafkaStreamsBuilder(, )  
.withProducerInterceptors()  
.withConsumerInterceptors()  .withTime()  
.withKafkaClientSupplier()  
.withMetricsReporter()  .build();{noformat}
 

 

 

This KIP can be seen as the continuity of the KIP-832.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13912) Generate metadata snapshot prior to metadata.version change

[David Arthur (Jira)]

David Arthur created KAFKA-13912:


 Summary: Generate metadata snapshot prior to metadata.version 
change
 Key: KAFKA-13912
 URL: https://issues.apache.org/jira/browse/KAFKA-13912
 Project: Kafka
  Issue Type: Sub-task
  Components: controller
Reporter: David Arthur
Assignee: David Arthur






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13830) Introduce metadata.version in KRaft

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-13830.
--
Resolution: Fixed

> Introduce metadata.version in KRaft
> ---
>
> Key: KAFKA-13830
> URL: https://issues.apache.org/jira/browse/KAFKA-13830
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.3.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[DISCUSS] KIP-837 Allow MultiCasting a Result Record.

[Sagar]

Hi All,

I would like to open a discussion thread on
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356.

Thanks!
Sagar.

Re: [VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Hi Guozhang, thanks for the feedback.


Guozhang wrote:
> Could you elaborate a bit on what does "load-processing-time-us" measure? I
> looked through the discussion thread and the KIP / JIRA but cannot find its
> definitions.

Yes. I updated the KIP. This is what I documented:
1. kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-avg
Reports the average amount of time it took for the broker to process
all pending records when there are pending records in the cluster
metadata partition. The time unit for this metric is microseconds.
2. kafka.server:type=broker-metadata-metrics,name=load-processing-time-us-max
Reports the maximum amount of time it took for the broker to process
all pending records when there are pending records in the cluster
metadata partition. The time unit for this metric is microseconds.
3. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
Reports the average byte size of the record batches in the cluster
metadata partition.
4. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
Reports the maximum byte size of the record batches in the cluster
metadata partition.

-José

Re: [DISCUSS] KIP-714: Client metrics and observability

[Jun Rao]

Hi, Magnus,

Thanks for the updated KIP. Just a couple of more comments.

50. To troubleshoot a particular client issue, I imagine that the client
needs to identify its client_instance_id. How does the client find this
out? Do we plan to include client_instance_id in the client log, expose it
as a metric or something else?

51. The KIP lists a bunch of metrics that need to be collected at the
client side. However, it seems quite a few useful java client metrics like
the following are missing.
buffer-total-bytes
buffer-available-bytes
bufferpool-wait-time
batch-size-avg
batch-size-max
io-wait-ratio
io-ratio

Thanks,

Jun

On Mon, Apr 4, 2022 at 10:01 AM Jun Rao  wrote:

> Hi, Xavier,
>
> Thanks for the reply.
>
> 28. It does seem that we have started using KafkaMetrics on the broker
> side. Then, my only concern is on the usage of Histogram in KafkaMetrics.
> Histogram in KafkaMetrics statically divides the value space into a fixed
> number of buckets and only returns values on the bucket boundary. So, the
> returned histogram value may never show up in a recorded value. Yammer
> Histogram, on the other hand, uses reservoir sampling. The reported value
> is always one of the recorded values. So, I am not sure that Histogram in
> KafkaMetrics is as good as Yammer Histogram. ClientMetricsPluginExportTime
> uses Histogram.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 31, 2022 at 5:21 PM Xavier Léauté 
> wrote:
>
>> >
>> > 28. On the broker, we typically use Yammer metrics. Only for metrics
>> that
>> > depend on Kafka metric features (e.g., quota), we use the Kafka metric.
>> > Yammer metrics have 4 types: gauge, meter, histogram and timer. meter
>> > calculates a rate, but also exposes an accumulated value.
>> >
>>
>> I don't see a good reason we should limit ourselves to Yammer metrics on
>> the broker. KafkaMetrics was written
>> to replace Yammer metrics and is used for all new components (clients,
>> streams, connect, etc.)
>> My understanding is that the original goal was to retire Yammer metrics in
>> the broker in favor of KafkaMetrics.
>> We just haven't done so out of backwards compatibility concerns.
>> There are other broker metrics such as group coordinator, transaction
>> state
>> manager, and various socket server metrics
>> already using KafkaMetrics that don't need specific Kafka metric features,
>> so I don't see why we should refrain from using
>> Kafka metrics on the broker unless there are real compatibility concerns
>> or
>> where implementation specifics could lead to confusion when comparing
>> metrics using different implementations.
>>
>> In my opinion we should encourage people to use KafkaMetrics going forward
>> on the broker as well, for two reasons:
>> a) yammer metrics is long deprecated and no longer maintained
>> b) yammer metrics are much less expressive
>> c) we don't have a proper API to expose yammer metrics outside of JMX
>> (MetricsReporter only exposes KafkaMetrics)
>>
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #941

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] guozhangwang merged pull request #412: MINOR: Fix DSL typo in streams docs

[GitBox]

guozhangwang merged PR #412:
URL: https://github.com/apache/kafka-site/pull/412


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Permission request to contribute

[Guozhang Wang]

Hello Alex,

I've added you to JIRA and I saw you're already in Wiki.


Guozhang

On Wed, May 18, 2022 at 7:27 AM Alexandre Garnier 
wrote:

> Hi there!
> I'd like to request permission to contribute to Kafka.
>
> Wiki ID: zigarn
> Jira ID: zigarn
>
> Thanks,
> --
> Alex
>


-- 
-- Guozhang

Re: Requesting Contribution Access

[Mickael Maison]

Hi Richard,

I've granted you access to the wiki and jira.
Thanks for your interest in Apache Kafka!

Mickael


On Wed, May 18, 2022 at 6:34 PM Richard Joerger  wrote:
>
> Hi Team,
>
> I’m looking to get access to Jira and the Wiki to help contribute.
>
> My IDs:
> jira: rjoerger
> wiki: rjoerger
>
>
> Thanks so much,
> Rich J.

Requesting Contribution Access

[Richard Joerger]

Hi Team,

I’m looking to get access to Jira and the Wiki to help contribute.

My IDs:
jira: rjoerger
wiki: rjoerger


Thanks so much,
Rich J.

[jira] [Created] (KAFKA-13911) Rate is calculated as NaN for minimum config values

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-13911:


 Summary: Rate is calculated as NaN for minimum config values
 Key: KAFKA-13911
 URL: https://issues.apache.org/jira/browse/KAFKA-13911
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya
Assignee: Divij Vaidya


Implementation of connection creation rate quotas in Kafka is dependent on two 
configurations:
 # 
[quota.window.num|https://kafka.apache.org/documentation.html#brokerconfigs_quota.window.num]
 # 
[quota.window.size.seconds|https://kafka.apache.org/documentation.html#brokerconfigs_quota.window.size.seconds]

The minimum possible values of these configuration is 1 as per the 
documentation. However, 1 as a minimum value for quota.window.num is invalid 
and leads to failure for calculation of rate as demonstrated below.

As a proof of the bug, the following unit test fails:
{code:java}
@Test
public void testUseWithMinimumPossibleConfiguration() {
final Rate r = new Rate();
MetricConfig config = new MetricConfig().samples(1).timeWindow(1, 
TimeUnit.SECONDS);
Time elapsed = new MockTime();
r.record(config, 1.0, elapsed.milliseconds());
elapsed.sleep(100);
r.record(config, 1.0, elapsed.milliseconds());
elapsed.sleep(1000);
final Double observedRate = r.measure(config, elapsed.milliseconds());
assertFalse(Double.isNaN(observedRate));
} {code}
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Jacot]

Hi,

I created a small KIP to strengthen the AlterPartition API in KRaft mode:
https://cwiki.apache.org/confluence/x/phmhD

Let me know what you think.

Best,
David

Re: Request contributor access

[Bill Bejeck]

Hi Patrik,

Done.  Thanks for your interest in Apache Kafka!

-Bill

On Wed, May 18, 2022 at 10:27 AM Patrik Nagy 
wrote:

> Hi,
>
> I would like to get my JIRA account added to the list of contributors in
> order to be able to assign a JIRA ticket to me for fixing.
>
> My JIRA username is pnagy.
>
> Thanks in advance,
> Patrik
>

Re: Permission request to contribute

[Bill Bejeck]

Hi Alex,

You're all set on Jira and the Wiki now.
Thanks for your interest in Apache Kafka!

-Bill

On Wed, May 18, 2022 at 10:27 AM Alexandre Garnier 
wrote:

> Hi there!
> I'd like to request permission to contribute to Kafka.
>
> Wiki ID: zigarn
> Jira ID: zigarn
>
> Thanks,
> --
> Alex
>

Permission request to contribute

[Alexandre Garnier]

Hi there!
I'd like to request permission to contribute to Kafka.

Wiki ID: zigarn
Jira ID: zigarn

Thanks,
-- 
Alex

Permission request to contribute

[Alexandre Garnier]

Hi there!
I'd like to request permission to contribute to Kafka.

Wiki ID: zigarn
Jira ID: zigarn

Thanks,
-- 
Alex

Request contributor access

[Patrik Nagy]

Hi,

I would like to get my JIRA account added to the list of contributors in
order to be able to assign a JIRA ticket to me for fixing.

My JIRA username is pnagy.

Thanks in advance,
Patrik

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #940

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 548313 lines...]
[2022-05-18T10:06:27.084Z] 
[2022-05-18T10:06:27.084Z] > Task :streams:javadoc
[2022-05-18T10:06:27.084Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/processor/StreamPartitioner.java:50:
 warning - Tag @link: reference not found: 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:28.210Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-05-18T10:06:29.382Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:443:
 warning - @statestore.cache.max.bytes is an unknown tag.
[2022-05-18T10:06:29.382Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:443:
 warning - @statestore.cache.max.bytes is an unknown tag.
[2022-05-18T10:06:29.382Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:443:
 warning - @statestore.cache.max.bytes is an unknown tag.
[2022-05-18T10:06:29.382Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:61:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-05-18T10:06:29.382Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:61:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-05-18T10:06:30.616Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-05-18T10:06:30.616Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-18T10:06:30.616Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-18T10:06:30.616Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/j

Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API

[Divij Vaidya]

+1 non binding.

Divij Vaidya



On Tue, May 17, 2022 at 6:16 PM Igor Soarez  wrote:

> Thanks for this KIP Mickael.
>
> +1 non binding
>
> --
> Igor
>
> On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote:
> > Hi Mickael,
> >
> > +1 (binding) from me.
> > Thanks for the KIP!
> >
> > Luke
> >
> > On Tue, May 17, 2022 at 9:30 PM Mickael Maison  >
> > wrote:
> >
> >> Hi,
> >>
> >> I'd like to start a vote on KIP-827. It proposes exposing the total
> >> and usable space of logdirs
> >> via the DescribeLogDirs API:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> >>
> >> Thanks,
> >> Mickael
> >>
>

[DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[Alexandre Garnier]

Hi everyone,

I created a KIP to add a config file option of reader/formatter for
kafka-console-(consumer|producer).sh tools.
https://cwiki.apache.org/confluence/x/bBqhD

Thanks for your feedback,
-- 
Alex

Re: Permission request to contribute

[Bruno Cadonna]

Hi Alexandre,

you should be all set now!

Thank you for your interest in Apache Kafka!

Best,
Bruno

On 18.05.22 09:40, Alexandre Garnier wrote:

Hi there!
I'd like to request permission to contribute to Kafka.

Wiki ID: zigarn
Jira ID: zigarn

Thanks,

Permission request to contribute

[Alexandre Garnier]

Hi there!
I'd like to request permission to contribute to Kafka.

Wiki ID: zigarn
Jira ID: zigarn

Thanks,
-- 
Alex

[jira] [Resolved] (KAFKA-13864) Change the visibility of a KafkaProducer and KafkaConsumer constructor

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

François Rosière resolved KAFKA-13864.
--
Resolution: Won't Do

> Change the visibility of a KafkaProducer and KafkaConsumer constructor
> --
>
> Key: KAFKA-13864
> URL: https://issues.apache.org/jira/browse/KAFKA-13864
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 3.1.0
>Reporter: François Rosière
>Assignee: François Rosière
>Priority: Major
>  Labels: needs-kip
>
> To allow implementing Spring managed interceptors for producers and consumers,
> [https://github.com/spring-projects/spring-kafka/issues/2244]
> The following changes should be applied in Kafka
> 1. Create 2 new constructors in _KafkaProducer_
> {code:java}
> public KafkaProducer(ProducerConfig config)
> public KafkaProducer(ProducerConfig config, Serializer keySerializer, 
> Serializer valueSerializer) {code}
> 2. Create a new constructor + increase the visibility of an existing 
> constructor in _KafkaConsumer_
> {code:java}
> public KafkaConsumer(ConsumerConfig config)
> public KafkaConsumer(ConsumerConfig config, Deserializer keyDeserializer, 
> Deserializer valueDeserializer){code}
> 3. Create 2 new constructors in _TopologyTestDriver_
> {code:java}
> public TopologyTestDriver(Topology topology, StreamsConfig config)
> public TopologyTestDriver(Topology topology, StreamsConfig config, Instant 
> initialWallClockTime) {code}
>  More details can be found in the related KIP
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578]
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[GitHub] [kafka-site] qingwei91 commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[GitBox]

qingwei91 commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1129651258

   Hi @mimaison , if you get a chance, can you review this again? I've 
addressed the issue you pointed out, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org