[jira] [Created] (KAFKA-16264) Expose `producer.id.expiration.check.interval.ms` as dynamic broker configuration

[Jorge Esteban Quilcate Otoya (Jira)]

Jorge Esteban Quilcate Otoya created KAFKA-16264:


 Summary: Expose `producer.id.expiration.check.interval.ms` as 
dynamic broker configuration
 Key: KAFKA-16264
 URL: https://issues.apache.org/jira/browse/KAFKA-16264
 Project: Kafka
  Issue Type: Improvement
Reporter: Jorge Esteban Quilcate Otoya
Assignee: Jorge Esteban Quilcate Otoya


Dealing with a scenario where too many producer ids lead to issues (e.g. high 
cpu utilization, see KAFKA-16229) put operators in need to flush producer ids 
more promptly than usual.

Currently, only the expiration timeout `producer.id.expiration.ms` is exposed 
as dynamic config. This is helpful (e.g. by reducing the timeout, less producer 
would eventually be kept in memory), but not enough if the evaluation frequency 
is not sufficiently short to flush producer ids before becoming an issue. Only 
by tuning both, the issue could be workaround.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] MINOR: Add docker in list of templates that get rendered [kafka-site]

[via GitHub]

omkreddy merged PR #574:
URL: https://github.com/apache/kafka-site/pull/574


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 3.7.0 RC4

[Vedarth Sharma]

Hey Stanislav,

Thanks for the release candidate.

+1 (non-binding)

I tested and verified the docker image artifact apache/kafka:3.7.0-rc4:-
- verified create topic, produce messages and consume messages flow when
running the docker image with
- default configs
- configs provided via env variables
- configs provided via file input
- verified the html documentation for docker image.
- ran the example docker compose files successfully.

All looks good for the docker image artifact!

Thanks and regards,
Vedarth


On Thu, Feb 15, 2024 at 10:58 PM Mickael Maison 
wrote:

> Hi Stanislav,
>
> Thanks for running the release.
>
> I did the following testing:
> - verified the check sums and signatures
> - ran ZooKeeper and KRaft quickstarts with Scala 2.13 binaries
> - ran a successful migration from ZooKeeper to KRaft
>
> We seem to be missing the upgrade notes for 3.7.0 in the docs. See
> https://kafka.apache.org/37/documentation.html#upgrade that still
> points to 3.6.0
> Before voting I'd like to see results from the system tests too.
>
> Thanks,
> Mickael
>
> On Thu, Feb 15, 2024 at 6:06 PM Andrew Schofield
>  wrote:
> >
> > +1 (non-binding). I used the staged binaries with Scala 2.13. I tried
> the new group coordinator
> > and consumer group protocol which is included with the Early Access
> release of KIP-848.
> > Also verified the availability of the new APIs. All working as expected.
> >
> > Thanks,
> > Andrew
> >
> > > On 15 Feb 2024, at 05:07, Paolo Patierno 
> wrote:
> > >
> > > +1 (non-binding). I used the staged binaries with Scala 2.13 and mostly
> > > focused on the ZooKeeper to KRaft migration with multiple tests.
> Everything
> > > works fine.
> > >
> > > Thanks
> > > Paolo
> > >
> > > On Mon, 12 Feb 2024, 22:06 Jakub Scholz,  wrote:
> > >
> > >> +1 (non-binding). I used the staged binaries with Scala 2.13 and the
> staged
> > >> Maven artifacts to run my tests. All seems to work fine. Thanks.
> > >>
> > >> Jakub
> > >>
> > >> On Fri, Feb 9, 2024 at 4:20 PM Stanislav Kozlovski
> > >>  wrote:
> > >>
> > >>> Hello Kafka users, developers and client-developers,
> > >>>
> > >>> This is the second candidate we are considering for release of Apache
> > >> Kafka
> > >>> 3.7.0.
> > >>>
> > >>> Major changes include:
> > >>> - Early Access to KIP-848 - the next generation of the consumer
> rebalance
> > >>> protocol
> > >>> - Early Access to KIP-858: Adding JBOD support to KRaft
> > >>> - KIP-714: Observability into Client metrics via a standardized
> interface
> > >>>
> > >>> Release notes for the 3.7.0 release:
> > >>>
> > >>>
> > >>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/RELEASE_NOTES.html
> > >>>
> > >>> *** Please download, test and vote by Thursday, February 15th, 9AM
> PST
> > >> ***
> > >>>
> > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>> https://kafka.apache.org/KEYS
> > >>>
> > >>> * Release artifacts to be voted upon (source and binary):
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/
> > >>>
> > >>> * Docker release artifact to be voted upon:
> > >>> apache/kafka:3.7.0-rc4
> > >>>
> > >>> * Maven artifacts to be voted upon:
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>
> > >>> * Javadoc:
> > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/javadoc/
> > >>>
> > >>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > >>> https://github.com/apache/kafka/releases/tag/3.7.0-rc4
> > >>>
> > >>> * Documentation:
> > >>> https://kafka.apache.org/37/documentation.html
> > >>>
> > >>> * Protocol:
> > >>> https://kafka.apache.org/37/protocol.html
> > >>>
> > >>> * Successful Jenkins builds for the 3.7 branch:
> > >>>
> > >>> Unit/integration tests: I am in the process of running and analyzing
> > >> these.
> > >>> System tests: I am in the process of running these.
> > >>>
> > >>> Expect a follow-up over the weekend
> > >>>
> > >>> * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > >>> Docker Build Test Pipeline:
> > >>> https://github.com/apache/kafka/actions/runs/7845614846
> > >>>
> > >>> /**
> > >>>
> > >>> Best,
> > >>> Stanislav
> > >>>
> > >>
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2650

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 466570 lines...]
[2024-02-16T12:42:58.727Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2024-02-16T12:42:58.727Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2024-02-16T12:42:58.727Z] > Task :connect:json:publishToMavenLocal
[2024-02-16T12:42:58.727Z] > Task :storage:storage-api:compileTestJava
[2024-02-16T12:42:58.727Z] > Task :storage:storage-api:testClasses
[2024-02-16T12:42:58.727Z] > Task :server:compileTestJava
[2024-02-16T12:42:58.727Z] > Task :server:testClasses
[2024-02-16T12:42:58.727Z] > Task :server-common:compileTestJava
[2024-02-16T12:42:58.727Z] > Task :server-common:testClasses
[2024-02-16T12:43:01.345Z] > Task :raft:compileTestJava
[2024-02-16T12:43:01.345Z] > Task :raft:testClasses
[2024-02-16T12:43:04.114Z] > Task :core:compileScala
[2024-02-16T12:43:05.732Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2024-02-16T12:43:05.732Z] > Task :group-coordinator:compileTestJava
[2024-02-16T12:43:05.732Z] > Task :group-coordinator:testClasses
[2024-02-16T12:43:05.732Z] 
[2024-02-16T12:43:05.732Z] > Task :clients:javadoc
[2024-02-16T12:43:05.732Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API";>KIP-554:
 Add Broker-side SCRAM Config API
[2024-02-16T12:43:05.732Z] 
[2024-02-16T12:43:05.732Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2024-02-16T12:43:05.732Z]  The type field in both files must match and must 
not change. The type field
[2024-02-16T12:43:05.732Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2024-02-16T12:43:05.732Z]  UserScramCredentialRecord. Do not change the type 
field."
[2024-02-16T12:43:05.732Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2024-02-16T12:43:06.928Z] 2 warnings
[2024-02-16T12:43:06.928Z] 
[2024-02-16T12:43:06.928Z] > Task :clients:javadocJar
[2024-02-16T12:43:08.320Z] > Task :clients:srcJar
[2024-02-16T12:43:08.320Z] > Task :clients:testJar
[2024-02-16T12:43:09.932Z] > Task :clients:testSrcJar
[2024-02-16T12:43:09.933Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2024-02-16T12:43:09.933Z] > Task :clients:publishToMavenLocal
[2024-02-16T12:43:09.933Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2024-02-16T12:43:09.933Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2024-02-16T12:43:09.933Z] > Task :connect:api:testClasses UP-TO-DATE
[2024-02-16T12:43:09.933Z] > Task :connect:api:testJar
[2024-02-16T12:43:09.933Z] > Task :connect:api:testSrcJar
[2024-02-16T12:43:09.933Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2024-02-16T12:43:09.933Z] > Task :connect:api:publishToMavenLocal
[2024-02-16T12:43:09.933Z] > Task :metadata:compileTestJava
[2024-02-16T12:43:09.933Z] > Task :metadata:testClasses
[2024-02-16T12:43:15.316Z] > Task :streams:javadoc
[2024-02-16T12:43:15.316Z] > Task :streams:javadocJar
[2024-02-16T12:43:15.316Z] > Task :streams:srcJar
[2024-02-16T12:43:15.316Z] > Task :streams:processTestResources UP-TO-DATE
[2024-02-16T12:44:01.724Z] > Task :core:classes
[2024-02-16T12:44:01.724Z] > Task :core:compileTestJava NO-SOURCE
[2024-02-16T12:44:29.810Z] > Task :core:compileTestScala
[2024-02-16T12:45:31.010Z] > Task :core:testClasses
[2024-02-16T12:46:01.273Z] > Task :streams:compileTestJava
[2024-02-16T12:47:34.322Z] > Task :streams:testClasses
[2024-02-16T12:47:34.322Z] > Task :streams:testJar
[2024-02-16T12:47:34.322Z] > Task :streams:testSrcJar
[2024-02-16T12:47:34.322Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2024-02-16T12:47:34.322Z] > Task :streams:publishToMavenLocal
[2024-02-16T12:47:34.322Z] 
[2024-02-16T12:47:34.322Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2024-02-16T12:47:34.322Z] 
[2024-02-16T12:47:34.322Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2024-02-16T12:47:34.322Z] 
[2024-02-16T12:47:34.322Z] For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
[2024-02-16T12:47:34.322Z] 
[2024-02-16T12:47:34.322Z] BUILD SUCCESSFUL in 5m 22s
[2024-02-16T12:47:34.322Z] 96 actionable tasks: 41 executed, 55 up-to-date
[2024-02-16T12:47:34.322Z] 
[2024-02-16T12:47:34.322Z] Publishing build scan...
[2024-02-16T12:47:34.322Z] https://ge.apac

Re: [VOTE] 3.7.0 RC4

[Federico Valeri]

Hi Stanislav, it's +1 (non binding) for me.

- Built from source using Java 17 and Scala 2.13.
- Ran all unit and integration tests.
- Used the staged Maven artifacts to test some client applications
against ZK and KRaft based clusters.
- Tested the Cluster-wide dynamic log adjustment for Kafka Connect
with a 3 nodes cluster.
- Tried the new Docker image.
- Ran system tests (most failed tests are related to streams, see report below).

Thanks
Fede

---

### flaky

[INFO:2024-02-16 00:47:39,263]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'reassign_partitions_test.py', 'cls_name':
'ReassignPartitionsTest', 'method_name': 'test_reassign_partitions',
'injected_args': {'bounce_brokers': False, 'metadata_quorum':
'ISOLATED_KRAFT', 'reassign_from_offset_zero': False,
'use_new_coordinator': False}}

### failed (single tests)

[INFO:2024-02-15 23:18:38,332]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'downgrade_test.py', 'cls_name': 'TestDowngrade',
'method_name': 'test_upgrade_and_downgrade', 'injected_args':
{'compression_types': 'none', 'static_membership': True, 'version':
'2.8.2'}}

[INFO:2024-02-15 23:30:22,380]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'network_degrade_test.py', 'cls_name':
'NetworkDegradeTest', 'method_name': 'test_rate', 'injected_args':
{'device_name': 'eth0', 'latency_ms': 50, 'rate_limit_kbit': 100,
'task_name': 'rate-1000-latency-50'}}

[INFO:2024-02-16 05:44:00,934]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name': 'test_version_probing_upgrade',
'injected_args': None}

# failed (14 tests with increasing from_version)

[INFO:2024-02-16 05:14:15,576]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_application_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name': 'test_app_upgrade',
'injected_args': {'bounce_type': 'full', 'from_version': '2.2.2',
'to_version': '3.7.0-SNAPSHOT'}}

# failed (18 tests with increasing from_version)

[INFO:2024-02-16 05:37:08,823]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args':
{'from_version': '0.10.0.1', 'to_version': '3.7.0-SNAPSHOT'}}

# failed (4 tests with increasing from_version)

[INFO:2024-02-16 05:41:10,539]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name':
'test_upgrade_downgrade_state_updater', 'injected_args':
{'from_version': '3.2.3', 'to_version': '3.7.0-SNAPSHOT', 'upgrade':
False}}








On Thu, Feb 15, 2024 at 3:39 PM Andrew Schofield
 wrote:
>
> +1 (non-binding). I used the staged binaries with Scala 2.13. I tried the new 
> group coordinator
> and consumer group protocol which is included with the Early Access release 
> of KIP-848.
> Also verified the availability of the new APIs. All working as expected.
>
> Thanks,
> Andrew
>
> > On 15 Feb 2024, at 05:07, Paolo Patierno  wrote:
> >
> > +1 (non-binding). I used the staged binaries with Scala 2.13 and mostly
> > focused on the ZooKeeper to KRaft migration with multiple tests. Everything
> > works fine.
> >
> > Thanks
> > Paolo
> >
> > On Mon, 12 Feb 2024, 22:06 Jakub Scholz,  wrote:
> >
> >> +1 (non-binding). I used the staged binaries with Scala 2.13 and the staged
> >> Maven artifacts to run my tests. All seems to work fine. Thanks.
> >>
> >> Jakub
> >>
> >> On Fri, Feb 9, 2024 at 4:20 PM Stanislav Kozlovski
> >>  wrote:
> >>
> >>> Hello Kafka users, developers and client-developers,
> >>>
> >>> This is the second candidate we are considering for release of Apache
> >> Kafka
> >>> 3.7.0.
> >>>
> >>> Major changes include:
> >>> - Early Access to KIP-848 - the next generation of the consumer rebalance
> >>> protocol
> >>> - Early Access to KIP-858: Adding JBOD support to KRaft
> >>> - KIP-714: Observability into Client metrics via a standardized interface
> >>>
> >>> Release notes for the 3.7.0 release:
> >>>
> >>>
> >> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/RELEASE_NOTES.html
> >>>
> >>> *** Please download, test and vote by Thursday, February 15th, 9AM PST
> >> ***
> >>>
> >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>> https://kafka.apache.org/KEYS
> >>>
> >>> * Release artifacts to be voted upon (source and binary):
> >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/
> >>>
> >>> * Docker release artifact to be voted upon:
> >>> apache/kafka:3.7.0-rc4
> >>>
> >>> * Maven artifacts to be voted upon:
> >>> https://repository.apache.org/

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Thanks for the feedback Jason. See my comments below.

On Mon, Feb 5, 2024 at 3:11 PM Jason Gustafson
 wrote:
> 1. When adding a voter, the KIP proposes to return a timeout error if the
> voter cannot catch up in time. It might be more useful to return a more
> specific error so that an operator can understand why the timeout occurred.
> Alternatively, perhaps we could keep the generic error but add an
> ErrorMessage field in the response to explain the issue. (At the same time,
> perhaps we can add ErrorMessage to the other new RPCs. I think this is
> becoming more or less standard in Kafka RPCs.)

I added the ErrorMessage field to the 3 admin client RPCs: AddVoter,
RemoveVoter and DescribeQuorrum. I didn't add it to the other,
internal, RPCs since I didn't think it would be useful in practice.
Let me know if you think we should add that field to the "internal"
RPCs.

> 2. Say that the voters are A, B, and C, and that C is offline. What error
> would be returned from RemoveVoters if we try to remove B before C is back
> online?

In this example, the case below would never succeed:
"5. Wait for the VotersRecord to commit using the majority of new
configuration."

I am planning to return a REQUEST_TIMED_OUT error for this case. A
future RemoveVoter request for the same replica id and uuid would also
return a REQUEST_TIMED_OUT if the VotersRecord
stays uncommitted because of:
"1. Wait until there are no uncommitted VotersRecord. Note that the
implementation may just return a REQUEST_TIMED_OUT error if there are
pending operations."

Note that this is after the leader (A or B) wrote VotersRecord that
removed B, so the cluster is unavailable. If C doesn't come back
online, eventually "check quorum" will fail and the leader (A or B)
will resign.

This is also an example of the HWM decreasing because of removing the voter.

We could add a heuristic to lower the impact of this example but I am
not sure if it is worth the implementation cost at the moment. For
example, the leader can check the LEO and lastFetchTimestamp of the
replicas (C in this example). What are these replicas in a bigger
voter set?

> 3. There is a note about a voter accepting a Vote request from a voter that
> is not in their voter set. I guess the same comment applies to
> BeginQuorumEpoch?

Yes. The other extension to the BeginQuorumEpoch request is that the
leader will resend the request (even if it was previously
acknowledged), if the voters fails to send a Fetch request within the
"check quorum" timeout. I had to change this because BeginQuorumEpoch
also includes the leader's endpoint information. Yet the receiving
replica doesn't persist the leader endpoint information into the
quorum-state file like it does for leader id, uuid and epoch.

> In other words, a voter should accept a new leader even
> if it is not in its local voter set.

Yes. There are two cases for this. 1. A voter has become leader before
another replica (observer or voter) has replicated the voter set. 2.
The active leader is the replica that is getting removed from the
voter set.

> The note in the reference says: "Thus,
> servers process incoming RPC requests without consulting their current
> configurations." On the other hand, when it comes to election, voters will
> become candidates based on the latest voter set from the log (regardless
> whether it is committed), and they will seek out votes only from the same
> voter set. Is that about right?

Yes. That's correct. When transitioning to prospective (pre-vote KIP),
when transitioning to candidate, or when deciding to send RPCs like
Vote (and BeginQuorumEpoch); the replica will do so based on the
latest voter set (VotersRecord) in the log.

Thanks
--
-José

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #95

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2651

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 347834 lines...]
[2024-02-16T16:57:04.886Z] 
[2024-02-16T16:57:04.886Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2024-02-16T16:57:04.886Z]  The type field in both files must match and must 
not change. The type field
[2024-02-16T16:57:04.886Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2024-02-16T16:57:04.886Z]  UserScramCredentialRecord. Do not change the type 
field."
[2024-02-16T16:57:04.886Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2024-02-16T16:57:04.886Z] 2 warnings
[2024-02-16T16:57:06.285Z] 
[2024-02-16T16:57:06.285Z] > Task :clients:javadocJar
[2024-02-16T16:57:06.285Z] > Task :clients:srcJar
[2024-02-16T16:57:07.497Z] > Task :clients:testJar
[2024-02-16T16:57:07.497Z] > Task :metadata:compileTestJava
[2024-02-16T16:57:07.497Z] > Task :metadata:testClasses
[2024-02-16T16:57:07.497Z] > Task :clients:testSrcJar
[2024-02-16T16:57:08.890Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2024-02-16T16:57:08.890Z] > Task :clients:publishToMavenLocal
[2024-02-16T16:57:08.890Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2024-02-16T16:57:08.890Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2024-02-16T16:57:08.890Z] > Task :connect:api:testClasses UP-TO-DATE
[2024-02-16T16:57:08.890Z] > Task :connect:api:testJar
[2024-02-16T16:57:08.890Z] > Task :connect:api:testSrcJar
[2024-02-16T16:57:08.890Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2024-02-16T16:57:08.890Z] > Task :connect:api:publishToMavenLocal
[2024-02-16T16:57:10.243Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2024-02-16T16:57:10.245Z] Skipping Kafka Streams archetype test for Java 21
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2024-02-16T16:57:14.634Z] > Task :streams:javadoc
[2024-02-16T16:57:14.634Z] > Task :streams:javadocJar
[2024-02-16T16:57:14.634Z] > Task :streams:srcJar
[2024-02-16T16:57:14.634Z] > Task :streams:processTestResources UP-TO-DATE
[2024-02-16T16:58:09.162Z] > Task :core:classes
[2024-02-16T16:58:09.162Z] > Task :core:compileTestJava NO-SOURCE
[2024-02-16T16:58:35.101Z] > Task :core:compileTestScala
[2024-02-16T16:59:33.769Z] > Task :core:testClasses
[2024-02-16T17:00:03.302Z] > Task :streams:compileTestJava
[2024-02-16T17:01:46.758Z] > Task :streams:testClasses
[2024-02-16T17:01:46.758Z] > Task :streams:testJar
[2024-02-16T17:01:46.758Z] > Task :streams:testSrcJar
[2024-02-16T17:01:46.758Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2024-02-16T17:01:46.758Z] > Task :streams:publishToMavenLocal
[2024-02-16T17:01:46.758Z] 
[2024-02-16T17:01:46.758Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2024-02-16T17:01:46.758Z] 
[2024-02-16T17:01:46.758Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2024-02-16T17:01:46.758Z] 
[2024-02-16T17:01:46.758Z] For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
[2024-02-16T17:01:46.758Z] 
[2024-02-16T17:01:46.758Z] BUILD SUCCESSFUL in 5m 40s
[2024-02-16T17:01:46.758Z] 96 actionable tasks: 41 executed, 55 up-to-date
[2024-02-16T17:01:46.758Z] 
[2024-02-16T17:01:46.758Z] Publishing build scan...
[2024-02-16T17:01:46.758Z] https://ge.apache.org/s/i3vleafiynvwe
[2024-02-16T17:01:46.758Z] 
[Pipeline] sh
[2024-02-16T17:01:50.810Z] + grep ^version= gradle.properties
[2024-02-16T17:01:50.810Z] + cut -d= -f 2
[Pipeline] dir
[2024-02-16T17:01:51.991Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2024-02-16T17:01:54.983Z] + mvn clean install -Dgpg.skip
[2024-02-16T17:01:56.523Z] [INFO] Scanning for projects...
[2024-02-16T17:01:56.523Z] [INFO] 

[2024-02-16T17:01:56.523Z] [INFO] Reactor Build Order:
[2024-02-16T17:01:56.523Z] [INFO] 
[2024-02-16T17:01:56.523Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2024-02-16T17:01:56.523Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2024-02-16T17:01:56.523Z] [INFO] 
[2024-02-16T17:01:56.523Z] [INFO] < 
org.apache.kafka:streams-quickstart >-

[jira] [Created] (KAFKA-16266) Introduce TransactionLastUpdateTimeMs tagged field to DescribeTransactionsResponse

[Yang Yu (Jira)]

Yang Yu created KAFKA-16266:
---

 Summary:  Introduce TransactionLastUpdateTimeMs tagged field to 
DescribeTransactionsResponse
 Key: KAFKA-16266
 URL: https://issues.apache.org/jira/browse/KAFKA-16266
 Project: Kafka
  Issue Type: Sub-task
Reporter: Yang Yu


Introduce TransactionLastUpdateTimeMs tagged field to 
DescribeTransactionsResponse. Make broker side changes to send this bit of 
information. Also, make changes to `kafka-transactions.sh --describe` tooling 
to display this new piece of information to the output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16265) Add durationFilter to ListTransactionsRequest

[Yang Yu (Jira)]

Yang Yu created KAFKA-16265:
---

 Summary: Add durationFilter to ListTransactionsRequest
 Key: KAFKA-16265
 URL: https://issues.apache.org/jira/browse/KAFKA-16265
 Project: Kafka
  Issue Type: Sub-task
Reporter: Yang Yu


* Add durationFilter field to ListTransactionsRequest, make corresponding 
server side changes
 * Make appropriate version bumps for ListTransactionsRequest and 
ListTransationsResponse
 * Add durationFilter option to kafka-transactions.sh --list



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jason and Jun,

I have updated the KIP to incorporate your feedback. Some of the changes are:

1. Removed security protocol from the listener definition. The local
replica is going to assume that the remote replica has the same
listener name to security group mapping.
2. Reverted most of the changes to QuorumStateData. In summary the KIP
only adds VotedUuid to this persistent data.
3. Explicitly mentioned in a few places that downgrades from
kraft.version 1 to kraft.version 0 is not supported.
4. Added an ErrorMessage field to all of the admin client RPCs used by
KRaft: AddVoter, RemoveVoter and DescribeQuorum.
5. Quoted the KIP-996: Pre-vote with respect to fencing voters that
were removed from the voter set affecting election.
6. Updated the admin client section to mention that AddVoter,
RemoveVoter and DescribeQuorum will be sent directly to the active
controller when using a controller bootstrap server.
7. Added why we rejected using controller mechanisms to the rejected
alternative section. This came up a few times so it is worth
documenting why it was rejected.

Thanks,
-- 
-José

Re: [VOTE] 3.7.0 RC4

[Stanislav Kozlovski]

Thanks all for the help in verifying.

I have updated
https://gist.github.com/stanislavkozlovski/820976fc7bfb5f4dcdf9742fd96a9982
with the system tests.
There were two builds ran, and across those - the following tests failed
two times in a row:


*kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest#test_replica_lagsArguments:{
"metadata_quorum": "ZK"}*Fails with the same error of
*`TimeoutError('Timed out waiting to reach non-zero number of replica
lags.')`*
I have scheduled a re-run of this specific test here ->
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6057

*kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
"compression_types": [ "zstd" ], "from_kafka_version": "2.4.1",
"to_message_format_version": null}*
Fails with the same error of
*`TimeoutError('Producer failed to produce messages for 20s.')`*
*kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
"compression_types": [ "lz4" ], "from_kafka_version": "3.0.2",
"to_message_format_version": null}*
Fails with the same error of *`TimeoutError('Producer failed to produce
messages for 20s.')`*

I have scheduled a re-run of this test here ->
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6058/

On Fri, Feb 16, 2024 at 12:15 PM Vedarth Sharma 
wrote:

> Hey Stanislav,
>
> Thanks for the release candidate.
>
> +1 (non-binding)
>
> I tested and verified the docker image artifact apache/kafka:3.7.0-rc4:-
> - verified create topic, produce messages and consume messages flow when
> running the docker image with
> - default configs
> - configs provided via env variables
> - configs provided via file input
> - verified the html documentation for docker image.
> - ran the example docker compose files successfully.
>
> All looks good for the docker image artifact!
>
> Thanks and regards,
> Vedarth
>
>
> On Thu, Feb 15, 2024 at 10:58 PM Mickael Maison 
> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for running the release.
> >
> > I did the following testing:
> > - verified the check sums and signatures
> > - ran ZooKeeper and KRaft quickstarts with Scala 2.13 binaries
> > - ran a successful migration from ZooKeeper to KRaft
> >
> > We seem to be missing the upgrade notes for 3.7.0 in the docs. See
> > https://kafka.apache.org/37/documentation.html#upgrade that still
> > points to 3.6.0
> > Before voting I'd like to see results from the system tests too.
> >
> > Thanks,
> > Mickael
> >
> > On Thu, Feb 15, 2024 at 6:06 PM Andrew Schofield
> >  wrote:
> > >
> > > +1 (non-binding). I used the staged binaries with Scala 2.13. I tried
> > the new group coordinator
> > > and consumer group protocol which is included with the Early Access
> > release of KIP-848.
> > > Also verified the availability of the new APIs. All working as
> expected.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 15 Feb 2024, at 05:07, Paolo Patierno 
> > wrote:
> > > >
> > > > +1 (non-binding). I used the staged binaries with Scala 2.13 and
> mostly
> > > > focused on the ZooKeeper to KRaft migration with multiple tests.
> > Everything
> > > > works fine.
> > > >
> > > > Thanks
> > > > Paolo
> > > >
> > > > On Mon, 12 Feb 2024, 22:06 Jakub Scholz,  wrote:
> > > >
> > > >> +1 (non-binding). I used the staged binaries with Scala 2.13 and the
> > staged
> > > >> Maven artifacts to run my tests. All seems to work fine. Thanks.
> > > >>
> > > >> Jakub
> > > >>
> > > >> On Fri, Feb 9, 2024 at 4:20 PM Stanislav Kozlovski
> > > >>  wrote:
> > > >>
> > > >>> Hello Kafka users, developers and client-developers,
> > > >>>
> > > >>> This is the second candidate we are considering for release of
> Apache
> > > >> Kafka
> > > >>> 3.7.0.
> > > >>>
> > > >>> Major changes include:
> > > >>> - Early Access to KIP-848 - the next generation of the consumer
> > rebalance
> > > >>> protocol
> > > >>> - Early Access to KIP-858: Adding JBOD support to KRaft
> > > >>> - KIP-714: Observability into Client metrics via a standardized
> > interface
> > > >>>
> > > >>> Release notes for the 3.7.0 release:
> > > >>>
> > > >>>
> > > >>
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/RELEASE_NOTES.html
> > > >>>
> > > >>> *** Please download, test and vote by Thursday, February 15th, 9AM
> > PST
> > > >> ***
> > > >>>
> > > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >>> https://kafka.apache.org/KEYS
> > > >>>
> > > >>> * Release artifacts to be voted upon (source and binary):
> > > >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/
> > > >>>
> > > >>> * Docker release artifact to be voted upon:
> > > >>> apache/kafka:3.7.0-rc4
> > > >>>
> > > >>> * Maven artifacts to be voted upon:
> > > >>>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>>
> > > >>> * Javadoc:
> > > >>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/javadoc/
> > > >>>
> > > >>> * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #96

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2652

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Jun Rao]

Thanks for the reply, Jose.

10. "The controller state machine will instead push the brokers'
kraft.version information to the KRaft client". If we do that, why do we
need KRaftVersionRecord?

15. Hmm, I thought controller.listener.names already provides the listener
name. It's a list so that we could support changing security protocols.

17.1 "1. They are implemented at two different layers of the protocol. The
Kafka controller is an application of the KRaft protocol. I wanted to
keep this distinction in this design. The controller API is going to
forward ControllerRegistrationRequest to the QuorumController and it
is going to forward UpdateVoter to the KafkaRaftClient."
Hmm, but the controller already pushes the brokers' kraft.version
information to the KRaft client.

"2. If the voter getting updated is not part of the
voter set the leader will reject the update."
Would it be simpler to just relax that? The KIP already relaxed some of the
checks during the vote.

"3. The other semantic difference is that in KRaft, the voter set
(which includes the replica ids and the endpoint) is based on
uncommitted data. While the controller state machine only sees
committed data."
Hmm, does that prevent the supported versions from being propagated through
ControllerRegistrationRequest?

17.2 It seems the main purpose of UpdateVoter is to change the endpoints.
Currently, this is handled through ControllerRegistrationRequest. Are we
going to have two ways of changing endpoints?

18.2 I thought controller.listener.names already provides the listener name.

18.3 When the new voter starts, does it send the
ControllerRegistrationRequest? If so, it already includes the supported
kraft.version.

19. Hmm, LeaderChangeMessage only has one voter set, not per partition. Is
that a problem in the future?

20.1 The following are the changes to EndQuorumEpochRequest.
PreferredSuccessors is still there.
+{ "name": "PreferredSuccessors", "type": "[]int32", "versions":
"0",
+  "about": "A sorted list of preferred successors to start the
election" },
+{ "name": "PreferredCandidates", "type": "[]ReplicaInfo",
"versions": "1+",
+  "about": "A sorted list of preferred successors to start the
election", "fields": [

20.2 "To all of these responses I have added endpoint information now that
the voter set and endpoints are dynamic."
This seems redundant. Shouldn't we just use 1 RPC (e.g.
ControllerRegistrationRequest) to handle endPoint changes?

22. Could we at least name the additional checkpoint file more consistently
with the existing one? e.g. kraft-bootstrap.checkpoint?

26. Since MinSupportedVersion and MaxSupportedVersion are stored in
RegisterBrokerRecord, it's redundant to store them again in VotersRecord,
right?

30. Currently, metadata.version controls all inter-broker RPCs, records and
some cluster-wide functionalities. What does kraft.version control? Is
there any RPC/record controlled by both features?

Thanks,

Jun

On Wed, Feb 14, 2024 at 11:12 AM José Armando García Sancio
 wrote:

> Hi Jun,
>
> Thanks for the feedback. Excuse the delay, it took me a while to
> properly address your detailed feedback. See my comments below.
>
> I am going to update the KIP as outlined in this email. I will send
> another email when I have made all of the changes.
>
> On Fri, Feb 2, 2024 at 10:54 AM Jun Rao  wrote:
> > 10. kraft.version: Functionality wise, this seems very similar to
> > metadata.version, which is to make sure that all brokers/controllers are
> on
> > a supported version before enabling a new feature. Could you explain why
> we
> > need a new one instead of just relying on metadata.version?
>
> Yes, they are trying to solve similar problems but they are doing them
> at a different abstraction layer. "metadata.version" uses
> FeatureLevelRecord which is a "data" record. I am trying to design the
> KRaft protocol (KafkaRaftClient implementation) to not assume a
> specific application (metadata and the controller). To support
> multiple applications possibly using KRaft means that we need to use
> control records to define KRaft's behavior. Instead of using
> FeatureLevelRecord which is a cluster metadata record, this KIP uses
> KRaftVersionRecord which will be a control record.
>
> The reason why we need to consider observers (brokers) when
> determining the kraft.version is because of snapshot (compaction).
> Since snapshots need to include all logical records in the log
> segments and observers (brokers) maintain the log on disk, this means
> that they need to be able to decode and encode both KRaftVersionRecord
> and VotersRecord before the leader is allowed to write them.
>
> > 11. Both the quorum-state file and controller.quorum.bootstrap.servers
> > contain endpoints. Which one takes precedence?
>
> I updated the KIP (section "Reference explanation / Endpoints
> information") after Jack's email. KRaft is going to keep two sets of
> endpoints.
>
> 1. Voters set. This set of endpoints is used 

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Artem Livshits]

Hi Rowland,

> I am not sure what you mean by guarantee,

A guarantee would be an elimination of complexity or a condition.  E.g. if
adding an explicit prepare RPC eliminated in-doubt transactions, or
eliminated a significant complexity in implementation.

> 1. Transactions that haven’t reached “prepared” state can be aborted via
timeout.

The argument is that it doesn't eliminate any conditions, it merely reduces
a subset of circumstances for the conditions to happen, but the conditions
still happen and must be handled.  The operator still needs to set up
monitoring for run-away transactions, there still needs to be an
"out-of-band" channel to resolve run-away transactions (i.e. the operation
would need a way that's not a part of the 2PC protocol to reconcile with
the application owner), there still needs to be tooling for resolving
run-away transactions.

On the downside, an explicit prepare RPC would have a performance hit on
the happy path in every single transaction.

-Artem

On Tue, Feb 6, 2024 at 7:35 PM Rowland Smith  wrote:

> Hi Artem,
>
> I am not sure what you mean by guarantee, but I am referring to a better
> operational experience. You mentioned this as the first benefit of an
> explicit "prepare" RPC in the KIP.
>
>
> 1. Transactions that haven’t reached “prepared” state can be aborted via
> timeout.
>
> However, in explaining why an explicit "prepare" RPC was not included in
> the design, you make no further mention of this benefit. So what I am
> saying is this benefit is quite significant operationally. Many client
> application failures may occur before the transaction reaches the prepared
> state, and the ability to automatically abort those transactions and
> unblock affected partitions without administrative intervention or fast
> restart of the client would be a worthwhile benefit. An explicit "prepare"
> RPC will also be needed by the XA implementation, so I would like to see it
> implemented for that reason. Otherwise, I will need to add this work to my
> KIP.
>
> - Rowland
>
> On Mon, Feb 5, 2024 at 9:35 PM Artem Livshits
>  wrote:
>
> > Hi Rowland,
> >
> > Thank you for your reply.  I think I understand what you're saying and
> just
> > tried to provide a quick summary.  The
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC#KIP939:SupportParticipationin2PC-Explicit%E2%80%9Cprepare%E2%80%9DRPC
> > actually goes into the details on what would be the benefits of adding an
> > explicit prepare RPC and why those won't really add any advantages such
> as
> > elimination the needs for monitoring, tooling or providing additional
> > guarantees.  Let me know if you think of a guarantee that prepare RPC
> would
> > provide.
> >
> > -Artem
> >
> > On Mon, Feb 5, 2024 at 6:22 PM Rowland Smith  wrote:
> >
> > > Hi Artem,
> > >
> > > I don't think that you understand what I am saying. In any transaction,
> > > there is work done before the call to prepareTranscation() and work
> done
> > > afterwards. Any work performed before the call to prepareTransaction()
> > can
> > > be aborted after a relatively short timeout if the client fails. It is
> > only
> > > after the prepareTransaction() call that a transaction becomes in-doubt
> > and
> > > must be remembered for a much longer period of time to allow the client
> > to
> > > recover and make the decision to either commit or abort. A considerable
> > > amount of time might be spent before prepareTransaction() is called,
> and
> > if
> > > the client fails in this period, relatively quick transaction abort
> would
> > > unblock any partitions and make the system fully available. So a
> prepare
> > > RPC would reduce the window where a client failure results in
> potentially
> > > long-lived blocking transactions.
> > >
> > > Here is the proposed sequence from the KIP with 2 added steps (4 and
> 5):
> > >
> > >
> > >1. Begin database transaction
> > >2. Begin Kafka transaction
> > >3. Produce data to Kafka
> > >4. Make updates to the database
> > >5. Repeat steps 3 and 4 as many times as necessary based on
> > application
> > >needs.
> > >6. Prepare Kafka transaction [currently implicit operation,
> expressed
> > as
> > >flush]
> > >7. Write produced data to the database
> > >8. Write offsets of produced data to the database
> > >9. Commit database transaction
> > >10. Commit Kafka transaction
> > >
> > >
> > > If the client application crashes before step 6, it is safe to abort
> the
> > > Kafka transaction after a relatively short timeout.
> > >
> > > I fully agree with a layered approach. However, the XA layer is going
> to
> > > require certain capabilities from the layer below it, and one of those
> > > capabilities is to be able to identify and report prepared transactions
> > > during recovery.
> > >
> > > - Rowland
> > >
> > > On Mon, Feb 5, 2024 at 12:46 AM Artem Livshits
> > >  wrote:
> > >
> > > > Hi Rowland,
> > > >
> > > > Thank you for

Re: [Discuss] KIP-1019: Expose method to determine Metric Measurability

[Matthias J. Sax]

Thanks for the KIP. Seems there is not much we need to discuss about it. 
Feel free to start a VOTE.


-Matthias

On 2/15/24 7:24 AM, Manikumar wrote:

LGTM, Thanks for the KIP.

On Thu, Feb 15, 2024 at 8:50 PM Doğuşcan Namal 
wrote:


LGTM thanks for the KIP.

+1(non-binding)

On Wed, 14 Feb 2024 at 15:22, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:


Hi Apoorv,
Thanks for the KIP. Looks like a useful change to tidy up the metrics

code.


Thanks,
Andrew


On 14 Feb 2024, at 14:55, Apoorv Mittal 

wrote:


Hi,
I would like to start discussion of a small KIP which fills a gap in
determining Kafka Metric measurability.

KIP-1019: Expose method to determine Metric Measurability
<



https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability



Regards,
Apoorv Mittal

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2653

[Apache Jenkins Server]

See