[jira] [Resolved] (KAFKA-13585) Fix `kafka.server.ReplicaManagerTest.testReplicaAlterLogDirsWithAndWithoutIds` flaky test

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13585.
-
Fix Version/s: 3.2.0
   3.1.1
 Reviewer: Jason Gustafson
   Resolution: Fixed

> Fix 
> `kafka.server.ReplicaManagerTest.testReplicaAlterLogDirsWithAndWithoutIds` 
> flaky test
> -
>
> Key: KAFKA-13585
> URL: https://issues.apache.org/jira/browse/KAFKA-13585
> Project: Kafka
>  Issue Type: Improvement
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Minor
> Fix For: 3.2.0, 3.1.1
>
>
> {noformat}
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:35)
>   at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:162)
>   at 
> app//kafka.server.ReplicaManagerTest.assertFetcherHasTopicId(ReplicaManagerTest.scala:3502)
>   at 
> app//kafka.server.ReplicaManagerTest.testReplicaAlterLogDirsWithAndWithoutIds(ReplicaManagerTest.scala:3572)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment

[Jason Gustafson]

Hey David,

Yeah, there might not be a simple option to address the scenario I
described. Other than a broker-side solution, another way we could fix it
is by adding additional metadata to the assignment. I do agree that it
might not be worth the additional complexity.  At least we should probably
update the KIP to describe the limitation.

@Hector In regard to the new consumer protocol, it's only just beyond
wishful thinking at this point. We are hoping to share some ideas with the
community in the near future though.

Best,
Jason

On Mon, Jan 24, 2022 at 3:06 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Hi David,
>
> Is the idea here to skip calling performAssignment(...) on the
> AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean
> parameter to the performAssignment(...) method? The reason I ask is because
> I raised KIP-795 a few weeks back, which aims to add a public API for
> AbstractCoordinator, which might change (or not) with this KIP.
>
> I see you also mentioned there's some discussions regarding a new consumer
> protocol. Is this being discussed somewhere else? I'm curious to know how
> would it work with other systems (like Kafka Connect or Schema Registry)
> that rely on the rebalance protocol to handle resource assignments.
>
> Apologies in advance if these questions are off-topic for the discussion
> at hand.
>
> Regards,
> Hector
>
> From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let the
> leader skip assignment
>
> Hey Jason,
>
> Thanks for your comments.
>
> Regarding your first point. Yes, you have it right. Let me complement
> the KIP to be clearer.
>
> Regarding your second point. That is right. New partitions would not
> be detected while the leader is down. It is definitely not ideal but that
> seems acceptable to me, at least as a first step. Adding partitions to
> a topic is an infrequent event so the likelihood of having it while the
> leader is down is rather low but that could happen.
>
> The only way to not suffer from this would be to monitor the metadata
> changes on the broker side. This implies that we would parse both the
> subscriptions and the assignments in order to have the full list of topics.
> I am not sure that it is worth doing it at the moment given that we are
> thinking about a new consumer protocol. What do you think?
>
> I suppose that we would need both in the long term as the current protocol
> is a bit weird at the moment so we need to fix it anyway. We could
> use this KIP to fix the protocol and do a subsequent KIP in the future for
> the server side monitoring if we need it.
>
> Best,
> David
>
> On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson
>  wrote:
> >
> > Hey David,
> >
> > Thanks for the proposal. This was a tricky bug and I think your approach
> is
> > probably the best way forward.
> >
> > It would be helpful to add a little more detail to the proposal. When the
> > coordinator detects that the static leader is returning, it will set
> > `skipAssignment` to true in the `JoinGroup` response. I believe the
> intent
> > is to return all member subscriptions in this response so that the leader
> > can monitor all topics subscribed in the group (which might be different
> > from the consumer's own subscription). The leader will then send an empty
> > `SyncGroup` request to collect its own assignment. Do I have that right?
> >
> > I think there might still be an edge case in this proposal (assuming I've
> > understood it correctly). In between the time that the leader shuts down
> > and is restarted, it is possible that new partitions are added to one of
> > the subscribed topics. The returning leader would not know about it
> > because it has no way to collect the full assignment. Do you think this
> is
> > a problem?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Jan 19, 2022 at 7:27 AM David Jacot  wrote:
> >
> > > Hi folks,
> > >
> > > I'd like to start a discussion for KIP-814: Static membership protocol
> > > should let the
> > > leader skip assignment. This is a small extension to the static
> > > membership protocol
> > > to address KAFKA-13435.
> > >
> > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw.
> > >
> > > Please let me know what you think.
> > >
> > > Best,
> > > David
> > >
>
>
>

Re: [VOTE] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Colin McCabe]

How about this:

We create a configuration key called early.start.listeners which contains a 
list of listener names. If this is not specified, its value defaults to just 
the controller listeners. Optionally, other listeners can be added too.

If super.users contains any user names, early start listeners will start 
immediately. In the beginning they only authorize users that are in 
super.users. All other listeners receive a new error code, 
AUTHORIZER_NOT_READY_ERROR. If super.users does not contain any user names, 
then early start listeners will not be treated differently than other listeners.

This will allow the controller listeners to get started immediately if the 
broker user is in super.users, which will speed up startup. It also will be 
useful for breaking chicken/egg cycles like needing to pull the SCRAM metadata 
to authorize pulling the SCRAM metadata.

There are still a few use cases where super.users won't be required, but it may 
be useful in many cases to have this early start functionality.

Leaving aside the preceding discussion, do you agree with starting up all 
endpoints (including non-early start ones) once we load a metadata snapshot? 
How feasible would it be for us to get a callback from the Raft layer the first 
time we caught up to the last stable offset? (we only want the callback the 
first time, not any other time). (I think the metadata shell also would want 
something like this, at least as an option).

best,
Colin


On Tue, Jan 25, 2022, at 13:34, Jason Gustafson wrote:
> Hi Colin,
>
> Thanks for the writeup. I had one question about bootstrapping. For the
> brokers, I understand that listener startup is delayed until the Authorizer
> is ready. However, I was not very clear how this would work for the
> controller listeners. We may need them to startup before the metadata log
> is ready so that a quorum can be established (as noted in the KIP). This
> works fine if we assume that the controller principals are among
> `super.users`. For requests forwarded from brokers, on the other hand, we
> need to ensure the ACLs have been loaded properly before we begin
> authorizing. The problem is that we currently use the same listener for
> quorum requests and for forwarded requests. So my question is how does the
> Authorizer communicate to the controller when it is safe to begin
> authorizing different request types?
>
> There are a couple ways I can see this working. First, we could allow the
> user to configure the listener used for forwarded requests separately. That
> would work with the existing `Authorizer.start` API. Alternatively, perhaps
> we could modify `Authorizer.start` to work with something more granular
> than `EndPoint`. This would allow the controller to begin accepting
> requests from the other quorum members before it is ready to authorize
> forwarded requests from clients.  Then we would need some way to let
> brokers know when the controller is ready to accept these forwarded
> requests (e.g. through an error code in the `Envelope` response).
>
> What do you think?
>
> Thanks,
> Jason
>
>
>
>
>
>
> On Wed, Jan 12, 2022 at 12:57 PM David Arthur
>  wrote:
>
>> +1 binding, thanks Colin!
>>
>> On Mon, Dec 13, 2021 at 7:47 PM Colin McCabe  wrote:
>>
>> > Hi all,
>> >
>> > I'd like to start the vote on KIP-801: Implement an Authorizer that
>> stores
>> > metadata in __cluster_metadata
>> >
>> > The KIP is here: https://cwiki.apache.org/confluence/x/h5KqCw
>> >
>> > The original DISCUSS thread is here:
>> >
>> > https://lists.apache.org/thread/3d5o7h17ztjztjhblx4fln0wbbs1rmdq
>> >
>> > Please take a look and vote if you can.
>> >
>> > best,
>> > Colin
>> >
>>
>>
>> --
>> -David
>>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #632

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 509358 lines...]
[2022-01-25T22:21:23.687Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2022-01-25T22:21:23.687Z] 
[2022-01-25T22:21:23.687Z] > Task :streams:processMessages
[2022-01-25T22:21:23.687Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2022-01-25T22:21:23.687Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-01-25T22:21:23.687Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2022-01-25T22:21:24.721Z] 
[2022-01-25T22:21:24.721Z] > Task :streams:compileJava UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :streams:classes UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :core:compileTestScala UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :core:testClasses UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task :streams:jar UP-TO-DATE
[2022-01-25T22:21:24.721Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-01-25T22:21:28.854Z] > Task :connect:api:javadoc
[2022-01-25T22:21:28.854Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task :connect:api:jar UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-01-25T22:21:28.854Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task :connect:json:jar UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-01-25T22:21:28.854Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-01-25T22:21:28.854Z] > Task :connect:json:publishToMavenLocal
[2022-01-25T22:21:28.854Z] > Task :connect:api:javadocJar
[2022-01-25T22:21:28.854Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-01-25T22:21:28.854Z] > Task :connect:api:testJar
[2022-01-25T22:21:28.854Z] > Task :connect:api:testSrcJar
[2022-01-25T22:21:28.854Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-01-25T22:21:28.854Z] > Task :connect:api:publishToMavenLocal
[2022-01-25T22:21:31.951Z] 
[2022-01-25T22:21:31.951Z] > Task :streams:javadoc
[2022-01-25T22:21:31.951Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-01-25T22:21:31.951Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-01-25T22:21:31.951Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-01-25T22:21:31.951Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-01-25T22:21:31.951Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2022-01-25T22:21:32.984Z] 5 warnings
[2022-01-25T22:21:32.984Z] 
[2022-01-25T22:21:32.984Z] > Task :streams:javadocJar
[2022-01-25T22:21:32.984Z] > Task :streams:compileTestJava UP-TO-DATE
[2022-01-25T22:21:32.984Z] > Task :streams:processTestResources UP-TO-DATE
[2022-01-25T22:21:32.984Z] > Task :streams:testClasses UP-TO-DATE
[2022-01-25T22:21:34.015Z] > Task :streams:testJar
[2022-01-25T22:21:34.015Z] > Task :streams:testSrcJar
[2022-01-25T22:21:34.015Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-01-25T22:21:34.015Z] > Task :streams:publishToMavenLocal
[2022-01-25T22:21:35.048Z] > Task :clients:javadoc
[2022-01-25T22:21:36.083Z] > Task :clients:javadocJar
[2022-01-25T22:21:37.116Z] 
[2022-01-25T22:21:37.116Z] > Task :clients:srcJar
[2022-01-25T22:21:37.116Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the 

Re: [VOTE] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Jason Gustafson]

Hi Colin,

Thanks for the writeup. I had one question about bootstrapping. For the
brokers, I understand that listener startup is delayed until the Authorizer
is ready. However, I was not very clear how this would work for the
controller listeners. We may need them to startup before the metadata log
is ready so that a quorum can be established (as noted in the KIP). This
works fine if we assume that the controller principals are among
`super.users`. For requests forwarded from brokers, on the other hand, we
need to ensure the ACLs have been loaded properly before we begin
authorizing. The problem is that we currently use the same listener for
quorum requests and for forwarded requests. So my question is how does the
Authorizer communicate to the controller when it is safe to begin
authorizing different request types?

There are a couple ways I can see this working. First, we could allow the
user to configure the listener used for forwarded requests separately. That
would work with the existing `Authorizer.start` API. Alternatively, perhaps
we could modify `Authorizer.start` to work with something more granular
than `EndPoint`. This would allow the controller to begin accepting
requests from the other quorum members before it is ready to authorize
forwarded requests from clients.  Then we would need some way to let
brokers know when the controller is ready to accept these forwarded
requests (e.g. through an error code in the `Envelope` response).

What do you think?

Thanks,
Jason






On Wed, Jan 12, 2022 at 12:57 PM David Arthur
 wrote:

> +1 binding, thanks Colin!
>
> On Mon, Dec 13, 2021 at 7:47 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I'd like to start the vote on KIP-801: Implement an Authorizer that
> stores
> > metadata in __cluster_metadata
> >
> > The KIP is here: https://cwiki.apache.org/confluence/x/h5KqCw
> >
> > The original DISCUSS thread is here:
> >
> > https://lists.apache.org/thread/3d5o7h17ztjztjhblx4fln0wbbs1rmdq
> >
> > Please take a look and vote if you can.
> >
> > best,
> > Colin
> >
>
>
> --
> -David
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #631

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

[Colin McCabe]

On Fri, Jan 21, 2022, at 11:07, José Armando García Sancio wrote:
> Hi all,
>
> The following suggestions are not strictly required to implement this
> KIP but what do we think about:
>
> 1. Changing the name of the AlterIsr RPC to AlterPartition RPC.
>
> 2. Change the name of the field "CurrentIsrVersion" to
> "PartitionEpoch". This is the name that we use in the KRaft
> implementation and the PartitionRecord.

+1 for both.

On Mon, Jan 24, 2022, at 14:59, José Armando García Sancio wrote:
> Okay. The current KIP only supports two election recovery states
> because the type used is a boolean. I am going to extend it to use an
> int8 and support the following states:
>
> 1. ELECTION_RECOVERY - Notify to the leader that it should recover the
> local log. It indicates to the followers that the leader is
> recovering.
> 2. ELECTION_RECOVERED - Notify to the leader that recovery is not
> needed. It indicates to the follower that the leader has recovered.

We already have many classes that are called "partition state." For example, 
PartitionStates.java on the client side, PartitionStateMachine.scala and 
TopicPartitionStateZNode in the old controller, RemotePartitionDeleteState.java 
under storage, and so forth. I don't object to adding another one, but let's 
make it very clear that it's LeaderRecoveryState not just a generic "partition 
state", to avoid confusion. Actually maybe we should call it 
LeaderRecoveryStateChange, since we'll need to have a "no change" entry in the 
enum.

On Fri, Jan 21, 2022, at 11:02, José Armando García Sancio wrote:
> I am mainly returning the recorded value for consistency with the
> existing fields in the AlterIsrResponse. The implementation on the
> broker (AlterIsrManager and Partition) takes advantage of this by
> updating it's local state using the response values if the
> "CurrentIsrVersion" validates.

I would argue that we should not add unused fields to RPCs and metadata records 
because we might want them in the future, or because it seems more 
"symmetrical," etc. We have a great mechanism for adding new stuff in the 
future: add a new field and specify the default as whatever the old behavior 
was.

So I would argue that we should not add this state to AlterIsr, either the 
request or the response. We already know that sending AlterIsr clears the 
recovery state, and if it succeeded then the state was cleared. If this changes 
in the future, we can add a new field that default to whatever we want.

Adding an RPC field that will only ever have one value is bad form. And 99% of 
the time, when you do finally decide to have more than one possible value, 
you'll find that what you originally wrote isn't adequate and you need to 
change the RPC, or the code, anyway. At least, that's my experience.

We should follow the YAGNI philosophy for metadata record fields and RPC 
fields. If you don't have a use case for something yet, wait until you do have 
a use case for it to add it.

best,
Colin

On Mon, Jan 24, 2022, at 14:59, José Armando García Sancio wrote:
> Thanks for the additional context regarding AlterIsrResponse.
>
> Jason wrote:
>> In regard to the naming of `IsLeaderRecovering`, I agree it still seems a
>> bit awkward. I kind of liked the idea of turning it into a `PartitionState`
>> field instead. That would also address the inconsistent type in the
>> `PartitionChangeRecord`.
>
> Okay. The current KIP only supports two election recovery states
> because the type used is a boolean. I am going to extend it to use an
> int8 and support the following states:
>
> 1. ELECTION_RECOVERY - Notify to the leader that it should recover the
> local log. It indicates to the followers that the leader is
> recovering.
> 2. ELECTION_RECOVERED - Notify to the leader that recovery is not
> needed. It indicates to the follower that the leader has recovered.
>
> In the current KIP is the partition state invariant is
> "!InElectionRecovery || size(ISR) == 1". With the above change the
> invariant will be "ElectionState == ELECTION_RECOVERED || size(ISR) ==
> 1".
>
> Thanks!
> -José

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #630

[Apache Jenkins Server]

See 

[DISCUSS] KIP-816: Topology changes without local state reset

[Nick Telford]

Hi everyone,

I'd like to start a discussion on Kafka Streams KIP-816 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-816%3A+Topology+changes+without+local+state+reset
)

This KIP outlines 3 possible solutions to the problem, and I plan to
whittle this down to a definitive solution based on this discussion.

Of the 3 proposed solutions:
* 'A' is probably the "correct" solution, but is also quite a significant
change.
* 'B' is the least invasive, but most "hacky" solution.
* 'C' requires a change to the wire protocol and will likely have
unintended consequences. C is also the least complete solution, and will
need significant additional work to make it work.

Please let me know if the Motivation and Background sections need more
clarity.

Regards,

Nick Telford

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #629

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 509369 lines...]
[2022-01-25T15:30:20.786Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-01-25T15:30:20.786Z] > Task :clients:publishToMavenLocal
[2022-01-25T15:30:20.786Z] 
[2022-01-25T15:30:20.786Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-01-25T15:30:20.786Z] 
[2022-01-25T15:30:20.786Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-01-25T15:30:20.786Z] 
[2022-01-25T15:30:20.787Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-01-25T15:30:20.787Z] 
[2022-01-25T15:30:20.787Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure correctness.
[2022-01-25T15:30:20.787Z] Please consult deprecation warnings for more details.
[2022-01-25T15:30:20.787Z] 
[2022-01-25T15:30:20.787Z] BUILD SUCCESSFUL in 39s
[2022-01-25T15:30:20.787Z] 77 actionable tasks: 34 executed, 43 up-to-date
[2022-01-25T15:30:21.136Z] 
[2022-01-25T15:30:21.136Z] ListOffsetsIntegrationTest > testLatestOffset() 
PASSED
[2022-01-25T15:30:21.136Z] 
[2022-01-25T15:30:21.136Z] ListOffsetsIntegrationTest > testEarliestOffset() 
STARTED
[Pipeline] sh
[2022-01-25T15:30:22.185Z] 
[2022-01-25T15:30:22.185Z] ListOffsetsIntegrationTest > testEarliestOffset() 
PASSED
[2022-01-25T15:30:22.185Z] 
[2022-01-25T15:30:22.185Z] FetchRequestBetweenDifferentIbpTest > 
testControllerOldToNewIBP() STARTED
[2022-01-25T15:30:24.403Z] + grep ^version= gradle.properties
[2022-01-25T15:30:24.403Z] + cut -d= -f 2
[Pipeline] dir
[2022-01-25T15:30:25.435Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-01-25T15:30:28.268Z] + mvn clean install -Dgpg.skip
[2022-01-25T15:30:30.658Z] [INFO] Scanning for projects...
[2022-01-25T15:30:30.658Z] [INFO] 

[2022-01-25T15:30:30.658Z] [INFO] Reactor Build Order:
[2022-01-25T15:30:30.658Z] [INFO] 
[2022-01-25T15:30:30.658Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2022-01-25T15:30:30.658Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2022-01-25T15:30:30.658Z] [INFO] 
[2022-01-25T15:30:30.658Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2022-01-25T15:30:30.658Z] [INFO] Building Kafka Streams :: Quickstart 
3.2.0-SNAPSHOT[1/2]
[2022-01-25T15:30:30.658Z] [INFO] [ pom 
]-
[2022-01-25T15:30:30.658Z] [INFO] 
[2022-01-25T15:30:30.658Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart ---
[2022-01-25T15:30:30.658Z] [INFO] 
[2022-01-25T15:30:30.658Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2022-01-25T15:30:31.768Z] [INFO] 
[2022-01-25T15:30:31.768Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart ---
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart ---
[2022-01-25T15:30:32.878Z] [INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.2.0-SNAPSHOT/streams-quickstart-3.2.0-SNAPSHOT.pom
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2022-01-25T15:30:32.878Z] [INFO] Building streams-quickstart-java 
3.2.0-SNAPSHOT[2/2]
[2022-01-25T15:30:32.878Z] [INFO] --[ maven-archetype 
]---
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart-java ---
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2022-01-25T15:30:32.878Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2022-01-25T15:30:32.878Z] [INFO] Copying 6 resources
[2022-01-25T15:30:32.878Z] [INFO] Copying 3 resources
[2022-01-25T15:30:32.878Z] [INFO] 
[2022-01-25T15:30:32.878Z] [INFO] --- maven-resources

[jira] [Created] (KAFKA-13617) Enhance delete-expired-group-metadata logging

[Nicolas Guyomar (Jira)]

Nicolas Guyomar created KAFKA-13617:
---

 Summary: Enhance delete-expired-group-metadata logging
 Key: KAFKA-13617
 URL: https://issues.apache.org/jira/browse/KAFKA-13617
 Project: Kafka
  Issue Type: Improvement
  Components: offset manager
Reporter: Nicolas Guyomar


Hi team,

When you try to understand why a consumer group offset was expired, the default 
INFO logging on the group coordinator does not give much : 

info(s"Removed $numOffsetsRemoved expired offsets in ${time.milliseconds() - 
currentTimestamp} milliseconds.")

[https://github.com/apache/kafka/blob/22d056c9b76c9bf8417d8701594d1fcee1c6a655/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L834]

 

Would it be possible to enhance this log with the actual group/topic/partition 
that was removed please ? 

Thank you



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13616) Log4j 1.X CVE-2022-23302/5/7 vulnerabilities

[Dominique Mongelli (Jira)]

Dominique Mongelli created KAFKA-13616:
--

 Summary: Log4j 1.X CVE-2022-23302/5/7 vulnerabilities
 Key: KAFKA-13616
 URL: https://issues.apache.org/jira/browse/KAFKA-13616
 Project: Kafka
  Issue Type: Bug
Reporter: Dominique Mongelli


Some log4j 1.x vulnerabilities have been disclosed recently:   
 * CVE-2022-23302: https://nvd.nist.gov/vuln/detail/CVE-2022-23302    
 * CVE-2022-23305 : https://nvd.nist.gov/vuln/detail/CVE-2022-23305    
 * CVE-2022-23307 : [https://nvd.nist.gov/vuln/detail/CVE-2022-23307]

We would like to know if kafka is affected by these vulnerabilities ?

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] KIP-784: Add top-level error code field to DescribeLogDirsResponse

[Mickael Maison]

Hi,

Since it's a pretty minor KIP, I'll do +1 (binding) myself and close the vote.

The vote has passed with:
- 3 +1 (binding) votes from Tom, David and Mickael
- 1 +1 (non binding) vote from Luke

Thanks,
Mickael


On Thu, Dec 16, 2021 at 10:07 AM David Jacot
 wrote:
>
> +1 (binding). Thanks for the KIP!
>
> On Mon, Dec 13, 2021 at 11:14 AM Mickael Maison
>  wrote:
> >
> > Bumping this thread another time.
> >
> > This is a very minor change to make DescribeLogDirsResponse consistent
> > with the other APIs.
> > Let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Nov 22, 2021 at 10:29 AM Tom Bentley  wrote:
> > >
> > > Hi Mickael,
> > >
> > > It's pretty low value, but I think consistency is a useful trait, and it's
> > > easily achievable here.
> > >
> > > +1 (binding).
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > >
> > > On Thu, Nov 18, 2021 at 2:56 PM Mickael Maison 
> > > wrote:
> > >
> > > > Bumping this thread.
> > > >
> > > > Let me know if you have any feedback.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Wed, Oct 27, 2021 at 3:25 PM Luke Chen  wrote:
> > > > >
> > > > > Hi Mickael,
> > > > > Thanks for the KIP.
> > > > > It's good to keep it consistent with others, to have top-level error
> > > > field.
> > > > >
> > > > > + 1 (non-binding)
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Wed, Oct 27, 2021 at 9:01 PM Mickael Maison 
> > > > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start the vote on this minor KIP.
> > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-784%3A+Add+top-level+error+code+field+to+DescribeLogDirsResponse
> > > > > >
> > > > > > Please take a look, vote or let me know if you have any feedback.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > >
> > > >

Re: [VOTE] KIP-799: Align behaviour for producer callbacks with documented behaviour

[Séamus Ó Ceanainn]

Hi all,

Bumping this voting thread, as it still needs one more binding vote to pass.

Regards,
Séamus.

On Mon, 6 Dec 2021 at 12:04, Mickael Maison 
wrote:

> Thanks for the KIP!
> +1 (binding)
>
> On Sat, Dec 4, 2021 at 3:49 AM Luke Chen  wrote:
> >
> > Hi Séamus,
> >
> > Thanks for the update.
> > Looks better now!
> >
> > Thank you.
> > Luke
> >
> > On Sat, Dec 4, 2021 at 12:57 AM Séamus Ó Ceanainn <
> > seamus.oceana...@zalando.ie> wrote:
> >
> > > Hey Luke,
> > >
> > > Thanks for the feedback. I've updated the relevant section to hopefully
> > > make it more clear from the KIP itself what placeholder value would be
> > > returned.
> > >
> > > Regards,
> > > Séamus.
> > >
> > > On Tue, 30 Nov 2021 at 09:52, Luke Chen  wrote:
> > >
> > > > Hi Séamus,
> > > > Thanks for the KIP!
> > > > We definitely want to keep the producer callback consistent for all
> types
> > > > of errors.
> > > >
> > > > Just one comment for the KIP:
> > > > In the "Proposed Changes" section, could you please "explicitly"
> describe
> > > > what placeholder you'll return, in addition to adding a hyperlink to
> > > other
> > > > places, to make it clear.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Tue, Nov 30, 2021 at 1:17 PM John Roesler 
> > > wrote:
> > > >
> > > > > Thanks, Séamus!
> > > > >
> > > > > I'm +1 (binding).
> > > > >
> > > > > On Mon, 2021-11-29 at 16:14 +, Séamus Ó Ceanainn wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a vote for KIP-799: Align behaviour for
> producer
> > > > > > callbacks with documented behaviour
> > > > > > <
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-799%3A+Align+behaviour+for+producer+callbacks+with+documented+behaviour
> > > > > >
> > > > > > .
> > > > > >
> > > > > > The KIP proposes a breaking change in the behaviour of producer
> > > client
> > > > > > callbacks. The breaking change would align the behaviour of
> callbacks
> > > > > with
> > > > > > the documented behaviour for the method, and makes it consistent
> with
> > > > > > similar methods for producer client interceptors.
> > > > > >
> > > > > > Regards,
> > > > > > Séamus.
> > > > >
> > > > >
> > > >
> > >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #170

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 2.8 #96

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-591: Add Kafka Streams config to set default state store

[Luke Chen]

Thanks Matthias!

I agree we could deprecate the existing ones, and add the one with
storeType parameter.

That is:
@deprecated
Stores#persistentSessionStore(...)
@deprecated
Stores#inMemorySessionStore(...)
@new added with an additional storeType parameter (IN_MEMORY or ROCKS_DB)
Stores#sessionStoreSupplier(StoreType storeType, ...)

Let's see what others think about it.

Thank you.
Luke

On Tue, Jan 25, 2022 at 4:01 PM Matthias J. Sax  wrote:

> Thanks,
>
> There is already `Stores.persistentSessionStore` and
> `Stores.inMemorySessionStore`. From a DSL code POV, I don't see large
> benefits to add a new one, but it also does not hurt.
>
> Do you propose to add the third one only, or to also deprecate the
> existing ones? In general, we should avoid to extend the API surface
> area, so it could be a good simplification is we plan to remove the
> existing ones?
>
> Btw: we could name the new method just `sessionStoreSupplier` for
> simplicity (especially, if we deprecate the existing ones)?
>
> Not sure what others think. I am fine adding it, if we deprecate the
> existing ones.
>
> -Matthias
>
>
> On 1/24/22 5:03 PM, Luke Chen wrote:
> > Hi Matthias,
> >
> > I didn't "save" the change. >.<
> > Anyway, you can refer to this WIP PR to have better understanding
> why/what
> > the new API is:
> >
> https://github.com/apache/kafka/pull/11705/files#diff-c552e58e01169886c5d8b8b149f5c8cd48ea1fc1c3d7b932d055d3df9a00e1b5R464-R477
> >
> > It's not necessary, actually, but it can make the implementation cleaner.
> > If you think this change is unnecessary and will make the `Stores` API
> more
> > complicated, it's fine to me.
> >
> > I'll update the KIP after we have a conclusion for it.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Jan 25, 2022 at 2:37 AM Matthias J. Sax 
> wrote:
> >
> >> I don't see the KIP update? Did you hit "save"?
> >>
> >> Also, the formatting in your email for the new methods is hard to read.
> >> Not sure atm why we need the API change? Can you elaborate? what does
> >>
> >>> I found it'd be better
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 1/24/22 2:29 AM, Luke Chen wrote:
> >>> Thanks for all your votes.
> >>>
> >>> During the implementation, I found it'd be better to have helper
> methods
> >> in
> >>> `Stores`, to be able to get the store supplier by the store type:
> >>>
> >>>
> >>>
> >>> *public static SessionBytesStoreSupplier
> >>> sessionStoreSupplierByStoreType()public static WindowBytesStoreSupplier
> >>> windowStoreSupplierByStoreType()public static
> KeyValueBytesStoreSupplier
> >>> keyValueStoreSupplierByStoreType()*
> >>>
> >>> I've also updated in the KIP.
> >>> Please let me know if you other thoughts.
> >>>
> >>> Also, welcome to vote for this KIP.
> >>>
> >>> Thank you.
> >>> Luke
> >>>
> >>>
> >>> On Fri, Jan 21, 2022 at 4:39 AM Walker Carlson
> >>>  wrote:
> >>>
>  +1 non binding
> 
>  On Thu, Jan 20, 2022 at 2:00 PM Matthias J. Sax 
> >> wrote:
> 
> > +1 (binding)
> >
> > On 1/20/22 10:52 AM, Guozhang Wang wrote:
> >> Thanks Luke! I'm +1 on the KIP.
> >>
> >>
> >> Guozhang
> >>
> >> On Wed, Jan 19, 2022 at 5:58 PM Luke Chen 
> wrote:
> >>
> >>> Hi devs,
> >>>
> >>> I'd like to start a vote for the KIP-591: Add Kafka Streams config
> to
> > set
> >>> default state store. The goal is to allow users to set a default
> >> store
> > in
> >>> the config, so it can apply to all the streams.
> >>>
> >>> Detailed description can be found here:
> >>>
> >>>
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-591%3A+Add+Kafka+Streams+config+to+set+default+state+store
> >>>
> >>>
> >>> Thank you.
> >>> Luke
> >>>
> >>
> >>
> >
> 
> >>>
> >>
> >
>

Re: [VOTE] KIP-591: Add Kafka Streams config to set default state store

[Matthias J. Sax]

Thanks,

There is already `Stores.persistentSessionStore` and 
`Stores.inMemorySessionStore`. From a DSL code POV, I don't see large 
benefits to add a new one, but it also does not hurt.


Do you propose to add the third one only, or to also deprecate the 
existing ones? In general, we should avoid to extend the API surface 
area, so it could be a good simplification is we plan to remove the 
existing ones?


Btw: we could name the new method just `sessionStoreSupplier` for 
simplicity (especially, if we deprecate the existing ones)?


Not sure what others think. I am fine adding it, if we deprecate the 
existing ones.


-Matthias


On 1/24/22 5:03 PM, Luke Chen wrote:

Hi Matthias,

I didn't "save" the change. >.<
Anyway, you can refer to this WIP PR to have better understanding why/what
the new API is:
https://github.com/apache/kafka/pull/11705/files#diff-c552e58e01169886c5d8b8b149f5c8cd48ea1fc1c3d7b932d055d3df9a00e1b5R464-R477

It's not necessary, actually, but it can make the implementation cleaner.
If you think this change is unnecessary and will make the `Stores` API more
complicated, it's fine to me.

I'll update the KIP after we have a conclusion for it.

Thank you.
Luke

On Tue, Jan 25, 2022 at 2:37 AM Matthias J. Sax  wrote:


I don't see the KIP update? Did you hit "save"?

Also, the formatting in your email for the new methods is hard to read.
Not sure atm why we need the API change? Can you elaborate? what does


I found it'd be better



-Matthias


On 1/24/22 2:29 AM, Luke Chen wrote:

Thanks for all your votes.

During the implementation, I found it'd be better to have helper methods

in

`Stores`, to be able to get the store supplier by the store type:



*public static SessionBytesStoreSupplier
sessionStoreSupplierByStoreType()public static WindowBytesStoreSupplier
windowStoreSupplierByStoreType()public static KeyValueBytesStoreSupplier
keyValueStoreSupplierByStoreType()*

I've also updated in the KIP.
Please let me know if you other thoughts.

Also, welcome to vote for this KIP.

Thank you.
Luke


On Fri, Jan 21, 2022 at 4:39 AM Walker Carlson
 wrote:


+1 non binding

On Thu, Jan 20, 2022 at 2:00 PM Matthias J. Sax 

wrote:



+1 (binding)

On 1/20/22 10:52 AM, Guozhang Wang wrote:

Thanks Luke! I'm +1 on the KIP.


Guozhang

On Wed, Jan 19, 2022 at 5:58 PM Luke Chen  wrote:


Hi devs,

I'd like to start a vote for the KIP-591: Add Kafka Streams config to

set

default state store. The goal is to allow users to set a default

store

in

the config, so it can apply to all the streams.

Detailed description can be found here:







https://cwiki.apache.org/confluence/display/KAFKA/KIP-591%3A+Add+Kafka+Streams+config+to+set+default+state+store



Thank you.
Luke