[jira] [Created] (KAFKA-16913) Support external schemas in JSONConverter

[Priyanka K U (Jira)]

Priyanka K U created KAFKA-16913:


 Summary: Support external schemas in JSONConverter
 Key: KAFKA-16913
 URL: https://issues.apache.org/jira/browse/KAFKA-16913
 Project: Kafka
  Issue Type: Improvement
  Components: connect
Reporter: Priyanka K U


 KIP-1054: Support external schemas in JSONConverter : 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+schemas+in+JSONConverter



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1054: Support external schemas in JSONConverter

[Priyanka K U]

Hi everyone,

I'd like to start a discussion of KIP-1054, which aims to Support external 
schemas in JSONConverter  to Kafka Connect: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+schemas+in+JSONConverter



Looking forward for your feedback.

Regards,
Priyanka

[jira] [Created] (KAFKA-16914) Add Shage Group dynamic and broker configs

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16914:
-

 Summary: Add Shage Group dynamic and broker configs
 Key: KAFKA-16914
 URL: https://issues.apache.org/jira/browse/KAFKA-16914
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 4.0.0
Reporter: Apoorv Mittal
Assignee: Abhinav Dixit
 Fix For: 4.0.0


Add the configs required for share group in KafkaConfig or equivalent.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-5261) Performance improvement of SimpleAclAuthorizer

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-5261.
---
Resolution: Won't Do

> Performance improvement of SimpleAclAuthorizer
> --
>
> Key: KAFKA-5261
> URL: https://issues.apache.org/jira/browse/KAFKA-5261
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.2.1
>Reporter: Stephane Maarek
>Priority: Major
>
> Currently, looking at the KafkaApis class, it seems that every request going 
> through Kafka is also going through an authorize check:
> {code}
>   private def authorize(session: Session, operation: Operation, resource: 
> Resource): Boolean =
> authorizer.forall(_.authorize(session, operation, resource))
> {code}
> The SimpleAclAuthorizer logic runs through checks which all look to be done 
> in linear time (except on first run) proportional to the number of acls on a 
> specific resource. This operation is re-run every time a client tries to use 
> a Kafka Api, especially on the very often called `handleProducerRequest` and  
> `handleFetchRequest`
> I believe a cache could be built to store the result of the authorize call, 
> possibly allowing more expensive authorize() calls to happen, and reducing 
> greatly the CPU usage in the long run. The cache would be invalidated every 
> time a change happens to aclCache
> Thoughts before I try giving it a go with a PR? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1050: Consistent error handling for Transactions

[Kaushik Raina]

Thank you Andrew for feedback

1. We are suggesting to only update subclasses of
o.a.k.common.errors.ApiException, which are used in transactions. All such
subclasses are mentioned in Exception table


2. "Producer-Recoverable" corresponds to the AbortableException. I have
updated comments on each exception type.

3. Yes, it's correct that by adding a "Retriable" exception, it simplifies
the determination of which errors can be retried internally. In the Exception
table

mentioned
in the "Proposed Changes" section, the "Expected Handling" column signifies
the handling for each error type. Please let me know if any further
clarification is needed.

4a. Yes, that is correct. For clarity, only one constructor has been
mentioned in the KIP. An ellipsis has been added as a placeholder,
indicating that there are additional functions in the class but they are
not explicitly specified.
4b. Updated in the KIP.

5. TopicAuthorizationException extends "Invalid Configuration". "Invalid
Configuration" type can be resolved either by dynamically updating the
configuration, which does not require a restart, or by statically updating
it by restarting the application. It is at the application's discretion how
they want to handle each "Invalid Configuration" type.

I have added Client side handling example

in
KIP. Hope that helps.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #20

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16915) Update leader change message

[Jira]

José Armando García Sancio created KAFKA-16915:
--

 Summary: Update leader change message
 Key: KAFKA-16915
 URL: https://issues.apache.org/jira/browse/KAFKA-16915
 Project: Kafka
  Issue Type: Sub-task
Reporter: José Armando García Sancio
 Fix For: 3.8.0


It would be good to include the schema changes as part of 3.8 event if they are 
not populated. It would make it compatible with kraft.version 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

[Mickael Maison]

Hi Chris,

Happy Friday! The KIP looks good to me. +1

Thanks,
Mickael

On Fri, Jan 26, 2024 at 8:41 PM Chris Egerton  wrote:
>
> Hi all,
>
> Happy Friday! I'd like to kick off discussion for KIP-1017, which (as the
> title suggests) proposes adding a health check endpoint for Kafka Connect:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect
>
> This is one of the longest-standing issues with Kafka Connect and I'm
> hoping we can finally put it in the ground soon. Looking forward to hearing
> people's thoughts!
>
> Cheers,
>
> Chris

Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

[Fan Yang]

Hi Priyanka,

My suggestion is that the we can place schema file at any location, such as 
file location, network location, etc. So, the new option could be just 
schema.location


Best,
Fan

From: Priyanka K U 
Sent: Friday, June 7, 2024 16:22
To: dev@kafka.apache.org 
Subject: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

Hi everyone,

I'd like to start a discussion of KIP-1054, which aims to Support external 
schemas in JSONConverter  to Kafka Connect: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+schemas+in+JSONConverter



Looking forward for your feedback.

Regards,
Priyanka

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

[Andrew Schofield]

Hi Chris,
This KIP looks good to me. I particularly like the explanation of how the 
result will specifically
check the worker health in ways that have previously caused trouble.

Thanks,
Andrew

> On 7 Jun 2024, at 16:18, Mickael Maison  wrote:
>
> Hi Chris,
>
> Happy Friday! The KIP looks good to me. +1
>
> Thanks,
> Mickael
>
> On Fri, Jan 26, 2024 at 8:41 PM Chris Egerton  wrote:
>>
>> Hi all,
>>
>> Happy Friday! I'd like to kick off discussion for KIP-1017, which (as the
>> title suggests) proposes adding a health check endpoint for Kafka Connect:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect
>>
>> This is one of the longest-standing issues with Kafka Connect and I'm
>> hoping we can finally put it in the ground soon. Looking forward to hearing
>> people's thoughts!
>>
>> Cheers,
>>
>> Chris

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #21

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16911) Kafka Streams topology optimization docs incomplete

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-16911.
-
Fix Version/s: 3.8.0
   Resolution: Fixed

> Kafka Streams topology optimization docs incomplete
> ---
>
> Key: KAFKA-16911
> URL: https://issues.apache.org/jira/browse/KAFKA-16911
> Project: Kafka
>  Issue Type: Improvement
>  Components: docs, streams
>Affects Versions: 3.4.0
>Reporter: Matthias J. Sax
>Assignee: James Galasyn
>Priority: Minor
> Fix For: 3.8.0
>
>
> The docs for topology optimization are incomplete: 
> [https://kafka.apache.org/37/documentation/streams/developer-guide/config-streams.html#topology-optimization]
> In 3.4.0 we added a new option via 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join+optimization+for+stream-stream+joins]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

[Greg Harris]

Hi Priyanka,

Thanks for the KIP! I think you captured the motivation well: The Converter
interface on its own implies a fairly large raw message size, and the
ecosystem's strategies for deduplicating schema information are complex. I
did have some questions/concerns.

1. Have you done a comparison between the external schemas approach and the
current schemas.enable approach with compression enabled? I think that's
currently the best alternative for users without an external schemas
service, so I'd like to get an idea of how impactful this feature is in an
example use-case.

2. This feature could be implemented without a KIP by creating and
distributing a custom JsonConverter. Could you add that as a rejected
alternative, with some justification for why it was rejected?

3. Recently we made an effort [1] to secure the FileConfigProvider and
DirectoryConfigProvider plugins, which each have the capability of reading
from the worker's disk. How do you think this could be made secure, such
that a REST API user can't have unrestricted access to the filesystem via
this plugin?

And to synthesize points 2 and 3: It is very reasonable for "dangerous"
features to be opted-in through installation of a custom Converter by
operators that understand the risks. It is unreasonable to deliver a
"dangerous" feature to operators without a simple way to opt-out, as the
upstream plugins are somewhat difficult to remove.

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-993%3A+Allow+restricting+files+accessed+by+File+and+Directory+ConfigProviders

On Fri, Jun 7, 2024 at 8:40 AM Fan Yang  wrote:

> Hi Priyanka,
>
> My suggestion is that the we can place schema file at any location, such
> as file location, network location, etc. So, the new option could be just
> schema.location
>
>
> Best,
> Fan
> 
> From: Priyanka K U 
> Sent: Friday, June 7, 2024 16:22
> To: dev@kafka.apache.org 
> Subject: [DISCUSS] KIP-1054: Support external schemas in JSONConverter
>
> Hi everyone,
>
> I'd like to start a discussion of KIP-1054, which aims to Support external
> schemas in JSONConverter  to Kafka Connect:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+schemas+in+JSONConverter
>
>
>
> Looking forward for your feedback.
>
> Regards,
> Priyanka
>

[DISCUSS] KIP-1053: Align the naming convention for config and default variables in *Config classes

[Eric Lu]

Hi,

I think that my last attempt at creating a discussion thread half-worked. I
wanted to open a discussion thread to discuss KIP-1053 about renaming the
constants to fit the naming convention. The idea is to deprecate the old
variables and introduce the new ones.

The main dilemma is whether it is truly worth it to rename constants in the
public interface to fit the alignment.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1053%3A+Align+the+naming+convention+for+config+and+default+variables+in+*Config+classes



Warm regards,

Eric

Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi,

Is this KIP focusing on Sink connectors? Meaning, is the idea to be able to 
decode JSON messages into Connect records using a schema read from disk?

If I look at this from the perspective of a source connector, not including the 
schema contents - or the ID of a schema from a schema registry - is exactly the 
same as not using schemas at all. Right?

From: dev@kafka.apache.org At: 06/07/24 14:16:13 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

Hi Priyanka,

Thanks for the KIP! I think you captured the motivation well: The Converter
interface on its own implies a fairly large raw message size, and the
ecosystem's strategies for deduplicating schema information are complex. I
did have some questions/concerns.

1. Have you done a comparison between the external schemas approach and the
current schemas.enable approach with compression enabled? I think that's
currently the best alternative for users without an external schemas
service, so I'd like to get an idea of how impactful this feature is in an
example use-case.

2. This feature could be implemented without a KIP by creating and
distributing a custom JsonConverter. Could you add that as a rejected
alternative, with some justification for why it was rejected?

3. Recently we made an effort [1] to secure the FileConfigProvider and
DirectoryConfigProvider plugins, which each have the capability of reading
from the worker's disk. How do you think this could be made secure, such
that a REST API user can't have unrestricted access to the filesystem via
this plugin?

And to synthesize points 2 and 3: It is very reasonable for "dangerous"
features to be opted-in through installation of a custom Converter by
operators that understand the risks. It is unreasonable to deliver a
"dangerous" feature to operators without a simple way to opt-out, as the
upstream plugins are somewhat difficult to remove.

Thanks,
Greg

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-993%3A+Allow+restricting+f
iles+accessed+by+File+and+Directory+ConfigProviders

On Fri, Jun 7, 2024 at 8:40 AM Fan Yang  wrote:

> Hi Priyanka,
>
> My suggestion is that the we can place schema file at any location, such
> as file location, network location, etc. So, the new option could be just
> schema.location
>
>
> Best,
> Fan
> 
> From: Priyanka K U 
> Sent: Friday, June 7, 2024 16:22
> To: dev@kafka.apache.org 
> Subject: [DISCUSS] KIP-1054: Support external schemas in JSONConverter
>
> Hi everyone,
>
> I'd like to start a discussion of KIP-1054, which aims to Support external
> schemas in JSONConverter  to Kafka Connect:
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+s
chemas+in+JSONConverter
>
>
>
> Looking forward for your feedback.
>
> Regards,
> Priyanka
>

Build hanging

[Sophie Blee-Goldman]

Seems like the build is currently broken -- specifically, a test is hanging
and causing it to abort after 7+ hours. There are many examples in the
current PRs, such as

Timed out after almost 8 hours:
1.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
2.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline

Still running after 6+ hours:
1.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/

It's pretty difficult to tell which test is hanging but it seems like one
of the commits in the last 1-2 days is the likely culprit. If anyone has an
idea of what may have caused this or is actively investigating, please let
everyone know.

Needless to say, this is rather urgent given the upcoming 3.8 code freeze.

Thanks,
Sophie

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #22

[Apache Jenkins Server]

See 

Re: Build hanging

[Haruki Okada]

Hi

I found that the hanging can be reproduced locally.
The blocking test is
"org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
It started to block after this commit (
https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
)

Thanks,

2024年6月8日(土) 8:30 Sophie Blee-Goldman :

> Seems like the build is currently broken -- specifically, a test is hanging
> and causing it to abort after 7+ hours. There are many examples in the
> current PRs, such as
>
> Timed out after almost 8 hours:
> 1.
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> 2.
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
>
> Still running after 6+ hours:
> 1.
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
>
> It's pretty difficult to tell which test is hanging but it seems like one
> of the commits in the last 1-2 days is the likely culprit. If anyone has an
> idea of what may have caused this or is actively investigating, please let
> everyone know.
>
> Needless to say, this is rather urgent given the upcoming 3.8 code freeze.
>
> Thanks,
> Sophie
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: Build hanging

[Luke Chen]

Hi Haruki,

Thanks for identifying this blocking test.
Could you help quickly open a PR to disable this test to unblock the CI
build?

Thanks.
Luke

On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada  wrote:

> Hi
>
> I found that the hanging can be reproduced locally.
> The blocking test is
>
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> It started to block after this commit (
>
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> )
>
> Thanks,
>
> 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
>
> > Seems like the build is currently broken -- specifically, a test is
> hanging
> > and causing it to abort after 7+ hours. There are many examples in the
> > current PRs, such as
> >
> > Timed out after almost 8 hours:
> > 1.
> >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > 2.
> >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> >
> > Still running after 6+ hours:
> > 1.
> >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> >
> > It's pretty difficult to tell which test is hanging but it seems like one
> > of the commits in the last 1-2 days is the likely culprit. If anyone has
> an
> > idea of what may have caused this or is actively investigating, please
> let
> > everyone know.
> >
> > Needless to say, this is rather urgent given the upcoming 3.8 code
> freeze.
> >
> > Thanks,
> > Sophie
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

[jira] [Created] (KAFKA-16916) ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will run forever

[Luke Chen (Jira)]

Luke Chen created KAFKA-16916:
-

 Summary: 
ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will run 
forever
 Key: KAFKA-16916
 URL: https://issues.apache.org/jira/browse/KAFKA-16916
 Project: Kafka
  Issue Type: Bug
Reporter: Luke Chen
Assignee: Haruki Okada


ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will run 
forever



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Build hanging

[Haruki Okada]

Hi Luke,

I see, but since this is likely due to AdminClient's behavior change, we
need to fix it anyways not only disabling test before 3.8 release.
Let's disable for now to unblock builds, and revert later if we can't solve
it until code freeze?

2024年6月8日(土) 9:31 Luke Chen :

> Hi Haruki,
>
> Thanks for identifying this blocking test.
> Could you help quickly open a PR to disable this test to unblock the CI
> build?
>
> Thanks.
> Luke
>
> On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada  wrote:
>
> > Hi
> >
> > I found that the hanging can be reproduced locally.
> > The blocking test is
> >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > It started to block after this commit (
> >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > )
> >
> > Thanks,
> >
> > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> >
> > > Seems like the build is currently broken -- specifically, a test is
> > hanging
> > > and causing it to abort after 7+ hours. There are many examples in the
> > > current PRs, such as
> > >
> > > Timed out after almost 8 hours:
> > > 1.
> > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > 2.
> > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > >
> > > Still running after 6+ hours:
> > > 1.
> > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > >
> > > It's pretty difficult to tell which test is hanging but it seems like
> one
> > > of the commits in the last 1-2 days is the likely culprit. If anyone
> has
> > an
> > > idea of what may have caused this or is actively investigating, please
> > let
> > > everyone know.
> > >
> > > Needless to say, this is rather urgent given the upcoming 3.8 code
> > freeze.
> > >
> > > Thanks,
> > > Sophie
> > >
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: Build hanging

[Luke Chen]

> Let's disable for now to unblock builds, and revert later if we can't
solve
it until code freeze?

That's exactly what I meant to do.
I've opened KAFKA-16916 
for this issue and assigned to you.
Welcome to unassign yourselves if you don't have time to fix the
adminClient behavior change issue.
But, let's disable it first.

Thanks.
Luke


On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada  wrote:

> Hi Luke,
>
> I see, but since this is likely due to AdminClient's behavior change, we
> need to fix it anyways not only disabling test before 3.8 release.
> Let's disable for now to unblock builds, and revert later if we can't solve
> it until code freeze?
>
> 2024年6月8日(土) 9:31 Luke Chen :
>
> > Hi Haruki,
> >
> > Thanks for identifying this blocking test.
> > Could you help quickly open a PR to disable this test to unblock the CI
> > build?
> >
> > Thanks.
> > Luke
> >
> > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada  wrote:
> >
> > > Hi
> > >
> > > I found that the hanging can be reproduced locally.
> > > The blocking test is
> > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > It started to block after this commit (
> > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > )
> > >
> > > Thanks,
> > >
> > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> > >
> > > > Seems like the build is currently broken -- specifically, a test is
> > > hanging
> > > > and causing it to abort after 7+ hours. There are many examples in
> the
> > > > current PRs, such as
> > > >
> > > > Timed out after almost 8 hours:
> > > > 1.
> > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > 2.
> > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > > >
> > > > Still running after 6+ hours:
> > > > 1.
> > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > > >
> > > > It's pretty difficult to tell which test is hanging but it seems like
> > one
> > > > of the commits in the last 1-2 days is the likely culprit. If anyone
> > has
> > > an
> > > > idea of what may have caused this or is actively investigating,
> please
> > > let
> > > > everyone know.
> > > >
> > > > Needless to say, this is rather urgent given the upcoming 3.8 code
> > > freeze.
> > > >
> > > > Thanks,
> > > > Sophie
> > > >
> > >
> > >
> > > --
> > > 
> > > Okada Haruki
> > > ocadar...@gmail.com
> > > 
> > >
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

Re: Build hanging

[Haruki Okada]

Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
Let's see if CI runs properly.

Thanks,

2024年6月8日(土) 10:01 Luke Chen :

> > Let's disable for now to unblock builds, and revert later if we can't
> solve
> it until code freeze?
>
> That's exactly what I meant to do.
> I've opened KAFKA-16916  >
> for this issue and assigned to you.
> Welcome to unassign yourselves if you don't have time to fix the
> adminClient behavior change issue.
> But, let's disable it first.
>
> Thanks.
> Luke
>
>
> On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada  wrote:
>
> > Hi Luke,
> >
> > I see, but since this is likely due to AdminClient's behavior change, we
> > need to fix it anyways not only disabling test before 3.8 release.
> > Let's disable for now to unblock builds, and revert later if we can't
> solve
> > it until code freeze?
> >
> > 2024年6月8日(土) 9:31 Luke Chen :
> >
> > > Hi Haruki,
> > >
> > > Thanks for identifying this blocking test.
> > > Could you help quickly open a PR to disable this test to unblock the CI
> > > build?
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada 
> wrote:
> > >
> > > > Hi
> > > >
> > > > I found that the hanging can be reproduced locally.
> > > > The blocking test is
> > > >
> > > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > > It started to block after this commit (
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > > )
> > > >
> > > > Thanks,
> > > >
> > > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> > > >
> > > > > Seems like the build is currently broken -- specifically, a test is
> > > > hanging
> > > > > and causing it to abort after 7+ hours. There are many examples in
> > the
> > > > > current PRs, such as
> > > > >
> > > > > Timed out after almost 8 hours:
> > > > > 1.
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > > 2.
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > > > >
> > > > > Still running after 6+ hours:
> > > > > 1.
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > > > >
> > > > > It's pretty difficult to tell which test is hanging but it seems
> like
> > > one
> > > > > of the commits in the last 1-2 days is the likely culprit. If
> anyone
> > > has
> > > > an
> > > > > idea of what may have caused this or is actively investigating,
> > please
> > > > let
> > > > > everyone know.
> > > > >
> > > > > Needless to say, this is rather urgent given the upcoming 3.8 code
> > > > freeze.
> > > > >
> > > > > Thanks,
> > > > > Sophie
> > > > >
> > > >
> > > >
> > > > --
> > > > 
> > > > Okada Haruki
> > > > ocadar...@gmail.com
> > > > 
> > > >
> > >
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: Build hanging

[Sophie Blee-Goldman]

Thanks for jumping on this guys, and nice find Haruki

I agree with Luke that we should disable the offending tests ASAP to
unblock other things, and file a 3.8 blocker to investigate this further.
Thanks for the PR Haruki and thanks for filing the ticket Luke -- I marked
it as a blocker for 3.8.0 so this doesn't slip.

By the way -- someone else was looking into this on the side and mentioned
another test they were suspicious of as well. It's
SaslClientsWithInvalidCredentialsTest.testKafkaAdminClientWithAuthenticationFailure

Does one of you have time to look into this test as well? Don't want to
have to wait another 8 hours for your PR to hit the timeout and abort
again. If not, I can spend a minute looking at this other test a bit later
tonight

On Fri, Jun 7, 2024 at 6:05 PM Haruki Okada  wrote:

> Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
> Let's see if CI runs properly.
>
> Thanks,
>
> 2024年6月8日(土) 10:01 Luke Chen :
>
> > > Let's disable for now to unblock builds, and revert later if we can't
> > solve
> > it until code freeze?
> >
> > That's exactly what I meant to do.
> > I've opened KAFKA-16916 <
> https://issues.apache.org/jira/browse/KAFKA-16916
> > >
> > for this issue and assigned to you.
> > Welcome to unassign yourselves if you don't have time to fix the
> > adminClient behavior change issue.
> > But, let's disable it first.
> >
> > Thanks.
> > Luke
> >
> >
> > On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada  wrote:
> >
> > > Hi Luke,
> > >
> > > I see, but since this is likely due to AdminClient's behavior change,
> we
> > > need to fix it anyways not only disabling test before 3.8 release.
> > > Let's disable for now to unblock builds, and revert later if we can't
> > solve
> > > it until code freeze?
> > >
> > > 2024年6月8日(土) 9:31 Luke Chen :
> > >
> > > > Hi Haruki,
> > > >
> > > > Thanks for identifying this blocking test.
> > > > Could you help quickly open a PR to disable this test to unblock the
> CI
> > > > build?
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada 
> > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I found that the hanging can be reproduced locally.
> > > > > The blocking test is
> > > > >
> > > > >
> > > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > > > It started to block after this commit (
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > > > )
> > > > >
> > > > > Thanks,
> > > > >
> > > > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> > > > >
> > > > > > Seems like the build is currently broken -- specifically, a test
> is
> > > > > hanging
> > > > > > and causing it to abort after 7+ hours. There are many examples
> in
> > > the
> > > > > > current PRs, such as
> > > > > >
> > > > > > Timed out after almost 8 hours:
> > > > > > 1.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > > > 2.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > > > > >
> > > > > > Still running after 6+ hours:
> > > > > > 1.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > > > > >
> > > > > > It's pretty difficult to tell which test is hanging but it seems
> > like
> > > > one
> > > > > > of the commits in the last 1-2 days is the likely culprit. If
> > anyone
> > > > has
> > > > > an
> > > > > > idea of what may have caused this or is actively investigating,
> > > please
> > > > > let
> > > > > > everyone know.
> > > > > >
> > > > > > Needless to say, this is rather urgent given the upcoming 3.8
> code
> > > > > freeze.
> > > > > >
> > > > > > Thanks,
> > > > > > Sophie
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > 
> > > > > Okada Haruki
> > > > > ocadar...@gmail.com
> > > > > 
> > > > >
> > > >
> > >
> > >
> > > --
> > > 
> > > Okada Haruki
> > > ocadar...@gmail.com
> > > 
> > >
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

Re: Build hanging

[Apoorv Mittal]

Hi,
Please find the fix for the issue:
https://github.com/apache/kafka/pull/16249

Regards,
Apoorv Mittal
+44 7721681581


On Sat, Jun 8, 2024 at 2:09 AM Sophie Blee-Goldman 
wrote:

> Thanks for jumping on this guys, and nice find Haruki
>
> I agree with Luke that we should disable the offending tests ASAP to
> unblock other things, and file a 3.8 blocker to investigate this further.
> Thanks for the PR Haruki and thanks for filing the ticket Luke -- I marked
> it as a blocker for 3.8.0 so this doesn't slip.
>
> By the way -- someone else was looking into this on the side and mentioned
> another test they were suspicious of as well. It's
>
> SaslClientsWithInvalidCredentialsTest.testKafkaAdminClientWithAuthenticationFailure
>
> Does one of you have time to look into this test as well? Don't want to
> have to wait another 8 hours for your PR to hit the timeout and abort
> again. If not, I can spend a minute looking at this other test a bit later
> tonight
>
> On Fri, Jun 7, 2024 at 6:05 PM Haruki Okada  wrote:
>
> > Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
> > Let's see if CI runs properly.
> >
> > Thanks,
> >
> > 2024年6月8日(土) 10:01 Luke Chen :
> >
> > > > Let's disable for now to unblock builds, and revert later if we can't
> > > solve
> > > it until code freeze?
> > >
> > > That's exactly what I meant to do.
> > > I've opened KAFKA-16916 <
> > https://issues.apache.org/jira/browse/KAFKA-16916
> > > >
> > > for this issue and assigned to you.
> > > Welcome to unassign yourselves if you don't have time to fix the
> > > adminClient behavior change issue.
> > > But, let's disable it first.
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada 
> wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > I see, but since this is likely due to AdminClient's behavior change,
> > we
> > > > need to fix it anyways not only disabling test before 3.8 release.
> > > > Let's disable for now to unblock builds, and revert later if we can't
> > > solve
> > > > it until code freeze?
> > > >
> > > > 2024年6月8日(土) 9:31 Luke Chen :
> > > >
> > > > > Hi Haruki,
> > > > >
> > > > > Thanks for identifying this blocking test.
> > > > > Could you help quickly open a PR to disable this test to unblock
> the
> > CI
> > > > > build?
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada 
> > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I found that the hanging can be reproduced locally.
> > > > > > The blocking test is
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > > > > It started to block after this commit (
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > > > > )
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> > > > > >
> > > > > > > Seems like the build is currently broken -- specifically, a
> test
> > is
> > > > > > hanging
> > > > > > > and causing it to abort after 7+ hours. There are many examples
> > in
> > > > the
> > > > > > > current PRs, such as
> > > > > > >
> > > > > > > Timed out after almost 8 hours:
> > > > > > > 1.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > > > > 2.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > > > > > >
> > > > > > > Still running after 6+ hours:
> > > > > > > 1.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > > > > > >
> > > > > > > It's pretty difficult to tell which test is hanging but it
> seems
> > > like
> > > > > one
> > > > > > > of the commits in the last 1-2 days is the likely culprit. If
> > > anyone
> > > > > has
> > > > > > an
> > > > > > > idea of what may have caused this or is actively investigating,
> > > > please
> > > > > > let
> > > > > > > everyone know.
> > > > > > >
> > > > > > > Needless to say, this is rather urgent given the upcoming 3.8
> > code
> > > > > > freeze.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sophie
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > 
> > > > > > Okada Haruki
> > > > > > ocadar...@gmail.com
> > > > > > 
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > 
> > > > Okada Haruki
> > > > ocadar...@gmail.com
> > > > 
> > > >
> > >
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>

Re: Build hanging

[Haruki Okada]

Oh, thank you for identifying the root cause quickly Apoorv.

I changed the assignee of KAFKA-16916 to you.


Thanks,

2024年6月8日(土) 10:13 Apoorv Mittal :

> Hi,
> Please find the fix for the issue:
> https://github.com/apache/kafka/pull/16249
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Sat, Jun 8, 2024 at 2:09 AM Sophie Blee-Goldman 
> wrote:
>
> > Thanks for jumping on this guys, and nice find Haruki
> >
> > I agree with Luke that we should disable the offending tests ASAP to
> > unblock other things, and file a 3.8 blocker to investigate this further.
> > Thanks for the PR Haruki and thanks for filing the ticket Luke -- I
> marked
> > it as a blocker for 3.8.0 so this doesn't slip.
> >
> > By the way -- someone else was looking into this on the side and
> mentioned
> > another test they were suspicious of as well. It's
> >
> >
> SaslClientsWithInvalidCredentialsTest.testKafkaAdminClientWithAuthenticationFailure
> >
> > Does one of you have time to look into this test as well? Don't want to
> > have to wait another 8 hours for your PR to hit the timeout and abort
> > again. If not, I can spend a minute looking at this other test a bit
> later
> > tonight
> >
> > On Fri, Jun 7, 2024 at 6:05 PM Haruki Okada  wrote:
> >
> > > Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
> > > Let's see if CI runs properly.
> > >
> > > Thanks,
> > >
> > > 2024年6月8日(土) 10:01 Luke Chen :
> > >
> > > > > Let's disable for now to unblock builds, and revert later if we
> can't
> > > > solve
> > > > it until code freeze?
> > > >
> > > > That's exactly what I meant to do.
> > > > I've opened KAFKA-16916 <
> > > https://issues.apache.org/jira/browse/KAFKA-16916
> > > > >
> > > > for this issue and assigned to you.
> > > > Welcome to unassign yourselves if you don't have time to fix the
> > > > adminClient behavior change issue.
> > > > But, let's disable it first.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada 
> > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > I see, but since this is likely due to AdminClient's behavior
> change,
> > > we
> > > > > need to fix it anyways not only disabling test before 3.8 release.
> > > > > Let's disable for now to unblock builds, and revert later if we
> can't
> > > > solve
> > > > > it until code freeze?
> > > > >
> > > > > 2024年6月8日(土) 9:31 Luke Chen :
> > > > >
> > > > > > Hi Haruki,
> > > > > >
> > > > > > Thanks for identifying this blocking test.
> > > > > > Could you help quickly open a PR to disable this test to unblock
> > the
> > > CI
> > > > > > build?
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada  >
> > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I found that the hanging can be reproduced locally.
> > > > > > > The blocking test is
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > > > > > It started to block after this commit (
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > > > > > )
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman :
> > > > > > >
> > > > > > > > Seems like the build is currently broken -- specifically, a
> > test
> > > is
> > > > > > > hanging
> > > > > > > > and causing it to abort after 7+ hours. There are many
> examples
> > > in
> > > > > the
> > > > > > > > current PRs, such as
> > > > > > > >
> > > > > > > > Timed out after almost 8 hours:
> > > > > > > > 1.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > > > > > 2.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
> > > > > > > >
> > > > > > > > Still running after 6+ hours:
> > > > > > > > 1.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16236/3/pipeline/
> > > > > > > >
> > > > > > > > It's pretty difficult to tell which test is hanging but it
> > seems
> > > > like
> > > > > > one
> > > > > > > > of the commits in the last 1-2 days is the likely culprit. If
> > > > anyone
> > > > > > has
> > > > > > > an
> > > > > > > > idea of what may have caused this or is actively
> investigating,
> > > > > please
> > > > > > > let
> > > > > > > > everyone know.
> > > > > > > >
> > > > > > > > Needless to say, this is rather urgent given the upcoming 3.8
> > > code
> > > > > > > freeze.
> > > > > > > >
> > > > > > > > Thanks,
> > > 

Re: Build hanging

[Sophie Blee-Goldman]

Thanks for the quick fix!

By the way, I took a quick look at the output for some aborted builds and
can confirm that this test is also hanging: SaslClientsWithInvalidCre
dentialsTest.testKafkaAdminClientWithAuthenticationFailure

It sounds like it probably has the same root cause as the test we already
disabled, but just to make sure, Apoorv can you confirm that your fix will
also address the SaslClientsWithInvalidCredentialsTest?

On Fri, Jun 7, 2024 at 6:18 PM Haruki Okada  wrote:

> Oh, thank you for identifying the root cause quickly Apoorv.
>
> I changed the assignee of KAFKA-16916 to you.
>
>
> Thanks,
>
> 2024年6月8日(土) 10:13 Apoorv Mittal :
>
> > Hi,
> > Please find the fix for the issue:
> > https://github.com/apache/kafka/pull/16249
> >
> > Regards,
> > Apoorv Mittal
> > +44 7721681581
> >
> >
> > On Sat, Jun 8, 2024 at 2:09 AM Sophie Blee-Goldman <
> sop...@responsive.dev>
> > wrote:
> >
> > > Thanks for jumping on this guys, and nice find Haruki
> > >
> > > I agree with Luke that we should disable the offending tests ASAP to
> > > unblock other things, and file a 3.8 blocker to investigate this
> further.
> > > Thanks for the PR Haruki and thanks for filing the ticket Luke -- I
> > marked
> > > it as a blocker for 3.8.0 so this doesn't slip.
> > >
> > > By the way -- someone else was looking into this on the side and
> > mentioned
> > > another test they were suspicious of as well. It's
> > >
> > >
> >
> SaslClientsWithInvalidCredentialsTest.testKafkaAdminClientWithAuthenticationFailure
> > >
> > > Does one of you have time to look into this test as well? Don't want to
> > > have to wait another 8 hours for your PR to hit the timeout and abort
> > > again. If not, I can spend a minute looking at this other test a bit
> > later
> > > tonight
> > >
> > > On Fri, Jun 7, 2024 at 6:05 PM Haruki Okada 
> wrote:
> > >
> > > > Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
> > > > Let's see if CI runs properly.
> > > >
> > > > Thanks,
> > > >
> > > > 2024年6月8日(土) 10:01 Luke Chen :
> > > >
> > > > > > Let's disable for now to unblock builds, and revert later if we
> > can't
> > > > > solve
> > > > > it until code freeze?
> > > > >
> > > > > That's exactly what I meant to do.
> > > > > I've opened KAFKA-16916 <
> > > > https://issues.apache.org/jira/browse/KAFKA-16916
> > > > > >
> > > > > for this issue and assigned to you.
> > > > > Welcome to unassign yourselves if you don't have time to fix the
> > > > > adminClient behavior change issue.
> > > > > But, let's disable it first.
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > >
> > > > > On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada 
> > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > I see, but since this is likely due to AdminClient's behavior
> > change,
> > > > we
> > > > > > need to fix it anyways not only disabling test before 3.8
> release.
> > > > > > Let's disable for now to unblock builds, and revert later if we
> > can't
> > > > > solve
> > > > > > it until code freeze?
> > > > > >
> > > > > > 2024年6月8日(土) 9:31 Luke Chen :
> > > > > >
> > > > > > > Hi Haruki,
> > > > > > >
> > > > > > > Thanks for identifying this blocking test.
> > > > > > > Could you help quickly open a PR to disable this test to
> unblock
> > > the
> > > > CI
> > > > > > > build?
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada <
> ocadar...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > I found that the hanging can be reproduced locally.
> > > > > > > > The blocking test is
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> > > > > > > > It started to block after this commit (
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> > > > > > > > )
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > 2024年6月8日(土) 8:30 Sophie Blee-Goldman  >:
> > > > > > > >
> > > > > > > > > Seems like the build is currently broken -- specifically, a
> > > test
> > > > is
> > > > > > > > hanging
> > > > > > > > > and causing it to abort after 7+ hours. There are many
> > examples
> > > > in
> > > > > > the
> > > > > > > > > current PRs, such as
> > > > > > > > >
> > > > > > > > > Timed out after almost 8 hours:
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/det

Re: Build hanging

[Apoorv Mittal]

Hi Sophie,
Yes the PR should also fix the tests in other class you mentioned. I have 
validated locally and attached details in the PR for successful run on local.

- Regards,
Apoorv Mittal

> On 8 Jun 2024, at 02:24, Sophie Blee-Goldman  wrote:
> 
> Thanks for the quick fix!
> 
> By the way, I took a quick look at the output for some aborted builds and
> can confirm that this test is also hanging: SaslClientsWithInvalidCre
> dentialsTest.testKafkaAdminClientWithAuthenticationFailure
> 
> It sounds like it probably has the same root cause as the test we already
> disabled, but just to make sure, Apoorv can you confirm that your fix will
> also address the SaslClientsWithInvalidCredentialsTest?
> 
>> On Fri, Jun 7, 2024 at 6:18 PM Haruki Okada  wrote:
>> 
>> Oh, thank you for identifying the root cause quickly Apoorv.
>> 
>> I changed the assignee of KAFKA-16916 to you.
>> 
>> 
>> Thanks,
>> 
>> 2024年6月8日(土) 10:13 Apoorv Mittal :
>> 
>>> Hi,
>>> Please find the fix for the issue:
>>> https://github.com/apache/kafka/pull/16249
>>> 
>>> Regards,
>>> Apoorv Mittal
>>> +44 7721681581
>>> 
>>> 
>>> On Sat, Jun 8, 2024 at 2:09 AM Sophie Blee-Goldman <
>> sop...@responsive.dev>
>>> wrote:
>>> 
 Thanks for jumping on this guys, and nice find Haruki
 
 I agree with Luke that we should disable the offending tests ASAP to
 unblock other things, and file a 3.8 blocker to investigate this
>> further.
 Thanks for the PR Haruki and thanks for filing the ticket Luke -- I
>>> marked
 it as a blocker for 3.8.0 so this doesn't slip.
 
 By the way -- someone else was looking into this on the side and
>>> mentioned
 another test they were suspicious of as well. It's
 
 
>>> 
>> SaslClientsWithInvalidCredentialsTest.testKafkaAdminClientWithAuthenticationFailure
 
 Does one of you have time to look into this test as well? Don't want to
 have to wait another 8 hours for your PR to hit the timeout and abort
 again. If not, I can spend a minute looking at this other test a bit
>>> later
 tonight
 
 On Fri, Jun 7, 2024 at 6:05 PM Haruki Okada 
>> wrote:
 
> Opened the PR h(ttps://github.com/apache/kafka/pull/16248)
> Let's see if CI runs properly.
> 
> Thanks,
> 
> 2024年6月8日(土) 10:01 Luke Chen :
> 
>>> Let's disable for now to unblock builds, and revert later if we
>>> can't
>> solve
>> it until code freeze?
>> 
>> That's exactly what I meant to do.
>> I've opened KAFKA-16916 <
> https://issues.apache.org/jira/browse/KAFKA-16916
>>> 
>> for this issue and assigned to you.
>> Welcome to unassign yourselves if you don't have time to fix the
>> adminClient behavior change issue.
>> But, let's disable it first.
>> 
>> Thanks.
>> Luke
>> 
>> 
>> On Sat, Jun 8, 2024 at 8:55 AM Haruki Okada 
 wrote:
>> 
>>> Hi Luke,
>>> 
>>> I see, but since this is likely due to AdminClient's behavior
>>> change,
> we
>>> need to fix it anyways not only disabling test before 3.8
>> release.
>>> Let's disable for now to unblock builds, and revert later if we
>>> can't
>> solve
>>> it until code freeze?
>>> 
>>> 2024年6月8日(土) 9:31 Luke Chen :
>>> 
 Hi Haruki,
 
 Thanks for identifying this blocking test.
 Could you help quickly open a PR to disable this test to
>> unblock
 the
> CI
 build?
 
 Thanks.
 Luke
 
 On Sat, Jun 8, 2024 at 8:20 AM Haruki Okada <
>> ocadar...@gmail.com
 
>> wrote:
 
> Hi
> 
> I found that the hanging can be reproduced locally.
> The blocking test is
> 
> 
 
>>> 
>> 
> 
 
>>> 
>> "org.apache.kafka.common.security.authenticator.ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials".
> It started to block after this commit (
> 
> 
 
>>> 
>> 
> 
 
>>> 
>> https://github.com/apache/kafka/commit/c01279b92acefd9135089588319910bac79bfd4c
> )
> 
> Thanks,
> 
> 2024年6月8日(土) 8:30 Sophie Blee-Goldman >> :
> 
>> Seems like the build is currently broken -- specifically, a
 test
> is
> hanging
>> and causing it to abort after 7+ hours. There are many
>>> examples
> in
>>> the
>> current PRs, such as
>> 
>> Timed out after almost 8 hours:
>> 1.
>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16238/1/pipeline/
>> 2.
>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16201/15/pipeline
>> 
>> Still running after 6+ hours:
>

[jira] [Created] (KAFKA-16917) Align the returned map type of KafkaAdminClient

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16917:
--

 Summary: Align the returned map type of KafkaAdminClient
 Key: KAFKA-16917
 URL: https://issues.apache.org/jira/browse/KAFKA-16917
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This jira includes following issues:
 # KafkaAdminClient#alterClientQuotas [0] return a immutable map, but other 
APIs return `HashMap`.
 # KafkaAdminClient#describeConfigs [1] has duplicate copy. the returned stuff 
from `Collectors.toMap` is a new HashMap already.

 

[0] 
[https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L4074]
[1] 
[https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L2763]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16916) ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will run forever

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16916.
---
Resolution: Fixed

> ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will 
> run forever
> --
>
> Key: KAFKA-16916
> URL: https://issues.apache.org/jira/browse/KAFKA-16916
> Project: Kafka
>  Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Apoorv Mittal
>Priority: Blocker
> Fix For: 3.8.0
>
>
> ClientAuthenticationFailureTest.testAdminClientWithInvalidCredentials will 
> run forever



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16918) TestUtils#assertFutureThrows should use future.get with timeout

[Luke Chen (Jira)]

Luke Chen created KAFKA-16918:
-

 Summary: TestUtils#assertFutureThrows should use future.get with 
timeout
 Key: KAFKA-16918
 URL: https://issues.apache.org/jira/browse/KAFKA-16918
 Project: Kafka
  Issue Type: Test
Reporter: Luke Chen


In KAFKA-16916, we had a test running forever. To avoid this issue happened 
again, we can use future.get with timeout in TestUtils#assertFutureThrows.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)