Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2133

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 305229 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAddTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() PASSED

Gradle 

[jira] [Created] (KAFKA-15400) Fix flaky RemoteIndexCache test

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15400:


 Summary: Fix flaky RemoteIndexCache test
 Key: KAFKA-15400
 URL: https://issues.apache.org/jira/browse/KAFKA-15400
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash


Build / JDK 8 and Scala 2.12 / testRemoveMultipleItems() – 
kafka.log.remote.RemoteIndexCacheTest


{noformat}
Errorjava.nio.file.NoSuchFileException: 
/tmp/kafka-RemoteIndexCacheTest2682821178858144340/tF88YIi7QG-EDPMx8jJfQA:foo-0/2147584984_axAb7u74Q02X0ySdo7Hjbw.txnindexStacktracejava.nio.file.NoSuchFileException:
 
/tmp/kafka-RemoteIndexCacheTest2682821178858144340/tF88YIi7QG-EDPMx8jJfQA:foo-0/2147584984_axAb7u74Q02X0ySdo7Hjbw.txnindex
   at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)   at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)at 
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
   at 
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at 
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
   at java.nio.file.Files.readAttributes(Files.java:1737)  at 
java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)  at 
java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)  at 
java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)   at 
java.nio.file.Files.walkFileTree(Files.java:2706)at 
java.nio.file.Files.walkFileTree(Files.java:2742)at 
org.apache.kafka.common.utils.Utils.delete(Utils.java:899)   at 
kafka.log.remote.RemoteIndexCacheTest.cleanup(RemoteIndexCacheTest.scala:94) at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
   at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
  at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
 at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
  at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:110)
  at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
  at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
 at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
 at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
   at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
   at 
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
   at 
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$24(ClassBasedTestDescriptor.java:510)
   at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:243)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:276)
   at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:276)
   at java.util.ArrayList.forEach(ArrayList.java:1259) at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:275)
 at 

[jira] [Created] (KAFKA-15399) Enable OffloadAndConsumeFromLeader test

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15399:


 Summary: Enable OffloadAndConsumeFromLeader test
 Key: KAFKA-15399
 URL: https://issues.apache.org/jira/browse/KAFKA-15399
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash


Build / JDK 17 and Scala 2.13 / initializationError – 
org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14780) Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14780.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay 
> deterministic
> 
>
> Key: KAFKA-14780
> URL: https://issues.apache.org/jira/browse/KAFKA-14780
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Alexandre Dupriez
>Assignee: Alexandre Dupriez
>Priority: Minor
> Fix For: 3.6.0
>
>
> The test {{RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay}} 
> relies on the actual system clock which makes it frequently fail on my poor 
> intellij setup.
>  
> The {{{}RefreshingHttpsJwks{}}}`component creates and uses a scheduled 
> executor service. We could expose the scheduling mechanism to be able to mock 
> its behaviour. One way to do could be to use the {{KafkaScheduler}} which has 
> a {{MockScheduler}} implementation which relies on {{MockTime}} instead of 
> the real time clock.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Roger Hoover]

Thanks.  I like that you're moving Kafka toward supporting this dual-write
pattern.  Each use case needs to consider the tradeoffs.  You already
summarized the pros very well in the KIP.  I would summarize the cons
as follows:

- you sacrifice availability - each write requires both DB and Kafka to be
available so I think your overall application availability is 1 - p(DB is
unavailable)*p(Kafka is unavailable).
- latency will be higher and throughput lower - each write requires both
writes to DB and Kafka while holding an exclusive lock in DB.
- you need to create a producer per unit of concurrency in your app which
has some overhead in the app and Kafka side (number of connections, poor
batching).  I assume the producers would need to be configured for low
latency (linger.ms=0)
- there's some complexity in managing stable transactional ids for each
producer/concurrency unit in your application.  With k8s deployment, you
may need to switch to something like a StatefulSet that gives each pod a
stable identity across restarts.  On top of that pod identity which you can
use as a prefix, you then assign unique transactional ids to each
concurrency unit (thread/goroutine).

On Wed, Aug 23, 2023 at 12:53 PM Artem Livshits
 wrote:

> Hi Roger,
>
> Thank you for the feedback.  You make a very good point that we also
> discussed internally.  Adding support for multiple concurrent
> transactions in one producer could be valuable but it seems to be a fairly
> large and independent change that would deserve a separate KIP.  If such
> support is added we could modify 2PC functionality to incorporate that.
>
> > Maybe not too bad but a bit of pain to manage these ids inside each
> process and across all application processes.
>
> I'm not sure if supporting multiple transactions in one producer would make
> id management simpler: we'd need to store a piece of data per transaction,
> so whether it's N producers with a single transaction or N transactions
> with a single producer, it's still roughly the same amount of data to
> manage.  In fact, managing transactional ids (current proposal) might be
> easier, because the id is controlled by the application and it knows how to
> complete the transaction after crash / restart; while a TID would be
> generated by Kafka and that would create a question of starting Kafka
> transaction, but not saving its TID and then crashing, then figuring out
> which transactions to abort and etc.
>
> > 2) creating a separate producer for each concurrency slot in the
> application
>
> This is a very valid concern.  Maybe we'd need to have some multiplexing of
> transactional logical "streams" over the same connection.  Seems like a
> separate KIP, though.
>
> > Otherwise, it seems you're left with single-threaded model per
> application process?
>
> That's a fair assessment.  Not necessarily exactly single-threaded per
> application, but a single producer per thread model (i.e. an application
> could have a pool of threads + producers to increase concurrency).
>
> -Artem
>
> On Tue, Aug 22, 2023 at 7:22 PM Roger Hoover 
> wrote:
>
> > Artem,
> >
> > Thanks for the reply.
> >
> > If I understand correctly, Kafka does not support concurrent transactions
> > from the same producer (transactional id).  I think this means that
> > applications that want to support in-process concurrency (say
> thread-level
> > concurrency with row-level DB locking) would need to manage separate
> > transactional ids and producers per thread and then store txn state
> > accordingly.   The potential usability downsides I see are
> > 1) managing a set of transactional ids for each application process that
> > scales up to it's max concurrency.  Maybe not too bad but a bit of pain
> to
> > manage these ids inside each process and across all application
> processes.
> > 2) creating a separate producer for each concurrency slot in the
> > application - this could create a lot more producers and resultant
> > connections to Kafka than the typical model of a single producer per
> > process.
> >
> > Otherwise, it seems you're left with single-threaded model per
> application
> > process?
> >
> > Thanks,
> >
> > Roger
> >
> > On Tue, Aug 22, 2023 at 5:11 PM Artem Livshits
> >  wrote:
> >
> > > Hi Roger, Arjun,
> > >
> > > Thank you for the questions.
> > > > It looks like the application must have stable transactional ids over
> > > time?
> > >
> > > The transactional id should uniquely identify a producer instance and
> > needs
> > > to be stable across the restarts.  If the transactional id is not
> stable
> > > across restarts, then zombie messages from a previous incarnation of
> the
> > > producer may violate atomicity.  If there are 2 producer instances
> > > concurrently producing data with the same transactional id, they are
> > going
> > > to constantly fence each other and most likely make little or no
> > progress.
> > >
> > > The name might be a little bit confusing as it may be mistaken for a
> > > transaction 

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Artem Livshits]

Hi Roger,

Thank you for the feedback.  You make a very good point that we also
discussed internally.  Adding support for multiple concurrent
transactions in one producer could be valuable but it seems to be a fairly
large and independent change that would deserve a separate KIP.  If such
support is added we could modify 2PC functionality to incorporate that.

> Maybe not too bad but a bit of pain to manage these ids inside each
process and across all application processes.

I'm not sure if supporting multiple transactions in one producer would make
id management simpler: we'd need to store a piece of data per transaction,
so whether it's N producers with a single transaction or N transactions
with a single producer, it's still roughly the same amount of data to
manage.  In fact, managing transactional ids (current proposal) might be
easier, because the id is controlled by the application and it knows how to
complete the transaction after crash / restart; while a TID would be
generated by Kafka and that would create a question of starting Kafka
transaction, but not saving its TID and then crashing, then figuring out
which transactions to abort and etc.

> 2) creating a separate producer for each concurrency slot in the
application

This is a very valid concern.  Maybe we'd need to have some multiplexing of
transactional logical "streams" over the same connection.  Seems like a
separate KIP, though.

> Otherwise, it seems you're left with single-threaded model per
application process?

That's a fair assessment.  Not necessarily exactly single-threaded per
application, but a single producer per thread model (i.e. an application
could have a pool of threads + producers to increase concurrency).

-Artem

On Tue, Aug 22, 2023 at 7:22 PM Roger Hoover  wrote:

> Artem,
>
> Thanks for the reply.
>
> If I understand correctly, Kafka does not support concurrent transactions
> from the same producer (transactional id).  I think this means that
> applications that want to support in-process concurrency (say thread-level
> concurrency with row-level DB locking) would need to manage separate
> transactional ids and producers per thread and then store txn state
> accordingly.   The potential usability downsides I see are
> 1) managing a set of transactional ids for each application process that
> scales up to it's max concurrency.  Maybe not too bad but a bit of pain to
> manage these ids inside each process and across all application processes.
> 2) creating a separate producer for each concurrency slot in the
> application - this could create a lot more producers and resultant
> connections to Kafka than the typical model of a single producer per
> process.
>
> Otherwise, it seems you're left with single-threaded model per application
> process?
>
> Thanks,
>
> Roger
>
> On Tue, Aug 22, 2023 at 5:11 PM Artem Livshits
>  wrote:
>
> > Hi Roger, Arjun,
> >
> > Thank you for the questions.
> > > It looks like the application must have stable transactional ids over
> > time?
> >
> > The transactional id should uniquely identify a producer instance and
> needs
> > to be stable across the restarts.  If the transactional id is not stable
> > across restarts, then zombie messages from a previous incarnation of the
> > producer may violate atomicity.  If there are 2 producer instances
> > concurrently producing data with the same transactional id, they are
> going
> > to constantly fence each other and most likely make little or no
> progress.
> >
> > The name might be a little bit confusing as it may be mistaken for a
> > transaction id / TID that uniquely identifies every transaction.  The
> name
> > and the semantics were defined in the original exactly-once-semantics
> (EoS)
> > proposal (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
> > )
> > and KIP-939 just build on top of that.
> >
> > > I'm curious to understand what happens if the producer dies, and does
> not
> > come up and recover the pending transaction within the transaction
> timeout
> > interval.
> >
> > If the producer / application never comes back, the transaction will
> remain
> > in prepared (a.k.a. "in-doubt") state until an operator forcefully
> > terminates the transaction.  That's why there is a new ACL is defined in
> > this proposal -- this functionality should only provided to applications
> > that implement proper recovery logic.
> >
> > -Artem
> >
> > On Tue, Aug 22, 2023 at 12:52 AM Arjun Satish 
> > wrote:
> >
> > > Hello Artem,
> > >
> > > Thanks for the KIP.
> > >
> > > I have the same question as Roger on concurrent writes, and an
> additional
> > > one on consumer behavior. Typically, transactions will timeout if not
> > > committed within some time interval. With the proposed changes in this
> > KIP,
> > > consumers cannot consume past the ongoing transaction. I'm curious to
> > > understand what happens if the producer dies, and does not come up and
> > > recover the 

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Greg Harris]

Hey Jack,

The design of this KIP is also consistent with the way header support
was added to Connect:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-440%3A+Extend+Connect+Converter+to+support+headers
I think making argument for precedent here is reasonable.

Hi Ismael,

Can you expand what you mean by "without breaking compatibility"? I
think the approach proposed here (a default method) would be backwards
compatible. If an implementation wishes to make use of the new
signature, they can override the new method and the version of Kafka
will determine which implementation is used without instance checking,
reflection, or exceptions.

I believe that when you pass a DTO, that some sort of instance
checking, reflection, or exceptions would be required for the
Partitioner to determine whether additional information is present.
For example, if we wished to add some information X to the partitioner
in the future, the caller could pass either a `PartitionInfo` or
`PartitionInfoWithX` DTO instance, and the callee could use an
`instanceof` check and a cast before accessing the X information. That
seems to be more machinery for the Partitioner implementation to
manage as compared to maintaining multiple methods, which may just be
one-line calls to other methods.

Please let me know if I've misunderstood your DTO design.

Thanks!
Greg

On Tue, Aug 22, 2023 at 9:33 PM Jack Tomy  wrote:
>
> Hi Ismael,
>
> That would be totally different from the pattern currently being followed
> in all the interfaces, for example serializer.
> I personally don't favour that either. Let's see if the community has any
> opinions on the same.
>
> Hey everyone, please share your thoughts on using a DTO instead of separate
> params for the interface.
>
> Thanks.
>
> On Mon, Aug 21, 2023 at 8:06 PM Ismael Juma  wrote:
>
> > Hi Jack,
> >
> > I mean a DTO. That means you can add additional parameters later without
> > breaking compatibility. The current proposal would result in yet another
> > method each time we need to add parameters.
> >
> > Ismael
> >
> > On Sun, Aug 20, 2023 at 4:53 AM Jack Tomy  wrote:
> >
> > > Hey Ismael,
> > >
> > > Are you suggesting to pass a param like a DTO or you are suggesting to
> > pass
> > > the record object?
> > >
> > > I would also like to hear other devs' opinions on this as I personally
> > > favour what is done currently.
> > >
> > > On Thu, Aug 17, 2023 at 9:34 AM Ismael Juma  wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for the KIP. The problem outlined here is a great example why we
> > > > should be using a record-like structure to pass the parameters to a
> > > method
> > > > like this. Then we can add more parameters without having to introduce
> > > new
> > > > methods. Have we considered this option?
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 7, 2023 at 5:26 AM Jack Tomy 
> > wrote:
> > > >
> > > > > Hey everyone.
> > > > >
> > > > > I would like to call for a vote on KIP-953: partition method to be
> > > > > overloaded to accept headers as well.
> > > > >
> > > > > KIP :
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424937
> > > > > Discussion thread :
> > > > > https://lists.apache.org/thread/0f20kvfqkmhdqrwcb8vqgqn80szcrcdd
> > > > >
> > > > > Thanks
> > > > > --
> > > > > Best Regards
> > > > > *Jack*
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > > *Jack*
> > >
> >
>
>
> --
> Best Regards
> *Jack*

[jira] [Created] (KAFKA-15398) Document Connect threat model

[Greg Harris (Jira)]

Greg Harris created KAFKA-15398:
---

 Summary: Document Connect threat model
 Key: KAFKA-15398
 URL: https://issues.apache.org/jira/browse/KAFKA-15398
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Greg Harris


Kafka Connect is a plugin framework, regularly requiring the installation of 
third-party code. This poses a security hazard for operators, who may be 
compromised by actively malicious plugins or well-intentioned plugins which are 
exploitable.

We should document the threat model that the Connect architecture uses, and 
make it clear to operators what trust and verification is required in order to 
operate Connect safely.

At a high level, this documentation may include:
 # Plugins are arbitrary code with unrestricted access to the filesystem, 
secrets, and network resources of the hosting Connect worker
 # The filesystem of the worker is trusted
 # Connector configurations passed via REST API are trusted
 # Plugins may have exploits triggered by certain configurations, or by 
external connections.
 # Exploits may also be present in plugins/drivers/dependencies used by Connect 
plugins, such as JDBC drivers
 # The default installation without REST API security is exploitable when run 
on an untrusted network.

Documenting this security model will also make it easier to discuss changing 
the model and improving the security architecture of Connect in the future.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

[Sagar]

Thanks Yash. LGTM

Thanks!
Sagar.

On Tue, Aug 22, 2023 at 6:04 PM Chris Egerton 
wrote:

> Hi Yash,
>
> Thanks for driving this, and for putting out a well-written KIP. LGTM!
>
> Cheers,
>
> Chris
>
> On Tue, Aug 22, 2023 at 6:13 AM Yash Mayya  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion thread for this KIP -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Deprecate+and+remove+Connect%27s+redundant+task+configurations+endpoint
> > .
> >
> > It proposes the deprecation and eventual removal of Kafka Connect's
> > redundant task configurations endpoint.
> >
> > Thanks,
> > Yash
> >
>

[jira] [Created] (KAFKA-15397) Deserializing produce requests may cause memory leaks when exceptions occur

[hudeqi (Jira)]

hudeqi created KAFKA-15397:
--

 Summary: Deserializing produce requests may cause memory leaks 
when exceptions occur
 Key: KAFKA-15397
 URL: https://issues.apache.org/jira/browse/KAFKA-15397
 Project: Kafka
  Issue Type: Bug
  Components: clients, core
Affects Versions: 3.5.1
Reporter: hudeqi
Assignee: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15396) Add a metric indicating the version of the current running kafka server

[hudeqi (Jira)]

hudeqi created KAFKA-15396:
--

 Summary: Add a metric indicating the version of the current 
running kafka server
 Key: KAFKA-15396
 URL: https://issues.apache.org/jira/browse/KAFKA-15396
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.5.1
Reporter: hudeqi
Assignee: hudeqi


At present, it is impossible to perceive the Kafka version that the broker is 
running from the perspective of metrics. If multiple Kafka versions are 
deployed in a cluster due to various reasons, it is difficult for us to 
intuitively understand the version distribution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [DISCUSS] KIP-939: Support Participation in 2PC

[guy]

Hi,

Nice idea, but you could maximise compatibility if you adhere to XA standard 
APIs rather than Kafka internal APIs.

We at Atomikos offer 2PC coordination and recovery and we are happy to help you 
design this, it's a service we usually offer for free to backend vendors / 
systems.

Let me know if you'd like to explore?

Guy


On 2023/08/17 06:39:57 Artem Livshits wrote:
> Hello,
>
>  This is a discussion thread for
>  
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC
>  .
>
>  The KIP proposes extending Kafka transaction support (that already uses 2PC
>  under the hood) to enable atomicity of dual writes to Kafka and an external
>  database, and helps to fix a long standing Flink issue.
>
>  An example of code that uses the dual write recipe with JDBC and should
>  work for most SQL databases is here
>  https://github.com/apache/kafka/pull/14231.
>
>  The FLIP for the sister fix in Flink is here
>  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710
>
>  -Artem


Sent with Spark

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2132

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 305483 lines...]
Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAddTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldUnassignTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > shouldUnassignTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
DefaultTaskManagerTest > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2131

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 327304 lines...]
Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAddTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAddTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotAssignAnyLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldRemoveTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveAssignedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldAssignTaskThatCanBeProcessed() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldNotRemoveUnlockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldReturnAndClearExceptionsOnDrainExceptions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > shouldUnassignTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 84 > 
DefaultTaskManagerTest > 

[jira] [Created] (KAFKA-15395) what's the default password for all the super user created in server.properties

[acharbha (Jira)]

acharbha created KAFKA-15395:


 Summary: what's the default password for all the super user 
created in server.properties
 Key: KAFKA-15395
 URL: https://issues.apache.org/jira/browse/KAFKA-15395
 Project: Kafka
  Issue Type: Test
  Components: security
Reporter: acharbha


what's the default password for all the super user created in server.properties
as we don't specify the password for the user anywhere:

cat opt/bitnami/kafka/config/server.properties | grep 
supersuper.users=User:admin;User:controller_user;User:dbaasadmin

how to check/test whether these users are having super user access or not? 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15394) Issue with Kafka ACLs: Unexpected Permissions for User

[acharbha (Jira)]

acharbha created KAFKA-15394:


 Summary: Issue with Kafka ACLs: Unexpected Permissions for User
 Key: KAFKA-15394
 URL: https://issues.apache.org/jira/browse/KAFKA-15394
 Project: Kafka
  Issue Type: Bug
  Components: security
 Environment: we are running Kafka on a Kubernetes cluster using helm..
Reporter: acharbha


Hello Community,

I'm facing an unexpected situation while working with Kafka ACLs. Despite 
having provided only read access permissions to the user "rbactest22sep," I've 
noticed that this user is still able to add ACLs for Kafka topics. Here are the 
details:

User: rbactest22sep

Current Permissions: Principal=User:rbactest22sep, Host=*, Operation=READ, 
PermissionType=ALLOW

I attempted to add an ACL for topic creation using the following command:
|kafka-acls{*}.{*}sh {*}--{*}bootstrap-server 
broker1{*}:{*}{*}9095{*}{*},{*}broker2{*}:{*}{*}9095{*}{*},{*}broker3{*}:{*}{*}9095{*}
 {*}--{*}command-config 
{*}/{*}bitnami{*}/{*}kafka{*}/{*}config{*}/{*}rbacuser{*}.{*}properties 
{*}--{*}add {*}--{*}allow-principal User{*}:{*}rbactest22sep {*}--{*}operation 
create {*}--{*}topic '*'
Adding ACLs *for* resource 
`ResourcePattern{*}({*}resourceType{*}={*}TOPIC{*},{*} name{*}=*,{*} 
patternType{*}={*}LITERAL{*}){*}`{*}:{*}
   {*}({*}principal{*}={*}User{*}:{*}rbactest22sep{*},{*} host{*}=*,{*} 
operation{*}={*}CREATE{*},{*} permissionType{*}={*}ALLOW{*}){*}|

 

Where content of /bitnami/kafka/config/rbacuser.properties as below:
|security{*}.{*}protocol{*}={*}SASL_SSL
sasl{*}.{*}mechanism{*}={*}SCRAM-SHA-256
sasl{*}.{*}jaas{*}.{*}config{*}={*}org{*}.{*}apache{*}.{*}kafka{*}.{*}common{*}.{*}security{*}.{*}scram{*}.{*}ScramLoginModule
 required username{*}={*}"rbactest22sep" password{*}={*}"mypass"{*};{*}
ssl{*}.{*}truststore{*}.{*}{*}type{*}{*}={*}JKS
ssl{*}.{*}truststore{*}.{*}location{*}=/{*}opt{*}/{*}bitnami{*}/{*}kafka{*}/{*}config{*}/{*}certs{*}/{*}kafka{*}.{*}truststore{*}.{*}jks
# Uncomment this line if your client truststore is password protected
ssl{*}.{*}truststore{*}.{*}password{*}={*}trustpass{*}.{*}com|

 

The command executed successfully, and the user gained the ability to create 
topics.

I'm puzzled by this behavior and would appreciate your insights into why this 
might be happening. Could this be related to Kafka configuration, ACL 
inheritance, or a misunderstanding of the permissions model?

Also, I'm under the assumption that we need to explicitly give the following 
permissions to a user to manage ACLs:

 
|DESCRIBE_ACLS {*}({*}{*}29{*}{*}){*} Describe Cluster
CREATE_ACLS {*}({*}{*}30{*}{*}){*} Alter Cluster
DELETE_ACLS {*}({*}{*}31{*}{*}){*} Alter Cluster|

Any guidance on how to troubleshoot and resolve this issue, as well as any 
clarifications on the necessary permissions for managing ACLs, would be greatly 
appreciated.

 

Thank you for your help!

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [REVIEW REQUEST] Move ReassignPartitionsCommandArgsTest to java

[Luke Chen]

Hi,

Sorry that we're mostly working on features for v3.6.0, which is expected
to be released in the following weeks.
I'll review your PR after releasing. (Please ping me then if I forget it!)

Also, it'd be good if the devs in the community can help on PR review when
available.
That'll help a lot.
Besides, PR review is also one kind of contribution, not just code
commitment.

Thanks.
Luke



On Tue, Aug 22, 2023 at 7:15 PM Николай Ижиков  wrote:

> Hello.
>
> Please, join the simple review)
> We have few steps left to completely rewrite ReassignPartitionsCommand in
> java.
>
> > 17 авг. 2023 г., в 17:16, Николай Ижиков 
> написал(а):
> >
> > Hello.
> >
> > I’m working on [1].
> > The goal of ticket is to rewire `ReassignPartitionCommand` in java.
> >
> > The PR that moves whole command is pretty big so it makes sense to split
> it.
> > I prepared the PR [2] that moves single test
> (ReassignPartitionsCommandArgsTest) to java.
> >
> > It relatively small and simple(touches only 3 files):
> >
> > To review - https://github.com/apache/kafka/pull/14217
> > Big PR  - https://github.com/apache/kafka/pull/13247
> >
> > Please, review.
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-14595
> > [2] https://github.com/apache/kafka/pull/14217
>
>

Re: Requesting permissions to contribute to Apache Kafka

[Luke Chen]

Hi Animesh,

Your accounts are all set.

Thanks.
Luke

On Tue, Aug 22, 2023 at 9:25 PM Animesh Kumar  wrote:

> Hi Team,
> Please provide access to contribute to Apache Kafka
> JIRA id -- akanimesh7
> Wiki Id -- akanimesh7
> --
> Animesh Kumar
> 8120004556
>

Issue with Kafka ACLs: Unexpected Permissions for User

[Bharath Achar]

Hello Community,



I'm facing an unexpected situation while working with Kafka ACLs. Despite 
having provided only read access permissions to the user "rbactest22sep," I've 
noticed that this user is still able to add ACLs for Kafka topics. Here are the 
details:



User: rbactest22sep

Current Permissions: Principal=User:rbactest22sep, Host=*, Operation=READ, 
PermissionType=ALLOW

I attempted to add an ACL for topic creation using the following command:



kafka-acls.sh --bootstrap-server broker1:9095,broker2:9095,broker3:9095 
--command-config /bitnami/kafka/config/rbacuser.properties --add 
--allow-principal User:rbactest22sep --operation create --topic '*'

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, 
patternType=LITERAL)`:

   (principal=User:rbactest22sep, host=*, operation=CREATE, 
permissionType=ALLOW)



Where content of /bitnami/kafka/config/rbacuser.properties as below:



security.protocol=SASL_SSL

sasl.mechanism=SCRAM-SHA-256

sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule 
required username="rbactest22sep" password="mypass";

ssl.truststore.type=JKS

ssl.truststore.location=/opt/bitnami/kafka/config/certs/kafka.truststore.jks

# Uncomment this line if your client truststore is password protected

ssl.truststore.password=trustpass.com



The command executed successfully, and the user gained the ability to create 
topics.



I'm puzzled by this behavior and would appreciate your insights into why this 
might be happening. Could this be related to Kafka configuration, ACL 
inheritance, or a misunderstanding of the permissions model?



Also, I'm under the assumption that we need to explicitly give the following 
permissions to a user to manage ACLs:



DESCRIBE_ACLS (29) Describe Cluster

CREATE_ACLS (30) Alter Cluster

DELETE_ACLS (31) Alter Cluster



Any guidance on how to troubleshoot and resolve this issue, as well as any 
clarifications on the necessary permissions for managing ACLs, would be greatly 
appreciated.



Thank you for your help!






--

Thanks,
Bharath Achar
IT Infrastructure and DevOps Engineer

Intel India pvt ltd, Bangalore.
Cell  ::  + 91 7259670196

bharath_ac...@outlook.com
--