Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.3 #154

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14713) Kafka Streams global table startup takes too long

[Tamas (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tamas resolved KAFKA-14713.
---
Fix Version/s: 3.4.0
   Resolution: Invalid

> Kafka Streams global table startup takes too long
> -
>
> Key: KAFKA-14713
> URL: https://issues.apache.org/jira/browse/KAFKA-14713
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.0.2
>Reporter: Tamas
>Priority: Critical
> Fix For: 3.4.0
>
>
> *Some context first*
> We have a spring based kafka streams application. This application is 
> listening to two topics. Let's call them apartment and visitor. The 
> apartments are stored in a global table, while the visitors are in the stream 
> we are processing, and at one point we are joining the visitor stream 
> together with the apartment table. In our test environment, both topics 
> contain 10 partitions.
> *Issue*
> At first deployment, everything goes fine, the global table is built and all 
> entries in the stream are processed.
> After everything is finished, we shut down the application, restart it and 
> send out a new set of visitors. The application seemingly does not respond.
> After some more debugging it turned out that it simply takes 5 minutes to 
> start up, because the global table takes 30 seconds (default value for the 
> global request timeout) to accept that there are no messages in the apartment 
> topics, for each and every partition. If we send out the list of apartments 
> as new messages, the application starts up immediately.
> To make matters worse, we have clients with 96 partitions, where the startup 
> time would be 48 minutes. Not having messages in the topics between 
> application shutdown and restart is a valid use case, so this is quite a big 
> problem.
> *Possible workarounds*
> We could reduce the request timeout, but since this value is not specific for 
> the global table initialization, but a global request timeout for a lot of 
> things, we do not know what else it will affect, so we are not very keen on 
> doing that. Even then, it would mean a 1.5 minute delay for this particular 
> client (more if we will have other use cases in the future where we will need 
> to use more global tables), which is far too much, considering that the 
> application would be able to otherwise start in about 20 seconds.
> *Potential solutions we see*
>  # Introduce a specific global table initialization timeout in 
> GlobalStateManagerImpl. Then we would be able to safely modify that value 
> without fear of making some other part of kafka unstable.
>  # Parallelize the initialization of the global table partitions in 
> GlobalStateManagerImpl: knowing that the delay at startup is constant instead 
> of linear with the number of partitions would be a huge help.
>  # As long as we receive a response, accept the empty map in the 
> KafkaConsumer, and continue instead of going into a busy-waiting loop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14726) Move/rewrite LogReadInfo, LogOffsetSnapshot, and LogStartOffsetIncrementReason to storage module.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-14726:
--

 Summary: Move/rewrite LogReadInfo, LogOffsetSnapshot, and 
LogStartOffsetIncrementReason to storage module.
 Key: KAFKA-14726
 URL: https://issues.apache.org/jira/browse/KAFKA-14726
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Satish Duggana
Assignee: Satish Duggana


Move/rewrite LogReadInfo, LogOffsetSnapshot, and LogStartOffsetIncrementReason 
to storage module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1588

[Apache Jenkins Server]

See 

Remote code execution flaw patched in Apache Kafka

[Turritopsis Dohrnii Teo En Ming]

Subject: Remote code execution flaw patched in Apache Kafka

Good day from Singapore,

Sharing this article. Hot off the press.

Article: Remote code execution flaw patched in Apache Kafka

Link: 
https://portswigger.net/daily-swig/remote-code-execution-flaw-patched-in-apache-kafka

Thank you.

Regards,

Mr. Turritopsis Dohrnii Teo En Ming
Targeted Individual in Singapore
Blogs:
https://tdtemcerts.blogspot.com
https://tdtemcerts.wordpress.com

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Chia-Ping Tsai]

On 2023/02/15 14:29:06 Ismael Juma wrote:
> Hi.
> 
> Is MessageReader the right name? The client apis tend to use the word
> `Record` instead. In fact, `readMessage` returns a `ProducerRecord`.

Make sense. “Record” is the better naming. I will update KIP to use “Record” 
instead of “message”

> 
> Also, I wonder if we it would be useful for the interface to support the
> usage of Serializers. That is, the interface could return ProducerRecord
> as long as a serializer was also configured. I am less sure about this one,
> but worth giving it some thought and if we think it's a bad idea, we should
> add it to the "Rejected Alternatives" section.

It seems to me the serializer should be a part of implementation of 
MessageReader. If the custom serializer is supported, users have to define two 
class (reader and serializer) to convert bytes (from input stream) to “correct 
records”.

I will add it to dejected alternatives

> 
> Ismael
> 
> On Wed, Feb 15, 2023 at 3:07 AM Mickael Maison 
> wrote:
> 
> > Hi Chia-Ping,
> >
> > Sorry nobody replied to your thread earlier!
> > I stumbled on this while looking at
> > https://issues.apache.org/jira/browse/KAFKA-14525 (migrating CLI tools
> > out of core). Your proposal still makes sense and would help migrating
> > the ConsoleProducer tool.
> >
> > Can we specify when we expect to delete kafka.common.MessageReader? I
> > guess in Kafka 4.0?
> > Apart from that, the KIP looks good and considering how simple it is,
> > you should be able to open a vote thread.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Jul 7, 2020 at 5:55 PM Chia-Ping Tsai  wrote:
> > >
> > > hi all,
> > >
> > > I would like to start the discussion for KIP-641.
> > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> > >
> > > Many thanks,
> > >
> > > Chia-Ping
> >
> 

Re: Kafka design docs

[Luke Chen]

Hi Arun,

The official doc should be good enough:
https://kafka.apache.org/documentation/#design
For detailed design doc, you can check the Kafka Improvement Proposals:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Thank you
Luke

On Thu, Feb 16, 2023 at 7:27 AM Arun Raju  wrote:

> Hi All,
>
> Just trying to understand if there are documents relating to high-level and
> low-level design that I can use to understand overall architecture. Please
> let me know.
>
> Arun
>

[jira] [Resolved] (KAFKA-14708) Remove kafka.examples.Consumer dependancy on ShutdownableThread

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14708.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Remove kafka.examples.Consumer dependancy on ShutdownableThread
> ---
>
> Key: KAFKA-14708
> URL: https://issues.apache.org/jira/browse/KAFKA-14708
> Project: Kafka
>  Issue Type: Task
>Reporter: Satish Duggana
>Priority: Major
> Fix For: 3.5.0
>
>
> Remove "kafka.examples.Consumer" dependency on ShutdownableThread. "examples" 
> module should be dependent only on public APIs but not to be dependent upon 
> server common/internal components. 
>  
> This is a followup item from the discussion 
> [here|https://github.com/apache/kafka/pull/13234#discussion_r1103848236]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-5756) Synchronization issue on flush

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-5756.
--
Resolution: Fixed

> Synchronization issue on flush
> --
>
> Key: KAFKA-5756
> URL: https://issues.apache.org/jira/browse/KAFKA-5756
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Oleg Kuznetsov
>Assignee: Greg Harris
>Priority: Major
> Fix For: 3.5.0
>
>
> Access to *OffsetStorageWriter#toFlush* is not synchronized in *doFlush()* 
> method, whereas this collection can be accessed from 2 different threads:
> - *WorkerSourceTask.execute()*, finally block
> - *SourceTaskOffsetCommitter*, from periodic flush task



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1587

[Apache Jenkins Server]

See 

Kafka design docs

[Arun Raju]

Hi All,

Just trying to understand if there are documents relating to high-level and
low-level design that I can use to understand overall architecture. Please
let me know.

Arun

[jira] [Resolved] (KAFKA-14664) Raft idle ratio is inaccurate

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-14664.
-
Resolution: Fixed

> Raft idle ratio is inaccurate
> -
>
> Key: KAFKA-14664
> URL: https://issues.apache.org/jira/browse/KAFKA-14664
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.5.0
>
>
> The `poll-idle-ratio-avg` metric is intended to track how idle the raft IO 
> thread is. When completely idle, it should measure 1. When saturated, it 
> should measure 0. The problem with the current measurements is that they are 
> treated equally with respect to time. For example, say we poll twice with the 
> following durations:
> Poll 1: 2s
> Poll 2: 0s
> Assume that the busy time is negligible, so 2s passes overall.
> In the first measurement, 2s is spent waiting, so we compute and record a 
> ratio of 1.0. In the second measurement, no time passes, and we record 0.0. 
> The idle ratio is then computed as the average of these two values (1.0 + 0.0 
> / 2 = 0.5), which suggests that the process was busy for 1s, which 
> overestimates the true busy time.
> Instead, we should sum up the time waiting over the full interval. 2s passes 
> total here and 2s is idle, so we should compute 1.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14725) Improve cancellation semantics for source tasks

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14725:
-

 Summary: Improve cancellation semantics for source tasks
 Key: KAFKA-14725
 URL: https://issues.apache.org/jira/browse/KAFKA-14725
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Chris Egerton


This came about during discussion on 
[https://github.com/apache/kafka/pull/13208/,] which addressed KAFKA-5756.

 

Right now, we make some effort to disable and shut down tasks that have been 
scheduled for shutdown but taken longer than the [graceful shutdown timeout 
period|https://kafka.apache.org/documentation.html#connectconfigs_task.shutdown.graceful.timeout.ms].

The logic for performing this disablement is contained in the {{cancel}} method 
for the 
[WorkerTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L129-L136]
 and its subclasses (at the time of writing, that would be the 
[AbstractWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java],
 
[WorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java],
 
[ExactlyOnceWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ExactlyOnceWorkerSourceTask.java],
 and 
[WorkerSinkTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java]
 classes). Right now we don't do much to interrupt in-progress operations, 
which may lead to zombie tasks lying around on a worker that have not yet 
relinquished resources like Kafka clients, file descriptors, or database 
connections despite being scheduled for shutdown.

We can and should make the cancellation logic for tasks more stringent, 
including but not limited to:
 * Interrupting the work thread for the task
 * Interrupting any in-progress offset commits
 * Preemptively shutting down any Kafka clients created for use by the task



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14724) Port tests in FetcherTest to FetchRequestManagerTest

[Kirk True (Jira)]

Kirk True created KAFKA-14724:
-

 Summary: Port tests in FetcherTest to FetchRequestManagerTest
 Key: KAFKA-14724
 URL: https://issues.apache.org/jira/browse/KAFKA-14724
 Project: Kafka
  Issue Type: Improvement
Reporter: Kirk True
Assignee: Kirk True


The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch 
records from the brokers. There is ongoing work to create a new consumer 
implementation with a significantly refactored threading model. The threading 
refactor work requires a similarly refactored {{{}Fetcher{}}}.

This task involves copying the relevant tests from {{FetcherTest}} and 
modifying them to fit a new unit test named {{{}FetchRequestManagerTest{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14723) Do not write expired store records to changelog

[Victoria Xia (Jira)]

Victoria Xia created KAFKA-14723:


 Summary: Do not write expired store records to changelog
 Key: KAFKA-14723
 URL: https://issues.apache.org/jira/browse/KAFKA-14723
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Victoria Xia


Window stores and versioned stores both have concepts of "retention" and 
"expiration." Records which are expired are not written to the store, e.g., 
[this 
example|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/streams/src/main/java/org/apache/kafka/streams/state/internals/AbstractRocksDBSegmentedBytesStore.java#L265-L266]
 for segments stores. However, these expired records are still written to the 
changelog topic, in the case of persistent stores. This does not cause any 
problems because the records are once again omitted from the store during 
restore, but it is inefficient. It'd be good to avoid writing expired records 
to the changelog topic in the first place. Another benefit is that doing so 
would allow us to simplify the restoration code for versioned stores (see 
[relevant 
discussion|https://github.com/apache/kafka/pull/13243#discussion_r1106568364]). 

The reason expired records are still written to the changelog topic is because 
the whether records are expired or not is only tracked at the innermost store 
layer, and not any of the outer store layers such as the changelogging layer. 
The innermost store layer keeps its own `observedStreamTime` which is advanced 
on calls to put() and during restoration, and uses this variable to determine 
when a record is expired. Because the return type from put() is void, the 
changelogging layer has no way to tell whether the inner store's put() actually 
put the record or dropped it as expired, and always writes to the changelog 
topic regardless.

In order to avoid this, we could:
 * update the put() interface to return a boolean indicating whether the record 
was actually put or not, or
 * move the logic for determining when a record is expired into an outer store 
layer, or
 * reorder/restructure the wrapped store layers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14722) Make BooleanSerde public

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-14722:
---

 Summary: Make BooleanSerde public
 Key: KAFKA-14722
 URL: https://issues.apache.org/jira/browse/KAFKA-14722
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


We introduce a "BooleanSerde via [https://github.com/apache/kafka/pull/13249] 
as internal class. We could make it public.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Need Jira Account

[Lovish Madhu]

Hi I would like to contribute to Kafka Open Source and I went through 4-5
tickets with the newbie label in the last couple of hours. But it looks
like I am not able to create jira as I am a non-ASF member. Can anyone
please help me with this?

username: lovish2404
email: lovish.madhu.2...@gmail.com
name: Lovish Madhu

Regards
Lovish

Re: Requesting permissions to contribute to Apache Kafka.

[Guozhang Wang]

Hello Zhangheng,

I've granted your permission on the Apache Kafka wiki space. You
should be able to create new KIP pages.


Guozhang

On Wed, Feb 15, 2023 at 6:02 AM hzh0425  wrote:
>
> Hello,
>
> My name is zhangheng huang. I want to start contributing to the Apache Kafka 
> project and want the permission to create KIPs
>
> JiraID: hzh0425
> WikiID: hzh0425
> email: hzh0...@apache.org
>
> Best regards,

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Chris Egerton]

Hi Tina,

It's looking better! A few thoughts:

I think we should clarify in the KIP that under these conditions, MM2 will
explicitly wipe properties from topic configs on the target cluster (via a
DELETE operation):
- The incremental API is used
- ConfigPropertyFilter::shouldReplicateConfigProperty returns true
- ConfigPropertyFilter::shouldReplicateSourceDefault returns false

I also think that extending the DefaultConfigPropertyFilter to allow users
to control which defaults (source or target) get applied on the target
cluster is worth adding to the KIP. This could be as simple as adding a
property "use.defaults.from" with accepted values "source" and "target", or
it could be something more granular like
"config.properties.source.default.exclude" which, similar to the
"config.properties.exclude" property, could take a list of regular
expressions of properties whose default values should not be propagated
from the source cluster to the target cluster (with a default value of
".*", to preserve existing behavior). I'm leaning toward keeping things
simple for now but both seem like viable options. And of course, if you
believe we should refrain from doing this, it's at least worth adding to
the rejected alternatives section.

Cheers,

Chris

On Wed, Feb 15, 2023 at 11:19 AM Gantigmaa Selenge 
wrote:

> Thank you Chris. I agree with what you have suggested.
> I have updated the KIP, please let me know if I missed anything or if there
> is any other question.
>
> Regards,
> Tina
>
> On Tue, Feb 14, 2023 at 4:40 PM Chris Egerton 
> wrote:
>
> > Hi Tina,
> >
> > While I agree that it's reasonable for users to want to favor the source
> > cluster's defaults over the target cluster's, I'm hesitant to change this
> > behavior in an opt-out fashion. IMO it's better to allow users to opt
> into
> > this (by adding a method to the ConfigPropertyFilter interface, and
> > possibly extending the DefaultConfigPropertyFilter with configuration
> > properties related to how it should handle source cluster defaults), but
> we
> > should try to preserve the existing behavior by default.
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Feb 13, 2023 at 5:10 PM Gantigmaa Selenge 
> > wrote:
> >
> > > Hi Chris
> > >
> > > My comment on the second point is not correct. Please ignore the part
> > about
> > > the config source (config source does set back to DEFAULT_CONFIG when
> > > deleting a config). I got diverted off the issue a little bit.
> > >
> > > With the legacy API, we do propagate deletion due to resetting all the
> > > configs on that target topic that are not being replicated. However
> with
> > > incrementalAlterConfigs API, this changes. If we delete a config that
> was
> > > previously altered on the source topic, the config on the target topic
> is
> > > left with the previous value as the default configs are not replicated.
> > The
> > > reason for favouring the source defaults was because it would set the
> > > config on the target topic with the source's default in this situation.
> > >
> > > Regards,
> > > Tina
> > >
> > > On Mon, Feb 13, 2023 at 8:15 AM Gantigmaa Selenge  >
> > > wrote:
> > >
> > > > Hi Chris and Luke,
> > > >
> > > > Thank you very much for your feedback.
> > > >
> > > > I have addressed some of the suggestions and would like to clarify a
> > few
> > > > things on the others:
> > > >
> > > > > 1) The current logic for syncing topic configs in MM2 is basically
> > > > fire-and-forget; all we do is log a warning message [1] if an attempt
> > > > fails. When "use.incremental.alter.configs" is set to "requested",
> > we'll
> > > > need to know whether attempts using the incremental API succeed or
> not,
> > > and
> > > > then adjust our behavior accordingly. Will the plan here be to block
> on
> > > the
> > > > success/failure of the first request before sending any more, or will
> > we
> > > > just switch over to the legacy API as soon as any request fails due
> to
> > > > targeting an incompatible broker, possibly after multiple requests
> with
> > > the
> > > > new API have been sent or at least queued up in the admin client?
> > > >
> > > > When it's set to "requested", I think the suggestion was to just
> switch
> > > > over to the legacy API as soon as any request fails due to
> > > > targeting an incompatible broker. We could keep using the legacy API
> > > until
> > > > the mirrormaker setting is changed by the user or the mirrormaker
> > > restarts
> > > > instead of trying the new API every time it syncs the configurations.
> > I'm
> > > > not sure which approach is the best here.
> > > >
> > > > > 2) We don't replicate default properties from the source cluster
> > right
> > > > now
> > > > [2].
> > > > Won't making ConfigPropertyFilter::shouldReplicateSourceDefault
> return
> > > true
> > > > by default change that behavior? If so, what's the motivation for
> > > favoring
> > > > defaults from the source cluster over defaults for the target
> cluster,
> > > > especially given that 

[jira] [Created] (KAFKA-14721) Kafka listener uses wrong login class

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-14721:


 Summary: Kafka listener uses wrong login class
 Key: KAFKA-14721
 URL: https://issues.apache.org/jira/browse/KAFKA-14721
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.1.2
Reporter: Daniel Urban


When trying to configure a single SASL_SSL listener with GSSAPI, Scram and 
OAuth support, we encounter an error at startup:
{code:java}
2023-02-15 13:26:04,250 ERROR kafka.server.KafkaServer: [main]: [KafkaServer 
id=104] Fatal error during KafkaServer startup. Prepare to shutdown
org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No 
serviceName defined in either JAAS or Kafka config
        at 
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at kafka.network.Processor.(SocketServer.scala:861) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at kafka.network.SocketServer.newProcessor(SocketServer.scala:442) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at 
kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:299)
 ~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190) 
~[scala-library-2.13.10.jar:?]
        at 
kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:297) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:262)
 ~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:259)
 ~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575) 
~[scala-library-2.13.10.jar:?]
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573) 
~[scala-library-2.13.10.jar:?]
        at scala.collection.AbstractIterable.foreach(Iterable.scala:933) 
~[scala-library-2.13.10.jar:?]
        at 
kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:259)
 ~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at kafka.network.SocketServer.startup(SocketServer.scala:131) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at kafka.server.KafkaServer.startup(KafkaServer.scala:310) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at kafka.Kafka$.main(Kafka.scala:109) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at com.cloudera.kafka.wrap.Kafka$.$anonfun$main$1(Kafka.scala:107) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at 
com.cloudera.kafka.wrap.Kafka$.$anonfun$main$1$adapted(Kafka.scala:107) 
~[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at com.cloudera.kafka.wrap.Kafka$.runMain(Kafka.scala:118) 
[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:110) 
[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
        at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala) 
[kafka_2.13-3.1.2.7.1.9.0-15.jar:?]
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either 
JAAS or Kafka config
        at 
org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:309)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:61)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        at 
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
 ~[kafka-clients-3.1.2.7.1.9.0-15.jar:?]
        ... 21 more{code}
Using the following configs in a Kafka broker:

jaas configuration file:
{code:java}
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true 
useKeyTab=true storeKey=true serviceName="kafka" 
keyTab="/var/KAFKA_BROKER/kafka.keytab" principal="kafka/hgiovr@SITE";
org.apache.kafka.common.security.scram.ScramLoginModule required;
};{code}
and the following properties:
{code:java}
listener.name.sasl_ssl.sasl.enabled.mechanisms=GSSAPI,SCRAM-SHA-256,SCRAM-SHA-512,OAUTHBEARER
listener.name.sasl_ssl.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
 required;

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #215

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 354693 lines...]
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-02-15T10:45:59.399Z] 
[2023-02-15T10:45:59.399Z] SaslSslAdminIntegrationTest > 
testCreateTopicsResponseMetadataAndConfig() PASSED
[2023-02-15T10:45:59.399Z] 
[2023-02-15T10:45:59.399Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() STARTED
[2023-02-15T10:46:23.325Z] 
[2023-02-15T10:46:23.325Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() PASSED
[2023-02-15T10:46:23.325Z] 
[2023-02-15T10:46:23.325Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() STARTED
[2023-02-15T10:46:49.346Z] 
[2023-02-15T10:46:49.346Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() PASSED
[2023-02-15T10:46:49.346Z] 
[2023-02-15T10:46:49.346Z] SaslSslAdminIntegrationTest > testAclOperations() 
STARTED
[2023-02-15T10:47:11.941Z] 
[2023-02-15T10:47:11.941Z] SaslSslAdminIntegrationTest > testAclOperations() 
PASSED
[2023-02-15T10:47:11.941Z] 
[2023-02-15T10:47:11.941Z] SaslSslAdminIntegrationTest > testAclOperations2() 
STARTED
[2023-02-15T10:47:34.499Z] 
[2023-02-15T10:47:34.499Z] SaslSslAdminIntegrationTest > testAclOperations2() 
PASSED
[2023-02-15T10:47:34.499Z] 
[2023-02-15T10:47:34.499Z] SaslSslAdminIntegrationTest > testAclDelete() STARTED
[2023-02-15T10:47:56.991Z] 
[2023-02-15T10:47:56.991Z] SaslSslAdminIntegrationTest > testAclDelete() PASSED
[2023-02-15T10:47:56.991Z] 
[2023-02-15T10:47:56.991Z] TransactionsTest > testBumpTransactionalEpoch() 
STARTED
[2023-02-15T10:48:45.796Z] 
[2023-02-15T10:48:45.796Z] TransactionsTest > testBumpTransactionalEpoch() 
PASSED
[2023-02-15T10:48:45.796Z] 
[2023-02-15T10:48:45.796Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() STARTED
[2023-02-15T10:48:50.620Z] 
[2023-02-15T10:48:50.620Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() PASSED
[2023-02-15T10:48:50.620Z] 
[2023-02-15T10:48:50.620Z] TransactionsTest > testBasicTransactions() STARTED
[2023-02-15T10:48:56.418Z] 
[2023-02-15T10:48:56.418Z] TransactionsTest > testBasicTransactions() PASSED
[2023-02-15T10:48:56.418Z] 
[2023-02-15T10:48:56.418Z] TransactionsTest > testSendOffsetsWithGroupId() 
STARTED
[2023-02-15T10:49:06.838Z] 
[2023-02-15T10:49:06.838Z] TransactionsTest > testSendOffsetsWithGroupId() 
PASSED
[2023-02-15T10:49:06.838Z] 
[2023-02-15T10:49:06.838Z] TransactionsTest > testFencingOnSendOffsets() STARTED
[2023-02-15T10:49:13.931Z] 
[2023-02-15T10:49:13.931Z] TransactionsTest > testFencingOnSendOffsets() PASSED
[2023-02-15T10:49:13.931Z] 
[2023-02-15T10:49:13.931Z] TransactionsTest > testFencingOnAddPartitions() 
STARTED
[2023-02-15T10:49:19.751Z] 
[2023-02-15T10:49:19.751Z] TransactionsTest > testFencingOnAddPartitions() 
PASSED
[2023-02-15T10:49:19.751Z] 
[2023-02-15T10:49:19.751Z] TransactionsTest > 
testFencingOnTransactionExpiration() STARTED
[2023-02-15T10:49:28.262Z] 
[2023-02-15T10:49:28.262Z] TransactionsTest > 
testFencingOnTransactionExpiration() PASSED
[2023-02-15T10:49:28.262Z] 
[2023-02-15T10:49:28.262Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() STARTED
[2023-02-15T10:49:33.433Z] 
[2023-02-15T10:49:33.433Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() PASSED
[2023-02-15T10:49:33.433Z] 
[2023-02-15T10:49:33.433Z] TransactionsTest > 
testOffsetMetadataInSendOffsetsToTransaction() STARTED
[2023-02-15T10:49:38.069Z] 
[2023-02-15T10:49:38.069Z] TransactionsTest > 
testOffsetMetadataInSendOffsetsToTransaction() PASSED
[2023-02-15T10:49:38.069Z] 
[2023-02-15T10:49:38.069Z] TransactionsTest > testInitTransactionsTimeout() 
STARTED
[2023-02-15T10:49:48.123Z] 
[2023-02-15T10:49:48.123Z] TransactionsTest > testInitTransactionsTimeout() 
PASSED
[2023-02-15T10:49:48.123Z] 
[2023-02-15T10:49:48.123Z] TransactionsTest > 
testConsecutivelyRunInitTransactions() STARTED
[2023-02-15T10:49:52.936Z] 
[2023-02-15T10:49:52.936Z] TransactionsTest > 
testConsecutivelyRunInitTransactions() PASSED
[2023-02-15T10:49:52.936Z] 
[2023-02-15T10:49:52.936Z] TransactionsTest > 
testReadCommittedConsumerShouldNotSeeUndecidedData() STARTED
[2023-02-15T10:49:59.985Z] 
[2023-02-15T10:49:59.985Z] TransactionsTest > 
testReadCommittedConsumerShouldNotSeeUndecidedData() PASSED
[2023-02-15T10:49:59.985Z] 
[2023-02-15T10:49:59.985Z] TransactionsTest > 
testSendOffsetsToTransactionTimeout() STARTED
[2023-02-15T10:50:12.373Z] 
[2023-02-15T10:50:12.373Z] TransactionsTest > 
testSendOffsetsToTransactionTimeout() PASSED
[2023-02-15T10:50:12.373Z] 
[2023-02-15T10:50:12.373Z] TransactionsTest > testFailureToFenceEpoch() STARTED
[2023-02-15T10:51:01.083Z] 
[2023-02-15T10:51:01.083Z] TransactionsTest > testFailureToFenceEpoch() PASSED
[2023-02-15T10:51:01.083Z] 
[2023-02-15T10:51:01.083Z] TransactionsTest > testFencingOnSend() STARTED
[2023-02-15T10:51:01.083Z] 
[2023-02-15T10:51:01.083Z] 

[GitHub] [kafka-site] mumrah merged pull request #488: Edit initial ZK migration doc

[via GitHub]

mumrah merged PR #488:
URL: https://github.com/apache/kafka-site/pull/488


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Gantigmaa Selenge]

Thank you Chris. I agree with what you have suggested.
I have updated the KIP, please let me know if I missed anything or if there
is any other question.

Regards,
Tina

On Tue, Feb 14, 2023 at 4:40 PM Chris Egerton 
wrote:

> Hi Tina,
>
> While I agree that it's reasonable for users to want to favor the source
> cluster's defaults over the target cluster's, I'm hesitant to change this
> behavior in an opt-out fashion. IMO it's better to allow users to opt into
> this (by adding a method to the ConfigPropertyFilter interface, and
> possibly extending the DefaultConfigPropertyFilter with configuration
> properties related to how it should handle source cluster defaults), but we
> should try to preserve the existing behavior by default.
>
> Cheers,
>
> Chris
>
> On Mon, Feb 13, 2023 at 5:10 PM Gantigmaa Selenge 
> wrote:
>
> > Hi Chris
> >
> > My comment on the second point is not correct. Please ignore the part
> about
> > the config source (config source does set back to DEFAULT_CONFIG when
> > deleting a config). I got diverted off the issue a little bit.
> >
> > With the legacy API, we do propagate deletion due to resetting all the
> > configs on that target topic that are not being replicated. However with
> > incrementalAlterConfigs API, this changes. If we delete a config that was
> > previously altered on the source topic, the config on the target topic is
> > left with the previous value as the default configs are not replicated.
> The
> > reason for favouring the source defaults was because it would set the
> > config on the target topic with the source's default in this situation.
> >
> > Regards,
> > Tina
> >
> > On Mon, Feb 13, 2023 at 8:15 AM Gantigmaa Selenge 
> > wrote:
> >
> > > Hi Chris and Luke,
> > >
> > > Thank you very much for your feedback.
> > >
> > > I have addressed some of the suggestions and would like to clarify a
> few
> > > things on the others:
> > >
> > > > 1) The current logic for syncing topic configs in MM2 is basically
> > > fire-and-forget; all we do is log a warning message [1] if an attempt
> > > fails. When "use.incremental.alter.configs" is set to "requested",
> we'll
> > > need to know whether attempts using the incremental API succeed or not,
> > and
> > > then adjust our behavior accordingly. Will the plan here be to block on
> > the
> > > success/failure of the first request before sending any more, or will
> we
> > > just switch over to the legacy API as soon as any request fails due to
> > > targeting an incompatible broker, possibly after multiple requests with
> > the
> > > new API have been sent or at least queued up in the admin client?
> > >
> > > When it's set to "requested", I think the suggestion was to just switch
> > > over to the legacy API as soon as any request fails due to
> > > targeting an incompatible broker. We could keep using the legacy API
> > until
> > > the mirrormaker setting is changed by the user or the mirrormaker
> > restarts
> > > instead of trying the new API every time it syncs the configurations.
> I'm
> > > not sure which approach is the best here.
> > >
> > > > 2) We don't replicate default properties from the source cluster
> right
> > > now
> > > [2].
> > > Won't making ConfigPropertyFilter::shouldReplicateSourceDefault return
> > true
> > > by default change that behavior? If so, what's the motivation for
> > favoring
> > > defaults from the source cluster over defaults for the target cluster,
> > > especially given that users may already be relying on the opposite?
> > >
> > > The concern was around what happens if deleting a topic config in
> > > the source cluster. I initially thought, because the legacy API resets
> > all
> > > the configs other than the ones being altered, it effectively
> propagates
> > > the deletions of a config in the source cluster. I thought by migrating
> > to
> > > incrementalAlterConfig API, the deletion would no longer get propagated
> > but
> > > looking into it again, it may not be the case.
> > >
> > > If the user deletes a config in the source cluster, it would be reset
> to
> > > the default value but the config source does not change back to
> > > DEFAULT_CONFIG. For example:
> > >
> > > ./kafka-configs.sh --alter --entity-type topics --entity-name
> > > quickstart-events --add-config retention.ms=72 --bootstrap-server
> > > localhost:9092
> > >
> > >
> > > /kafka-configs.sh --describe --entity-type topics --entity-name
> > > quickstart-events --bootstrap-server localhost:9092
> > >
> > > Dynamic configs for topic quickstart-events are:
> > >
> > >   retention.ms=72 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:
> > > retention.ms=72}
> > >
> > >
> > > ./kafka-configs.sh --alter --entity-type topics --entity-name
> > > quickstart-events --delete-config retention.ms  --bootstrap-server
> > > localhost:9092
> > >
> > >
> > > /kafka-configs.sh --describe --entity-type topics --entity-name
> > > quickstart-events --bootstrap-server localhost:9092
> > >
> > >   

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

[Chris Egerton]

Hi all,

Thanks to everyone who's voted so far! Just wanted to bump this thread and
see if we could get a few more votes; currently we're at +3 non-binding
and +1 binding. Hoping we can get this approved, reviewed, and merged in
time for 3.5.0.

Cheers,

Chris

On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison 
wrote:

> Thanks Chris for the KIP, this is a much needed feature!
>
> +1 (binding)
>
>
> On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
>  wrote:
> >
> > +1 (non binding)
> >
> > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya  wrote:
> >
> > > Hi Chris,
> > >
> > > I'm +1 (non-binding). Thanks again for proposing this extremely
> > > valuable addition to Kafka Connect!
> > >
> > > Thanks,
> > > Yash
> > >
> > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-875, which adds support for
> viewing
> > > and
> > > > manipulating the offsets of connectors to the Kafka Connect REST API.
> > > >
> > > > The KIP:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > >
> > > > The discussion thread:
> > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
>

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Ismael Juma]

Hi.

Is MessageReader the right name? The client apis tend to use the word
`Record` instead. In fact, `readMessage` returns a `ProducerRecord`.

Also, I wonder if we it would be useful for the interface to support the
usage of Serializers. That is, the interface could return ProducerRecord
as long as a serializer was also configured. I am less sure about this one,
but worth giving it some thought and if we think it's a bad idea, we should
add it to the "Rejected Alternatives" section.

Ismael

On Wed, Feb 15, 2023 at 3:07 AM Mickael Maison 
wrote:

> Hi Chia-Ping,
>
> Sorry nobody replied to your thread earlier!
> I stumbled on this while looking at
> https://issues.apache.org/jira/browse/KAFKA-14525 (migrating CLI tools
> out of core). Your proposal still makes sense and would help migrating
> the ConsoleProducer tool.
>
> Can we specify when we expect to delete kafka.common.MessageReader? I
> guess in Kafka 4.0?
> Apart from that, the KIP looks good and considering how simple it is,
> you should be able to open a vote thread.
>
> Thanks,
> Mickael
>
> On Tue, Jul 7, 2020 at 5:55 PM Chia-Ping Tsai  wrote:
> >
> > hi all,
> >
> > I would like to start the discussion for KIP-641.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> >
> > Many thanks,
> >
> > Chia-Ping
>

Requesting permissions to contribute to Apache Kafka.

[hzh0425]

Hello,

My name is zhangheng huang. I want to start contributing to the Apache Kafka 
project and want the permission to create KIPs

JiraID: hzh0425
WikiID: hzh0425
email: hzh0...@apache.org

Best regards,

[jira] [Created] (KAFKA-14720) Tools migration guidelines

[Federico Valeri (Jira)]

Federico Valeri created KAFKA-14720:
---

 Summary: Tools migration guidelines
 Key: KAFKA-14720
 URL: https://issues.apache.org/jira/browse/KAFKA-14720
 Project: Kafka
  Issue Type: Improvement
Reporter: Federico Valeri


The tools migration effort is ongoing and being tracked in KAFKA-14525. This is 
part of a bigger initiative to split the core module into multiple modules 
(e.g. storage, network, security, tools), which is being tracked in KAFKA-14524.

The plan is to migrate tools and related classes in a fully compatible way from 
kafka.tools and kafka.admin packages (core module) to org.apache.kafka.tools 
package (tools module).

While kicking off this activity, we identified a number of potential 
compatibility issues:

* Missing wrapper: some tools do not have a wrapper script. There are system 
tests that directly refer to the tool's fully qualified class name (FQCN) and 
expect the old package name when running on old Kafka releases. They are often 
used for troubleshooting or automation through the “kafka-run-class.sh” script 
which takes the FQCN as input parameter.
* SPI argument: some tools have arguments for setting a custom SPI 
implementation to be used in place of the default implementation. Any custom 
SPI implementation depends on the old package name.
* Broken tool: some tools do not work on supported releases.
* Core dependency: some tools require access to non-trivial core classes that 
should be migrated first.

See KIP-906 for more information.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Mickael Maison]

Hi Chia-Ping,

Sorry nobody replied to your thread earlier!
I stumbled on this while looking at
https://issues.apache.org/jira/browse/KAFKA-14525 (migrating CLI tools
out of core). Your proposal still makes sense and would help migrating
the ConsoleProducer tool.

Can we specify when we expect to delete kafka.common.MessageReader? I
guess in Kafka 4.0?
Apart from that, the KIP looks good and considering how simple it is,
you should be able to open a vote thread.

Thanks,
Mickael

On Tue, Jul 7, 2020 at 5:55 PM Chia-Ping Tsai  wrote:
>
> hi all,
>
> I would like to start the discussion for KIP-641.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
>
> Many thanks,
>
> Chia-Ping

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #141

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 440904 lines...]
[2023-02-15T10:56:34.886Z] > Task :core:compileJava NO-SOURCE
[2023-02-15T10:56:34.886Z] > Task :connect:json:compileJava UP-TO-DATE
[2023-02-15T10:56:34.886Z] > Task :connect:json:classes UP-TO-DATE
[2023-02-15T10:56:34.886Z] > Task :connect:json:javadoc SKIPPED
[2023-02-15T10:56:34.886Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2023-02-15T10:56:34.886Z] > Task :connect:json:javadocJar
[2023-02-15T10:56:35.803Z] > Task :clients:compileTestJava UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :clients:testClasses UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :connect:json:testJar
[2023-02-15T10:56:35.803Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :raft:testClasses UP-TO-DATE
[2023-02-15T10:56:35.803Z] > Task :connect:json:testSrcJar
[2023-02-15T10:56:36.721Z] > Task :streams:copyDependantLibs
[2023-02-15T10:56:36.721Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:56:36.721Z] > Task :streams:jar UP-TO-DATE
[2023-02-15T10:56:36.721Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:56:36.721Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-02-15T10:56:36.721Z] > Task :metadata:testClasses UP-TO-DATE
[2023-02-15T10:56:41.266Z] > Task :connect:api:javadoc
[2023-02-15T10:56:41.266Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-02-15T10:56:41.266Z] > Task :connect:api:jar UP-TO-DATE
[2023-02-15T10:56:41.266Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-02-15T10:56:41.266Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:56:41.266Z] > Task :connect:json:jar UP-TO-DATE
[2023-02-15T10:56:41.266Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:56:42.184Z] > Task :connect:api:javadocJar
[2023-02-15T10:56:42.184Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-02-15T10:56:42.184Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-02-15T10:56:42.184Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-02-15T10:56:42.184Z] > Task :connect:json:publishToMavenLocal
[2023-02-15T10:56:42.184Z] > Task :connect:api:testJar
[2023-02-15T10:56:43.107Z] > Task :connect:api:testSrcJar
[2023-02-15T10:56:43.107Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-02-15T10:56:43.107Z] > Task :connect:api:publishToMavenLocal
[2023-02-15T10:56:44.025Z] > Task :streams:javadoc
[2023-02-15T10:56:44.025Z] > Task :streams:javadocJar
[2023-02-15T10:56:46.606Z] 
[2023-02-15T10:56:46.606Z] > Task :clients:javadoc
[2023-02-15T10:56:46.606Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2023-02-15T10:56:47.523Z] 1 warning
[2023-02-15T10:56:48.578Z] 
[2023-02-15T10:56:48.578Z] > Task :clients:javadocJar
[2023-02-15T10:56:51.159Z] > Task :clients:testJar
[2023-02-15T10:56:52.076Z] > Task :clients:testSrcJar
[2023-02-15T10:56:53.006Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-02-15T10:56:53.006Z] > Task :clients:publishToMavenLocal
[2023-02-15T10:57:04.873Z] > Task :core:compileScala
[2023-02-15T10:58:25.143Z] > Task :core:classes
[2023-02-15T10:58:25.143Z] > Task :core:compileTestJava NO-SOURCE
[2023-02-15T10:58:55.377Z] > Task :core:compileTestScala
[2023-02-15T10:59:43.923Z] > Task :core:testClasses
[2023-02-15T11:00:00.207Z] > Task :streams:compileTestJava
[2023-02-15T11:00:00.207Z] > Task :streams:testClasses
[2023-02-15T11:00:00.207Z] > Task :streams:testJar
[2023-02-15T11:00:01.149Z] > Task :streams:testSrcJar
[2023-02-15T11:00:01.149Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-02-15T11:00:01.149Z] > Task :streams:publishToMavenLocal
[2023-02-15T11:00:01.149Z] 
[2023-02-15T11:00:01.149Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-02-15T11:00:01.149Z] 
[2023-02-15T11:00:01.149Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-02-15T11:00:01.149Z] 
[2023-02-15T11:00:01.149Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-02-15T11:00:01.149Z] 
[2023-02-15T11:00:01.149Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2023-02-15T11:00:01.149Z] Please consult deprecation warnings for more details.
[2023-02-15T11:00:01.149Z] 
[2023-02-15T11:00:01.149Z] BUILD SUCCESSFUL in 3m 52s
[2023-02-15T11:00:01.149Z] 77 actionable tasks: 39 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #108

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 407604 lines...]
[2023-02-15T10:54:09.280Z] 
[2023-02-15T10:54:09.280Z] AclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls() STARTED
[2023-02-15T10:54:09.280Z] 
[2023-02-15T10:54:09.280Z] AclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls() PASSED
[2023-02-15T10:54:09.280Z] 
[2023-02-15T10:54:09.280Z] AclAuthorizerTest > testAddAclsOnWildcardResource() 
STARTED
[2023-02-15T10:54:10.198Z] 
[2023-02-15T10:54:10.198Z] AclAuthorizerTest > testAddAclsOnWildcardResource() 
PASSED
[2023-02-15T10:54:10.198Z] 
[2023-02-15T10:54:10.198Z] AclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2() STARTED
[2023-02-15T10:54:11.115Z] 
[2023-02-15T10:54:11.115Z] AclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2() PASSED
[2023-02-15T10:54:11.115Z] 
[2023-02-15T10:54:11.115Z] AclAuthorizerTest > 
testAuthorizerZkConfigFromPrefixOverrides() STARTED
[2023-02-15T10:54:11.115Z] 
[2023-02-15T10:54:11.115Z] AclAuthorizerTest > 
testAuthorizerZkConfigFromPrefixOverrides() PASSED
[2023-02-15T10:54:11.115Z] 
[2023-02-15T10:54:11.115Z] AclAuthorizerTest > testAclsFilter() STARTED
[2023-02-15T10:54:12.113Z] 
[2023-02-15T10:54:12.113Z] AclAuthorizerTest > testAclsFilter() PASSED
[2023-02-15T10:54:12.113Z] 
[2023-02-15T10:54:12.113Z] AclAuthorizerTest > testAclManagementAPIs() STARTED
[2023-02-15T10:54:12.113Z] 
[2023-02-15T10:54:12.113Z] AclAuthorizerTest > testAclManagementAPIs() PASSED
[2023-02-15T10:54:12.113Z] 
[2023-02-15T10:54:12.113Z] AclAuthorizerTest > testWildCardAcls() STARTED
[2023-02-15T10:54:13.031Z] 
[2023-02-15T10:54:13.031Z] AclAuthorizerTest > testWildCardAcls() PASSED
[2023-02-15T10:54:13.031Z] 
[2023-02-15T10:54:13.031Z] AclAuthorizerTest > testCreateDeleteTiming() STARTED
[2023-02-15T10:54:13.948Z] 
[2023-02-15T10:54:13.948Z] AclAuthorizerTest > testCreateDeleteTiming() PASSED
[2023-02-15T10:54:13.948Z] 
[2023-02-15T10:54:13.948Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllHostAce() STARTED
[2023-02-15T10:54:14.865Z] 
[2023-02-15T10:54:14.865Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllHostAce() PASSED
[2023-02-15T10:54:14.865Z] 
[2023-02-15T10:54:14.865Z] AclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2() STARTED
[2023-02-15T10:54:14.865Z] 
[2023-02-15T10:54:14.865Z] AclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2() PASSED
[2023-02-15T10:54:14.865Z] 
[2023-02-15T10:54:14.865Z] AclAuthorizerTest > testTopicAcl() STARTED
[2023-02-15T10:54:15.783Z] 
[2023-02-15T10:54:15.783Z] AclAuthorizerTest > testTopicAcl() PASSED
[2023-02-15T10:54:15.783Z] 
[2023-02-15T10:54:15.783Z] AclAuthorizerTest > testSuperUserHasAccess() STARTED
[2023-02-15T10:54:15.783Z] 
[2023-02-15T10:54:15.783Z] AclAuthorizerTest > testSuperUserHasAccess() PASSED
[2023-02-15T10:54:15.783Z] 
[2023-02-15T10:54:15.783Z] AclAuthorizerTest > 
testDeleteAclOnPrefixedResource() STARTED
[2023-02-15T10:54:16.781Z] 
[2023-02-15T10:54:16.781Z] AclAuthorizerTest > 
testDeleteAclOnPrefixedResource() PASSED
[2023-02-15T10:54:16.781Z] 
[2023-02-15T10:54:16.781Z] AclAuthorizerTest > testDenyTakesPrecedence() STARTED
[2023-02-15T10:54:16.781Z] 
[2023-02-15T10:54:16.781Z] AclAuthorizerTest > testDenyTakesPrecedence() PASSED
[2023-02-15T10:54:16.781Z] 
[2023-02-15T10:54:16.781Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeIsolationUnrelatedDenyWontDominateAllow() STARTED
[2023-02-15T10:54:17.699Z] 
[2023-02-15T10:54:17.699Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeIsolationUnrelatedDenyWontDominateAllow() PASSED
[2023-02-15T10:54:17.699Z] 
[2023-02-15T10:54:17.699Z] AclAuthorizerTest > 
testSingleCharacterResourceAcls() STARTED
[2023-02-15T10:54:17.699Z] 
[2023-02-15T10:54:17.699Z] AclAuthorizerTest > 
testSingleCharacterResourceAcls() PASSED
[2023-02-15T10:54:17.699Z] 
[2023-02-15T10:54:17.699Z] AclAuthorizerTest > testNoAclFoundOverride() STARTED
[2023-02-15T10:54:18.616Z] 
[2023-02-15T10:54:18.616Z] AclAuthorizerTest > testNoAclFoundOverride() PASSED
[2023-02-15T10:54:18.616Z] 
[2023-02-15T10:54:18.616Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWildcardResourceDenyDominate() STARTED
[2023-02-15T10:54:19.533Z] 
[2023-02-15T10:54:19.533Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWildcardResourceDenyDominate() PASSED
[2023-02-15T10:54:19.533Z] 
[2023-02-15T10:54:19.533Z] AclAuthorizerTest > testEmptyAclThrowsException() 
STARTED
[2023-02-15T10:54:19.533Z] 
[2023-02-15T10:54:19.533Z] AclAuthorizerTest > testEmptyAclThrowsException() 
PASSED
[2023-02-15T10:54:19.533Z] 
[2023-02-15T10:54:19.533Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeNoAclFoundOverride() STARTED
[2023-02-15T10:54:20.450Z] 
[2023-02-15T10:54:20.450Z] AclAuthorizerTest > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #97

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 304510 lines...]
[2023-02-15T10:18:48.890Z] 1 warning
[2023-02-15T10:18:48.890Z] 
[2023-02-15T10:18:48.890Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task :connect:api:jar UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:18:48.890Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task :connect:json:jar UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-02-15T10:18:48.890Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-02-15T10:18:48.890Z] > Task :connect:json:publishToMavenLocal
[2023-02-15T10:18:48.890Z] > Task :connect:api:javadocJar
[2023-02-15T10:18:48.890Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-02-15T10:18:48.890Z] > Task :connect:api:testJar
[2023-02-15T10:18:48.890Z] > Task :connect:api:testSrcJar
[2023-02-15T10:18:48.890Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-02-15T10:18:48.890Z] > Task :connect:api:publishToMavenLocal
[2023-02-15T10:18:50.933Z] 
[2023-02-15T10:18:50.933Z] > Task :streams:javadoc
[2023-02-15T10:18:50.933Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2023-02-15T10:18:50.933Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2023-02-15T10:18:50.933Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2023-02-15T10:18:50.934Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2023-02-15T10:18:50.934Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2023-02-15T10:18:51.852Z] 12 warnings
[2023-02-15T10:18:51.852Z] 
[2023-02-15T10:18:51.852Z] > Task :streams:javadocJar
[2023-02-15T10:18:51.852Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-02-15T10:18:51.852Z] > Task :streams:testClasses UP-TO-DATE
[2023-02-15T10:18:52.772Z] 
[2023-02-15T10:18:52.772Z] > Task :clients:javadoc
[2023-02-15T10:18:52.772Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 

[DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

[Christo Lolov]

Hello!

I would like to start a discussion for KIP-902: Upgrade Zookeeper to 3.8.1. The 
Zookeeper version currently used in Kafka reached its end of life in December 
2022. You can find the KIP at 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784

Thanks in advance for the reviews.

Christo

[jira] [Resolved] (KAFKA-14716) Connect schema does not allow struct default values

[Daniel Urban (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Urban resolved KAFKA-14716.
--
Resolution: Duplicate

> Connect schema does not allow struct default values
> ---
>
> Key: KAFKA-14716
> URL: https://issues.apache.org/jira/browse/KAFKA-14716
> Project: Kafka
>  Issue Type: Bug
>Reporter: Daniel Urban
>Assignee: Daniel Urban
>Priority: Major
>
> The ConnectSchema API should allow specifying a composite (struct) default 
> value for a field, but with the current API, it is impossible to do so.
>  # There is a circular dependency between creating a struct as a default 
> value and creating the schema which holds it as the default value. The Struct 
> constructor expects a Schema object, and the default value setter of 
> SchemaBuilder checks schema conformity by using the 
> ConnectSchema.validateValue, which in turn uses ConnectSchema.equals. This 
> can only be bypassed if the struct references a SchemaBuilder instance, and 
> defaultValue is called on that builder instance, but this goes against the 
> Struct docs stating that "Struct objects must specify a complete \{@link 
> Schema} up front".
>  # ConnectSchema.equals is not prepared to be used with other Schema 
> implementations, so equals checks between ConnectSchema and SchemaBuilder 
> instances will always fail. This is only causing an issue if equals has to be 
> used for schema conformity checks.
> Code examples:
> Working code (mind that the schema referenced by the Struct is a 
> SchemaBuilder, and it is mutated after the Struct is constructed):
> {code:java}
> @Test
> public void testCompositeDefault() {
> SchemaBuilder nestedSchema = SchemaBuilder.struct()
> .field("bar", Schema.STRING_SCHEMA);
> Struct nestedDefault = new Struct(nestedSchema);
> nestedDefault.put("bar", "default_value");
> Schema schema = SchemaBuilder.struct()
> .field("foo",
> nestedSchema
> .defaultValue(nestedDefault)
> .build()
> )
> .build();
> } {code}
> Not working code (but better aligned with the current API and docs - 2 
> separate Schema instances used by the Struct and the field, only diff is the 
> default value between the 2):
> {code:java}
>  @Test
> public void testCompositeDefault() {
> Schema nestedSchema = SchemaBuilder.struct()
> .field("bar", Schema.STRING_SCHEMA)
> .build();
> Struct nestedDefault = new Struct(nestedSchema);
> nestedDefault.put("bar", "default_value");
> Schema schema = SchemaBuilder.struct()
> .field("foo",
> SchemaBuilder
> .struct()
> .field("bar", Schema.STRING_SCHEMA)
> .defaultValue(nestedDefault)
> .build()
> )
> .build();
> }{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)