Re: [DISCUSS] KIP-763: Range queries with open endpoints

[Boyang Chen]

Thanks Patrick for the KIP and sorry for being late to the party here. One
thing I would like to understand is, what's the expected behavior when a
user specifies a null key in previous versions? In the new version which
uses null key for open-ended range queries, could it potentially hide bugs
in the user's code when their intention was to do a bounded range query but
accidentally set the key to null?

Boyang

On Wed, Jul 21, 2021 at 2:59 AM Patrick Stuedi 
wrote:

> Thanks John, Bruno for the valuable feedback!
>
> John: I had a quick look at the SessionStore and WindowStore interface.
> While it looks like we should be able to apply similar ideas to those
> stores, the actual interfaces are slightly different and expanding them for
> open ranges may need a bit more thinking. In that sense, and to make sure
> we will be converging with this KIP, I'd prefer to not expand the scope of
> the KIP . As Bruno suggested we can always propose changes to the
> SessionStore and WindowStore in a separate KIP. I'll add a subsection in
> the rejected alternatives.
>
> @Bruno: good point, I'll add an example. Yes, all() will be equivalent to
> range(null, null) and the implementation of all() will be a call to
> range(null, null).
>
> -Patrick
>
> On Wed, Jul 21, 2021 at 9:12 AM Bruno Cadonna  wrote:
>
> > Thanks for the KIP, Patrick!
> >
> > I agree with John that your KIP is well motivated.
> >
> > I have just one minor feedback. Could you add store.range(null, null) to
> > the example snippets with the comment that this is equivalent to all()
> > (it is equivalent, right?)? This question about equivalence between
> > range(null, null) and all() came up in my mind when I read the KIP and I
> > think I am not the only one.
> >
> > Regarding expanding the KIP to SessionStore and WindowStore, I think you
> > should not expand scope of the KIP. We can do the changes to the
> > SessionStore and WindowStore in a separate KIP. But it is your call!
> >
> >
> > Best,
> > Bruno
> >
> > On 21.07.21 00:18, John Roesler wrote:
> > > Thanks for the KIP, Patrick!
> > >
> > > I think your KIP is well motivated. It's definitely a bummer
> > > to have to iterate over the full store as a workaround for
> > > open-ended ranges.
> > >
> > > I agree with your pragmatic approach here. We have recently
> > > had a couple of other contributions to the store APIs
> > > (prefix and reverseRange), and the complexity of adding
> > > those new methods far exceeded what anyone expected. I'd be
> > > in favor of being conservative with new store APIs until we
> > > are able to reign in that complexity somehow. Since your
> > > proposed semantics seem reasonable, I'm in favor.
> > >
> > > While reviewing your KIP, it struck me that we have several
> > > range-like APIs on:
> > > * SessionStore
> > > (
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlySessionStore.java
> > )
> > > * WindowStore
> > > (
> >
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlyWindowStore.java
> > )
> > > as well.
> > >
> > > Do you think it would be a good idea to also propose a
> > > similar change on those APIs? It might be nice (for exactly
> > > the same reasons you documented), but it also increases the
> > > scope of your work. There is one extra wrinkle I can see:
> > > SessionStore has versions of the range methods that take a
> > > `long` instead of an `Instant`, so `null` wouldn't work for
> > > them.
> > >
> > > If you prefer to keep your KIP narrow in scope to just the
> > > KeyValueStore, I'd also support you. In that case, it might
> > > be a good idea to simply mention in the "Rejected
> > > Alternatives" that you decided not to address those other
> > > store APIs at this time. That way, people later on won't
> > > have to wonder why those other methods didn't get updated.
> > >
> > > Other than that, I have no concerns!
> > >
> > > Thank you,
> > > John
> > >
> > > On Mon, 2021-07-19 at 13:22 +0200, Patrick Stuedi wrote:
> > >> Hi everyone,
> > >>
> > >> I would like to start the discussion for KIP-763: Range queries with
> > open
> > >> endpoints.
> > >>
> > >> The KIP can be found here:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints
> > >>
> > >> Any feedback will be highly appreciated.
> > >>
> > >> Many thanks,
> > >>   Patrick
> > >
> > >
> >
>

Re: [DISCUSS] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

[Sophie Blee-Goldman]

1) I agree that we should just distribute the bytes evenly, at least for
now. It's simpler to understand and
we can always change it later, plus it makes sense to keep this aligned
with how the cache works today

2) +1 to being conservative in the generous sense, it's just not something
we can predict with any degree
of accuracy and even if we could, the appropriate value is going to differ
wildly across applications and use
cases. We might want to just pick some multiple of the default cache size,
and maybe do some research on
other relevant defaults or sizes (default JVM heap, size of available
memory in common hosts eg EC2
instances, etc). We don't need to worry as much about erring on the side of
too big, since other configs like
the max.poll.records will help somewhat to keep it from exploding.

4) 100%, I always found the *cache.max.bytes.buffering* config name to be
incredibly confusing. Deprecating this in
favor of "*statestore.cache.max.bytes*" and aligning it to the new input
buffer config sounds good to me to include here.

5) The KIP should list all relevant public-facing changes, including
metadata like the config's "Importance". Personally
I would recommend Medium, or even High if we're really worried about the
default being wrong for a lot of users

Thanks for the KIP, besides those few things that Guozhang brought up and
the config importance, everything SGTM

-Sophie

On Thu, Aug 26, 2021 at 2:41 PM Guozhang Wang  wrote:

> 1) I meant for your proposed solution. I.e. to distribute the configured
> bytes among threads evenly.
>
> 2) I was actually thinking about making the default a large enough value so
> that we would not introduce performance regression: thinking about a use
> case with many partitions and each record may be large, then effectively we
> would only start pausing when the total bytes buffered is pretty large. If
> we set the default value to small, we would be "more aggressive" on pausing
> which may impact throughput.
>
> 3) Yes exactly, this would naturally be at the "partition-group" class
> since that represents the task's all input partitions.
>
> 4) This is just a bold thought, I'm interested to see other's thoughts.
>
>
> Guozhang
>
> On Mon, Aug 23, 2021 at 4:10 AM Sagar  wrote:
>
> > Thanks Guozhang.
> >
> > 1) Just for my confirmation, when you say we should proceed with the even
> > distribution of bytes, are you referring to the Proposed Solution in the
> > KIP or the option you had considered in the JIRA?
> > 2) Default value for the config is something that I missed. I agree we
> > can't have really large values as it might be detrimental to the
> > performance. Maybe, as a starting point, we assume that only 1 Stream
> Task
> > is running so what could be the ideal value in such a scenario? Somewhere
> > around 10MB similar to the caching config?
> > 3) When you say,  *a task level metric indicating the current totally
> > aggregated metrics, * you mean the bytes aggregated at a task level?
> > 4) I am ok with the name change, but would like to know others' thoughts.
> >
> > Thanks!
> > Sagar.
> >
> > On Sun, Aug 22, 2021 at 11:54 PM Guozhang Wang 
> wrote:
> >
> > > Thanks Sagar for writing this PR.
> > >
> > > I think twice about the options that have been proposed in
> > > https://issues.apache.org/jira/browse/KAFKA-13152, and feel that at
> the
> > > moment it's simpler to just do the even distribution of the configured
> > > total bytes. My rationale is that right now we have a static tasks ->
> > > threads mapping, and hence each partition would only be fetched by a
> > single
> > > thread / consumer at a given time. If in the future we break that
> static
> > > mapping into dynamic mapping, then we would not be able to do this even
> > > distribution. Instead we would have other threads polling from consumer
> > > only, and those threads would be responsible for checking the config
> and
> > > pause non-empty partitions if it goes beyond the threshold. But since
> at
> > > that time we would not change the config but just how it would be
> > > implemented behind the scenes we would not need another KIP to change
> it.
> > >
> > > Some more comments:
> > >
> > > 1. We need to discuss a bit about the default value of this new config.
> > > Personally I think we need to be a bit conservative with large values
> so
> > > that it would not have any perf regression compared with old configs
> > > especially with large topology and large number of partitions.
> > > 2. I looked at the existing metrics, and do not have corresponding
> > sensors.
> > > How about also adding a task level metric indicating the current
> totally
> > > aggregated metrics. The reason I do not suggest this metric on the
> > > per-thread level is that in the future we may break the static mapping
> of
> > > tasks -> threads.
> > >
> > > [optional] As an orthogonal thought, I'm thinking maybe we can rename
> the
> > > other "*cache.max.bytes.buffering*" as "statestore.cache.max.bytes"

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #436

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 489794 lines...]
[2021-08-28T05:02:01.178Z] > Task :raft:testClasses UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :connect:json:testJar
[2021-08-28T05:02:01.178Z] > Task :connect:json:testSrcJar
[2021-08-28T05:02:01.178Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :metadata:testClasses UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-08-28T05:02:01.178Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-08-28T05:02:01.178Z] 
[2021-08-28T05:02:01.178Z] > Task :streams:processMessages
[2021-08-28T05:02:01.178Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-08-28T05:02:01.178Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-28T05:02:01.178Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-08-28T05:02:01.178Z] 
[2021-08-28T05:02:01.178Z] > Task :core:compileScala UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :core:classes UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :core:compileTestJava NO-SOURCE
[2021-08-28T05:02:01.178Z] > Task :streams:compileJava UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :streams:classes UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :streams:jar UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-08-28T05:02:01.178Z] > Task :core:compileTestScala UP-TO-DATE
[2021-08-28T05:02:01.178Z] > Task :core:testClasses UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task :connect:api:javadoc
[2021-08-28T05:02:04.940Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task :connect:api:jar UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-08-28T05:02:04.940Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task :connect:json:jar UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-08-28T05:02:04.940Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-08-28T05:02:04.940Z] > Task :connect:json:publishToMavenLocal
[2021-08-28T05:02:04.940Z] > Task :connect:api:javadocJar
[2021-08-28T05:02:04.940Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-08-28T05:02:04.940Z] > Task :connect:api:testJar
[2021-08-28T05:02:04.940Z] > Task :connect:api:testSrcJar
[2021-08-28T05:02:04.940Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-08-28T05:02:04.940Z] > Task :connect:api:publishToMavenLocal
[2021-08-28T05:02:07.631Z] > Task :streams:javadoc
[2021-08-28T05:02:07.631Z] > Task :streams:javadocJar
[2021-08-28T05:02:07.631Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-08-28T05:02:07.631Z] > Task :streams:testClasses UP-TO-DATE
[2021-08-28T05:02:08.592Z] > Task :streams:testJar
[2021-08-28T05:02:08.592Z] > Task :streams:testSrcJar
[2021-08-28T05:02:08.592Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-08-28T05:02:08.592Z] > Task :streams:publishToMavenLocal
[2021-08-28T05:02:09.553Z] > Task :clients:javadoc
[2021-08-28T05:02:09.553Z] > Task :clients:javadocJar
[2021-08-28T05:02:10.689Z] 
[2021-08-28T05:02:10.689Z] > Task :clients:srcJar
[2021-08-28T05:02:10.689Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-08-28T05:02:10.689Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-28T05:02:10.689Z] 
[2021-08-28T05:02:10.689Z] > Task :clients:testJar
[2021-08-28T05:02:11.651Z] > Task :clients:testSrcJar
[2021-

[jira] [Resolved] (KAFKA-13128) Flaky Test StoreQueryIntegrationTest.shouldQueryStoresAfterAddingAndRemovingStreamThread

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-13128.

Resolution: Fixed

> Flaky Test 
> StoreQueryIntegrationTest.shouldQueryStoresAfterAddingAndRemovingStreamThread
> 
>
> Key: KAFKA-13128
> URL: https://issues.apache.org/jira/browse/KAFKA-13128
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.0.0, 2.8.1
>Reporter: A. Sophie Blee-Goldman
>Assignee: Walker Carlson
>Priority: Blocker
>  Labels: flaky-test
> Fix For: 3.1.0
>
>
> h3. Stacktrace
> java.lang.AssertionError: Expected: is not null but: was null 
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) 
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
>   at 
> org.apache.kafka.streams.integration.StoreQueryIntegrationTest.lambda$shouldQueryStoresAfterAddingAndRemovingStreamThread$19(StoreQueryIntegrationTest.java:461)
>   at 
> org.apache.kafka.streams.integration.StoreQueryIntegrationTest.until(StoreQueryIntegrationTest.java:506)
>   at 
> org.apache.kafka.streams.integration.StoreQueryIntegrationTest.shouldQueryStoresAfterAddingAndRemovingStreamThread(StoreQueryIntegrationTest.java:455)
>  
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-11085/5/testReport/org.apache.kafka.streams.integration/StoreQueryIntegrationTest/Build___JDK_16_and_Scala_2_13___shouldQueryStoresAfterAddingAndRemovingStreamThread_2/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12963) Improve error message for Class cast exception

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-12963.

Fix Version/s: 3.1.0
   Resolution: Fixed

> Improve error message for Class cast exception
> --
>
> Key: KAFKA-12963
> URL: https://issues.apache.org/jira/browse/KAFKA-12963
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 2.7.0
>Reporter: Rasmus Helbig Hansen
>Assignee: Andrew Lapidas
>Priority: Minor
> Fix For: 3.1.0
>
>
> After a topology change and starting the application again, we got this type 
> of error message:
>  [g9z-StreamThread-1] ERROR 
> org.apache.kafka.streams.processor.internals.TaskManager  - stream-thread 
> [g9z-StreamThread-1] Failed to process stream task 1_12 due to the following 
> error:
>  org.apache.kafka.streams.errors.StreamsException: ClassCastException 
> invoking Processor. Do the Processor's input types match the deserialized 
> types? Check the Serde setup and change the default Serdes in StreamConfig or 
> provide correct Serdes via method parameters. Make sure the Processor can 
> accept the deserialized input of type key: org.acme.SomeKey, and value: 
> org.acme.SomeValue.
>  Note that although incorrect Serdes are a common cause of error, the cast 
> exception might have another cause (in user code, for example). For example, 
> if a processor wires in a store, but casts the generics incorrectly, a class 
> cast exception could be raised during processing, but the cause would not be 
> wrong Serdes.
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:185)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:273)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:252)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:219)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:172)
>  at 
> org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:55)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorAdapter.process(ProcessorAdapter.java:71)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:181)
>  at 
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:181)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:273)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:252)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:219)
>  at 
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:86)
>  at 
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:703)
>  at 
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)
>  at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:703)
>  at 
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1105)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:647)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)
>  Caused by: java.lang.ClassCastException: class org.acme.SomeValue cannot be 
> cast to class org.acme.OtherValue (org.acme.SomeValue and org.acme.OtherValue 
> are in unnamed module of loader 'app')
>  at 
> org.apache.kafka.streams.kstream.internals.KStreamKStreamJoin$KStreamKStreamJoinProcessor.process(KStreamKStreamJoin.java:112)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorAdapter.process(ProcessorAdapter.java:71)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:181)
>  at 
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:883)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:181)
>  ... 20 more
>  [g9z-Stream

[jira] [Created] (KAFKA-13246) StoreQueryIntegrationTest#shouldQueryStoresAfterAddingAndRemovingStreamThread does not gate on stream state well

[Walker Carlson (Jira)]

Walker Carlson created KAFKA-13246:
--

 Summary: 
StoreQueryIntegrationTest#shouldQueryStoresAfterAddingAndRemovingStreamThread 
does not gate on stream state well
 Key: KAFKA-13246
 URL: https://issues.apache.org/jira/browse/KAFKA-13246
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Walker Carlson


StoreQueryIntegrationTest#shouldQueryStoresAfterAddingAndRemovingStreamThread 
should be improved by waiting for the client to go to rebalancing or running 
after adding and removing a thread. It should also wait until running before 
querying the state store 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #73

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #435

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 489690 lines...]
[2021-08-27T21:49:16.134Z] > Task :raft:testClasses UP-TO-DATE
[2021-08-27T21:49:16.134Z] > Task :connect:json:testJar
[2021-08-27T21:49:16.134Z] > Task :connect:json:testSrcJar
[2021-08-27T21:49:16.134Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-08-27T21:49:16.134Z] > Task :metadata:testClasses UP-TO-DATE
[2021-08-27T21:49:16.134Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-08-27T21:49:16.134Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-08-27T21:49:16.134Z] 
[2021-08-27T21:49:16.134Z] > Task :streams:processMessages
[2021-08-27T21:49:16.134Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-08-27T21:49:16.134Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-27T21:49:16.134Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-08-27T21:49:16.134Z] 
[2021-08-27T21:49:16.134Z] > Task :streams:compileJava UP-TO-DATE
[2021-08-27T21:49:16.134Z] > Task :streams:classes UP-TO-DATE
[2021-08-27T21:49:16.134Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-08-27T21:49:17.095Z] > Task :streams:copyDependantLibs
[2021-08-27T21:49:17.095Z] > Task :streams:jar UP-TO-DATE
[2021-08-27T21:49:17.095Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-08-27T21:49:19.954Z] > Task :connect:api:javadoc
[2021-08-27T21:49:19.954Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task :connect:api:jar UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-08-27T21:49:19.954Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task :connect:json:jar UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-08-27T21:49:19.954Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-08-27T21:49:19.954Z] > Task :connect:json:publishToMavenLocal
[2021-08-27T21:49:19.954Z] > Task :connect:api:javadocJar
[2021-08-27T21:49:19.954Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-08-27T21:49:19.954Z] > Task :connect:api:testJar
[2021-08-27T21:49:19.954Z] > Task :connect:api:testSrcJar
[2021-08-27T21:49:19.954Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-08-27T21:49:19.954Z] > Task :connect:api:publishToMavenLocal
[2021-08-27T21:49:23.600Z] > Task :streams:javadoc
[2021-08-27T21:49:23.600Z] > Task :streams:javadocJar
[2021-08-27T21:49:24.557Z] > Task :clients:javadoc
[2021-08-27T21:49:25.514Z] > Task :clients:javadocJar
[2021-08-27T21:49:25.514Z] 
[2021-08-27T21:49:25.514Z] > Task :clients:srcJar
[2021-08-27T21:49:25.514Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-08-27T21:49:25.514Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-27T21:49:26.471Z] 
[2021-08-27T21:49:26.471Z] > Task :clients:testJar
[2021-08-27T21:49:27.428Z] > Task :clients:testSrcJar
[2021-08-27T21:49:27.428Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-08-27T21:49:27.428Z] > Task :clients:publishToMavenLocal
[2021-08-27T21:49:47.295Z] > Task :core:compileScala
[2021-08-27T21:50:55.394Z] > Task :core:classes
[2021-08-27T21:50:55.394Z] > Task :core:compileTestJava NO-SOURCE
[2021-08-27T21:51:21.881Z] > Task :core:compileTestScala
[2021-08-27T21:52:03.736Z] > Task :core:testClasses
[2021-08-27T21:52:18.024Z] > Task :streams:compileTestJava
[2021-08-27T21:52:18.024Z] > Task :streams:testClasses
[2021-08-27T21:52:18.024Z] > Task :streams:testJar
[2021-08-27T21:52:18.024Z] > Task :streams:testSrcJar
[2021-08-27T21:52:18.024Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal

Re: [DISCUSS] KIP-771: KRaft broker should not expose controller metrics

[Ron Dagostino]

Thanks for the KIP, Ryan.  I agree this makes sense.  It also reflects the
state of affairs right now: KRaft nodes that do not have the controller
role currently do not expose these metrics.  Assuming this KIP ends up
being accepted, we would then close KAFKA-13140 and its associated PR
https://github.com/apache/kafka/pull/11133.

Ron

On Fri, Aug 27, 2021 at 5:19 PM Ryan Dielhenn
 wrote:

> Hello kafka devs,
>
> I would like to start a discussion on a KIP I have created to change how
> controller metrics are exposed for KRaft brokers.
>
> Here is the KIP:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+771%3A+KRaft+brokers+should+not+expose+controller+metrics
>
> Regards,
> Ryan Dielhenn
>

[DISCUSS] KIP-771: KRaft broker should not expose controller metrics

[Ryan Dielhenn]

Hello kafka devs,

I would like to start a discussion on a KIP I have created to change how
controller metrics are exposed for KRaft brokers.

Here is the KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP+771%3A+KRaft+brokers+should+not+expose+controller+metrics

Regards,
Ryan Dielhenn

[jira] [Created] (KAFKA-13245) KIP 771: Change behavior of how controller metrics are exposed when using KRaft.

[Ryan Dielhenn (Jira)]

Ryan Dielhenn created KAFKA-13245:
-

 Summary: KIP 771: Change behavior of how controller metrics are 
exposed when using KRaft.
 Key: KAFKA-13245
 URL: https://issues.apache.org/jira/browse/KAFKA-13245
 Project: Kafka
  Issue Type: Improvement
Reporter: Ryan Dielhenn
Assignee: Ryan Dielhenn


Zookeeper brokers expose 0 for controller metrics in the case of controller 
failover. KRaft brokers should not since they will never be elected as 
controller and since this would introduce a non-negligible performance impact.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #72

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #434

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 43 lines...]
[2021-08-27T19:42:44.087Z] 
[2021-08-27T19:42:44.087Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
STARTED
[2021-08-27T19:42:46.491Z] 
[2021-08-27T19:42:46.491Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
PASSED
[2021-08-27T19:42:46.491Z] 
[2021-08-27T19:42:46.491Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() STARTED
[2021-08-27T19:42:47.062Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2021-08-27T19:42:47.063Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-08-27T19:42:50.682Z] 
[2021-08-27T19:42:50.682Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() PASSED
[2021-08-27T19:42:50.682Z] 
[2021-08-27T19:42:50.682Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() STARTED
[2021-08-27T19:42:56.334Z] 
[2021-08-27T19:42:56.334Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() PASSED
[2021-08-27T19:42:56.334Z] 
[2021-08-27T19:42:56.334Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() STARTED
[2021-08-27T19:43:00.294Z] 
[2021-08-27T19:43:00.294Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() PASSED
[2021-08-27T19:43:00.294Z] 
[2021-08-27T19:43:00.294Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() STARTED
[2021-08-27T19:43:02.565Z] 
[2021-08-27T19:43:02.565Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() PASSED
[2021-08-27T19:43:02.565Z] 
[2021-08-27T19:43:02.565Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() STARTED
[2021-08-27T19:43:06.621Z] 
[2021-08-27T19:43:06.621Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() PASSED
[2021-08-27T19:43:06.621Z] 
[2021-08-27T19:43:06.621Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() STARTED
[2021-08-27T19:43:09.439Z] 
[2021-08-27T19:43:09.439Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() PASSED
[2021-08-27T19:43:09.439Z] 
[2021-08-27T19:43:09.439Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() STARTED
[2021-08-27T19:43:13.608Z] 
[2021-08-27T19:43:13.608Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() PASSED
[2021-08-27T19:43:13.608Z] 
[2021-08-27T19:43:13.608Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() STARTED
[2021-08-27T19:43:15.828Z] 
[2021-08-27T19:43:15.828Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() PASSED
[2021-08-27T19:43:15.828Z] 
[2021-08-27T19:43:15.828Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithNonExistingFeatureZNode()
 STARTED
[2021-08-27T19:43:18.671Z] 
[2021-08-27T19:43:18.671Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithNonExistingFeatureZNode()
 PASSED
[2021-08-27T19:43:18.671Z] 
[2021-08-27T19:43:18.671Z] ControllerIntegrationTest > testAlterIsrErrors() 
STARTED
[2021-08-27T19:43:20.588Z] 
[2021-08-27T19:43:20.588Z] ControllerIntegrationTest > testAlterIsrErrors() 
PASSED
[2021-08-27T19:43:20.588Z] 
[2021-08-27T19:43:20.588Z] ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection() STARTED
[2021-08-27T19:43:28.126Z] 
[2021-08-27T19:43:28.126Z] ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection() PASSED
[2021-08-27T19:43:28.126Z] 
[2021-08-27T19:43:28.126Z] ControllerIntegrationTest > testTopicCreation() 
STARTED
[2021-08-27T19:43:29.933Z] 
[2021-08-27T19:43:29.933Z] ControllerIntegrationTest > testTopicCreation() 
PASSED
[2021-08-27T19:43:29.933Z] 
[2021-08-27T19:43:29.933Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithEnabledExistingFeatureZNode()
 STARTED
[2021-08-27T19:43:32.546Z] 
[2021-08-27T19:43:32.546Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithEnabledExistingFeatureZNode()
 PASSED
[2021-08-27T19:43:32.546Z] 
[2021-08-27T19:43:32.546Z] ControllerIntegrationTest > 
testControllerMoveOnTopicDeletion() STARTED
[2021-08-27T19:43:34.695Z] 
[2021-08-27T19:43:34.695Z] ControllerIntegrationTest > 
testControllerMoveOnTopicDeletion() PASSED
[2021-08-27T19:43:34.695Z] 
[2021-08-27T19:43:34.695Z] ControllerIntegrationTest > 
testPartitionReassignment() STARTED
[2021-08-27T19:43:38.255Z] 
[202

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #433

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 489922 lines...]
[2021-08-27T17:31:30.746Z] ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch() STARTED
[2021-08-27T17:31:34.432Z] 
[2021-08-27T17:31:34.432Z] ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch() PASSED
[2021-08-27T17:31:34.432Z] 
[2021-08-27T17:31:34.432Z] ControllerIntegrationTest > testIdempotentAlterIsr() 
STARTED
[2021-08-27T17:31:37.290Z] 
[2021-08-27T17:31:37.290Z] ControllerIntegrationTest > testIdempotentAlterIsr() 
PASSED
[2021-08-27T17:31:37.290Z] 
[2021-08-27T17:31:37.290Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled() STARTED
[2021-08-27T17:31:41.126Z] 
[2021-08-27T17:31:41.126Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled() PASSED
[2021-08-27T17:31:41.126Z] 
[2021-08-27T17:31:41.126Z] ControllerIntegrationTest > 
testControllerMoveOnPartitionReassignment() STARTED
[2021-08-27T17:31:43.810Z] 
[2021-08-27T17:31:43.810Z] ControllerIntegrationTest > 
testControllerMoveOnPartitionReassignment() PASSED
[2021-08-27T17:31:43.810Z] 
[2021-08-27T17:31:43.810Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithEnabledExistingFeatureZNode()
 STARTED
[2021-08-27T17:31:45.629Z] 
[2021-08-27T17:31:45.629Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithEnabledExistingFeatureZNode()
 PASSED
[2021-08-27T17:31:45.629Z] 
[2021-08-27T17:31:45.629Z] ControllerIntegrationTest > 
testControllerMoveOnTopicCreation() STARTED
[2021-08-27T17:31:48.313Z] 
[2021-08-27T17:31:48.313Z] ControllerIntegrationTest > 
testControllerMoveOnTopicCreation() PASSED
[2021-08-27T17:31:48.313Z] 
[2021-08-27T17:31:48.313Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithDisabledExistingFeatureZNode()
 STARTED
[2021-08-27T17:31:50.104Z] 
[2021-08-27T17:31:50.104Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithDisabledExistingFeatureZNode()
 PASSED
[2021-08-27T17:31:50.104Z] 
[2021-08-27T17:31:50.104Z] ControllerIntegrationTest > 
testControllerRejectControlledShutdownRequestWithStaleBrokerEpoch() STARTED
[2021-08-27T17:31:52.788Z] 
[2021-08-27T17:31:52.788Z] ControllerIntegrationTest > 
testControllerRejectControlledShutdownRequestWithStaleBrokerEpoch() PASSED
[2021-08-27T17:31:52.788Z] 
[2021-08-27T17:31:52.788Z] ControllerIntegrationTest > 
testBackToBackPreferredReplicaLeaderElections() STARTED
[2021-08-27T17:31:59.901Z] 
[2021-08-27T17:31:59.901Z] ControllerIntegrationTest > 
testBackToBackPreferredReplicaLeaderElections() PASSED
[2021-08-27T17:31:59.901Z] 
[2021-08-27T17:31:59.901Z] ControllerIntegrationTest > 
testTopicIdUpgradeAfterReassigningPartitions() STARTED
[2021-08-27T17:32:08.434Z] 
[2021-08-27T17:32:08.434Z] ControllerIntegrationTest > 
testTopicIdUpgradeAfterReassigningPartitions() PASSED
[2021-08-27T17:32:08.434Z] 
[2021-08-27T17:32:08.434Z] ControllerIntegrationTest > 
testTopicIdPersistsThroughControllerReelection() STARTED
[2021-08-27T17:32:12.625Z] 
[2021-08-27T17:32:12.625Z] ControllerIntegrationTest > 
testTopicIdPersistsThroughControllerReelection() PASSED
[2021-08-27T17:32:12.625Z] 
[2021-08-27T17:32:12.625Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithNonExistingFeatureZNode()
 STARTED
[2021-08-27T17:32:15.299Z] 
[2021-08-27T17:32:15.299Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithNonExistingFeatureZNode()
 PASSED
[2021-08-27T17:32:15.299Z] 
[2021-08-27T17:32:15.299Z] ControllerIntegrationTest > testEmptyCluster() 
STARTED
[2021-08-27T17:32:17.972Z] 
[2021-08-27T17:32:17.972Z] ControllerIntegrationTest > testEmptyCluster() PASSED
[2021-08-27T17:32:17.972Z] 
[2021-08-27T17:32:17.972Z] ControllerIntegrationTest > 
testControllerMoveOnPreferredReplicaElection() STARTED
[2021-08-27T17:32:19.755Z] 
[2021-08-27T17:32:19.755Z] ControllerIntegrationTest > 
testControllerMoveOnPreferredReplicaElection() PASSED
[2021-08-27T17:32:19.755Z] 
[2021-08-27T17:32:19.755Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElection() STARTED
[2021-08-27T17:32:24.439Z] 
[2021-08-27T17:32:24.439Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElection() PASSED
[2021-08-27T17:32:24.439Z] 
[2021-08-27T17:32:24.439Z] ControllerIntegrationTest > 
testMetadataPropagationOnBrokerChange() STARTED
[2021-08-27T17:32:28.532Z] 
[2021-08-27T17:32:28.532Z] ControllerIntegrationTest > 
testMetadataPropagationOnBrokerChange() PASSED
[2021-08-27T17:32:28.532Z] 
[2021-08-27T17:32:28.532Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithDisabledExistingFeatureZNode()
 STARTED
[2021-08-27T

[jira] [Created] (KAFKA-13244) Control Which Brokers Host Partitions of Newly Created Topics

[Michael Jaschob (Jira)]

Michael Jaschob created KAFKA-13244:
---

 Summary: Control Which Brokers Host Partitions of Newly Created 
Topics
 Key: KAFKA-13244
 URL: https://issues.apache.org/jira/browse/KAFKA-13244
 Project: Kafka
  Issue Type: New Feature
  Components: admin
Reporter: Michael Jaschob


When new topics are created through the admin interface, the Kafka controller 
creates the partition assignments for these topics according to the algorithm 
defined in {{AdminUtils.assignReplicasToBrokers}}. All (alive) brokers in the 
cluster become candidates for hosting partitions.

However, sometimes cluster administrators don't want certain brokers to host 
partitions for newly created topics. This can be for a variety of reasons, e.g.:
 * a broker may be on a host that is slated for retirement and about to be shut 
down
 * a broker may be acting as a canary of a new Kafka version and is being 
tested with a controlled set of topic-partitions, and should be restricted from 
hosting any other topics
 * similarly, a broker may in use for administrative operations, such as 
dumping log segments, and consequently may be unstable and ideally as 
little-used as possible

One solution is for the cluster admins to not give out CREATE permissions to 
clients, and instead provide a service/abstraction for creating new topics with 
explicit partition assignments. Unfortunately, in a world with Kafka Streams 
applications, this is too restrictive: Kafka Streams clients expect to use the 
Kafka tooling to create topics when their topology changes, or when they need 
to reset their apps.

Since there is currently there is no way to control which brokers are eligible 
to be assigned newly created partitions, I am proposing a new configuration 
parameter, {{create.topic.broker.filter.policy.class.name}} to allow cluster 
administrators to provide a pluggable class to control whether a broker should 
be allowed to host new partitions. The class implements a simple interface to 
give a binary yes/no verdict on each broker:
{code:java}
boolean isAllowedToHostPartitions(Broker broker);
{code}
This follows a pattern similar to {{create.topic.policy.class.name}} and 
{{alter.topic.policy.class.name}}, which also provide cluster-level control 
over other aspects of topic creation.

I have a PR ready which I'll submit after creating this ticket. I'm not sure if 
this feature idea needs a KIP - technically, it's a change to configuration, 
but on the other hand it's a pretty small change that aligns closely to 
existing functionality (create/alter topic policies). Please let me know if I 
should follow the KIP process.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13243) Differentiate metric latency measured in millis and nanos

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-13243:
-

 Summary: Differentiate metric latency measured in millis and nanos
 Key: KAFKA-13243
 URL: https://issues.apache.org/jira/browse/KAFKA-13243
 Project: Kafka
  Issue Type: Improvement
  Components: metrics
Reporter: Guozhang Wang


Today most of the client latency metrics are measured in millis, and some in 
nanos. For those measured in nanos we usually differentiate them by having a 
`-ns` suffix in the metric names, e.g. `io-wait-time-ns-avg` and 
`io-time-ns-avg`. But there are a few that we obviously forgot to follow this 
pattern, e.g. `io-wait-time-total`: it is inconsistent where `avg` has `-ns` 
suffix and `total` has not. I did a quick search and found just two of them:

* bufferpool-wait-time-total : bufferpool-wait-time-ns-total
* io-wait-time-total: io-wait-time-ns-total

We should change their name accordingly with the `-ns` suffix as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-13079) Forgotten Topics in Fetch Requests may incorrectly use topic IDs

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13079.
-
Fix Version/s: 3.1.0
 Reviewer: David Jacot
   Resolution: Fixed

> Forgotten Topics in Fetch Requests may incorrectly use topic IDs
> 
>
> Key: KAFKA-13079
> URL: https://issues.apache.org/jira/browse/KAFKA-13079
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 3.1.0
>
>
> In the new code for Fetch, we only check if the topics contained in the 
> session have IDs to decide whether to send a version < 13 (topic names) or 
> version 13+ (topic IDs) request. However, if we have an empty session that 
> previously did not use IDs, we will try to send a request to forget the 
> topics. Since all topics in the session (none) were not missing topic ids, we 
> will send a version 13 request. This request will have the Zero UUID and fail.
> The result is that we close the session and mark any partitions in it as 
> errored, but the message is confusing and the request is not correct. We 
> should somehow also track forgotten topics when deciding what version to use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13242) KRaft Controller doesn't handle UpdateFeaturesRequest

[dengziming (Jira)]

dengziming created KAFKA-13242:
--

 Summary: KRaft Controller doesn't handle UpdateFeaturesRequest
 Key: KAFKA-13242
 URL: https://issues.apache.org/jira/browse/KAFKA-13242
 Project: Kafka
  Issue Type: Bug
Reporter: dengziming






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #432

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #431

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 488910 lines...]
[2021-08-27T10:27:37.072Z] ControllerIntegrationTest > testIdempotentAlterIsr() 
PASSED
[2021-08-27T10:27:37.072Z] 
[2021-08-27T10:27:37.072Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled() STARTED
[2021-08-27T10:27:40.717Z] 
[2021-08-27T10:27:40.718Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled() PASSED
[2021-08-27T10:27:40.718Z] 
[2021-08-27T10:27:40.718Z] ControllerIntegrationTest > 
testControllerMoveOnPartitionReassignment() STARTED
[2021-08-27T10:27:43.400Z] 
[2021-08-27T10:27:43.400Z] ControllerIntegrationTest > 
testControllerMoveOnPartitionReassignment() PASSED
[2021-08-27T10:27:43.400Z] 
[2021-08-27T10:27:43.400Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithEnabledExistingFeatureZNode()
 STARTED
[2021-08-27T10:27:46.079Z] 
[2021-08-27T10:27:46.079Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithEnabledExistingFeatureZNode()
 PASSED
[2021-08-27T10:27:46.079Z] 
[2021-08-27T10:27:46.079Z] ControllerIntegrationTest > 
testControllerMoveOnTopicCreation() STARTED
[2021-08-27T10:27:49.725Z] 
[2021-08-27T10:27:49.725Z] ControllerIntegrationTest > 
testControllerMoveOnTopicCreation() PASSED
[2021-08-27T10:27:49.725Z] 
[2021-08-27T10:27:49.725Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithDisabledExistingFeatureZNode()
 STARTED
[2021-08-27T10:27:51.513Z] 
[2021-08-27T10:27:51.513Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithDisabledExistingFeatureZNode()
 PASSED
[2021-08-27T10:27:51.513Z] 
[2021-08-27T10:27:51.513Z] ControllerIntegrationTest > 
testControllerRejectControlledShutdownRequestWithStaleBrokerEpoch() STARTED
[2021-08-27T10:27:54.194Z] 
[2021-08-27T10:27:54.194Z] ControllerIntegrationTest > 
testControllerRejectControlledShutdownRequestWithStaleBrokerEpoch() PASSED
[2021-08-27T10:27:54.194Z] 
[2021-08-27T10:27:54.194Z] ControllerIntegrationTest > 
testBackToBackPreferredReplicaLeaderElections() STARTED
[2021-08-27T10:28:01.489Z] 
[2021-08-27T10:28:01.489Z] ControllerIntegrationTest > 
testBackToBackPreferredReplicaLeaderElections() PASSED
[2021-08-27T10:28:01.489Z] 
[2021-08-27T10:28:01.489Z] ControllerIntegrationTest > 
testTopicIdUpgradeAfterReassigningPartitions() STARTED
[2021-08-27T10:28:10.039Z] 
[2021-08-27T10:28:10.039Z] ControllerIntegrationTest > 
testTopicIdUpgradeAfterReassigningPartitions() PASSED
[2021-08-27T10:28:10.039Z] 
[2021-08-27T10:28:10.039Z] ControllerIntegrationTest > 
testTopicIdPersistsThroughControllerReelection() STARTED
[2021-08-27T10:28:16.061Z] 
[2021-08-27T10:28:16.061Z] ControllerIntegrationTest > 
testTopicIdPersistsThroughControllerReelection() PASSED
[2021-08-27T10:28:16.061Z] 
[2021-08-27T10:28:16.061Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithNonExistingFeatureZNode()
 STARTED
[2021-08-27T10:28:17.850Z] 
[2021-08-27T10:28:17.850Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsDisabledWithNonExistingFeatureZNode()
 PASSED
[2021-08-27T10:28:17.850Z] 
[2021-08-27T10:28:17.850Z] ControllerIntegrationTest > testEmptyCluster() 
STARTED
[2021-08-27T10:28:19.812Z] 
[2021-08-27T10:28:19.812Z] ControllerIntegrationTest > testEmptyCluster() PASSED
[2021-08-27T10:28:19.812Z] 
[2021-08-27T10:28:19.812Z] ControllerIntegrationTest > 
testControllerMoveOnPreferredReplicaElection() STARTED
[2021-08-27T10:28:22.492Z] 
[2021-08-27T10:28:22.492Z] ControllerIntegrationTest > 
testControllerMoveOnPreferredReplicaElection() PASSED
[2021-08-27T10:28:22.492Z] 
[2021-08-27T10:28:22.492Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElection() STARTED
[2021-08-27T10:28:27.185Z] 
[2021-08-27T10:28:27.185Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElection() PASSED
[2021-08-27T10:28:27.185Z] 
[2021-08-27T10:28:27.185Z] ControllerIntegrationTest > 
testMetadataPropagationOnBrokerChange() STARTED
[2021-08-27T10:28:33.035Z] 
[2021-08-27T10:28:33.035Z] ControllerIntegrationTest > 
testMetadataPropagationOnBrokerChange() PASSED
[2021-08-27T10:28:33.035Z] 
[2021-08-27T10:28:33.035Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithDisabledExistingFeatureZNode()
 STARTED
[2021-08-27T10:28:34.824Z] 
[2021-08-27T10:28:34.824Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithDisabledExistingFeatureZNode()
 PASSED
[2021-08-27T10:28:34.824Z] 
[2021-08-27T10:28:34.824Z] ControllerIntegrationTest > 
testMetadataPropagationForOfflineReplicas() STARTED
[2021-08-27T10:28:45.156Z] 
[2021-08-27T10:28:45.156Z] ControllerIntegrationTest

[jira] [Created] (KAFKA-13241) Name resolution should be disabled during SASL authentication

[AndrewDi (Jira)]

AndrewDi created KAFKA-13241:


 Summary: Name resolution should be disabled during SASL 
authentication
 Key: KAFKA-13241
 URL: https://issues.apache.org/jira/browse/KAFKA-13241
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 2.8.0
 Environment: Redhat linux
Reporter: AndrewDi


{code:java}
 LoginManager loginManager = loginManagers.get(clientSaslMechanism);
authenticatorCreator = () -> buildClientAuthenticator(configs,
saslCallbackHandlers.get(clientSaslMechanism),
id,
socket.getInetAddress().getHostName(),
loginManager.serviceName(),
transportLayer,
subjects.get(clientSaslMechanism));{code}
When using SASL authentication, kafka will always try to do hostname resolution 
when build client authenticator, this is unnecessary, if we use ip to connect 
to kafka server, and didn't config kafka server hostname resolution, kafka 
client will suck here for about 10s, and then timeout with 
java.net.UnknownHostException but client can auth success anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13240) HTTP TRACE should be disabled in Connect

[Viktor Somogyi-Vass (Jira)]

Viktor Somogyi-Vass created KAFKA-13240:
---

 Summary: HTTP TRACE should be disabled in Connect
 Key: KAFKA-13240
 URL: https://issues.apache.org/jira/browse/KAFKA-13240
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


Modern browsers mostly disable HTTP TRACE to prevent XST (cross-site tracking) 
attacks.  Because of this usually this type of attack isn't too prevalent these 
days but since it isn't disabled in Connect it may open up possible ways of 
attacks (and constantly pops up in security scans :) ). Therefore we'd like to 
disable it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #121

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 2.8 #71

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #430

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 417188 lines...]
[2021-08-27T08:23:30.703Z] 
[2021-08-27T08:23:30.703Z] TopicCommandIntegrationTest > 
testListTopicsWithIncludeList() PASSED
[2021-08-27T08:23:30.703Z] 
[2021-08-27T08:23:30.703Z] TopicCommandIntegrationTest > testTopicDeletion() 
STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] > Task :streams:integrationTest
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2021-08-27T08:23:31.660Z] 
[2021-08-27T08:23:31.660Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2021-08-27T08:23:50.913Z] 
[2021-08-27T08:23:50.913Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2021-08-27T08:23:50.913Z] 
[2021-08-27T08:23:50.913Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] > Task :core:integrationTest
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > testTopicDeletion() 
PASSED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() STARTED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() PASSED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() STARTED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() PASSED
[2021-08-27T08:24:01.227Z] 
[2021-08-27T08:24:01.227Z] TopicCommandIntegrationTest > 
testDescribeWhenTopicDoesntExist() STARTED
[2021-08-27T08:24:17.840Z] 
[2021-08-27T08:24:17.840Z] TopicCommandIntegrationTest > 
testDescribeWhenTopicDoesntExist() PASSED
[2021-08-27T08:24:17.840Z] 
[2021-08-27T08:24:17.840Z] TopicCommandIntegrationTest > 
testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized() STARTED
[2021-08-27T08:24:20.521Z] 
[2021-08-27T08:24:20.521Z] > Task :streams:integrationTest
[2021-08-27T08:24:20.521Z] 
[2021-08-27T08:24:20.521Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2021-08-27T08:24:20.521Z] 
[2021-08-27T08:24:20.521Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2021-08-27T08:24:24.053Z] 
[2021-08-27T08:24:24.053Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2021-08-27T08:24:24.053Z] 
[2021-08-27T08:24:24.053Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorMan