Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[Luke Chen]

Hi Alexandre,

Thanks for the KIP.

Some comments:

1. Usually, in "Proposed Changes" section, we won't put PR link there.
Maybe you can take KIP-824

as a reference.

2. Also, it'd be better you can provide some example reader-config and
formatter-config file.
And how they work within the script.

3. If user provide both `--reader-config or --formatter-config` and
`--property` at the same time, how will we handle this case?
Could you add that into the KIP?

Thank you.
Luke


On Sun, May 29, 2022 at 2:32 AM Alexandre Garnier 
wrote:

> Hi!
>
> Thanks for the feedback.
> It's a good point, I updated KIP accordingly and did put the
> dot-separated option in rejected alternatives.
>
> Le ven. 27 mai 2022 à 10:22, deng ziming  a
> écrit :
> >
> > Thanks for the KIP, this is a good improvement. I only have one minor
> suggestion.
> >
> > Currently many command line tools supports config file argument, but
> their name style is not unified, for example, most newly added tools are
> using --command-config, but ConsoleConsumer use —consumer.config。 I think
> we should unify the naming style from now on, I recommend us to use
> --reader-config and --formatter-config for the newly added arguments.
> >
> > --
> > Best,
> > Ziming
> >
> >
> > > On May 26, 2022, at 4:36 PM, Alexandre Garnier 
> wrote:
> > >
> > > Hello everyone,
> > >
> > > Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD?
> > > It is a straightforward improvement without any impact on existing
> users,
> > > so not much to discuss besides maybe the option name.
> > >
> > > --
> > > Alex
> > >
> > >
> > > Le mer. 18 mai 2022 à 10:44, Alexandre Garnier  a
> écrit :
> > >
> > >> Hi everyone,
> > >>
> > >> I created a KIP to add a config file option of reader/formatter for
> > >> kafka-console-(consumer|producer).sh tools.
> > >> https://cwiki.apache.org/confluence/x/bBqhD
> > >>
> > >> Thanks for your feedback,
> > >> --
> > >> Alex
> > >>
> >
>

[GitHub] [kafka-site] showuon opened a new pull request, #416: MINOR: add java 8 deprecation note and java 17 support

[GitBox]

showuon opened a new pull request, #416:
URL: https://github.com/apache/kafka-site/pull/416

   refer to : https://github.com/apache/kafka/pull/12261


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[John Roesler]

Thanks for the KIP, Alexandre!

I’m +1 (binding)

-John

On Wed, Jun 8, 2022, at 20:54, deng ziming wrote:
> Thank you for this KIP,
>
> +1 (non-binding)
>
> -- 
> Best,
> Ziming
>
>> On Jun 7, 2022, at 8:53 PM, Alexandre Garnier  wrote:
>> 
>> Hi!
>> 
>> A little reminder to vote for this KIP.
>> 
>> Thanks.
>> 
>> 
>> Le mer. 1 juin 2022 à 10:58, Alexandre Garnier  a écrit :
>>> 
>>> Hi everyone!
>>> 
>>> I propose to start voting for KIP-840:
>>> https://cwiki.apache.org/confluence/x/bBqhD
>>> 
>>> Thanks,
>>> --
>>> Alex

Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[deng ziming]

Thank you for this KIP,

+1 (non-binding)

-- 
Best,
Ziming

> On Jun 7, 2022, at 8:53 PM, Alexandre Garnier  wrote:
> 
> Hi!
> 
> A little reminder to vote for this KIP.
> 
> Thanks.
> 
> 
> Le mer. 1 juin 2022 à 10:58, Alexandre Garnier  a écrit :
>> 
>> Hi everyone!
>> 
>> I propose to start voting for KIP-840:
>> https://cwiki.apache.org/confluence/x/bBqhD
>> 
>> Thanks,
>> --
>> Alex

Re: [VOTE] KIP-843: Adding metricOrElseCreate method to Metrics

[Sagar]

Hi all,

I’m closing the voting on this one. We already had 3 binding votes on this
from Guozhang, Bruno and John.

Thanks everyone!

Sagar.

On Sun, 5 Jun 2022 at 1:53 PM, Sagar  wrote:

> Hi All,
>
> I am planning to use the name of the new function as addMetricIfAbsent.
> Let me know if there are any other suggestions/objections to this.
>
> Also, @Ismael, I agree with Guozhang about not deprecating the existing
> functions. Plz let me know if you have any reservations about that.
>
> If we all agree to these 2 points, then I would mark the KIP as voted.
>
> Thanks!
> Sagar.
>
> On Wed, Jun 1, 2022 at 11:47 PM Guozhang Wang  wrote:
>
>> I think `addMetricIfAbsent` is a good function name, and like John said
>> people in the Java world are familiar with its return value semantics as
>> well.
>>
>> Regarding deprecating the existing functions, I feel it is not necessary
>> just for the function semantics difference between `sensors` and `metrics`
>> (Ismael may chime in here if you have other good reasons).
>>
>>
>> Guozhang
>>
>> On Wed, Jun 1, 2022 at 9:50 AM Sagar  wrote:
>>
>> > So, we have multiple options in terms of names, at this point I actually
>> > liked John's suggestion to use addMetricIfAbsent or something along
>> those
>> > lines.
>> >
>> > Regarding the deprecation of sensor/metric method, I am not sure...
>> Would
>> > like to know others' thoughts.
>> >
>> > Thanks!
>> > Sagar.
>> >
>> > On Wed, Jun 1, 2022 at 2:28 AM Guozhang Wang 
>> wrote:
>> >
>> > > Hey Ismael, just checking do you mean the `metric` method instead?
>> > >
>> > > On Tue, May 31, 2022 at 1:45 PM Ismael Juma 
>> wrote:
>> > >
>> > > > Should we deprecate the `sensor` method? One other thing to take
>> into
>> > > > account is that these methods are meant to be used like a dsl for
>> > > > configuring sensors and metrics. So brevity is a plus (but clarity
>> is
>> > > > critical still).
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Tue, May 31, 2022 at 11:09 AM John Roesler 
>> > > wrote:
>> > > >
>> > > > > Generally, I agree with Ismael that having a new, weird name will
>> > make
>> > > it
>> > > > > hard to keep them straight. Then again, we need to make them
>> > different
>> > > to
>> > > > > prevent confusion about their semantics. To be clear, I'll be a +1
>> > > > > regardless of how we break this dilemma.
>> > > > >
>> > > > > One suggestion: We currently have addMetric to add a new metric.
>> We
>> > can
>> > > > > take some inspiration from the Java Map interface and call this
>> new
>> > > > method
>> > > > > `addMetricIfAbsent`. Having the same prefix should help discovery,
>> > and
>> > > > > following the Map convention should help confusion.
>> > > > >
>> > > > > Thanks all,
>> > > > > -John
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, May 31, 2022, at 12:13, Sagar wrote:
>> > > > > > Oh yeah there's another metric function which is get-only. I
>> think
>> > we
>> > > > > > should go ahead with getOrCreateMetric.
>> > > > > >
>> > > > > > Thanks!
>> > > > > > Sagar.
>> > > > > >
>> > > > > > On Tue, May 31, 2022 at 10:02 PM Guozhang Wang <
>> wangg...@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > >> I'd prefer the getOrCreateMetric function name, since for the
>> > > > existing "
>> > > > > >> sensor(String name)" function that only takes a single `String`
>> > > > > parameter,
>> > > > > >> its semantics is already "get or create". Whereas the existing
>> > > > > >> "metric(MetricName)" function's semantics is "get" only. So in
>> my
>> > > > mind,
>> > > > > the
>> > > > > >> inconsistent conventions in function signatures already exist
>> > today.
>> > > > And
>> > > > > >> with the other option we would need to educate users that "all
>> the
>> > > > > `sensor`
>> > > > > >> functions are get-or-create, but, please remember that the
>> > `metric`
>> > > > > >> function with just the metric name is get-only, while other
>> > `metric`
>> > > > > >> overrides with more parameters are get-or-create", which I
>> think
>> > is
>> > > > even
>> > > > > >> more confusing.
>> > > > > >>
>> > > > > >>
>> > > > > >> Guozhang
>> > > > > >>
>> > > > > >>
>> > > > > >> On Mon, May 30, 2022 at 9:51 PM Sagar <
>> sagarmeansoc...@gmail.com>
>> > > > > wrote:
>> > > > > >>
>> > > > > >> > Hi Ismael,
>> > > > > >> >
>> > > > > >> > I guess in that case, we will have to go with the name
>> *metric*-
>> > > > > similar
>> > > > > >> to
>> > > > > >> > *sensor* - which David pointed out above because I think
>> that's
>> > > the
>> > > > > >> closest
>> > > > > >> > method which either gets or creates a new sensor. Current
>> > > addMetric
>> > > > in
>> > > > > >> the
>> > > > > >> > Metrics class throw an IllegalArguementException when the
>> metric
>> > > > > already
>> > > > > >> > exists and that's why I still think getOrCreateMetric still
>> > > > signifies
>> > > > > the
>> > > > > >> > action correctly. Or how about addOrGetMetric or
>> getOrAddMetric,
>> > > > just
>> > > > > >> > replacing 

[jira] [Created] (KAFKA-13972) Reassignment cancellation causes stray replicas

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13972:
---

 Summary: Reassignment cancellation causes stray replicas
 Key: KAFKA-13972
 URL: https://issues.apache.org/jira/browse/KAFKA-13972
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


A stray replica is one that is left behind on a broker after the partition has 
been reassigned to other brokers or the partition has been deleted. We found 
one case where this can happen is after a cancelled reassignment. When a 
reassignment is cancelled, the controller sends `StopReplica` requests to any 
of the adding replicas, but it does not necessarily bump the leader epoch. 
Following 
[KIP-570|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-570%3A+Add+leader+epoch+in+StopReplicaRequest],]
 brokers will ignore `StopReplica` requests if the leader epoch matches the 
current partition leader epoch. So we need to bump the epoch whenever we need 
to ensure that `StopReplica` will be received.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Jorge Esteban Quilcate Otoya]

Thanks, Chris!

Please, find my comments below:

On Tue, 7 Jun 2022 at 04:39, Chris Egerton  wrote:

> Hi Jorge,
>
> Thanks! Sorry for the delay; here are my thoughts:
>
> 1. Under the "Accessing multiple values by deep-scan" header it's stated
> that "If deep-scan is used, it must have only one field after the asterisk
> level.". However, in example 3 for the Cast SMT and other examples for
> other SMTs, the spec contains a field of "*.child.k2", which appears to
> have two fields after the asterisk level. I may be misunderstanding the
> proposal, but it seems like the two contradict each other.
>

Thanks for catching this. I have clarified it by removing this restriction.
Also, have extended the deep-scan scenarios.


>
> 2. I'm a little unclear on why we need the special handling for arrays
> where, for an array field "a", the field name "a" can be treated as either
> the array itself, or every element in the array. Is there a reason we can't
> use the field name "a.*" to handle the latter case, and "a" to handle the
> former?
>

Agree, this is confusing. I like the `a.*` approach to access array items.
I have added this to the proposal.


>
> 3. How would a user specify that they'd like to access a field with the
> literal name "*"?
>

Good one. I'm proposing an approach similar to how it's proposed to escape
dots, with a double-asterisk. Curious on your thoughts around this.


>
> 4. For the Cast SMT, do you think it might bite some people if fields that
> can't be cast correctly are silently ignored? I'm imagining the case where
> none of the fields in a multi-path expression can be cast correctly and it
> ends up eating half of someone's day to track down why their SMT isn't
> doing anything.
>

If I understand correctly, this challenge could be relevant across SMTs.
At the moment, most/all? SMTs just silently ignore.
Was thinking about adding a flag `field.on.path.not.found` to either ignore
or fail when no paths are found. What do your think?


>
> 5. For the ExtractField and ValueToKey SMTs, what happens if a deep-scan
> field name is used, but only one field is found? Is the resulting field
> still an array, or is it just the single field that was found? (FWIW I'm
> leaning towards keeping it an array just to keep schemas consistent in a
> pipeline in case the number of fields found fluctuates across records.)
>
> Agree. Will clarify that an array is always produced even for 1 or 0
fields are found.


> 6. (Nit) For the HeaderFrom SMT, it's stated that "As this SMT affects only
> existing fields, additional configurations will not be required.". Given
> the new "field.syntax.version" property, this part should probably be
> removed.
>
Agree.


>
> 7. Is recursive descent intentionally excluded? That was an important part
> of Joshua's KIP and his feedback on this KIP; I think it's worth pursuing
> if we can.
>

My understanding from Joshua's feedback is that by including support for
deep-scan, we are already covering the recursive functionality. Though, I
may be missing something.

>
>
> Cheers,
>
> Chris
>
> On Tue, May 24, 2022 at 3:49 AM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Thanks, Chris!
> >
> > I have updated the KIP with the rejected alternatives updated. Also, I
> have
> > drafted the support for arrays and deep scans as part of the proposed
> > notation to make it more complete, even though these can be implemented
> in
> > multiple PRs.
> >
> > Looking forward to your feedback.
> >
> > Cheers,
> > Jorge.
> >
> > On Sat, 21 May 2022 at 17:39, Chris Egerton 
> > wrote:
> >
> > > Hi Jorge,
> > >
> > > I really appreciate the effort you've made to simplify the syntax and
> > > feature set of a JSONPath-based approach as much as possible. I'm still
> > > hesitant to continue with it, though.
> > >
> > > 1. The syntax is much less friendly. Just compare "top.mid.bottom" to
> > > "$['top']['mid']['bottom']"... not everyone uses JSONPath or even JSON,
> > and
> > > the learning curve for the former is going to be steeper. The examples
> in
> > > the new KIP draft you published demonstrate this pretty well, and this
> is
> > > without diving into the details of what escape syntax would look like.
> > >
> > > 2. The three new major features that this syntax adds (array accesses,
> > deep
> > > scans, and multi-value paths) could all be added pretty easily to the
> dot
> > > notation syntax without introducing brackets and dollar signs. Array
> > > accesses can be described using the same syntax as struct/map field
> > access,
> > > deep scans can be described using '*', and multi-value paths can be
> > > described by referencing the name of a field that's expected to have
> > > children. These are all top-of-the-head ideas and can probably be
> > refined,
> > > but hopefully they demonstrate that we can keep the syntax simple
> without
> > > sacrificing features. Of course, the question of leaving room for
> future
> > > features might arise... given that these 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #988

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 482875 lines...]
[2022-06-08T20:57:24.643Z] 
[2022-06-08T20:57:24.643Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi STARTED
[2022-06-08T20:58:10.178Z] 
[2022-06-08T20:58:10.178Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi PASSED
[2022-06-08T20:58:10.178Z] 
[2022-06-08T20:58:10.178Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi STARTED
[2022-06-08T20:59:03.410Z] 
[2022-06-08T20:59:03.410Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi PASSED
[2022-06-08T20:59:03.410Z] 
[2022-06-08T20:59:03.410Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigrateInMemoryWindowStoreToTimestampedWindowStoreUsingPapi STARTED
[2022-06-08T20:59:46.569Z] 
[2022-06-08T20:59:46.569Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigrateInMemoryWindowStoreToTimestampedWindowStoreUsingPapi PASSED
[2022-06-08T20:59:46.569Z] 
[2022-06-08T20:59:46.569Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-06-08T20:59:47.594Z] 
[2022-06-08T20:59:47.594Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-06-08T20:59:47.594Z] 
[2022-06-08T20:59:47.594Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2022-06-08T20:59:50.222Z] 
[2022-06-08T20:59:50.222Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2022-06-08T20:59:50.222Z] 
[2022-06-08T20:59:50.222Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2022-06-08T20:59:52.850Z] 
[2022-06-08T20:59:52.850Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2022-06-08T20:59:52.850Z] 
[2022-06-08T20:59:52.850Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-06-08T20:59:57.551Z] 
[2022-06-08T20:59:57.551Z] 
org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2022-06-08T20:59:57.551Z] 
[2022-06-08T20:59:57.551Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceThreads STARTED
[2022-06-08T20:59:59.306Z] 
[2022-06-08T20:59:59.307Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-365619bc-51a4-4335-9f9b-64129263e9bf-StreamThread-1"
[2022-06-08T21:00:02.305Z] 
[2022-06-08T21:00:02.305Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-365619bc-51a4-4335-9f9b-64129263e9bf-StreamThread-2"
[2022-06-08T21:00:05.993Z] 
[2022-06-08T21:00:05.993Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceThreads PASSED
[2022-06-08T21:00:05.993Z] 
[2022-06-08T21:00:05.993Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException STARTED
[2022-06-08T21:00:07.182Z] 
[2022-06-08T21:00:07.182Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException PASSED
[2022-06-08T21:00:07.182Z] 
[2022-06-08T21:00:07.182Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler STARTED
[2022-06-08T21:00:13.969Z] 
[2022-06-08T21:00:13.969Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler PASSED
[2022-06-08T21:00:13.969Z] 
[2022-06-08T21:00:13.969Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread STARTED
[2022-06-08T21:00:14.908Z] 
[2022-06-08T21:00:14.908Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-8780c662-b788-42cb-9934-d6bb3a3d95b8-GlobalStreamThread"
[2022-06-08T21:00:14.908Z] 
[2022-06-08T21:00:14.908Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread PASSED

[jira] [Resolved] (KAFKA-13955) Fix failing KRaftClusterTest tests

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-13955.

Resolution: Fixed

> Fix failing KRaftClusterTest tests
> --
>
> Key: KAFKA-13955
> URL: https://issues.apache.org/jira/browse/KAFKA-13955
> Project: Kafka
>  Issue Type: Test
>Reporter: Luke Chen
>Assignee: dengziming
>Priority: Major
>
> Tests are failing with timeout exception
> java.util.concurrent.TimeoutException: 
> testCreateClusterAndPerformReassignment() timed out after 120 seconds
>  
> Failing tests:
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testIncrementalAlterConfigs()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testSetLog4jConfigurations()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testLegacyAlterConfigs()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testCreateClusterAndPerformReassignment()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testUnregisterBroker()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testCreateClusterAndCreateAndManyTopics()
> Build / JDK 8 and Scala 2.12 / 
> kafka.server.KRaftClusterTest.testCreateClusterAndCreateListDeleteTopic()



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13971) Atomicity violations caused by improper usage of ConcurrentHashMap

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-13971:


 Summary: Atomicity violations caused by improper usage of 
ConcurrentHashMap
 Key: KAFKA-13971
 URL: https://issues.apache.org/jira/browse/KAFKA-13971
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya


*Code:* 
[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConfigTransformer.java#L81-L84]
 

*Problem*
We are using a {{{}ConcurrentHashMap{}}}, but usage of {{{}get(){}}}, {{null}} 
check, and {{put()}} may not be thread-safe at lines: 81, 82, and 84. Two 
threads can perform this same check at the same time and one thread can 
overwrite the value written by the other thread.

*Fix*
Consider replacing {{put()}} with {{putIfAbsent()}} to help prevent accidental 
overwriting. {{putIfAbsent()}} puts the value only if the {{ConcurrentHashMap}} 
does not contain the key and therefore avoids overwriting the value written 
there by the other thread's {{{}putIfAbsent(){}}}.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13944) Shutting down broker can be elected as partition leader in KRaft

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13944.
-
Fix Version/s: 3.3.0
 Assignee: David Jacot  (was: Jose Armando Garcia Sancio)
   Resolution: Fixed

This has been fixed by https://github.com/apache/kafka/pull/12240.

> Shutting down broker can be elected as partition leader in KRaft
> 
>
> Key: KAFKA-13944
> URL: https://issues.apache.org/jira/browse/KAFKA-13944
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: David Jacot
>Priority: Major
>  Labels: kip-500
> Fix For: 3.3.0
>
>
> When a broker requests shutdown, it transitions to the CONTROLLED_SHUTDOWN 
> state in the controller. It is possible for the broker to remain unfenced in 
> this state until the controlled shutdown completes. When doing an election, 
> the only thing we generally check is that the broker is unfenced, so this 
> means we can elect a broker that is in controlled shutdown. 
> Here are a few snippets from a recent system test in which this occurred:
> {code:java}
> // broker 2 starts controlled shutdown
> [2022-05-26 21:17:26,451] INFO [Controller 3001] Unfenced broker 2 has 
> requested and been granted a controlled shutdown. 
> (org.apache.kafka.controller.BrokerHeartbeatManager)
>  
> // there is only one replica, so we set leader to -1
> [2022-05-26 21:17:26,452] DEBUG [Controller 3001] partition change for _foo-1 
> with topic ID _iUQ72T_R4mmZgI3WrsyXw: leader: 2 -> -1, leaderEpoch: 0 -> 1, 
> partitionEpoch: 0 -> 1 (org.apache.kafka.controller.ReplicationControlManager)
> // controlled shutdown cannot complete immediately
> [2022-05-26 21:17:26,529] DEBUG [Controller 3001] The request from broker 2 
> to shut down can not yet be granted because the lowest active offset 177 is 
> not greater than the broker's shutdown offset 244. 
> (org.apache.kafka.controller.BrokerHeartbeatManager)
> [2022-05-26 21:17:26,530] DEBUG [Controller 3001] Updated the controlled 
> shutdown offset for broker 2 to 244. 
> (org.apache.kafka.controller.BrokerHeartbeatManager)
> // later on we elect leader 2 again
> [2022-05-26 21:17:27,703] DEBUG [Controller 3001] partition change for _foo-1 
> with topic ID _iUQ72T_R4mmZgI3WrsyXw: leader: -1 -> 2, leaderEpoch: 1 -> 2, 
> partitionEpoch: 1 -> 2 (org.apache.kafka.controller.ReplicationControlManager)
> // now controlled shutdown is stuck because of the newly elected leader
> [2022-05-26 21:17:28,531] DEBUG [Controller 3001] Broker 2 is in controlled 
> shutdown state, but can not shut down because more leaders still need to be 
> moved. (org.apache.kafka.controller.BrokerHeartbeatManager)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13948) Don't catch and re-throw exception unnecessarily

[Christo Lolov (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christo Lolov resolved KAFKA-13948.
---
Resolution: Not A Problem

> Don't catch and re-throw exception unnecessarily
> 
>
> Key: KAFKA-13948
> URL: https://issues.apache.org/jira/browse/KAFKA-13948
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Divij Vaidya
>Assignee: Christo Lolov
>Priority: Trivial
>  Labels: Newbie, Newcomer, newbie
>
> An exception is caught and re-thrown within the catch clause without any 
> other operations being performed on the exception, making the catch clause 
> unnecessary.
> Code should be changed here: 
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorStateManager.java



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Should we automatically close stale PRs?

[Viktor Somogyi-Vass]

>One thing that might make sense to do maybe is to add frequent contributors
>with the "triage" role, so they could label PRs they reviewed and they can
>be taken by committers for a further review and potential merge. What do
>you think?

In addition to labeling commits as stale I think the opposite as said above
(triage and label new PRs) is a very good idea too. We often try to review
each other's commits in the team before/after publishing them upstream.
Such commits I think would filter the incoming PRs well and make overall
quality better.
Would it be possible to do this? I'd be happy to be triager.

On Sun, Feb 27, 2022 at 4:23 AM Guozhang Wang  wrote:

> Hey David,
>
> Just reviving on this thread, do you have some final decision on this now
> with all the feedbacks received so far?
>
> On Sun, Feb 13, 2022 at 8:41 PM Ismael Juma  wrote:
>
> > Hi David,
> >
> > I think it's a good idea to use the bot for auto closing stale PRs. The
> > ideal flow would be:
> >
> > 1. Write a comment and add stale label
> > 2. If user responds saying that the PR is still valid, the stale label is
> > removed
> > 3. Otherwise, the PR is closed
> >
> > Thanks,
> > Ismael
> >
> > On Sat, Feb 5, 2022, 2:22 AM David Jacot  wrote:
> >
> > > Hi team,
> > >
> > > I find our ever growing back of PRs a little frustrating, don't
> > > you? I just made a pass over all the list and a huge chunk
> > > of the PRs are abandoned, outdated or irrelevant with the
> > > current code base. For instance, we still have PRs opened
> > > back in 2015.
> > >
> > > There is not a Github Action [1] for automatically marking
> > > PRs as stale and to automatically close them as well. How
> > > would the community feel about enabling this? I think that
> > > we could mark a PR as stable after one year and close it
> > > a month after if there are no new activities. Reopening a
> > > closed PR is really easy so there is no real arm is closing
> > > it.
> > >
> > > [1] https://github.com/actions/stale
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-830: Allow disabling JMX Reporter

[Mickael Maison]

Hi,

Thanks everybody for the votes and feedback!

The vote passes with:
- 3 +1 (binding) from Tom, Ismael and Luke
- 1 +1 (non-binding) from Federico

Mickael

On Wed, Jun 8, 2022 at 12:23 PM Luke Chen  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP.
> It's a good improvement.
> Although it's a little weird that we introduce a "new deprecated" config,
> it seems like this is the best solution we can have now.
>
> +1 (binding) from me.
>
> Thank you.
> Luke
>
> On Thu, May 26, 2022 at 12:21 AM Ismael Juma  wrote:
>
> > Thanks, +1 (binding).
> >
> > Ismael
> >
> > On Thu, May 19, 2022 at 5:04 AM Mickael Maison 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > That's a good idea, I've updated the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, May 17, 2022 at 4:17 PM Federico Valeri 
> > > wrote:
> > > >
> > > > +1 (non binding)
> > > >
> > > > Thanks.
> > > >
> > > > On Tue, May 17, 2022 at 3:47 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'd like to start a vote on KIP-830 which proposes a method for a
> > > > > method for disabling JMXReporter and making JMXReporter behave like
> > > > > other reporters in the next major release when it will need to be
> > > > > explicitly listed in metric.reporters to be enabled.
> > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-830%3A+Allow+disabling+JMX+Reporter
> > > > >
> > > > > Let me know if you have feedback,
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > >
> >

Re: Upgrade log4j to log4j2

[Luke Chen]

Hi Nicola,

Here's the plan for log4j and log4j2 in Kafka project:
- adopt reload4j in Kafka 3.2.0
- switch to log4j2 to Kafka 4.0.0

Thank you.
Luke

On Tue, Jun 7, 2022 at 11:18 PM Nozza, Nicola
 wrote:

> Hi,
>
> I'm contacting you to please ask for an ETA, if it is planned, for the
> release in the following link:
>
> [KAFKA-9366] Upgrade log4j to log4j2 - ASF JIRA (apache.org)<
> https://issues.apache.org/jira/browse/KAFKA-9366>
>
> Hope to hear from you soon,
>
> Thanks,
>
> Nicola Nozza
>
> 
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy. Your privacy is important to us.
> Accenture uses your personal data only in compliance with data protection
> laws. For further information on how Accenture processes your personal
> data, please see our privacy statement at
> https://www.accenture.com/us-en/privacy-policy.
>
> __
>
> www.accenture.com
>

[jira] [Created] (KAFKA-13970) TopicAdmin topic creation should be retried on TimeoutException

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-13970:


 Summary: TopicAdmin topic creation should be retried on 
TimeoutException
 Key: KAFKA-13970
 URL: https://issues.apache.org/jira/browse/KAFKA-13970
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Daniel Urban


org.apache.kafka.connect.util.TopicAdmin#createTopicsWithRetry handles the case 
when there aren't enough brokers in the cluster to create a topic with the 
expected replication factor. This logic should also handle the case when there 
are 0 brokers in the cluster, and should retry in that case.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-830: Allow disabling JMX Reporter

[Luke Chen]

Hi Mickael,

Thanks for the KIP.
It's a good improvement.
Although it's a little weird that we introduce a "new deprecated" config,
it seems like this is the best solution we can have now.

+1 (binding) from me.

Thank you.
Luke

On Thu, May 26, 2022 at 12:21 AM Ismael Juma  wrote:

> Thanks, +1 (binding).
>
> Ismael
>
> On Thu, May 19, 2022 at 5:04 AM Mickael Maison 
> wrote:
>
> > Hi Ismael,
> >
> > That's a good idea, I've updated the KIP.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 17, 2022 at 4:17 PM Federico Valeri 
> > wrote:
> > >
> > > +1 (non binding)
> > >
> > > Thanks.
> > >
> > > On Tue, May 17, 2022 at 3:47 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'd like to start a vote on KIP-830 which proposes a method for a
> > > > method for disabling JMXReporter and making JMXReporter behave like
> > > > other reporters in the next major release when it will need to be
> > > > explicitly listed in metric.reporters to be enabled.
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-830%3A+Allow+disabling+JMX+Reporter
> > > >
> > > > Let me know if you have feedback,
> > > >
> > > > Thanks,
> > > > Mickael
> >
>

[jira] [Created] (KAFKA-13969) CVE-2022-24823 in netty 4.1.76.Final

[Dominique Mongelli (Jira)]

Dominique Mongelli created KAFKA-13969:
--

 Summary: CVE-2022-24823 in netty 4.1.76.Final
 Key: KAFKA-13969
 URL: https://issues.apache.org/jira/browse/KAFKA-13969
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Dominique Mongelli


Netty reported a new MEDIUM CVE: 
[https://github.com/netty/netty/security/advisories/GHSA-269q-hmxg-m83q]

NVD: [https://nvd.nist.gov/vuln/detail/CVE-2022-24823]

It is fixed in netty 4.1.77.Final.

It should be noted that this CVE impacts applications running on Java 6 or 
lower.

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #986

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 636268 lines...]
[2022-06-08T07:47:12.697Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = false] PASSED
[2022-06-08T07:47:12.697Z] 
[2022-06-08T07:47:12.697Z] 
org.apache.kafka.streams.integration.TaskMetadataIntegrationTest > 
shouldReportCorrectEndOffsetInformation STARTED
[2022-06-08T07:47:13.728Z] 
[2022-06-08T07:47:13.728Z] 
org.apache.kafka.streams.integration.TaskMetadataIntegrationTest > 
shouldReportCorrectEndOffsetInformation PASSED
[2022-06-08T07:47:13.728Z] 
[2022-06-08T07:47:13.728Z] 
org.apache.kafka.streams.integration.TaskMetadataIntegrationTest > 
shouldReportCorrectCommittedOffsetInformation STARTED
[2022-06-08T07:47:15.659Z] 
[2022-06-08T07:47:15.659Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException PASSED
[2022-06-08T07:47:15.659Z] 
[2022-06-08T07:47:15.659Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler STARTED
[2022-06-08T07:47:16.068Z] 
[2022-06-08T07:47:16.069Z] 
org.apache.kafka.streams.integration.TaskMetadataIntegrationTest > 
shouldReportCorrectCommittedOffsetInformation PASSED
[2022-06-08T07:47:21.433Z] streams-12: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-9: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-10: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-8: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-13: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-6: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-14: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-11: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:21.434Z] streams-7: SMOKE-TEST-CLIENT-CLOSED
[2022-06-08T07:47:22.373Z] 
[2022-06-08T07:47:22.373Z] 642 tests completed, 1 failed
[2022-06-08T07:47:23.365Z] 
[2022-06-08T07:47:23.365Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler PASSED
[2022-06-08T07:47:23.365Z] 
[2022-06-08T07:47:23.366Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread STARTED
[2022-06-08T07:47:27.446Z] There were failing tests. See the report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/streams/build/reports/tests/integrationTest/index.html
[2022-06-08T07:47:28.326Z] 
[2022-06-08T07:47:28.326Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-6a9d6eb9-066d-47c2-8922-a1ecf81a5165-GlobalStreamThread"
[2022-06-08T07:47:28.326Z] 
[2022-06-08T07:47:28.326Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread PASSED
[2022-06-08T07:47:28.326Z] 
[2022-06-08T07:47:28.326Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalArgumentException STARTED
[2022-06-08T07:47:29.296Z] 
[2022-06-08T07:47:29.296Z] BUILD SUCCESSFUL in 2h 51m 46s
[2022-06-08T07:47:29.296Z] 209 actionable tasks: 113 executed, 96 up-to-date
[2022-06-08T07:47:29.296Z] 
[2022-06-08T07:47:29.296Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2022-06-08-04-55-43.html
[2022-06-08T07:47:29.296Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2022-06-08T07:47:30.160Z] Recording test results
[2022-06-08T07:47:32.190Z] 
[2022-06-08T07:47:32.190Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalArgumentException PASSED
[2022-06-08T07:47:32.190Z] 
[2022-06-08T07:47:32.190Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread STARTED
[2022-06-08T07:47:38.117Z] 
[2022-06-08T07:47:38.117Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-22af385c-e4fa-4e79-92c2-1f389e85ae22-StreamThread-1"
[2022-06-08T07:47:38.117Z] 
[2022-06-08T07:47:38.117Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread