Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1615

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14757) Kafka Cooperative Sticky Assignor results in significant duplicate consumption

[Siddharth Anand (Jira)]

Siddharth Anand created KAFKA-14757:
---

 Summary: Kafka Cooperative Sticky Assignor results in significant 
duplicate consumption
 Key: KAFKA-14757
 URL: https://issues.apache.org/jira/browse/KAFKA-14757
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 3.1.1
 Environment: AWS MSK (broker) and Spring Kafka (2.8.7) for use in 
Spring Boot consumers.
Reporter: Siddharth Anand


Details may be found within the linked document:

[Kafka Cooperative Sticky Assignor Issue : Duplicate Consumption | 
[https://docs.google.com/document/d/1E7qAwGOpF8jo_YhF4NwUx9CXxUGJmT8OhHEqIg7-GfI/edit?usp=sharing]]

In a nutshell, we noticed that the Cooperative Sticky Assignor resulted in 
significant duplicate message consumption. During last year's F1 Grand Prix 
events and World Cup soccer events, our company's Kafka-based platform received 
live-traffic. This live traffic, coupled with autoscaled consumers resulted in 
as much as 70% duplicate message consumption at the Kafka consumers. 

In December 2022, we ran a synthetic load test to confirm that duplicate 
message consumption occurs during consumer scale out/in and Kafka partition 
rebalancing when using the Cooperative Sticky Assignor. This issue does not 
occur when using the Range Assignor.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14756) improve exactly-once-demo example and ExactlyOnceMessageProcessor

[Luke Chen (Jira)]

Luke Chen created KAFKA-14756:
-

 Summary: improve exactly-once-demo example and 
ExactlyOnceMessageProcessor
 Key: KAFKA-14756
 URL: https://issues.apache.org/jira/browse/KAFKA-14756
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


while running both the examples, I saw flood of logs output because we print 
one line for message sent, and one line for message received. In 
java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

Also, we should add more comments to both example files

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14755) improve java-producer-consumer-demo

[Luke Chen (Jira)]

Luke Chen created KAFKA-14755:
-

 Summary: improve java-producer-consumer-demo
 Key: KAFKA-14755
 URL: https://issues.apache.org/jira/browse/KAFKA-14755
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


while running both the examples, I saw flood of logs output because we print 
one line for message sent, and one line for message received. In 
java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

One more thing, in exactly-once-demo.java, there are clear class java doc in 
the demo file, but there's nothing in java-producer-consumer-demo.java. We 
should also improve that.

 

Lastly, the main thread in java-producer-consumer-demo won't exit if timed out 
waiting for demo producer and consumer to finish. You'll see exception thrown, 
but the application keeps waiting there.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1614

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14753) improve producer under example package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14753:
-

 Summary: improve producer under example package
 Key: KAFKA-14753
 URL: https://issues.apache.org/jira/browse/KAFKA-14753
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


I found the producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14754) improve consumer under example package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14754:
-

 Summary: improve consumer under example package
 Key: KAFKA-14754
 URL: https://issues.apache.org/jira/browse/KAFKA-14754
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


I found the producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14752) improve kafka examples under examples package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14752:
-

 Summary: improve kafka examples under examples package
 Key: KAFKA-14752
 URL: https://issues.apache.org/jira/browse/KAFKA-14752
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Kafka provided some examples under "examples" package. Currently we provided
 * java-producer-consumer-demo, which is to produce 1 records and then 
consume all of them
 * exactly-once-demo, which is to produce records -> consume -> process  -> 
consume.

Among them, the base component is producer and consumer. However, I found the 
producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 

Furthermore, while running both the examples, I saw flood of logs output 
because we print one line for message sent, and one line for message received. 
In java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

One more thing, in exactly-once-demo.java, there are clear class java doc in 
the demo file, but there's nothing in java-producer-consumer-demo.java. We 
should also improve that.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Hey Ismael,

Thanks for the feedback! The proposal is not to retry automatically but
relies on the user polling the NetworkClient (basically, consumer.poll) to
reattempt the bootstrap. If bootstrapping fails, a NetworkException
(retriable) will be thrown.

Thanks!
P



On Thu, Feb 23, 2023 at 1:34 PM Ismael Juma  wrote:

> Thanks for the KIP. Not sure if I missed it, but how long will we retry for
> and when do we give up and propagate the failure to the user?
>
> Ismael
>
> On Thu, Feb 23, 2023 at 9:30 AM Philip Nee  wrote:
>
> > Hi all!
> >
> > I want to start a discussion thread about how we can handle client
> > bootstrap failure due DNS lookup.  This requires a bit of behavioral
> > change, so a KIP is proposed and attached to this email. Let me know what
> > you think!
> >
> >
> > *A small remark here*: *As the title of this KIP might sound
> > familiar/similar to KIP-899, it is not the same.*
> >
> > *In Summary:* I want to propose a KIP to change the existing bootstrap
> > (upon instantiation) strategy because it is reasonable to allow clients
> to
> > retry
> >
> > *KIP: *
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+Allow+Clients+to+Rebootstrap+Upon+Failed+DNS+Resolution
> >
> > Thanks!
> > Philip
> >
>

[jira] [Created] (KAFKA-14751) Official website CONTACT page IRC channel link change

[Koma Zhang (Jira)]

Koma Zhang created KAFKA-14751:
--

 Summary: Official website CONTACT page IRC channel link change 
 Key: KAFKA-14751
 URL: https://issues.apache.org/jira/browse/KAFKA-14751
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Koma Zhang
Assignee: Koma Zhang


"chat.freenode.net" this link should be change to 
"https://webchat.freenode.net/";



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14750) Sink connector fails if a source topic matching irs topics.regex gets deleted

[Sergei Morozov (Jira)]

Sergei Morozov created KAFKA-14750:
--

 Summary: Sink connector fails if a source topic matching irs 
topics.regex gets deleted
 Key: KAFKA-14750
 URL: https://issues.apache.org/jira/browse/KAFKA-14750
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 3.3.1
Reporter: Sergei Morozov


Steps to reproduce:
 # In {{config/connect-standalone.properties}} set (use the version 
corresponding to the version of Apache Kafka):
plugin.path=libs/connect-file-3.3.1.jar
 # In {{config/connect-file-sink.properties}} remove the {{topics=}} line and 
add this one: 
topics.regex=connect-test-.*
 # Start zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
 # Start the cluster:
bin/kafka-server-start.sh config/server.properties
 # Start the file sink connector:
bin/connect-standalone.sh config/connect-standalone.properties 
config/connect-file-sink.properties
 # Create topics for the sink connector to subscribe to:
for i in \{0..2}; do
  for j in {$(($i * 100))..$(( ($i + 1) * 100 - 1 ))}; do
    bin/kafka-topics.sh \
        --bootstrap-server localhost:9092 \
        --create \
        --topic connect-test-$j
  done &
done
wait
 # Wait until all the created topics are assigned to the connector. Check the 
number of partitions to be > 0 in the output of:
bin/kafka-consumer-groups.sh \
    --bootstrap-server localhost:9092 \
    --group connect-local-file-sink \
    --describe --members
 # Delete the created topics
for i in \{0..2}; do
  for j in {$(($i * 100))..$(( ($i + 1) * 100 - 1 ))}; do
    bin/kafka-topics.sh \
        --bootstrap-server localhost:9092 \
        --delete \
        --topic connect-test-$j
    echo Created topic connect-test-$j.
  done &
done
wait
 # Observe the connector to fail with the following error:
{quote}org.apache.kafka.common.errors.TimeoutException: Timeout of 6ms 
expired before the position for partition connect-test-211-0 could be determined
{quote}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14749) Re-enable 'spotlessScalaCheck' task (in Jenkinsfile)

[Jira]

Dejan Stojadinović created KAFKA-14749:
--

 Summary: Re-enable 'spotlessScalaCheck' task (in Jenkinsfile)
 Key: KAFKA-14749
 URL: https://issues.apache.org/jira/browse/KAFKA-14749
 Project: Kafka
  Issue Type: Task
  Components: build
Reporter: Dejan Stojadinović
 Fix For: 4.0.0


{*}Description{*}:

We were forced to remove 'spotlessScalaCheck' (see KAFKA-14728) but we should 
bring it back when circumstances change (i.e. when Apache Kafka 4.0 drops 
support for Java 8).

Related github issue comment: 
[https://github.com/apache/kafka/pull/13263#issuecomment-1441825913]

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14728) Remove 'spotlessScalaCheck' task (out of Jenkinsfile)

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dejan Stojadinović resolved KAFKA-14728.

Resolution: Fixed

Solution is merged into trunk, hence resolvig as *fixed*.

> Remove 'spotlessScalaCheck' task (out of Jenkinsfile)
> -
>
> Key: KAFKA-14728
> URL: https://issues.apache.org/jira/browse/KAFKA-14728
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Reporter: Dejan Stojadinović
>Assignee: Dejan Stojadinović
>Priority: Major
>
> *Note*: this ticket blocks Gradle major version upgrade (_*7 -->> 8*_): 
> KAFKA-14680
> *Rationale:*
> * build works fine in trunk with Gradle 7.6 and spotless gradle plugin 6.13.0 
> for all currently used JDK versions (that is: JDK 8 / JDK 11 / JDK 17)
> * however, recent spotless gradle plugin versions (6.14.\+) support only JDK 
> 11\+ versions: 
> https://github.com/diffplug/spotless/blob/main/plugin-gradle/CHANGES.md#6140---2023-01-26
>  
> * at the moment Kafka build powered by Gradle 8.0 breaks when combined with 
> all relevant spotless gradle plugin versions (from 6.13.0 to 6.15.0); _github 
> issue is created [here|https://github.com/diffplug/spotless/issues/1572]_
> * given a fact that Kafka still needs to support JDK 8 builds (until Kafka 
> version 4.0) it is reasonable to simply remove spotless checks out of 
> Jenkinsfile (and re-introduce them when the time comes).
> For even more details see GitHub discussion here: 
> https://github.com/apache/kafka/pull/13205#issuecomment-1431575644
> Note: spotless plugin configuration in build.gradle is out of this ticket 
> scope.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1613

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Ismael Juma]

Thanks for the KIP. Not sure if I missed it, but how long will we retry for
and when do we give up and propagate the failure to the user?

Ismael

On Thu, Feb 23, 2023 at 9:30 AM Philip Nee  wrote:

> Hi all!
>
> I want to start a discussion thread about how we can handle client
> bootstrap failure due DNS lookup.  This requires a bit of behavioral
> change, so a KIP is proposed and attached to this email. Let me know what
> you think!
>
>
> *A small remark here*: *As the title of this KIP might sound
> familiar/similar to KIP-899, it is not the same.*
>
> *In Summary:* I want to propose a KIP to change the existing bootstrap
> (upon instantiation) strategy because it is reasonable to allow clients to
> retry
>
> *KIP: *
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+Allow+Clients+to+Rebootstrap+Upon+Failed+DNS+Resolution
>
> Thanks!
> Philip
>

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Hey Ivan,

(Also, thanks John!)

Looping you in just for transparency. Let me know what do you think.

Thanks!
P


On Thu, Feb 23, 2023 at 12:03 PM John Roesler  wrote:

> Thanks for the KIP, Philip!
>
> I think your proposal makes sense. I suspect the reason that we previously
> did the DNS resolution in the constructor is to fail fast if the name is
> wrong. On the other hand, it's generally a hassle to do failure-prone or
> slow operations in a constructor, so I'm in favor of moving it to poll.
>
> I'm also in favor of throwing NetworkException (or some other
> RetriableException), since failing to resolve the DNS entry for the brokers
> shouldn't poison the state of the client, and it should be fine for users
> to retry if they want to.
>
> I actually do think there might be some overlap with KIP-899. If we go
> ahead and move DNS resolution to poll, then KIP-899 becomes just a question
> of whether we should call poll at other points after the first resolution.
> It seems like these could potentially be merged into one proposal, or you
> and Ivan could coordinate on symbiotic KIPs.
>
> Thanks again,
> -John
>
> On 2023/02/23 17:29:23 Philip Nee wrote:
> > Hi all!
> >
> > I want to start a discussion thread about how we can handle client
> > bootstrap failure due DNS lookup.  This requires a bit of behavioral
> > change, so a KIP is proposed and attached to this email. Let me know what
> > you think!
> >
> >
> > *A small remark here*: *As the title of this KIP might sound
> > familiar/similar to KIP-899, it is not the same.*
> >
> > *In Summary:* I want to propose a KIP to change the existing bootstrap
> > (upon instantiation) strategy because it is reasonable to allow clients
> to
> > retry
> >
> > *KIP: *
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+Allow+Clients+to+Rebootstrap+Upon+Failed+DNS+Resolution
> >
> > Thanks!
> > Philip
> >
>

Re: [DISCUSS] KIP-907: Add Boolean Serde to public interface

[spacrocket]

Hello Matthias,

Thanks for looking into my first KIP. I’ve updated the confluence document and 
soon I will start voting.

Kind regards
Jakub

> On Feb 23, 2023, at 8:23 AM, Matthias J. Sax  wrote:
> 
> Thanks for the KIP.
> 
> Overall LGMT.
> 
> One comment: Both `BooleanSerializer` and `BooleanDeserializer` are also new 
> classes that are added and should be listed explicitly similar to 
> `BooleanSerde` in the "Public Interfaces" section of the KIP.
> 
> 
> -Matthias
> 
> On 2/21/23 10:52 AM, SpacRocket wrote:
>> Hello Everyone,
>> I’d like to get a discussion going for KIP-907:
>> KIP-907: Add Boolean Serde to public interface - Apache Kafka - Apache 
>> Software Foundation 
>> 
>> cwiki.apache.org 
>> 
>>  favicon.ico 
>> 
>> 
>> Which adds Boolean Serde to the public interface.
>> The KIP contains the details how I want to do this and what internal code I 
>> need to change.
>> Looking forward to the group’s feedback! :)
>> Kind regards
>> Jakub

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[John Roesler]

Thanks for the KIP, Philip!

I think your proposal makes sense. I suspect the reason that we previously did 
the DNS resolution in the constructor is to fail fast if the name is wrong. On 
the other hand, it's generally a hassle to do failure-prone or slow operations 
in a constructor, so I'm in favor of moving it to poll.

I'm also in favor of throwing NetworkException (or some other 
RetriableException), since failing to resolve the DNS entry for the brokers 
shouldn't poison the state of the client, and it should be fine for users to 
retry if they want to.

I actually do think there might be some overlap with KIP-899. If we go ahead 
and move DNS resolution to poll, then KIP-899 becomes just a question of 
whether we should call poll at other points after the first resolution. It 
seems like these could potentially be merged into one proposal, or you and Ivan 
could coordinate on symbiotic KIPs.

Thanks again,
-John

On 2023/02/23 17:29:23 Philip Nee wrote:
> Hi all!
> 
> I want to start a discussion thread about how we can handle client
> bootstrap failure due DNS lookup.  This requires a bit of behavioral
> change, so a KIP is proposed and attached to this email. Let me know what
> you think!
> 
> 
> *A small remark here*: *As the title of this KIP might sound
> familiar/similar to KIP-899, it is not the same.*
> 
> *In Summary:* I want to propose a KIP to change the existing bootstrap
> (upon instantiation) strategy because it is reasonable to allow clients to
> retry
> 
> *KIP: *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+Allow+Clients+to+Rebootstrap+Upon+Failed+DNS+Resolution
> 
> Thanks!
> Philip
> 

Re: [DISCUSS] KIP-907: Add Boolean Serde to public interface

[SpacRocket]

Hello Matthias,

Thanks for looking into my KIP and for an advice. I’ve added BooleanSerializer 
and BooleanDeserializer to the Public Interfaces.

Kind regards
Jakub

> On Feb 23, 2023, at 8:23 AM, Matthias J. Sax  wrote:
> 
> Thanks for the KIP.
> 
> Overall LGMT.
> 
> One comment: Both `BooleanSerializer` and `BooleanDeserializer` are also new 
> classes that are added and should be listed explicitly similar to 
> `BooleanSerde` in the "Public Interfaces" section of the KIP.
> 
> 
> -Matthias
> 
> On 2/21/23 10:52 AM, SpacRocket wrote:
>> Hello Everyone,
>> I’d like to get a discussion going for KIP-907:
>> KIP-907: Add Boolean Serde to public interface - Apache Kafka - Apache 
>> Software Foundation 
>> 
>> cwiki.apache.org 
>> 
>>  favicon.ico 
>> 
>> 
>> Which adds Boolean Serde to the public interface.
>> The KIP contains the details how I want to do this and what internal code I 
>> need to change.
>> Looking forward to the group’s feedback! :)
>> Kind regards
>> Jakub

[jira] [Created] (KAFKA-14748) Relax non-null FK left-join requirement

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-14748:
---

 Summary: Relax non-null FK left-join requirement
 Key: KAFKA-14748
 URL: https://issues.apache.org/jira/browse/KAFKA-14748
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


Kafka Streams enforces a strict non-null-key policy in the DSL across all 
key-dependent operations (like aggregations and joins).

This also applies to FK-joins, in particular to the ForeignKeyExtractor. If it 
returns `null`, it's treated as invalid. For left-joins, it might make sense to 
still accept a `null`, and add the left-hand record with an empty 
right-hand-side to the result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-890: Transactions Server Side Defense

[Justine Olshan]

Yup -- those are the main changes!

On Thu, Feb 23, 2023 at 9:44 AM Guozhang Wang 
wrote:

> Thanks Justine. I checked the diff between the two versions on wiki,
> seems the major changes are:
>
> 1) Move the `verifyOnly` field of the request into each transaction
> and hence we no longer have any top-level primitive fields.
> 2) Add a top-level `errorCode` field in the response.
>
> Is that summary right?
>
>
> Guozhang
>
> On Wed, Feb 22, 2023 at 4:51 PM Justine Olshan
>  wrote:
> >
> > Hey all,
> >
> > I've updated the KIP to slightly change some of the request and response
> > specs for AddPartitionsToTxn. Nothing huge, but some points came up
> during
> > PR review.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
> >
> > Thanks,
> > Justine
> >
> > On Fri, Feb 3, 2023 at 8:40 AM Justine Olshan 
> wrote:
> >
> > > Thanks everyone! I'm going to close the vote.
> > > The KIP is accepted with five binding votes from Jason, Guozhang,
> > > Matthias, David (and me), and two non-binding votes from Colt and
> Artem.
> > >
> > > Thanks again,
> > > Justine
> > >
> > > On Thu, Feb 2, 2023 at 11:41 PM David Jacot
> 
> > > wrote:
> > >
> > >> Thanks for the KIP, Justine. +1 (binding)
> > >>
> > >> On Fri, Feb 3, 2023 at 1:36 AM Matthias J. Sax 
> wrote:
> > >>
> > >> > Thanks for the KIP!
> > >> >
> > >> > +1 (binding)
> > >> >
> > >> >
> > >> > On 2/2/23 4:18 PM, Artem Livshits wrote:
> > >> > > (non-binding) +1.  Looking forward to the implementation and
> fixing
> > >> the
> > >> > > issues that we've got.
> > >> > >
> > >> > > -Artem
> > >> > >
> > >> > > On Mon, Jan 23, 2023 at 2:25 PM Guozhang Wang <
> > >> > guozhang.wang...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Thanks Justine, I have no further comments on the KIP. +1.
> > >> > >>
> > >> > >> On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson
> > >> > >>  wrote:
> > >> > >>>
> > >> > >>> +1. Thanks Justine!
> > >> > >>>
> > >> > >>> -Jason
> > >> > >>>
> > >> > >>> On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy <
> c...@littlehorse.io>
> > >> > >> wrote:
> > >> > >>>
> > >> >  (non-binding) +1. Thank you for the KIP, Justine! I've read
> it; it
> > >> > >> makes
> > >> >  sense to me and I am excited for the implementation.
> > >> > 
> > >> >  Colt McNealy
> > >> >  *Founder, LittleHorse.io*
> > >> > 
> > >> > 
> > >> >  On Tue, Jan 10, 2023 at 10:46 AM Justine Olshan
> > >> >   wrote:
> > >> > 
> > >> > > Hi everyone,
> > >> > >
> > >> > > I would like to start a vote on KIP-890 which aims to prevent
> some
> > >> > >> of the
> > >> > > common causes of hanging transactions and make other general
> > >> > >> improvements
> > >> > > to transactions in Kafka.
> > >> > >
> > >> > >
> > >> > >
> > >> > 
> > >> > >>
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
> > >> > >
> > >> > > Please take a look if you haven't already and vote!
> > >> > >
> > >> > > Justine
> > >> > >
> > >> > 
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1612

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14747) FK join should record discarded subscription responses

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-14747:
---

 Summary: FK join should record discarded subscription responses
 Key: KAFKA-14747
 URL: https://issues.apache.org/jira/browse/KAFKA-14747
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


FK-join are subject to a race condition. If the left-hand side record is 
updated, a subscription is sent to the right-hand side (including a hash value 
of the left-hand side record), and the right-hand side might send back join 
responses (also including the original hash). The left hand side only processed 
the responses if the hash matches, because a different hash implies that the 
left hand side row was updated in the mean-time (including sending a new 
subscription to the right hand side), and thus the data is stale and the 
response should not be processed.

A similar thing can happen on a right hand side update that triggers a 
response, that might be dropped if the left hand side row was updated in 
parallel.

While the behavior is correct, we don't record if this happens. We should 
consider to record this using the existing "dropped record" sensor or maybe add 
a new sensor.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14295) FetchMessageConversionsPerSec meter not recorded

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-14295.

Fix Version/s: 3.5.0
   Resolution: Fixed

> FetchMessageConversionsPerSec meter not recorded
> 
>
> Key: KAFKA-14295
> URL: https://issues.apache.org/jira/browse/KAFKA-14295
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Mao
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.5.0
>
>
> The broker topic metric FetchMessageConversionsPerSec doesn't get recorded on 
> a fetch message conversion.
> The bug is that we pass in a callback that expects a MultiRecordsSend in 
> KafkaApis:
> {code:java}
> def updateConversionStats(send: Send): Unit = {
>   send match {
> case send: MultiRecordsSend if send.recordConversionStats != null =>
>   send.recordConversionStats.asScala.toMap.foreach {
> case (tp, stats) => updateRecordConversionStats(request, tp, stats)
>   }
> case _ =>
>   }
> } {code}
> But we call this callback with a NetworkSend in the SocketServer:
> {code:java}
> selector.completedSends.forEach { send =>
>   try {
> val response = inflightResponses.remove(send.destinationId).getOrElse {
>   throw new IllegalStateException(s"Send for ${send.destinationId} 
> completed, but not in `inflightResponses`")
> }
> updateRequestMetrics(response)
> // Invoke send completion callback
> response.onComplete.foreach(onComplete => onComplete(send))
> ...{code}
> Note that Selector.completedSends returns a collection of NetworkSend



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-890: Transactions Server Side Defense

[Guozhang Wang]

Thanks Justine. I checked the diff between the two versions on wiki,
seems the major changes are:

1) Move the `verifyOnly` field of the request into each transaction
and hence we no longer have any top-level primitive fields.
2) Add a top-level `errorCode` field in the response.

Is that summary right?


Guozhang

On Wed, Feb 22, 2023 at 4:51 PM Justine Olshan
 wrote:
>
> Hey all,
>
> I've updated the KIP to slightly change some of the request and response
> specs for AddPartitionsToTxn. Nothing huge, but some points came up during
> PR review.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
>
> Thanks,
> Justine
>
> On Fri, Feb 3, 2023 at 8:40 AM Justine Olshan  wrote:
>
> > Thanks everyone! I'm going to close the vote.
> > The KIP is accepted with five binding votes from Jason, Guozhang,
> > Matthias, David (and me), and two non-binding votes from Colt and Artem.
> >
> > Thanks again,
> > Justine
> >
> > On Thu, Feb 2, 2023 at 11:41 PM David Jacot 
> > wrote:
> >
> >> Thanks for the KIP, Justine. +1 (binding)
> >>
> >> On Fri, Feb 3, 2023 at 1:36 AM Matthias J. Sax  wrote:
> >>
> >> > Thanks for the KIP!
> >> >
> >> > +1 (binding)
> >> >
> >> >
> >> > On 2/2/23 4:18 PM, Artem Livshits wrote:
> >> > > (non-binding) +1.  Looking forward to the implementation and fixing
> >> the
> >> > > issues that we've got.
> >> > >
> >> > > -Artem
> >> > >
> >> > > On Mon, Jan 23, 2023 at 2:25 PM Guozhang Wang <
> >> > guozhang.wang...@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Thanks Justine, I have no further comments on the KIP. +1.
> >> > >>
> >> > >> On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson
> >> > >>  wrote:
> >> > >>>
> >> > >>> +1. Thanks Justine!
> >> > >>>
> >> > >>> -Jason
> >> > >>>
> >> > >>> On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy 
> >> > >> wrote:
> >> > >>>
> >> >  (non-binding) +1. Thank you for the KIP, Justine! I've read it; it
> >> > >> makes
> >> >  sense to me and I am excited for the implementation.
> >> > 
> >> >  Colt McNealy
> >> >  *Founder, LittleHorse.io*
> >> > 
> >> > 
> >> >  On Tue, Jan 10, 2023 at 10:46 AM Justine Olshan
> >> >   wrote:
> >> > 
> >> > > Hi everyone,
> >> > >
> >> > > I would like to start a vote on KIP-890 which aims to prevent some
> >> > >> of the
> >> > > common causes of hanging transactions and make other general
> >> > >> improvements
> >> > > to transactions in Kafka.
> >> > >
> >> > >
> >> > >
> >> > 
> >> > >>
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
> >> > >
> >> > > Please take a look if you haven't already and vote!
> >> > >
> >> > > Justine
> >> > >
> >> > 
> >> > >>
> >> > >
> >> >
> >>
> >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #158

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 502534 lines...]
[2023-02-23T17:28:35.077Z] 
[2023-02-23T17:28:35.077Z] 1: Task failed with an exception.
[2023-02-23T17:28:35.077Z] ---
[2023-02-23T17:28:35.077Z] * What went wrong:
[2023-02-23T17:28:35.077Z] Execution failed for task 
':streams:upgrade-system-tests-0110:unitTest'.
[2023-02-23T17:28:35.077Z] > Process 'Gradle Test Executor 38' finished with 
non-zero exit value 1
[2023-02-23T17:28:35.077Z]   This problem might be caused by incorrect test 
process configuration.
[2023-02-23T17:28:35.077Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.4.2/userguide/java_testing.html#sec:test_execution
[2023-02-23T17:28:35.077Z] 
[2023-02-23T17:28:35.077Z] * Try:
[2023-02-23T17:28:35.077Z] > Run with --stacktrace option to get the stack 
trace.
[2023-02-23T17:28:35.077Z] > Run with --info or --debug option to get more log 
output.
[2023-02-23T17:28:35.077Z] > Run with --scan to get full insights.
[2023-02-23T17:28:35.077Z] 
==
[2023-02-23T17:28:35.077Z] 
[2023-02-23T17:28:35.077Z] 2: Task failed with an exception.
[2023-02-23T17:28:35.077Z] ---
[2023-02-23T17:28:35.077Z] * What went wrong:
[2023-02-23T17:28:35.077Z] Execution failed for task 
':streams:upgrade-system-tests-26:integrationTest'.
[2023-02-23T17:28:35.077Z] > Process 'Gradle Test Executor 49' finished with 
non-zero exit value 1
[2023-02-23T17:28:35.078Z]   This problem might be caused by incorrect test 
process configuration.
[2023-02-23T17:28:35.078Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.4.2/userguide/java_testing.html#sec:test_execution
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] * Try:
[2023-02-23T17:28:35.078Z] > Run with --stacktrace option to get the stack 
trace.
[2023-02-23T17:28:35.078Z] > Run with --info or --debug option to get more log 
output.
[2023-02-23T17:28:35.078Z] > Run with --scan to get full insights.
[2023-02-23T17:28:35.078Z] 
==
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] 3: Task failed with an exception.
[2023-02-23T17:28:35.078Z] ---
[2023-02-23T17:28:35.078Z] * What went wrong:
[2023-02-23T17:28:35.078Z] Execution failed for task ':core:integrationTest'.
[2023-02-23T17:28:35.078Z] > Process 'Gradle Test Executor 141' finished with 
non-zero exit value 1
[2023-02-23T17:28:35.078Z]   This problem might be caused by incorrect test 
process configuration.
[2023-02-23T17:28:35.078Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.4.2/userguide/java_testing.html#sec:test_execution
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] * Try:
[2023-02-23T17:28:35.078Z] > Run with --stacktrace option to get the stack 
trace.
[2023-02-23T17:28:35.078Z] > Run with --info or --debug option to get more log 
output.
[2023-02-23T17:28:35.078Z] > Run with --scan to get full insights.
[2023-02-23T17:28:35.078Z] 
==
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] * Get more help at https://help.gradle.org
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] BUILD FAILED in 2h 50m 12s
[2023-02-23T17:28:35.078Z] 212 actionable tasks: 115 executed, 97 up-to-date
[2023-02-23T17:28:35.078Z] 
[2023-02-23T17:28:35.078Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.3/build/reports/profile/profile-2023-02-23-14-38-26.html
[2023-02-23T17:28:35.078Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.13
[2023-02-23T17:28:50.254Z] 
[2023-02-23T17:28:50.254Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] PASSED
[2023-02-23T17:28:50.254Z] 
[2023-02-23T17:28:50.254Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] STARTED
[2023-02-23T17:28:50.254Z] 
[2023-02-23T17:28:50.254Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testSelfJoin[caching enabled = true] PASSED
[2023-02-23T17:28:50.254Z] 
[2023-02-23T17:28:50.254Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] STARTED
[2023-02-23T17:28:57.172Z] 
[2023-02-23T17:28:57.172Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
test

[jira] [Resolved] (KAFKA-14744) NPE while converting OffsetFetch from version < 8 to version >= 8

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-14744.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> NPE while converting OffsetFetch from version < 8 to version >= 8
> -
>
> Key: KAFKA-14744
> URL: https://issues.apache.org/jira/browse/KAFKA-14744
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.5.0
>
>
> While refactoring the OffsetFetch handling in KafkaApis, we introduced a 
> NullPointerException (NPE). The NPE arises when the FetchOffset API is called 
> with a client using a version older than version 8 and using null for the 
> topics to signal that all topic-partition offsets must be returned. This 
> means that this bug mainly impacts admin tools. The consumer does not use 
> null.
> This NPE is here: 
> https://github.com/apache/kafka/commit/24a86423e9907b751d98fddc7196332feea2b48d#diff-0f2f19fd03e2fc5aa9618c607b432ea72e5aaa53866f07444269f38cb537f3feR237.
> We missed this during the refactor because we had no tests in place to test 
> this mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Hi all!

I want to start a discussion thread about how we can handle client
bootstrap failure due DNS lookup.  This requires a bit of behavioral
change, so a KIP is proposed and attached to this email. Let me know what
you think!


*A small remark here*: *As the title of this KIP might sound
familiar/similar to KIP-899, it is not the same.*

*In Summary:* I want to propose a KIP to change the existing bootstrap
(upon instantiation) strategy because it is reasonable to allow clients to
retry

*KIP: *
https://cwiki.apache.org/confluence/display/KAFKA/KIP-909%3A+Allow+Clients+to+Rebootstrap+Upon+Failed+DNS+Resolution

Thanks!
Philip

Re: Hosting Kafka Videos on ASF YouTube channel

[Joe Brockmeier]

Actually adding Brian to CC now...

On Thu, Feb 23, 2023 at 12:12 PM Joe Brockmeier  wrote:

> Hi Bill,
>
> On Thu, Feb 23, 2023 at 10:48 AM Bill Bejeck  wrote:
>
>> It took a little time, but I have download links to share with you for
>> the 4 Kafka Streams videos we want to host on the ASF Youtube channel, as
>> discussed on this thread
>> .  The
>> "What is Apache Kafka " video has
>> been edited to remove any vendor reference.  I'll also have a download link
>> to share for that video soon.
>>
>> Where is the best place to share the links with you?  I'd prefer not to
>> share them with you directly vs. the public.
>>
>
> I've stepped aside from the VP marketing role, Brian Proffitt is now
> heading up M&P, so he can help guide you on the uploads. (Brian, lemme know
> if you have questions or whatnot.)
>
> Did you mean you'd prefer *to* share them directly vs. publicly? Either
> way, you can work out with Brian/press@ how to share and upload them.
>
> Thanks,
>
> Joe
>

Re: Hosting Kafka Videos on ASF YouTube channel

[Joe Brockmeier]

Hi Bill,

On Thu, Feb 23, 2023 at 10:48 AM Bill Bejeck  wrote:

> It took a little time, but I have download links to share with you for the
> 4 Kafka Streams videos we want to host on the ASF Youtube channel, as
> discussed on this thread
> .  The "What
> is Apache Kafka " video has been
> edited to remove any vendor reference.  I'll also have a download link to
> share for that video soon.
>
> Where is the best place to share the links with you?  I'd prefer not to
> share them with you directly vs. the public.
>

I've stepped aside from the VP marketing role, Brian Proffitt is now
heading up M&P, so he can help guide you on the uploads. (Brian, lemme know
if you have questions or whatnot.)

Did you mean you'd prefer *to* share them directly vs. publicly? Either
way, you can work out with Brian/press@ how to share and upload them.

Thanks,

Joe

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #88

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 523981 lines...]
[2023-02-23T16:26:39.651Z] 
[2023-02-23T16:26:39.651Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetDataAndStat() PASSED
[2023-02-23T16:26:39.651Z] 
[2023-02-23T16:26:39.651Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testReassignPartitionsInProgress() STARTED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testReassignPartitionsInProgress() PASSED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() STARTED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() PASSED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateTopLevelPaths() STARTED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2023-02-23T16:26:40.685Z] 
[2023-02-23T16:26:40.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-02-23T16:26:41.924Z] 
[2023-02-23T16:26:41.924Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-02-23T16:26:42.866Z] 
[2023-02-23T16:26:42.866Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-02-23T16:26:42.866Z] 
[2023-02-23T16:26:42.866Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-02-23T16:26:42.866Z] 
[2023-02-23T16:26:42.866Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-02-23T16:26:42.866Z] 
[2023-02-23T16:26:42.866Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testAclMethods() STARTED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testAclMethods() PASSED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-02-23T16:26:44.074Z] 
[2023-02-23T16:26:44.074Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-02-23T16:26:45.017Z] 
[2023-02-23T16:26:45.017Z] Gradle Test Run :core:integrationT

[jira] [Resolved] (KAFKA-5827) Allow configuring Kafka sink connectors to start processing records from the end of topics

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-5827.

Resolution: Done

> Allow configuring Kafka sink connectors to start processing records from the 
> end of topics
> --
>
> Key: KAFKA-5827
> URL: https://issues.apache.org/jira/browse/KAFKA-5827
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Behrang Saeedzadeh
>Priority: Major
>
> As far as I can see, Kafka connectors start exporting data of a topic from 
> the beginning of its partitions. We have a topic that contains a few million 
> old records that we don't need but we would like to start exporting new 
> records that are added to the topic.
> Basically:
> * When the connector is started for the first time and it does not have a 
> current offset stored, it should start consuming data from the end of topic 
> partitions
> * When the connector is restarted and has a current offset for partitions 
> stored somewhere, it should start from those offsets



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-5827) Allow configuring Kafka sink connectors to start processing records from the end of topics

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris reopened KAFKA-5827:


> Allow configuring Kafka sink connectors to start processing records from the 
> end of topics
> --
>
> Key: KAFKA-5827
> URL: https://issues.apache.org/jira/browse/KAFKA-5827
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Behrang Saeedzadeh
>Priority: Major
>
> As far as I can see, Kafka connectors start exporting data of a topic from 
> the beginning of its partitions. We have a topic that contains a few million 
> old records that we don't need but we would like to start exporting new 
> records that are added to the topic.
> Basically:
> * When the connector is started for the first time and it does not have a 
> current offset stored, it should start consuming data from the end of topic 
> partitions
> * When the connector is restarted and has a current offset for partitions 
> stored somewhere, it should start from those offsets



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14746) Throwing in Connector.taskConfigs generates a lot of logs

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14746:
--

 Summary: Throwing in Connector.taskConfigs generates a lot of logs
 Key: KAFKA-14746
 URL: https://issues.apache.org/jira/browse/KAFKA-14746
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Mickael Maison


If a Connector throws in its taskConfigs() method, the runtime ends up retrying 
using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which is a fixed 
value (250ms). For each retry, the runtime prints the connector configuration 
and the enriched configuration so this can quickly generate a lot of logs.

There is some value in throwing in taskConfigs() as it allows to fail fast in 
case the connector is given bad credentials for example.

The way Connectors are expected to work today is to instead always create tasks 
and let each task fail in case the configuration is wrong. We should document 
that and make it clear in the javadoc that throwing in taskConfigs is not 
recommended.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Hosting Kafka Videos on ASF YouTube channel

[Bill Bejeck]

Joe,

It took a little time, but I have download links to share with you for the
4 Kafka Streams videos we want to host on the ASF Youtube channel, as
discussed on this thread
.  The "What
is Apache Kafka " video has been
edited to remove any vendor reference.  I'll also have a download link to
share for that video soon.

Where is the best place to share the links with you?  I'd prefer not to
share them with you directly vs. the public.

Thanks,
Bill

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1611

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 523327 lines...]
[2023-02-23T15:39:43.612Z] [INFO] Total time:  2.134 s
[2023-02-23T15:39:43.612Z] [INFO] Finished at: 2023-02-23T15:39:43Z
[2023-02-23T15:39:43.612Z] [INFO] 

[Pipeline] dir
[2023-02-23T15:39:44.289Z] Running in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2023-02-23T15:39:47.164Z] + mvn compile
[2023-02-23T15:39:49.289Z] [INFO] Scanning for projects...
[2023-02-23T15:39:49.289Z] [INFO] 
[2023-02-23T15:39:49.289Z] [INFO] -< 
streams.examples:streams.examples >--
[2023-02-23T15:39:49.289Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2023-02-23T15:39:49.289Z] [INFO] [ jar 
]-
[2023-02-23T15:39:49.289Z] [INFO] 
[2023-02-23T15:39:49.289Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2023-02-23T15:39:50.301Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2023-02-23T15:39:50.301Z] [INFO] Copying 1 resource
[2023-02-23T15:39:50.301Z] [INFO] 
[2023-02-23T15:39:50.301Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2023-02-23T15:39:51.316Z] [INFO] Changes detected - recompiling the module!
[2023-02-23T15:39:51.316Z] [INFO] Compiling 3 source files to 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2023-02-23T15:39:53.441Z] [INFO] 

[2023-02-23T15:39:53.441Z] [INFO] BUILD SUCCESS
[2023-02-23T15:39:53.441Z] [INFO] 

[2023-02-23T15:39:53.441Z] [INFO] Total time:  4.434 s
[2023-02-23T15:39:53.441Z] [INFO] Finished at: 2023-02-23T15:39:52Z
[2023-02-23T15:39:53.441Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-02-23T15:39:54.846Z] > Task :core:compileScala
[2023-02-23T15:41:33.748Z] > Task :core:classes
[2023-02-23T15:41:33.748Z] > Task :core:compileTestJava NO-SOURCE
[2023-02-23T15:42:04.253Z] > Task :core:compileTestScala
[2023-02-23T15:43:30.502Z] > Task :core:testClasses
[2023-02-23T15:43:30.502Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-02-23T15:43:30.502Z] > Task :streams:testClasses UP-TO-DATE
[2023-02-23T15:43:31.515Z] > Task :streams:testJar
[2023-02-23T15:43:32.700Z] > Task :streams:testSrcJar
[2023-02-23T15:43:32.700Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-02-23T15:43:32.700Z] > Task :streams:publishToMavenLocal
[2023-02-23T15:43:32.700Z] 
[2023-02-23T15:43:32.700Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-02-23T15:43:32.700Z] 
[2023-02-23T15:43:32.700Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-02-23T15:43:32.700Z] 
[2023-02-23T15:43:32.700Z] See 
https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings
[2023-02-23T15:43:32.700Z] 
[2023-02-23T15:43:32.700Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2023-02-23T15:43:32.700Z] Please consult deprecation warnings for more details.
[2023-02-23T15:43:32.700Z] 
[2023-02-23T15:43:32.700Z] BUILD SUCCESSFUL in 4m 43s
[2023-02-23T15:43:32.700Z] 86 actionable tasks: 35 executed, 51 up-to-date
[Pipeline] sh
[2023-02-23T15:43:36.134Z] + grep ^version= gradle.properties
[2023-02-23T15:43:36.134Z] + cut -d= -f 2
[Pipeline] dir
[2023-02-23T15:43:37.151Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-02-23T15:43:40.028Z] + mvn clean install -Dgpg.skip
[2023-02-23T15:43:42.228Z] [INFO] Scanning for projects...
[2023-02-23T15:43:42.228Z] [INFO] 

[2023-02-23T15:43:42.228Z] [INFO] Reactor Build Order:
[2023-02-23T15:43:42.228Z] [INFO] 
[2023-02-23T15:43:42.228Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2023-02-23T15:43:42.228Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2023-02-23T15:43:42.228Z] [INFO] 
[2023-02-23T15:43:42.228Z] [IN

[jira] [Created] (KAFKA-14745) MirrorSourceConnector keeps creating ReplicationPolicy instances

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14745:
--

 Summary: MirrorSourceConnector keeps creating ReplicationPolicy 
instances
 Key: KAFKA-14745
 URL: https://issues.apache.org/jira/browse/KAFKA-14745
 Project: Kafka
  Issue Type: Improvement
  Components: mirrormaker
Reporter: Mickael Maison
Assignee: Mickael Maison


In MirrorSourceConnector.findTargetTopicPartitions() we call 
MirrorSourceConfig.checkpointsTopic() for each remote topic or all topics when 
using IdentityReplicationPolicy.

The issue is that checkpointsTopic() calls 
MirrorSourceConfig.replicationPolicy() which always creates a new instance of 
the ReplicationPolicy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

[Divij Vaidya]

Thanks for the KIP Christo.

Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
hence I completely agree with the motivation. Your experiments have
demonstrated that the new version of Zk is stable at scale and the backward
compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
version.

Vote +1 (non binding)

--
Divij Vaidya



On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov 
wrote:

> Hello!
>
> I would like to start the vote for KIP-902, which upgrades Zookeeper to
> version 3.8.1.
>
> The KIP can be found at
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>
> The discussion thread is
> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
>
> Thanks
> Christo

Re: Message conversion and TLS

[David Mao]

My understanding of the produce message conversions metric is that the
metric is specifically for message format up-conversion which is
independent of TLS being enabled.

It’s also worth noting that the zero-copy sendfile is not used on the
produce path, only the fetch path.

On Thu, Feb 23, 2023 at 4:56 AM Sergio Daniel Troiano
 wrote:

> Hey lads,
>
> I was thinking about message conversions and zero copy, as we know TLS will
> break the zero copy as the encrypt/decrypt happens in userland, correct?
>
> For example  ProduceMessageConversionsPerSec  will be triggered when a
> message must be converted (for example compressing at topic level).
>
> Should we consider the TLS layer a conversion as well? My concern is we are
> "hiding" this metric, so let's suppose I have my cluster using TLS (it
> means zero-copy is already "broken") then I activate the compression at
> topic level.
> Now I will see the values in ProduceMessageConversionsPerSec  as the
> conversion is happening (tracking the compression at topic level) but the
> reality is it was already happening before because of the TLS, so in terms
> of performance it's a kind of illusion as it seems the conversion is not
> costing any extra CPU cycles.
>
> If you think it is a good idea maybe we could include the count on the
> ProduceMessageConversionsPerSec when the TLS is enabled, what do you think?
>
> Thanks in advance.
> Sergio Troiano
>

Re: [VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Gantigmaa Selenge]

Thank you very much everyone!

Regards,
Tina

On Wed, Feb 22, 2023 at 6:01 PM Mickael Maison 
wrote:

> Thanks for the KIP!
>
> +1 binding
>
> On Wed, Feb 22, 2023 at 6:00 PM Chris Egerton 
> wrote:
> >
> > +1 (binding). Thanks again, Tina!
> >
> > On Wed, Feb 22, 2023 at 2:02 AM Luke Chen  wrote:
> >
> > > +1 (binding) from me.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Jan 26, 2023 at 11:24 PM Federico Valeri  >
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > Thanks.
> > > >
> > > > On Thu, Jan 26, 2023 at 2:30 PM Gantigmaa Selenge <
> gsele...@redhat.com>
> > > > wrote:
> > > > >
> > > > > Hi
> > > > >
> > > > > I'd like to call for a vote on KIP-894, which updates MirrorMaker
> to
> > > use
> > > > > IncrementalAlterConfigs API to sync topic configurations between
> > > > clusters.
> > > > >
> > > > > The KIP:
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
> > > > >
> > > > > The discussion thread:
> > > > > https://lists.apache.org/thread/4chr4s5vbd9rqhml2tk60fsghojwo6bb
> > > > >
> > > > > Thank you
> > > > > Tina
> > > >
> > >
>
>

[VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

[Christo Lolov]

Hello!

I would like to start the vote for KIP-902, which upgrades Zookeeper to version 
3.8.1.

The KIP can be found at 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784

The discussion thread is 
https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1

Thanks
Christo

Re: [DISCUSS] KIP-908: Add description field to connector configuration

[Chris Egerton]

Actually, I misspoke--a rebalance isn't triggered when an existing
connector's config is updated. Assuming the set of workers remains stable,
a rebalance is only necessary when a new connector is created, an existing
one is deleted, or a new set of task configs is generated.

This weakens the point about unnecessary rebalances when a connector's
description is updated, but doesn't entirely address it. Spurious
rebalances may still be triggered if a new set of task configs is
generated, which for reasons outlined above, is fairly likely.

On Thu, Feb 23, 2023 at 7:41 AM Chris Egerton  wrote:

> Hi Mickael,
>
> Thanks for the KIP!
>
> While it's tempting to add this field to the out-of-the-box connector
> config def, I'm a little hesitant, for two reasons.
>
> 1. Adding this directly to the connector config could have unintended
> consequences on the actual data processing by the connector. Any time a
> connector's config is modified, the Connector object running for it is
> restarted with that new config. In most cases this is a trivial operation
> since we have incremental rebalancing enabled by default, the connector can
> (and probably should!) generate task configs that are functionally
> identical to the ones it last generated, and most (though not all)
> Connector classes are fairly lightweight and leave the real work to their
> Task class. However, due to KAFKA-9228 [1], it's not just common practice
> for connectors to perform transparent passthrough of most of their configs
> when generating task configs, it's actually necessary to work around a bug
> in the runtime. As a result, tweaking the description of a connector would
> be fairly likely to result in a full restart of all of its tasks, in
> addition to triggering two rebalances (which may not be so lightweight if
> users are still running with eager rebalancing... which, sadly, I've heard
> is still happening today).
>
> 2. The motivation section mentions some information that might go into the
> description field, such as the team that owns the connector and emergency
> contact info. It seems like this info might benefit from a little more
> structure if we're trying to design for programmatic access by GUIs and
> CLIs (which I'm assuming is the case, since I can't imagine a human being
> getting much use out of the raw output from the GET
> /connector-plugins/{name}/config and PUT
> /connector-plugins/{name}/config/validate endpoints). This might also make
> it easier to add custom validation logic around what kind of information is
> present via REST extension.
>
>
> With these thoughts in mind, what do you think about adding a new generic
> "tags" object to connectors that can support arbitrary user-provided
> key/value pairs? If using the POST /connectors endpoint, it might look
> something like this:
>
> {
>   "name": "my-connector",
>   "config": {
> "connector.class": "MirrorSource",
> "tasks.max": "908"
>   },
>   "tags": {
> "team": "data-infra",
> "phone": "12345678901",
> "email": "din...@example.org"
>   }
> }
>
> And, to allow users to modify connector tags after one has been created,
> we might introduce a /connectors/{name}/tags endpoint with PUT/PATCH/DELETE
> verbs that writes tag info for a connector to the config topic, but without
> altering the actual connector config (allowing us to skip a rebalance
> altogether).
>
> One other thing we could consider is allowing cluster administrators to
> require that connectors are created with a certain set of tags, or even add
> per-tag regex validation. They might specify something like
> "connector.required.tags = team, phone, email" or
> "connector.tags.phone.regex = [0-9]{11}" in a worker config file. But this
> is probably overboard for now, especially since it's already possible to
> accomplish via a REST extension.
>
> Finally, I'm also wondering if, in pursuit of expanding Connect's
> out-of-the-box support for dealing with connector metadata, we might want
> to expose created/last-updated times for connector configurations. We
> definitely don't have to do that in this KIP but if you agree that this
> would be useful, we should probably keep it in mind to avoid painting
> ourselves into a corner. This is why I'm thinking of using "tags" instead
> of something a little more generic like "metadata", BTW.
>
> [1] -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-908%3A+Add+description+field+to+connector+configuration
>
> Cheers,
>
> Chris
>
> On Thu, Feb 23, 2023 at 4:52 AM Mickael Maison 
> wrote:
>
>> Hi,
>>
>> I created a very small KIP to add a description field to connectors:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-908%3A+Add+description+field+to+connector+configuration
>>
>> Let me know if you have any feedback.
>>
>> Thanks,
>> Mickael
>>
>

[jira] [Resolved] (KAFKA-13690) Flaky test EosIntegrationTest.shouldWriteLatestOffsetsToCheckpointOnShutdown[at_least_once]

[Christo Lolov (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christo Lolov resolved KAFKA-13690.
---
Resolution: Fixed

> Flaky test 
> EosIntegrationTest.shouldWriteLatestOffsetsToCheckpointOnShutdown[at_least_once]
> ---
>
> Key: KAFKA-13690
> URL: https://issues.apache.org/jira/browse/KAFKA-13690
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Major
>
> The _at_least_once_ version of the 
> "{*}EosIntegrationTest.shouldWriteLatestOffsetsToCheckpointOnShutdown"{*} 
> test is occasionally failing with
> h3. Error Message
> java.lang.AssertionError: The committed records do not match what expected 
> Expected: <[KeyValue(0, 0), KeyValue(0, 1), KeyValue(0, 3), KeyValue(0, 6), 
> KeyValue(0, 10), KeyValue(0, 15), KeyValue(0, 21), KeyValue(0, 28), 
> KeyValue(0, 36), KeyValue(0, 45)]> but: was <[KeyValue(0, 0), KeyValue(0, 1), 
> KeyValue(0, 3), KeyValue(0, 6), KeyValue(0, 10), KeyValue(0, 10), KeyValue(0, 
> 11), KeyValue(0, 13), KeyValue(0, 16), KeyValue(0, 20), KeyValue(0, 25), 
> KeyValue(0, 31), KeyValue(0, 38)]>
>  
> Seems we are receiving more than the expected records.
> ...of course, this is an ALOS flavor of the {*}EOS{*}IntegrationTest, so 
> perhaps we shouldn't be running this variant at all? Not sure if this 
> explains the exact output we receive but it certainly seems suspicious
>  
> Added at_least_once in [https://github.com/apache/kafka/pull/11283]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13966) Flaky test `QuorumControllerTest.testUnregisterBroker`

[Christo Lolov (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christo Lolov resolved KAFKA-13966.
---
Resolution: Fixed

> Flaky test `QuorumControllerTest.testUnregisterBroker`
> --
>
> Key: KAFKA-13966
> URL: https://issues.apache.org/jira/browse/KAFKA-13966
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: David Arthur
>Priority: Major
>
> We have seen the following assertion failure in 
> `QuorumControllerTest.testUnregisterBroker`:
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <2> but was: <0>
>   at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
>   at 
> org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)
>   at 
> org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:161)
>   at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:628)
>   at 
> org.apache.kafka.controller.QuorumControllerTest.testUnregisterBroker(QuorumControllerTest.java:494)
>  {code}
> I reproduced it by running the test in a loop. It looks like what happens is 
> that the BrokerRegistration request is able to get interleaved between the 
> leader change event and the write of the bootstrap metadata. Something like 
> this:
>  # handleLeaderChange() start
>  # appendWriteEvent(registerBroker)
>  # appendWriteEvent(bootstrapMetadata)
>  # handleLeaderChange() finish
>  # registerBroker() -> writes broker registration to log
>  # bootstrapMetadata() -> writes bootstrap metadata to log



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1610

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 531321 lines...]
[2023-02-23T12:56:17.038Z] 
[2023-02-23T12:56:17.038Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ReplicationUtilsTest > testUpdateLeaderAndIsr() PASSED
[2023-02-23T12:56:17.038Z] 
[2023-02-23T12:56:17.038Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
STARTED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
PASSED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() STARTED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() PASSED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
STARTED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
PASSED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetChildrenExistingZNode() STARTED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetChildrenExistingZNode() PASSED
[2023-02-23T12:56:18.229Z] 
[2023-02-23T12:56:18.229Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnection() STARTED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnection() PASSED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() STARTED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() PASSED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetAclExistingZNode() STARTED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetAclExistingZNode() PASSED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSessionExpiryDuringClose() STARTED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSessionExpiryDuringClose() PASSED
[2023-02-23T12:56:19.169Z] 
[2023-02-23T12:56:19.169Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testReinitializeAfterAuthFailure() STARTED
[2023-02-23T12:56:22.322Z] 
[2023-02-23T12:56:22.322Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testReinitializeAfterAuthFailure() PASSED
[2023-02-23T12:56:22.322Z] 
[2023-02-23T12:56:22.322Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSetAclNonExistentZNode() STARTED
[2023-02-23T12:56:22.322Z] 
[2023-02-23T12:56:22.322Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSetAclNonExistentZNode() PASSED
[2023-02-23T12:56:22.322Z] 
[2023-02-23T12:56:22.322Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnectionLossRequestTermination() 
STARTED
[2023-02-23T12:56:27.801Z] 
[2023-02-23T12:56:27.801Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 183 > DynamicBrokerReconfigurationTest > testTrustStoreAlter(String) > 
kafka.server.DynamicBrokerReconfigurationTest.testTrustStoreAlter(String)[1] 
PASSED
[2023-02-23T12:56:27.801Z] 
[2023-02-23T12:56:27.801Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 183 > DynamicBrokerReconfigurationTest > testTrustStoreAlter(String) > 
kafka.server.DynamicBrokerReconfigurationTest.testTrustStoreAlter(String)[2] 
STARTED
[2023-02-23T12:56:33.719Z] 
[2023-02-23T12:56:33.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnectionLossRequestTermination() 
PASSED
[2023-02-23T12:56:3

Message conversion and TLS

[Sergio Daniel Troiano]

Hey lads,

I was thinking about message conversions and zero copy, as we know TLS will
break the zero copy as the encrypt/decrypt happens in userland, correct?

For example  ProduceMessageConversionsPerSec  will be triggered when a
message must be converted (for example compressing at topic level).

Should we consider the TLS layer a conversion as well? My concern is we are
"hiding" this metric, so let's suppose I have my cluster using TLS (it
means zero-copy is already "broken") then I activate the compression at
topic level.
Now I will see the values in ProduceMessageConversionsPerSec  as the
conversion is happening (tracking the compression at topic level) but the
reality is it was already happening before because of the TLS, so in terms
of performance it's a kind of illusion as it seems the conversion is not
costing any extra CPU cycles.

If you think it is a good idea maybe we could include the count on the
ProduceMessageConversionsPerSec when the TLS is enabled, what do you think?

Thanks in advance.
Sergio Troiano

Re: [DISCUSS] KIP-908: Add description field to connector configuration

[Chris Egerton]

Hi Mickael,

Thanks for the KIP!

While it's tempting to add this field to the out-of-the-box connector
config def, I'm a little hesitant, for two reasons.

1. Adding this directly to the connector config could have unintended
consequences on the actual data processing by the connector. Any time a
connector's config is modified, the Connector object running for it is
restarted with that new config. In most cases this is a trivial operation
since we have incremental rebalancing enabled by default, the connector can
(and probably should!) generate task configs that are functionally
identical to the ones it last generated, and most (though not all)
Connector classes are fairly lightweight and leave the real work to their
Task class. However, due to KAFKA-9228 [1], it's not just common practice
for connectors to perform transparent passthrough of most of their configs
when generating task configs, it's actually necessary to work around a bug
in the runtime. As a result, tweaking the description of a connector would
be fairly likely to result in a full restart of all of its tasks, in
addition to triggering two rebalances (which may not be so lightweight if
users are still running with eager rebalancing... which, sadly, I've heard
is still happening today).

2. The motivation section mentions some information that might go into the
description field, such as the team that owns the connector and emergency
contact info. It seems like this info might benefit from a little more
structure if we're trying to design for programmatic access by GUIs and
CLIs (which I'm assuming is the case, since I can't imagine a human being
getting much use out of the raw output from the GET
/connector-plugins/{name}/config and PUT
/connector-plugins/{name}/config/validate endpoints). This might also make
it easier to add custom validation logic around what kind of information is
present via REST extension.


With these thoughts in mind, what do you think about adding a new generic
"tags" object to connectors that can support arbitrary user-provided
key/value pairs? If using the POST /connectors endpoint, it might look
something like this:

{
  "name": "my-connector",
  "config": {
"connector.class": "MirrorSource",
"tasks.max": "908"
  },
  "tags": {
"team": "data-infra",
"phone": "12345678901",
"email": "din...@example.org"
  }
}

And, to allow users to modify connector tags after one has been created, we
might introduce a /connectors/{name}/tags endpoint with PUT/PATCH/DELETE
verbs that writes tag info for a connector to the config topic, but without
altering the actual connector config (allowing us to skip a rebalance
altogether).

One other thing we could consider is allowing cluster administrators to
require that connectors are created with a certain set of tags, or even add
per-tag regex validation. They might specify something like
"connector.required.tags = team, phone, email" or
"connector.tags.phone.regex = [0-9]{11}" in a worker config file. But this
is probably overboard for now, especially since it's already possible to
accomplish via a REST extension.

Finally, I'm also wondering if, in pursuit of expanding Connect's
out-of-the-box support for dealing with connector metadata, we might want
to expose created/last-updated times for connector configurations. We
definitely don't have to do that in this KIP but if you agree that this
would be useful, we should probably keep it in mind to avoid painting
ourselves into a corner. This is why I'm thinking of using "tags" instead
of something a little more generic like "metadata", BTW.

[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-908%3A+Add+description+field+to+connector+configuration

Cheers,

Chris

On Thu, Feb 23, 2023 at 4:52 AM Mickael Maison 
wrote:

> Hi,
>
> I created a very small KIP to add a description field to connectors:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-908%3A+Add+description+field+to+connector+configuration
>
> Let me know if you have any feedback.
>
> Thanks,
> Mickael
>

[jira] [Created] (KAFKA-14744) NPE while converting OffsetFetch from < 8 to >= 8

[David Jacot (Jira)]

David Jacot created KAFKA-14744:
---

 Summary: NPE while converting OffsetFetch from < 8 to >= 8
 Key: KAFKA-14744
 URL: https://issues.apache.org/jira/browse/KAFKA-14744
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.0
Reporter: David Jacot
Assignee: David Jacot


This NPE is here: 
https://github.com/apache/kafka/commit/24a86423e9907b751d98fddc7196332feea2b48d#diff-0f2f19fd03e2fc5aa9618c607b432ea72e5aaa53866f07444269f38cb537f3feR237.
 `topics` can be `null` to signal that all topics must be fetched.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #87

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14743) MessageConversionsTimeMs metric is not updated

[Luke Chen (Jira)]

Luke Chen created KAFKA-14743:
-

 Summary: MessageConversionsTimeMs metric is not updated
 Key: KAFKA-14743
 URL: https://issues.apache.org/jira/browse/KAFKA-14743
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.3.1, 3.4.0
Reporter: Luke Chen
Assignee: Luke Chen
 Attachments: image-2023-02-23-18-09-24-916.png

During message conversion, there are 2 metrics we should update as doc written:

!image-2023-02-23-18-09-24-916.png|width=652,height=121!

 

In KAFKA-14295, it is addressing the issue in FetchMessageConversionsPerSec 
metric. This ticket will address the issue in 
*kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request=fetch.*

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-908: Add description field to connector configuration

[Mickael Maison]

Hi,

I created a very small KIP to add a description field to connectors:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-908%3A+Add+description+field+to+connector+configuration

Let me know if you have any feedback.

Thanks,
Mickael

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1609

[Apache Jenkins Server]

See