Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #138

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 519365 lines...]
[2023-05-24T06:40:22.738Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2023-05-24T06:40:22.740Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-05-24T06:41:22.842Z] > Task :core:classes
[2023-05-24T06:41:22.842Z] > Task :core:compileTestJava NO-SOURCE
[2023-05-24T06:41:39.694Z] > Task :core:compileTestScala
[2023-05-24T06:43:14.584Z] > Task :core:testClasses
[2023-05-24T06:43:28.743Z] > Task :streams:compileTestJava
[2023-05-24T06:43:28.743Z] > Task :streams:testClasses
[2023-05-24T06:43:29.713Z] > Task :streams:testJar
[2023-05-24T06:43:29.713Z] > Task :streams:testSrcJar
[2023-05-24T06:43:29.713Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-05-24T06:43:29.713Z] > Task :streams:publishToMavenLocal
[2023-05-24T06:43:29.713Z] 
[2023-05-24T06:43:29.713Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-05-24T06:43:29.713Z] 
[2023-05-24T06:43:29.713Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-05-24T06:43:29.713Z] 
[2023-05-24T06:43:29.713Z] See 
https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings
[2023-05-24T06:43:29.713Z] 
[2023-05-24T06:43:29.713Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2023-05-24T06:43:29.713Z] Please consult deprecation warnings for more details.
[2023-05-24T06:43:29.713Z] 
[2023-05-24T06:43:29.713Z] BUILD SUCCESSFUL in 4m 57s
[2023-05-24T06:43:29.713Z] 81 actionable tasks: 37 executed, 44 up-to-date
[Pipeline] sh
[2023-05-24T06:43:32.668Z] + grep ^version= gradle.properties
[2023-05-24T06:43:32.668Z] + cut -d= -f 2
[Pipeline] dir
[2023-05-24T06:43:33.389Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-05-24T06:43:35.624Z] + mvn clean install -Dgpg.skip
[2023-05-24T06:43:38.337Z] [INFO] Scanning for projects...
[2023-05-24T06:43:38.337Z] [INFO] 

[2023-05-24T06:43:38.337Z] [INFO] Reactor Build Order:
[2023-05-24T06:43:38.337Z] [INFO] 
[2023-05-24T06:43:38.337Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2023-05-24T06:43:38.337Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2023-05-24T06:43:38.337Z] [INFO] 
[2023-05-24T06:43:38.337Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2023-05-24T06:43:38.337Z] [INFO] Building Kafka Streams :: Quickstart 
3.4.1-SNAPSHOT[1/2]
[2023-05-24T06:43:38.337Z] [INFO]   from pom.xml
[2023-05-24T06:43:38.337Z] [INFO] [ pom 
]-
[2023-05-24T06:43:38.337Z] [INFO] 
[2023-05-24T06:43:38.337Z] [INFO] --- clean:3.0.0:clean (default-clean) @ 
streams-quickstart ---
[2023-05-24T06:43:39.307Z] [INFO] 
[2023-05-24T06:43:39.307Z] [INFO] --- remote-resources:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2023-05-24T06:43:40.270Z] [INFO] 
[2023-05-24T06:43:40.270Z] [INFO] --- site:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [INFO] --- gpg:1.6:sign (sign-artifacts) @ 
streams-quickstart ---
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [INFO] --- install:2.5.2:install (default-install) @ 
streams-quickstart ---
[2023-05-24T06:43:41.223Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.4/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.4.1-SNAPSHOT/streams-quickstart-3.4.1-SNAPSHOT.pom
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2023-05-24T06:43:41.223Z] [INFO] Building streams-quickstart-java 
3.4.1-SNAPSHOT[2/2]
[2023-05-24T06:43:41.223Z] [INFO]   from java/pom.xml
[2023-05-24T06:43:41.223Z] [INFO] --[ maven-archetype 
]---
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [INFO] --- clean:3.0.0:clean (default-clean) @ 
streams-quickstart-java ---
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [INFO] --- remote-resources:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2023-05-24T06:43:41.223Z] [INFO] 
[2023-05-24T06:43:41.223Z] [I

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1863

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #137

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.5 #4

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15015) Binaries contain 2 versions of reload4j

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15015.
---
Fix Version/s: 3.5.0
   3.4.1
   Resolution: Fixed

> Binaries contain 2 versions of reload4j
> ---
>
> Key: KAFKA-15015
> URL: https://issues.apache.org/jira/browse/KAFKA-15015
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0, 3.4.1
>Reporter: Mickael Maison
>Assignee: Atul Sharma
>Priority: Major
> Fix For: 3.5.0, 3.4.1
>
>
> These releases ship 2 versions of reload4j:
> - reload4j-1.2.19.jar
> - reload4j-1.2.25.jar



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1862

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1861

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 476726 lines...]
[2023-05-23T20:01:56.855Z] 
[2023-05-23T20:01:56.855Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > VersionedKeyValueStoreIntegrationTest > shouldRestore PASSED
[2023-05-23T20:01:56.855Z] 
[2023-05-23T20:01:56.855Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > VersionedKeyValueStoreIntegrationTest > 
shouldPutGetAndDelete STARTED
[2023-05-23T20:01:56.855Z] 
[2023-05-23T20:01:56.855Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > VersionedKeyValueStoreIntegrationTest > 
shouldPutGetAndDelete PASSED
[2023-05-23T20:01:56.855Z] 
[2023-05-23T20:01:56.855Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned STARTED
[2023-05-23T20:02:46.661Z] 
[2023-05-23T20:02:46.661Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED
[2023-05-23T20:02:46.661Z] 
[2023-05-23T20:02:46.661Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED
[2023-05-23T20:02:52.935Z] 
[2023-05-23T20:02:52.935Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED
[2023-05-23T20:02:53.874Z] 
[2023-05-23T20:02:53.874Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers STARTED
[2023-05-23T20:03:15.082Z] 
[2023-05-23T20:03:15.082Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2023-05-23T20:03:15.082Z] 
[2023-05-23T20:03:15.082Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2023-05-23T20:03:27.452Z] 
[2023-05-23T20:03:27.452Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2023-05-23T20:03:27.452Z] 
[2023-05-23T20:03:27.452Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2023-05-23T20:03:27.452Z] 
[2023-05-23T20:03:27.452Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2023-05-23T20:03:27.452Z] 
[2023-05-23T20:03:27.452Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2023-05-23T20:03:34.929Z] 
[2023-05-23T20:03:34.929Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2023-05-23T20:03:34.929Z] 
[2023-05-23T20:03:34.929Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2023-05-23T20:03:36.037Z] 
[2023-05-23T20:03:36.037Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2023-05-23T20:03:36.037Z] 
[2023-05-23T20:03:36.037Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2023-05-23T20:03:37.238Z] 
[2023-05-23T20:03:37.238Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2023-05-23T20:03:37.238Z] 
[2023-05-23T20:03:37.238Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2023-05-23T20:04:08.394Z] 
[2023-05-23T20:04:08.394Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount PASSED
[2023-05-23T20:04:08.394Z] 
[2023-05-23T20:04:08.394Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2023-05-23T20:04:33.466Z] 
[2023-05-23T20:04:33.466Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorLarg

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

[Hao Li]

Thanks John! Yeah. The ConvergenceTest looks very helpful. I will add it to
the test plan. I will also add tests to verify the new optimizer will
produce a balanced assignment which has no worse cross AZ cost than the
existing assignor.

Hao

On Mon, May 22, 2023 at 3:39 PM John Roesler  wrote:

> Hi Hao,
>
> Thanks for the KIP!
>
> Overall, I think this is a great idea. I always wanted to circle back
> after the Smooth Scaling KIP to put a proper optimization algorithm into
> place. I think this has the promise to really improve the quality of the
> balanced assignments we produce.
>
> Thanks for providing the details about the MaxCut/MinFlow algorithm. It
> seems like a good choice for me, assuming we choose the right scaling
> factors for the weights we add to the graph. Unfortunately, I don't think
> that there's a good way to see how easy or hard this is going to be until
> we actually implement it and test it.
>
> That leads to the only real piece of feedback I had on the KIP, which is
> the testing portion. You mentioned system/integration/unit tests, but
> there's not too much information about what those tests will do. I'd like
> to suggest that we invest in more simulation testing specifically, similar
> to what we did in
> https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java
> .
>
> In fact, it seems like we _could_ write the simulation up front, and then
> implement the algorithm in a dummy way and just see whether it passes the
> simulations or not, before actually integrating it with Kafka Streams.
>
> Basically, I'd be +1 on this KIP today, but I'd feel confident about it if
> we had a little more detail regarding how we are going to verify that the
> new optimizer is actually going to produce more optimal plans than the
> existing assigner we have today.
>
> Thanks again!
> -John
>
> On 2023/05/22 16:49:22 Hao Li wrote:
> > Hi Colt,
> >
> > Thanks for the comments.
> >
> > > and I struggle to see how the algorithm isn't at least O(N) where N is
> > the number of Tasks...?
> >
> > For O(E^2 * (CU)) complexity, C and U can be viewed as constant. Number
> of
> > edges E is T * N where T is the number of clients and N is the number of
> > Tasks. This is because a task can be assigned to any client so there will
> > be an edge between every task and every client. The total complexity
> would
> > be O(T * N) if we want to be more specific.
> >
> > > But if the leaders for each partition are spread across multiple zones,
> > how will you handle that?
> >
> > This is what the min-cost flow solution is trying to solve? i.e. Find an
> > assignment of tasks to clients where across AZ traffic can be minimized.
> > But there are some constraints to the solution and one of them is we need
> > to balance task assignment first (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams#KIP925:RackawaretaskassignmentinKafkaStreams-Designforrackawareassignment
> ).
> > So in your example of three tasks' partitions being in the same AZ of a
> > client, if there are other clients, we still want to balance the tasks to
> > other clients even if putting all tasks to a single client can result in
> 0
> > cross AZ traffic. In
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams#KIP925:RackawaretaskassignmentinKafkaStreams-Algorithm
> > section, the algorithm will try to find a min-cost solution based on
> > balanced assignment instead of pure min-cost.
> >
> > Thanks,
> > Hao
> >
> > On Tue, May 9, 2023 at 5:55 PM Colt McNealy  wrote:
> >
> > > Hello Hao,
> > >
> > > First of all, THANK YOU for putting this together. I had been hoping
> > > someone might bring something like this forward. A few comments:
> > >
> > > **1: Runtime Complexity
> > > > Klein’s cycle canceling algorithm can solve the min-cost flow
> problem in
> > > O(E^2CU) time where C is max cost and U is max capacity. In our
> particular
> > > case, C is 1 and U is at most 3 (A task can have at most 3 topics
> including
> > > changelog topic?). So the algorithm runs in O(E^2) time for our case.
> > >
> > > A Task can have multiple input topics, and also multiple state stores,
> and
> > > multiple output topics. The most common case is three topics as you
> > > described, but this is not necessarily guaranteed. Also, math is one
> of my
> > > weak points, but to me O(E^2) is equivalent to O(1), and I struggle to
> see
> > > how the algorithm isn't at least O(N) where N is the number of
> Tasks...?
> > >
> > > **2: Broker-Side Partition Assignments
> > > Consider the case with just three topics in a Task (one input, one
> output,
> > > one changelog). If all three partition leaders are in the same Rack (or
> > > better yet, the same broker), then we could get massive savings by
> > > assigning the Task to that Rack/availability zone. But if th

[jira] [Created] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-15018:
-

 Summary: Potential tombstone offsets corruption for exactly-once 
source connectors
 Key: KAFKA-15018
 URL: https://issues.apache.org/jira/browse/KAFKA-15018
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 3.3.2, 3.3.1, 3.4.0, 3.3.0, 3.5.0, 3.4.1
Reporter: Chris Egerton


When exactly-once support is enabled for source connectors, source offsets can 
potentially be written to two different offsets topics: a topic specific to the 
connector, and the global offsets topic (which was used for all connectors 
prior to KIP-618 / version 3.3.0).

Precedence is given to offsets in the per-connector offsets topic, but if none 
are found for a given partition, then the global offsets topic is used as a 
fallback.

When committing offsets, a transaction is used to ensure that source records 
and source offsets are written to the Kafka cluster targeted by the source 
connector. This transaction only includes the connector-specific offsets topic. 
Writes to the global offsets topic take place after writes to the 
connector-specific offsets topic have completed successfully, and if they fail, 
a warning message is logged, but no other action is taken.

Normally, this ensures that, for offsets committed by exactly-once-supported 
source connectors, the per-connector offsets topic is at least as up-to-date as 
the global offsets topic, and sometimes even ahead.

However, for tombstone offsets, we lose that guarantee. If a tombstone offset 
is successfully written to the per-connector offsets topic, but cannot be 
written to the global offsets topic, then the global offsets topic will still 
contain that source offset, but the per-connector topic will not. Due to the 
fallback-on-global logic used by the worker, if a task requests offsets for one 
of the tombstoned partitions, the worker will provide it with the offsets 
present in the global offsets topic, instead of indicating to the task that no 
offsets can be found.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi everyone,

Someone suggested at the recent Kafka Summit that it may be useful
to have a video call to discuss remaining concerns.

I'm proposing we have a video call Monday 29th May 16:30-17:00 UTC.

If you'd like to join, please reply to the thread or to me directly so
I can send you a link.

Please also do let me know if you'd like to attend but the proposed
time does not work for you.

Thanks,

--
Igor

Re: KafkaConsumer refactor proposal

[Philip Nee]

Hi all!

I've added the "KIP-848" and "ctr" tags to the relevant PRs.

see: https://github.com/apache/kafka/pulls?q=label%3Actr+sort%3Acreated-asc

On Tue, May 9, 2023 at 10:29 AM Philip Nee  wrote:

> Hey Divij,
>
> Firstly, THANK YOU for reaching out and providing the feedback.  At this
> moment, I don't have all of the answers to your questions at the moment,
> but I hope my responses could be helpful in moving the project forward.
>
> 1. There isn't a prioritization strategy, and the network request buffer
> is just FIFO. Basically, all requests are buffered in the network queue and
> sent out at the end of the event loop where we poll the client.
> Presumably, your concern is that the member can get constantly kicked out
> of the group due to not sending out a heartbeat soon enough under high load
> or network congestion.  I think that's a fair point.  On the other hand, we
> design the background thread (or we would like the design to be...) to be
> lightweight and focus on network communication, such that the client can
> quickly go through all the requests without consuming too many resources.
> In the case of network congestion, I think there could be other reasons
> that cause members to fail, so I don't know if we need the prioritize the
> requests.
>
> 2. Yes, as of now, the fetch offset request is being de-duped, and we
> would do that for other requests as well.  There's also a RequestState
> class that helps request managers track retry backoff (exponential
> backoff), which prevents members from spamming brokers.
>
> 3. There's no implementation of the backpressure for both the application
> and background event queue, but here is my thought:
>  - For ApplicationEventQueue: I can think of a few ways to implement this.
> 1. user handles a RetriableError and re-enqueues the event. or 2. Having
> the application to buffer the events, but then we are just implementing a
> queue on top of the queue.  What's your suggestion here?
>  - For BackgroundEventQueue: The main events are fetches, errors, and
> rebalance triggers.  For fetching, what's in my mind is some kind of memory
> bound mechanism like KIP-81
> 
>  (it
> hasn't been implemented).  For rebalancing events or errors, there is not
> much we could do aside from pausing the request managers.  What do you
> think?
>
> 4. See above.
>
> 5. I'll add an Observability section to the one-pager. My thinking is
> similar to yours: 1. memory and CPU pressure and 2. event latency.
>
> I would love to incorporate some of your past experience from TinkerPop
> into this design, so thank you again for sharing your feedback!
>
> Thank you!
> P
>
> On Tue, May 9, 2023 at 9:50 AM Divij Vaidya 
> wrote:
>
>> Hey Philip
>>
>> Thank you for writing this down.
>>
>> Overall, the ideal of decoupling the responsibility amongst threads and
>> communication using queues looks good. I have a few questions
>> associated with handling failure scenarios.
>>
>> 1. I observed that there is no concept of prioritization in the background
>> thread. In case of high load, should a heartbeat handler be prioritized
>> over listOffset handler?
>> 2. Idempotent requests could be "bulked" together, e.g. if we have two
>> list
>> offsets in the queue, we should be able to call listOffset once and
>> duplicate the response for the requests
>> 3. What would the back pressure handling behaviour from the
>> applicationEventQueue look like? This is bound to occur during
>> intermittent
>> network disruption
>> 4. Same as 3 but for backgroundEventQueue. If the client thread is doing
>> some I/O since the user is blocking it in their business logic, then the
>> queue may get full. What happens then? How do we recover and cancel the
>> resources such as waiting sync APIs?
>> 5. Can we add observability/metrics in the design? As an example, the end
>> user might measure latency from the client thread but the request may
>> actually be spending time in the queue waiting to be picked by the
>> background thread. In such cases, user should have the visibility to
>> identify that a need to scale the background thread processing is
>> required.
>>
>> --
>> Divij Vaidya
>>
>>
>>
>> On Sun, Sep 18, 2022 at 11:24 PM Philip Nee  wrote:
>>
>> > On Sun, Sep 18, 2022 at 6:03 AM Luke Chen  wrote:
>> >
>> > > Hi Philip,
>> > >
>> > > Thanks for the write-up.
>> > >
>> > Also thank you for taking the time to read the proposal.  Very grateful.
>> >
>> > > Some questions:
>> > >
>> > > 1. You said: If we don't need a coordinator, the background thread
>> will
>> > > stay in the *initialized* state.
>> > > But in the definition of *initialized, *it said:
>> > > *initialized*: The background thread completes initialization, and the
>> > loop
>> > > has started.
>> > > Does that mean, even if we don't need a coordinator, we still create a
>> > > coordinator and run the loop? And why?
>> > >
>> > If we don't

Re: [VOTE] 3.5.0 RC0

[David Arthur]

Mickael, we have some migration fixes on trunk, is it okay to cherry-pick
these to 3.5?

KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
KAFKA-15009 Handle new ACLs in KRaft snapshot during migration

There is another issue KAFKA-15010 that I'm also investigating to determine
the impact and likelihood of seeing it in practice. This one may be a
significant migration blocker

Cheers,
David

On Tue, May 23, 2023 at 9:57 AM Mickael Maison 
wrote:

> Hi Christo,
>
> Yes this is expected. This happens when nested fields also accept
> optional tagged fields. The tables list all fields, so they may
> include _tagged_fields multiple times.
> Clearly the layout of this page could be improved, if you have ideas
> how to describe the protocol in a better way, feel free to share them.
>
> Thanks,
> Mickael
>
> On Tue, May 23, 2023 at 3:50 PM Mickael Maison 
> wrote:
> >
> > Hi Josep,
> >
> > Good catch! I opened a PR to fix this:
> > https://github.com/apache/kafka-site/pull/514
> >
> > Thanks,
> > Mickael
> >
> >
> > On Tue, May 23, 2023 at 3:36 PM Christo Lolov 
> wrote:
> > >
> > > Hey Mickael!
> > >
> > > I am giving a +1 (non-binding) for this candidate release.
> > >
> > > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on
> Intel
> > > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > > * Ran unit and integration tests on Intel and ARM machines.
> > > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> > > machines.
> > >
> > > Question:
> > > * I went through https://kafka.apache.org/35/protocol.html and there
> are
> > > quite a few repetitive __tagged_fileds fields within the same
> structures -
> > > is this expected?
> > >
> > > On Tue, 23 May 2023 at 12:01, Josep Prat 
> > > wrote:
> > >
> > > > Hi Mickael,
> > > > I just wanted to point out that I think the documentation you
> recently
> > > > merged on Kafka site regarding the 3.5.0 version has a problem when
> it
> > > > states the version number and the sub-menu that links to previous
> versions.
> > > > Left a comment here:
> > > >
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > > >
> > > > Best,
> > > >
> > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat 
> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I can +1 this candidate. I verified the following:
> > > > > - Built from source with Java 17 and Scala 2.13
> > > > > - Signatures and hashes of the artifacts generated
> > > > > - Navigated through Javadoc including links to JDK classes
> > > > > - Run the unit tests
> > > > > - Run integration tests
> > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > >
> > > > > Best,
> > > > >
> > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison <
> mimai...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Hello Kafka users, developers and client-developers,
> > > > >>
> > > > >> This is the first candidate for release of Apache Kafka 3.5.0.
> Some of
> > > > the
> > > > >> major features include:
> > > > >> - KIP-710: Full support for distributed mode in dedicated
> MirrorMaker
> > > > >> 2.0 clusters
> > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > > > >> - KIP-889: Versioned State Stores
> > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic
> configurations
> > > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> for
> > > > >> Kafka Brokers
> > > > >>
> > > > >> Release notes for the 3.5.0 release:
> > > > >>
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > > >>
> > > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > > > >>
> > > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > >> https://kafka.apache.org/KEYS
> > > > >>
> > > > >> * Release artifacts to be voted upon (source and binary):
> > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > > >>
> > > > >> * Maven artifacts to be voted upon:
> > > > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >>
> > > > >> * Javadoc:
> > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > > >>
> > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > > >>
> > > > >> The PR adding the 35 documentation is not merged yet
> > > > >> (https://github.com/apache/kafka-site/pull/513)
> > > > >> * Documentation:
> > > > >> https://kafka.apache.org/35/documentation.html
> > > > >> * Protocol:
> > > > >> https://kafka.apache.org/35/protocol.html
> > > > >>
> > > > >> * Successful Jenkins builds for the 3.5 branch:
> > > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > > > >> working with INFRA to sort it out:
> > > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > > >> System tests: The build is still running, I'l

Re: [DISCUSS] KIP-932: Queues for Kafka

[Ashish Chhabria]

This is such a great proposal. Parallel/Cooperative consumption is a long
missing pattern in Apache Kafka and it is a viable solution for the common
head-of-line-blocking problem. Developers have long attempted to solve this
with bespoke consumer proxies that state manage the inflight payloads, so
this would definitely be helpful to have in Apache Kafka.

On Mon, 22 May 2023 at 05:09, Andrew Otto  wrote:

> Wow looks very cool.
>
> One Q.
>
> In the Future Work session:
> > The concept can be extended to give key-based ordering so that partial
> ordering and fine-grained sharing can be achieved at the same time.
>
> I think this will be a pretty important feature needed to make shared
> consumer groups useful.  Perhaps it would be worth considering how this
> could be implemented and mentioning it in the KIP?
>
>
>
>
>
>
> On Mon, May 22, 2023 at 6:20 AM Stanislav Kozlovski
>  wrote:
>
> > Hey Andrew!
> >
> > Kudos on the proposal. It is greatly written - a joy to read. It is
> > definitely an interesting solution to the queueing problem - I would not
> > have guessed we could solve it like this. Thank you for working on this.
> >
> > Happy to get the discussion started - I have a few comments/questions on
> > first read:
> >
> > 1. Tiered Storage
> >
> > I notice no mention of Tiered Storage (KIP-405). Does that complicate the
> > design, especially when fetching historical data? It would be good to
> have
> > at least one sentence mentioning it, even if it doesn't impact it. Right
> > now I'm unsure if it was considered.
> >
> > 2. SSO initialized to the latest offset
> >
> > > "By default, the SSO for each share-partition is initialized to the
> > latest offset for the corresponding topic-partitions."
> >
> > Have we considered allowing this to be configurable to latest/earliest?
> > This would be consistent with the auto.offset.reset config of vanilla
> > consumer groups.
> > Thinking from a user's perspective, it sounds valid to want to start from
> > the start of a topic when starting a share group. Historical processing
> > comes to mind
> >
> > 3. Durable Storage
> >
> > The KIP mentions that "The cluster records this information durably",
> which
> > implies that it saves it somewhere. Does the ShareCoordinator have its
> own
> > topic? Would it be compacted?
> >
> > In particular, I am interested in what such a topic's retention would be
> > like. The vanilla consumer offsets topic has some special retention
> > semantics (KIP-211) where we start counting the retention after the
> > consumer group becomes empty (inactive) - the default being 7 days. Need
> to
> > make sure the retention here isn't too short either, as the offsets topic
> > originally had 24 hours of retention and that proved problematic.
> >
> > In general, some extra detail about the persistence would be greatly
> > appreciated!
> >
> > 4. Batch Acknowledgement
> >
> > > "In the situation where some records in a batch have been released or
> > rejected separately, subsequent fetches of those records are more likely
> to
> > have gaps."
> >
> > Can we expand a bit more on this edge case? I am interested in learning
> > what gets returned on subsequent fetch requests.
> > In particular, - how does this work with compression? As far as I
> remember,
> > we can compress the whole batch there, which might make individual record
> > filtering tricky.
> >
> > 5. Member Management
> >
> > How is consumer group member management handled? I didn't see any
> specific
> > mention - is it the same as a vanilla group?
> > In particular - how will bad consumers be handled?
> >
> > I guess I see two cases:
> > 1. bad consumer that doesn't even heartbeat
> > 2. bad consumer that heartbeats well but for some reason every message
> > processing times out. e.g imagine it was network partitioned from some
> > third-party system that is a critical part of its message processing loop
> >
> > One evident problem I can foresee in production systems is one (or a few)
> > slow consumer applications bringing the SSO/SEO advancement down to a
> > crawl.
> > Imagine an example where the same consumer app always hits the timeout
> > limit - what would the behavior be in such a case? Do we keep that
> consumer
> > app indefinitely (if so, do we run the risk of having it invalidate
> > completely valid messages)? Are there any equivalents to the consumer
> group
> > rebalances which fence off such bad consumers?
> >
> > 6. Processing Semantics (exactly once)
> >
> > > The delivery counts are only maintained approximately and the Acquired
> > state is not persisted.
> >
> > Does this introduce the risk of zombie consumers on
> share-partition-leader
> > failure? i.e restarting and giving another consumer the acquired state
> for
> > the same record
> >
> > I notice that the KIP says:
> > > Finally, this KIP does not include support for acknowledging delivery
> > using transactions for exactly-once semantics.
> > at the very end. It would be helpful to address th

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Christo,

Thank you for your interest in this KIP.

Indeed, I'd like to open up voting ASAP.
I'm hoping there will still be a bit more feedback,
but if not I'll probably request a vote next week or so.

Do you have any concerns or suggestions regarding this KIP?

I'll have a look at your KIP too, it looks interesting.

Best,

--
Igor

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Dániel Urbán]

Hi Chris,

Thank you for your comments!
1. This approach is OK for me. I thought that the "sample" configs in the
repo do not fall into the same category as the default of a new config.
Adding a commented line instead should be ok, and the future opt-out change
sounds good to me.

2. Besides the MDC, there are 2 other places where the flow context
information could be added:
- Some of the Connect internal classes use their .toString methods when
logging (e.g. WorkerSinkTask has a line like this: log.info("{} Executing
sink task", this);). These toString implementations contain the connector
name, and nothing else, so in MM2 dedicated mode, adding the flow would
make these lines unique.
- Connect worker thread names. Currently, they contain the connector
name/task ID, but to make them unique, the flow should be added in MM2
distributed mode.
In my draft PR, I changed both of these, but for the sake of backward
compatibility, I made the new toString/thread name dependent on the new,
suggested configuration flag.
If we go with a new MDC key, but we still want to change the toString
methods and the thread names, we still might need an extra flag to turn on
the new behavior.
AFAIK the toString calls are all made inside a proper logging context, so
changing the toString methods don't add much value. I think that the thread
name changes are useful, giving more context for log lines written outside
of a log context.

In short, I think that MDC + thread name changes are necessary to make MM2
dedicated logs useful for diagnostics. If we keep both, then maybe having a
single config to control both at the same time is better than having a new
MDC key (configured separately in the log pattern) and a config flag (set
separately in the properties file) for the thread name change.

Thanks,
Daniel

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #2

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] mimaison merged pull request #514: MINOR: Fix version in 35 docs

[via GitHub]

mimaison merged PR #514:
URL: https://github.com/apache/kafka-site/pull/514


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15017) New ClientQuotas are not written to ZK from snapshot

[David Arthur (Jira)]

David Arthur created KAFKA-15017:


 Summary: New ClientQuotas are not written to ZK from snapshot 
 Key: KAFKA-15017
 URL: https://issues.apache.org/jira/browse/KAFKA-15017
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.5.0
Reporter: David Arthur


Similar issue to KAFKA-15009



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.5.0 RC0

[Mickael Maison]

Hi Christo,

Yes this is expected. This happens when nested fields also accept
optional tagged fields. The tables list all fields, so they may
include _tagged_fields multiple times.
Clearly the layout of this page could be improved, if you have ideas
how to describe the protocol in a better way, feel free to share them.

Thanks,
Mickael

On Tue, May 23, 2023 at 3:50 PM Mickael Maison  wrote:
>
> Hi Josep,
>
> Good catch! I opened a PR to fix this:
> https://github.com/apache/kafka-site/pull/514
>
> Thanks,
> Mickael
>
>
> On Tue, May 23, 2023 at 3:36 PM Christo Lolov  wrote:
> >
> > Hey Mickael!
> >
> > I am giving a +1 (non-binding) for this candidate release.
> >
> > * Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
> > (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> > * Ran unit and integration tests on Intel and ARM machines.
> > * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> > machines.
> >
> > Question:
> > * I went through https://kafka.apache.org/35/protocol.html and there are
> > quite a few repetitive __tagged_fileds fields within the same structures -
> > is this expected?
> >
> > On Tue, 23 May 2023 at 12:01, Josep Prat 
> > wrote:
> >
> > > Hi Mickael,
> > > I just wanted to point out that I think the documentation you recently
> > > merged on Kafka site regarding the 3.5.0 version has a problem when it
> > > states the version number and the sub-menu that links to previous 
> > > versions.
> > > Left a comment here:
> > > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> > >
> > > Best,
> > >
> > > On Tue, May 23, 2023 at 9:29 AM Josep Prat  wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > I can +1 this candidate. I verified the following:
> > > > - Built from source with Java 17 and Scala 2.13
> > > > - Signatures and hashes of the artifacts generated
> > > > - Navigated through Javadoc including links to JDK classes
> > > > - Run the unit tests
> > > > - Run integration tests
> > > > - Run the quickstart in KRaft and Zookeeper mode
> > > >
> > > > Best,
> > > >
> > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison 
> > > > wrote:
> > > >
> > > >> Hello Kafka users, developers and client-developers,
> > > >>
> > > >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> > > the
> > > >> major features include:
> > > >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > > >> 2.0 clusters
> > > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > > >> - KIP-889: Versioned State Stores
> > > >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > > >> Kafka Brokers
> > > >>
> > > >> Release notes for the 3.5.0 release:
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > > >>
> > > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > > >>
> > > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >> https://kafka.apache.org/KEYS
> > > >>
> > > >> * Release artifacts to be voted upon (source and binary):
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > > >>
> > > >> * Maven artifacts to be voted upon:
> > > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>
> > > >> * Javadoc:
> > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > > >>
> > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > > >>
> > > >> The PR adding the 35 documentation is not merged yet
> > > >> (https://github.com/apache/kafka-site/pull/513)
> > > >> * Documentation:
> > > >> https://kafka.apache.org/35/documentation.html
> > > >> * Protocol:
> > > >> https://kafka.apache.org/35/protocol.html
> > > >>
> > > >> * Successful Jenkins builds for the 3.5 branch:
> > > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > > >> working with INFRA to sort it out:
> > > >> https://issues.apache.org/jira/browse/INFRA-24577
> > > >> System tests: The build is still running, I'll send an update once I
> > > >> have the results
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >
> > > >
> > > > --
> > > > [image: Aiven] 
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.p...@aiven.io   |   +491715557497
> > > > aiven.io    |
> > > > 
> > > >    <
> > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >

Re: [VOTE] 3.5.0 RC0

[Mickael Maison]

Hi Josep,

Good catch! I opened a PR to fix this:
https://github.com/apache/kafka-site/pull/514

Thanks,
Mickael


On Tue, May 23, 2023 at 3:36 PM Christo Lolov  wrote:
>
> Hey Mickael!
>
> I am giving a +1 (non-binding) for this candidate release.
>
> * Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
> (m5.4xlarge) and ARM (m6g.4xlarge) machines.
> * Ran unit and integration tests on Intel and ARM machines.
> * Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
> machines.
>
> Question:
> * I went through https://kafka.apache.org/35/protocol.html and there are
> quite a few repetitive __tagged_fileds fields within the same structures -
> is this expected?
>
> On Tue, 23 May 2023 at 12:01, Josep Prat 
> wrote:
>
> > Hi Mickael,
> > I just wanted to point out that I think the documentation you recently
> > merged on Kafka site regarding the 3.5.0 version has a problem when it
> > states the version number and the sub-menu that links to previous versions.
> > Left a comment here:
> > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
> >
> > Best,
> >
> > On Tue, May 23, 2023 at 9:29 AM Josep Prat  wrote:
> >
> > > Hi Mickael,
> > >
> > > I can +1 this candidate. I verified the following:
> > > - Built from source with Java 17 and Scala 2.13
> > > - Signatures and hashes of the artifacts generated
> > > - Navigated through Javadoc including links to JDK classes
> > > - Run the unit tests
> > > - Run integration tests
> > > - Run the quickstart in KRaft and Zookeeper mode
> > >
> > > Best,
> > >
> > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison 
> > > wrote:
> > >
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> > the
> > >> major features include:
> > >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > >> 2.0 clusters
> > >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > >> - KIP-887: Add ConfigProvider to make use of environment variables
> > >> - KIP-889: Versioned State Stores
> > >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > >> Kafka Brokers
> > >>
> > >> Release notes for the 3.5.0 release:
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> > >>
> > >> *** Please download, test and vote by Friday, May 26, 5pm PT
> > >>
> > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> https://kafka.apache.org/KEYS
> > >>
> > >> * Release artifacts to be voted upon (source and binary):
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> > >>
> > >> * Maven artifacts to be voted upon:
> > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>
> > >> * Javadoc:
> > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> > >>
> > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> > >>
> > >> The PR adding the 35 documentation is not merged yet
> > >> (https://github.com/apache/kafka-site/pull/513)
> > >> * Documentation:
> > >> https://kafka.apache.org/35/documentation.html
> > >> * Protocol:
> > >> https://kafka.apache.org/35/protocol.html
> > >>
> > >> * Successful Jenkins builds for the 3.5 branch:
> > >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> > >> working with INFRA to sort it out:
> > >> https://issues.apache.org/jira/browse/INFRA-24577
> > >> System tests: The build is still running, I'll send an update once I
> > >> have the results
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |
> > > 
> > >    <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |    > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >

Re: [VOTE] 3.5.0 RC0

[Christo Lolov]

Hey Mickael!

I am giving a +1 (non-binding) for this candidate release.

* Built from the binary tar.gz source with Java 17 and Scala 2.13 on Intel
(m5.4xlarge) and ARM (m6g.4xlarge) machines.
* Ran unit and integration tests on Intel and ARM machines.
* Ran the Quickstart in both Zookeeper and KRaft modes on Intel and ARM
machines.

Question:
* I went through https://kafka.apache.org/35/protocol.html and there are
quite a few repetitive __tagged_fileds fields within the same structures -
is this expected?

On Tue, 23 May 2023 at 12:01, Josep Prat 
wrote:

> Hi Mickael,
> I just wanted to point out that I think the documentation you recently
> merged on Kafka site regarding the 3.5.0 version has a problem when it
> states the version number and the sub-menu that links to previous versions.
> Left a comment here:
> https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939
>
> Best,
>
> On Tue, May 23, 2023 at 9:29 AM Josep Prat  wrote:
>
> > Hi Mickael,
> >
> > I can +1 this candidate. I verified the following:
> > - Built from source with Java 17 and Scala 2.13
> > - Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> >
> > Best,
> >
> > On Mon, May 22, 2023 at 5:30 PM Mickael Maison 
> > wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the first candidate for release of Apache Kafka 3.5.0. Some of
> the
> >> major features include:
> >> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> >> 2.0 clusters
> >> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> >> - KIP-887: Add ConfigProvider to make use of environment variables
> >> - KIP-889: Versioned State Stores
> >> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> >> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> >> Kafka Brokers
> >>
> >> Release notes for the 3.5.0 release:
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Friday, May 26, 5pm PT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
> >>
> >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
> >>
> >> The PR adding the 35 documentation is not merged yet
> >> (https://github.com/apache/kafka-site/pull/513)
> >> * Documentation:
> >> https://kafka.apache.org/35/documentation.html
> >> * Protocol:
> >> https://kafka.apache.org/35/protocol.html
> >>
> >> * Successful Jenkins builds for the 3.5 branch:
> >> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> >> working with INFRA to sort it out:
> >> https://issues.apache.org/jira/browse/INFRA-24577
> >> System tests: The build is still running, I'll send an update once I
> >> have the results
> >>
> >> Thanks,
> >> Mickael
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

[GitHub] [kafka-site] mimaison commented on a diff in pull request #513: MINOR: Add Kafka 3.5 documentation and javadoc

[via GitHub]

mimaison commented on code in PR #513:
URL: https://github.com/apache/kafka-site/pull/513#discussion_r1202345570


##
35/documentation.html:
##
@@ -0,0 +1,111 @@
+
+
+
+
+
+
+
+
+
+ 
+  
+<
+  
+  
+
+  
+  
+
+
+Documentation
+Kafka 3.4 Documentation

Review Comment:
   Good catch! I opened https://github.com/apache/kafka-site/pull/514



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison opened a new pull request, #514: MINOR: Fix version in 35 docs

[via GitHub]

mimaison opened a new pull request, #514:
URL: https://github.com/apache/kafka-site/pull/514

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #136

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-934: Add DeleteTopicPolicy

[Jorge Esteban Quilcate Otoya]

Thanks Christo!

On Tue, 23 May 2023 at 13:02, Christo Lolov  wrote:

> Heya Jorge,
>
> Thank you for the KIP!
>
> This feature sounds great to me since I have encountered problems with
> this, so I am supporting it. Do you have any idea why the previous KIPs
> were abandoned - I went through the email conversations and pull requests,
> but I didn't find a good reason?
>

Looking at the mailing thread, my understanding is:
In the case of KIP-170, it was retired as part of the KIP-201 when it was
figured they were overlapping. (around here[1]?)
For KIP-201, it's not clear. It was not stated within the mailing list
(last mail on Aug 2019), but on the PR that I found it was abandoned[3].
Maybe @Mickael Maison  may have more background?

[1] https://lists.apache.org/thread/6nkm706o7qjlz2ld7v8xdhf1421brh7o
[2] https://lists.apache.org/thread/cklw5fr5j1nhxyp5hpkfc1orm8r2tsmn
[3] https://github.com/apache/kafka/pull/4281#issuecomment-1035154386


>
> Best,
> Christo
>
> On Mon, 22 May 2023 at 17:19, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > I'd like to start a discussion for KIP-934 <
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-934%3A+Add+DeleteTopicPolicy
> > >
> > which proposes adding a new policy for when deleting topics.
> >
> > There have been related KIPs in the past that haven't been accepted and
> > seem retired/abandoned as outlined in the motivation.
> > The scope of this KIP intends to be more specific to get most of the
> > benefits from previous discussions; and if previous KIPs are resurrected,
> > should still be possible to do it if this one is adopted.
> >
> > Looking forward to your feedback!
> >
> > Thanks,
> > Jorge.
> >
>

Re: [VOTE] 3.4.1 RC1

[Luke Chen]

Hi Mickael,

Thanks for pointing this out.
Yes, I forgot to update the LICENSE-binary file.
I'll create another RC build to patch it.

Thanks Chris and Greg for pointing out these 2 issues.
I've backported into v3.4 branch. Will include them in next RC build.

Thanks Fede and Josep for helping verify this RC build.

Luke

On Tue, May 23, 2023 at 7:13 PM Mickael Maison 
wrote:

> Hi Luke,
>
> I've had to update the LICENSE-binary file when running the 3.5.0
> release and I noticed the file also needs updates in 3.4. If I run the
> validation logic described in
> https://issues.apache.org/jira/browse/KAFKA-12622, I get the
> following:
> audience-annotations-0.13.0 is missing in license file
> jackson-annotations-2.13.5 is missing in license file
> jackson-core-2.13.5 is missing in license file
> jackson-databind-2.13.5 is missing in license file
> jackson-dataformat-csv-2.13.5 is missing in license file
> jackson-datatype-jdk8-2.13.5 is missing in license file
> jackson-jaxrs-base-2.13.5 is missing in license file
> jackson-jaxrs-json-provider-2.13.5 is missing in license file
> jackson-module-jaxb-annotations-2.13.5 is missing in license file
> jackson-module-scala_2.13-2.13.5 is missing in license file
> javassist-3.29.2-GA is missing in license file
> javax.activation-api-1.2.0 is missing in license file
> jaxb-api-2.3.1 is missing in license file
> jersey-client-2.39.1 is missing in license file
> jersey-common-2.39.1 is missing in license file
> jersey-container-servlet-2.39.1 is missing in license file
> jersey-container-servlet-core-2.39.1 is missing in license file
> jersey-hk2-2.39.1 is missing in license file
> jersey-server-2.39.1 is missing in license file
> jetty-client-9.4.51.v20230217 is missing in license file
> jetty-continuation-9.4.51.v20230217 is missing in license file
> jetty-http-9.4.51.v20230217 is missing in license file
> jetty-io-9.4.51.v20230217 is missing in license file
> jetty-security-9.4.51.v20230217 is missing in license file
> jetty-server-9.4.51.v20230217 is missing in license file
> jetty-servlet-9.4.51.v20230217 is missing in license file
> jetty-servlets-9.4.51.v20230217 is missing in license file
> jetty-util-9.4.51.v20230217 is missing in license file
> jetty-util-ajax-9.4.51.v20230217 is missing in license file
> jline-3.22.0 is missing in license file
> netty-buffer-4.1.92.Final is missing in license file
> netty-codec-4.1.92.Final is missing in license file
> netty-common-4.1.92.Final is missing in license file
> netty-handler-4.1.92.Final is missing in license file
> netty-resolver-4.1.92.Final is missing in license file
> netty-transport-4.1.92.Final is missing in license file
> netty-transport-classes-epoll-4.1.92.Final is missing in license file
> netty-transport-native-epoll-4.1.92.Final is missing in license file
> netty-transport-native-unix-common-4.1.92.Final is missing in license file
> reload4j-1.2.25 is missing in license file
> scala-collection-compat_2.13-2.10.0 is missing in license file
>
> I think this needs to be addressed before releasing 3.4.1.
>
> Thanks,
> Mickael
>
> On Mon, May 22, 2023 at 11:08 PM Greg Harris
>  wrote:
> >
> > Also I'll point it out here so that everyone is aware, but I don't
> > think it necessarily warrants a new RC on it's own:
> >
> > This patch: https://github.com/apache/kafka/pull/13005 is present on
> > trunk/3.5.0, but is not present in 3.4.0/3.4.1-RC1.
> > It is not a regression strictly, since it was already a problem in 3.4.0.
> >
> > Impact: when any MM2 connector starts, it attempts to create a topic.
> > if that topic exists (as it will on every connector restart), it
> > prints an ERROR log with a TopicExistsException.
> > This is annoying and causes ERROR log noise, but does not otherwise
> > affect the operation or correctness of the connectors.
> >
> > If something else forces a new RC, perhaps consider rolling this into
> > the next RC.
> >
> > Thanks,
> > Greg
> >
> > On Mon, May 22, 2023 at 1:47 PM Greg Harris 
> wrote:
> > >
> > > Hi Luke,
> > >
> > > I performed a test upgrade of MM2 from 3.4.0 to 3.4.1-RC1, and
> > > verified that the new offset translation logic worked as intended.
> > >
> > > Steps I took to verify:
> > > - Downloaded 3.4.0_2.13 from the Kafka website
> > > - Formatted and started two 3.4.0 1-node clusters, and configured MM2
> > > to mirror data and consumer group offsets between those clusters.
> > > - I used the console producer to produce data in batches to the source
> > > cluster, and verified that similar amounts of data were present in the
> > > destination.
> > > - Verified that pre-upgrade, offset translation occurs smoothly.
> > > - Stopped MM2 3.4.0 and started 3.4.1-RC1 with the same configuration.
> > > - Verified that post-upgrade, offset translation occurs in steps and
> > > can be performed prior to the latest offset sync.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Mon, May 22, 2023 at 11:37 AM Josep Prat
>  wrote:
> > > >
> > > > Hi Luke,
> > > >
> > > > I can

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.5 #1

[Apache Jenkins Server]

See 

Re: [VOTE] 3.4.1 RC1

[Mickael Maison]

Hi Luke,

I've had to update the LICENSE-binary file when running the 3.5.0
release and I noticed the file also needs updates in 3.4. If I run the
validation logic described in
https://issues.apache.org/jira/browse/KAFKA-12622, I get the
following:
audience-annotations-0.13.0 is missing in license file
jackson-annotations-2.13.5 is missing in license file
jackson-core-2.13.5 is missing in license file
jackson-databind-2.13.5 is missing in license file
jackson-dataformat-csv-2.13.5 is missing in license file
jackson-datatype-jdk8-2.13.5 is missing in license file
jackson-jaxrs-base-2.13.5 is missing in license file
jackson-jaxrs-json-provider-2.13.5 is missing in license file
jackson-module-jaxb-annotations-2.13.5 is missing in license file
jackson-module-scala_2.13-2.13.5 is missing in license file
javassist-3.29.2-GA is missing in license file
javax.activation-api-1.2.0 is missing in license file
jaxb-api-2.3.1 is missing in license file
jersey-client-2.39.1 is missing in license file
jersey-common-2.39.1 is missing in license file
jersey-container-servlet-2.39.1 is missing in license file
jersey-container-servlet-core-2.39.1 is missing in license file
jersey-hk2-2.39.1 is missing in license file
jersey-server-2.39.1 is missing in license file
jetty-client-9.4.51.v20230217 is missing in license file
jetty-continuation-9.4.51.v20230217 is missing in license file
jetty-http-9.4.51.v20230217 is missing in license file
jetty-io-9.4.51.v20230217 is missing in license file
jetty-security-9.4.51.v20230217 is missing in license file
jetty-server-9.4.51.v20230217 is missing in license file
jetty-servlet-9.4.51.v20230217 is missing in license file
jetty-servlets-9.4.51.v20230217 is missing in license file
jetty-util-9.4.51.v20230217 is missing in license file
jetty-util-ajax-9.4.51.v20230217 is missing in license file
jline-3.22.0 is missing in license file
netty-buffer-4.1.92.Final is missing in license file
netty-codec-4.1.92.Final is missing in license file
netty-common-4.1.92.Final is missing in license file
netty-handler-4.1.92.Final is missing in license file
netty-resolver-4.1.92.Final is missing in license file
netty-transport-4.1.92.Final is missing in license file
netty-transport-classes-epoll-4.1.92.Final is missing in license file
netty-transport-native-epoll-4.1.92.Final is missing in license file
netty-transport-native-unix-common-4.1.92.Final is missing in license file
reload4j-1.2.25 is missing in license file
scala-collection-compat_2.13-2.10.0 is missing in license file

I think this needs to be addressed before releasing 3.4.1.

Thanks,
Mickael

On Mon, May 22, 2023 at 11:08 PM Greg Harris
 wrote:
>
> Also I'll point it out here so that everyone is aware, but I don't
> think it necessarily warrants a new RC on it's own:
>
> This patch: https://github.com/apache/kafka/pull/13005 is present on
> trunk/3.5.0, but is not present in 3.4.0/3.4.1-RC1.
> It is not a regression strictly, since it was already a problem in 3.4.0.
>
> Impact: when any MM2 connector starts, it attempts to create a topic.
> if that topic exists (as it will on every connector restart), it
> prints an ERROR log with a TopicExistsException.
> This is annoying and causes ERROR log noise, but does not otherwise
> affect the operation or correctness of the connectors.
>
> If something else forces a new RC, perhaps consider rolling this into
> the next RC.
>
> Thanks,
> Greg
>
> On Mon, May 22, 2023 at 1:47 PM Greg Harris  wrote:
> >
> > Hi Luke,
> >
> > I performed a test upgrade of MM2 from 3.4.0 to 3.4.1-RC1, and
> > verified that the new offset translation logic worked as intended.
> >
> > Steps I took to verify:
> > - Downloaded 3.4.0_2.13 from the Kafka website
> > - Formatted and started two 3.4.0 1-node clusters, and configured MM2
> > to mirror data and consumer group offsets between those clusters.
> > - I used the console producer to produce data in batches to the source
> > cluster, and verified that similar amounts of data were present in the
> > destination.
> > - Verified that pre-upgrade, offset translation occurs smoothly.
> > - Stopped MM2 3.4.0 and started 3.4.1-RC1 with the same configuration.
> > - Verified that post-upgrade, offset translation occurs in steps and
> > can be performed prior to the latest offset sync.
> >
> > Thanks!
> > Greg
> >
> > On Mon, May 22, 2023 at 11:37 AM Josep Prat  
> > wrote:
> > >
> > > Hi Luke,
> > >
> > > I can confirm the
> > > *org.apache.kafka.common.utils.UtilsTest.testToLogDateTimeFormat*
> > > consistent failure when not running on a system set on UTC-0 timezone.
> > >
> > > I verified:
> > > - Built from source with Java 11 and Scala 2.13
> > > - Checked signatures and hashes
> > > - Check Javadoc and checked links to JDK javadoc are functional
> > > - Run the unit tests (works on UTC-0 environments)
> > > - Run integration tests
> > > - Run the quickstart in KRaft and Zookeeper mode
> > >
> > > Best,
> > >
> > > On Mon, May 22, 2023 at 7:04 PM Chris Egerton 
> > > wrote:
>

[jira] [Resolved] (KAFKA-7497) Kafka Streams should support self-join on streams

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-7497.

Resolution: Fixed

> Kafka Streams should support self-join on streams
> -
>
> Key: KAFKA-7497
> URL: https://issues.apache.org/jira/browse/KAFKA-7497
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Robin Moffatt
>Priority: Major
>  Labels: needs-kip
>
> There are valid reasons to want to join a stream to itself, but Kafka Streams 
> does not currently support this ({{Invalid topology: Topic foo has already 
> been registered by another source.}}).  To perform the join requires creating 
> a second stream as a clone of the first, and then doing a join between the 
> two. This is a clunky workaround and results in unnecessary duplication of 
> data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14173) TopologyTestDriver does not use mock wall clock time when sending test records

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14173.
-
Resolution: Not A Problem

> TopologyTestDriver does not use mock wall clock time when sending test records
> --
>
> Key: KAFKA-14173
> URL: https://issues.apache.org/jira/browse/KAFKA-14173
> Project: Kafka
>  Issue Type: Bug
>  Components: streams-test-utils
>Affects Versions: 2.3.1
>Reporter: Guido Josquin
>Priority: Minor
>
> I am trying to test a stream-stream join with `TopologyTestDriver`. My goal 
> is to confirm that my topology performs the following left join correctly.
> {code:java}
> bills
>   .leftJoin(payments)(
> {
>   case (billValue, null) => billValue
>   case (billValue, paymentValue) => (billValue.toInt - 
> paymentValue.toInt).toString
> },
> JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(100))
>   )
>   .to("debt")
> {code}
>  
> In other words, if we see a `bill` and a `payment` within 100ms, the payment 
> should be subtracted from the bill. If we do not see a payment, the debt is 
> simply the bill.
> Here is the test code.
> {code:java}
> val simpleLeftJoinTopology = new SimpleLeftJoinTopology
> val driver = new TopologyTestDriver(simpleLeftJoinTopology.topology)
> val serde = Serdes.stringSerde
> val bills = driver.createInputTopic("bills", serde.serializer, 
> serde.serializer)
> val payments = driver.createInputTopic("payments", serde.serializer, 
> serde.serializer)
> val debt = driver.createOutputTopic("debt", serde.deserializer, 
> serde.deserializer)
> bills.pipeInput("fred", "100")
> bills.pipeInput("george", "20")
> payments.pipeInput("fred", "95")
> // When in doubt, sleep twice
> driver.advanceWallClockTime(Duration.ofMillis(500))
> Thread.sleep(500)
> // Send a new record to cause the previous window to be closed
> payments.pipeInput("percy", "0")
> val keyValues = debt.readKeyValuesToList()
> keyValues should contain theSameElementsAs Seq(
>   // This record is present
>   new KeyValue[String, String]("fred", "5"),
>   // This record is missing
>   new KeyValue[String, String]("george", "20")
> )
> {code}
> Full code available at [https://github.com/Oduig/kstreams-left-join-example]
> Is seems that advancing the wall clock time, sleeping, or sending an extra 
> record, never triggers the join condition when data only arrives on the left 
> side. It is possible to circumvent this by passing an explicit event time 
> with each test record. (See 
> https://stackoverflow.com/questions/73443812/using-kafka-streams-topologytestdriver-how-to-test-left-join-between-two-strea/73540161#73540161)
>  
> However, the behavior deviates from a real Kafka broker. With a real broker, 
> if we do not send an event, it uses the wall clock time of the broker 
> instead. The behavior under test should be the same: 
> `driver.advanceWallClockTime` should provide the default time to be used for 
> `TestTopic.pipeInput`, when no other time is specified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.5.0 RC0

[Josep Prat]

Hi Mickael,
I just wanted to point out that I think the documentation you recently
merged on Kafka site regarding the 3.5.0 version has a problem when it
states the version number and the sub-menu that links to previous versions.
Left a comment here:
https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939

Best,

On Tue, May 23, 2023 at 9:29 AM Josep Prat  wrote:

> Hi Mickael,
>
> I can +1 this candidate. I verified the following:
> - Built from source with Java 17 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
>
> Best,
>
> On Mon, May 22, 2023 at 5:30 PM Mickael Maison 
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
>> major features include:
>> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
>> 2.0 clusters
>> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
>> - KIP-887: Add ConfigProvider to make use of environment variables
>> - KIP-889: Versioned State Stores
>> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
>> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
>> Kafka Brokers
>>
>> Release notes for the 3.5.0 release:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, May 26, 5pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>>
>> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
>> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>>
>> The PR adding the 35 documentation is not merged yet
>> (https://github.com/apache/kafka-site/pull/513)
>> * Documentation:
>> https://kafka.apache.org/35/documentation.html
>> * Protocol:
>> https://kafka.apache.org/35/protocol.html
>>
>> * Successful Jenkins builds for the 3.5 branch:
>> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
>> working with INFRA to sort it out:
>> https://issues.apache.org/jira/browse/INFRA-24577
>> System tests: The build is still running, I'll send an update once I
>> have the results
>>
>> Thanks,
>> Mickael
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Resolved] (KAFKA-10575) StateRestoreListener#onRestoreEnd should always be triggered

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-10575.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> StateRestoreListener#onRestoreEnd should always be triggered
> 
>
> Key: KAFKA-10575
> URL: https://issues.apache.org/jira/browse/KAFKA-10575
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 3.5.0
>
>
> Today we only trigger `StateRestoreListener#onRestoreEnd` when we complete 
> the restoration of an active task and transit it to the running state. 
> However the restoration can also be stopped when the restoring task gets 
> closed (because it gets migrated to another client, for example). We should 
> also trigger the callback indicating its progress when the restoration 
> stopped in any scenarios.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Christo Lolov]

Hello!

This proposal will address problems with configuration dependencies which I
run into very frequently, so I am fully supporting the development of this
feature!

Best,
Christo

On Mon, 22 May 2023 at 17:18, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi everyone,
>
> I'd like to start a discussion for KIP-935 <
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations
> >
> which proposes extend AlterConfigPolicy with existing configuration to
> enable more complex policies.
>
> There have been related KIPs in the past that haven't been accepted and
> seem retired/abandoned as outlined in the motivation.
> The scope of this KIP intends to be more specific to get most of the
> benefits from previous discussions; and if previous KIPs are resurrected,
> should still be possible to do it if this one is adopted.
>
> Looking forward to your feedback!
>
> Thanks,
> Jorge.
>

Re: [DISCUSS] KIP-934: Add DeleteTopicPolicy

[Christo Lolov]

Heya Jorge,

Thank you for the KIP!

This feature sounds great to me since I have encountered problems with
this, so I am supporting it. Do you have any idea why the previous KIPs
were abandoned - I went through the email conversations and pull requests,
but I didn't find a good reason?

Best,
Christo

On Mon, 22 May 2023 at 17:19, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi everyone,
>
> I'd like to start a discussion for KIP-934 <
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-934%3A+Add+DeleteTopicPolicy
> >
> which proposes adding a new policy for when deleting topics.
>
> There have been related KIPs in the past that haven't been accepted and
> seem retired/abandoned as outlined in the motivation.
> The scope of this KIP intends to be more specific to get most of the
> benefits from previous discussions; and if previous KIPs are resurrected,
> should still be possible to do it if this one is adopted.
>
> Looking forward to your feedback!
>
> Thanks,
> Jorge.
>

[GitHub] [kafka-site] jlprat commented on a diff in pull request #513: MINOR: Add Kafka 3.5 documentation and javadoc

[via GitHub]

jlprat commented on code in PR #513:
URL: https://github.com/apache/kafka-site/pull/513#discussion_r1201740082


##
35/documentation.html:
##
@@ -0,0 +1,111 @@
+
+
+
+
+
+
+
+
+
+ 
+  
+<
+  
+  
+
+  
+  
+
+
+Documentation
+Kafka 3.4 Documentation

Review Comment:
   Shouldn't this read 3.5?
   



##
35/documentation.html:
##
@@ -0,0 +1,111 @@
+
+
+
+
+
+
+
+
+
+ 
+  
+<
+  
+  
+
+  
+  
+
+
+Documentation
+Kafka 3.4 Documentation
+Prior releases: 0.7.x, 
+0.8.0, 
+0.8.1.X, 
+0.8.2.X, 
+0.9.0.X, 
+0.10.0.X, 
+0.10.1.X, 
+0.10.2.X, 
+0.11.0.X, 
+1.0.X, 
+1.1.X, 
+2.0.X, 
+2.1.X, 
+2.2.X, 
+2.3.X, 
+2.4.X, 
+2.5.X, 
+2.6.X, 
+2.7.X,
+2.8.X,
+3.0.X.
+3.1.X.
+3.2.X.
+3.3.X.

Review Comment:
   And here a link to 3.4 release



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison merged pull request #513: MINOR: Add Kafka 3.5 documentation and javadoc

[via GitHub]

mimaison merged PR #513:
URL: https://github.com/apache/kafka-site/pull/513


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 3.5.0 RC0

[Josep Prat]

Hi Mickael,

I can +1 this candidate. I verified the following:
- Built from source with Java 17 and Scala 2.13
- Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the unit tests
- Run integration tests
- Run the quickstart in KRaft and Zookeeper mode

Best,

On Mon, May 22, 2023 at 5:30 PM Mickael Maison  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.5.0. Some of the
> major features include:
> - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> 2.0 clusters
> - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> - KIP-887: Add ConfigProvider to make use of environment variables
> - KIP-889: Versioned State Stores
> - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> Kafka Brokers
>
> Release notes for the 3.5.0 release:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, May 26, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/
>
> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> https://github.com/apache/kafka/releases/tag/3.5.0-rc0
>
> The PR adding the 35 documentation is not merged yet
> (https://github.com/apache/kafka-site/pull/513)
> * Documentation:
> https://kafka.apache.org/35/documentation.html
> * Protocol:
> https://kafka.apache.org/35/protocol.html
>
> * Successful Jenkins builds for the 3.5 branch:
> Unit/integration tests: Jenkins is not detecting the 3.5 branch,
> working with INFRA to sort it out:
> https://issues.apache.org/jira/browse/INFRA-24577
> System tests: The build is still running, I'll send an update once I
> have the results
>
> Thanks,
> Mickael
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B