Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Lucas Brutschy]

Hi Walker,

thanks for your responses. That makes sense. I guess there is always
the option to make the implementation more configurable later on, if
users request it. Also thanks for the clarifications. From my side,
the KIP is good to go.

Cheers,
Lucas

On Wed, May 24, 2023 at 11:54 PM Victoria Xia
 wrote:
>
> Thanks for the updates, Walker! Looks great, though I do have a couple
> questions about the latest updates:
>
>1. The new example says that without stream-side buffering, "ex" and
>"fy" are possible join results. How could those join results happen? The
>example versioned table suggests that table record "x" has timestamp 2, and
>table record "y" has timestamp 3. If stream record "e" has timestamp 1,
>then it can never be joined against record "x", and similarly for stream
>record "f" with timestamp 2 being joined against "y".
>2. I see in your replies above that "If table is not materialized it
>will materialize it as versioned" but I don't see this called out in the
>KIP -- seems worth calling out. Also, what will the history retention for
>the versioned table be? Will it be the same as the join grace period, or
>will it be greater?
>
>
> And some additional thoughts:
>
> Sounds like there are a few things users should watch out for when enabling
> the stream-side buffer:
>
>- Records will get "stuck" if there are no newer records to advance
>stream time.
>- If there are large gaps between the timestamps of stream-side records,
>then it's possible that versioned store history retention will have expired
>by the time a record is evicted from the join buffer, leading to a join
>"miss." For example, if the join grace period and table history retention
>are both 10, and records come in the order:
>
>table side t0 with ts=0
>stream side s1 with ts=1 <-- enters buffer
>table side t10 with ts=10
>table side t20 with ts=20
>stream side s21 with ts=21 <-- evicts record s1 from buffer, but
>versioned store no longer contains data for ts=1 due to history retention
>having elapsed
>
>This will result in the join result (s1, null) even though it should've
>been (s1, t0), due to t0 having been expired from the versioned store
>already.
>- Out-of-order records from the stream-side will be reordered, and late
>records will be dropped.
>
> I don't think any of these are reasons to not go forward with this KIP, but
> it'd be good to call them out in the eventual documentation to decrease the
> chance users get tripped up.
>
> > We could maybe do an improvement later to advance stream time from table
> side as well, but that might be debatable as we might get more late records.
>
> Yes, the likelihood of late records increases but also the likelihood of
> "join misses" due to versioned store history retention having elapsed
> decreases, which feels important for certain use cases. Either way, agreed
> that it can be a discussion for the future as incorporating this would
> substantially complicate the implementation.
>
> Also a couple nits:
>
>- The KIP currently says "We recently added versioned tables which allow
>the table side of the a join [...] but it is not taken advantage of in
>joins," but this doesn't seem true? If the table of a stream-table join is
>versioned, then the DSL's stream-table join processor will automatically
>perform timestamped lookups into the table, in order to take advantage of
>the new timestamp-aware store to provide better join semantics.
>- The KIP mentions "grace period" for versioned stores in a number of
>places but I think you actually mean "history retention"? The two happen to
>be the same today (it is not an option for users to configure the two
>separately) but this need not be true in the future. "History retention"
>governs how far back in time reads may occur, which is the relevant
>parameter for performing lookups as part of the stream-table join. "Grace
>period" in the context of versioned stores refers to how far back in time
>out-of-order writes may occur, which probably isn't directly relevant for
>introducing a stream-side buffer, though it's also possible I've overlooked
>something. (As a bonus, switching from "table grace period" in the KIP to
>"table history retention" also helps to clarify/distinguish that it's a
>different parameter from the "join grace period," which I could see being
>confusing to readers. :) )
>
>
> Cheers,
> Victoria
>
> On Thu, May 18, 2023 at 1:43 PM Walker Carlson
>  wrote:
>
> > Hey all,
> >
> > Thanks for the comments, they gave me a lot to think about. I'll try to
> > address them all inorder. I have made some updates to the kip related to
> > them, but I mention where below.
> >
> > Lucas
> >
> > Good idea about the example. I added a simple one.
> >
> > 1) I have thought about including options for the underlying buffer
> > configuration.

[GitHub] [kafka-site] mimaison opened a new pull request, #516: MINOR: Add blog for 3.5.0 release

[via GitHub]

mimaison opened a new pull request, #516:
URL: https://github.com/apache/kafka-site/pull/516

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-921 OpenJDK CRaC support

[Radim Vansa]

Hi Divij,

I have prototyped this using Quarkus Superheroes [1], a demo application 
consisting of several microservices that communicate with each other 
using both HTTP and Kafka. I wanted to add the ability to transparently 
checkpoint and restore this application - while the regular startup 
takes seconds, the restore could bring this application online in the 
order of tens of milliseconds.


I agree that the change will not help Kafka itself to get any faster; it 
will enable CRaC for the whole application that, amongst other 
technologies, uses Kafka. You're saying that the clients are not 
supposed to be re-created quickly, I hope that a use case where the app 
is scaled down if it's idle e.g. 60 seconds and then needs to be started 
on a request (to serve it ASAP) would make sense to you. It's really not 
about Kafka per-se - it's about the needs of those who consume it. Of 
course, I'll be glad for any comments pointing out difficulties e.g. if 
the producer is replicated.


An alternative, and less transparent approach, would handle this in the 
integration layer. However from my experience this can be problematic if 
the integration layer provides Kafka API directly, losing control over 
the instance - it's not possible to simply shutdown the client and 
reopen the instance, and some sort of proxy would be needed that 
prevents access to this closed instance. And besides complexity, proxy 
means degraded performance.


Another motivation to push changes as far down the dependency tree is 
the fan-out of these changes: we don't want to target Quarkus 
specifically, but other frameworks (Spring Boot, ...) and stand-alone 
applications as well. By keeping it low level we can concentrate the 
maintenance efforts to once place.


Thank you for spending time reviewing the proposal and let me know if I 
can clarify this further.


Radim


[1] https://quarkus.io/quarkus-workshops/super-heroes/

On 24. 05. 23 17:13, Divij Vaidya wrote:

Caution: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


Hey Radim

After reading the KIP, I am still not sure about the motivation for this
change. The bottleneck in starting a producer on Kafka is setup of the
network connection with the broker (since it performs SSL + AuthN). From
what I understand, checkpoint and restore is not going to help with that.
Also, Kafka clients are supposed to be long running clients which aren't
supposed to be destroyed and re-created quickly in a short span. I fail to
understand the benefit of using CRaC for the producer. Perhaps I am missing
something very obvious here, but please help me understand more.

I was wondering if you are aware of any Kafka user use cases which could be
positively impacted by this change? Adding these use cases to the
motivation will greatly help in convincing the community about the impact
of this change.

A use case that I can think of is to aid debugging of a faulty cluster. If
we get the ability to snapshot a broker when it is mis-behaving, then we
could re-use the snapshot to re-create the exact production setup in a
local environment where we can test things. But that would be a larger
change since capturing the state of a broker is much more than in-memory
Java objects. Perhaps, you can try approaching the usefulness of this new
technology from this lens?

--
Divij Vaidya



On Thu, May 11, 2023 at 2:33 PM Radim Vansa  wrote:


Thank you, Divij. I'll give it more time and remind the list in that
timeframe, then.

Radim

On 11. 05. 23 13:47, Divij Vaidya wrote:

Caution: This email originated from outside of the organization. Do not

click links or open attachments unless you recognize the sender and know
the content is safe.


Hey Radim

One of the reasons for the slowdown is preparation of upcoming releases
(the community is currently in code freeze/resolve release blockers mode)
and preparation for Kafka Summit next week. I would suggest giving

another

2-3 weeks for folks to chime in. I would personally visit this KIP in the
last week of May.

--
Divij Vaidya



On Thu, May 11, 2023 at 1:34 PM Radim Vansa 

wrote:

Hello all,

it seems that this KIP did not sparkle much interest, not sure if people
just don't care or whether there are any objections against the
proposal. What should be the next step, I don't think it has been
discussed enough to proceed with voting.

Cheers,

Radim

On 27. 04. 23 8:39, Radim Vansa wrote:

Caution: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe.


Thank you for those questions, as I've mentioned, my knowledge of Kafka
is quite limited so these are the things that need careful thinking!
Comments inline.

On 26. 04. 23 16:28, Mickael Maison wrote:

Hi Radim,

Thanks for the KIP! CRaC is an interesting project and it could be a
useful feature in Kafk

Re: [VOTE] 3.4.1 RC2

[Luke Chen]

Thanks all for helping verify the RC2.

But my team member found a bug that in KRaft mode, we'll log every commit
message in info level, with `handleSnapshot: xxx`.
That means, in KRaft mode, the active controller log will be flooded with
this message every 500ms (default no-op message commit interval).

I've opened a PR to fix it: https://github.com/apache/kafka/pull/13763.
After this PR merged, I'll create another RC.

Thank you.
Luke

On Thu, May 25, 2023 at 2:53 AM Josep Prat 
wrote:

> Hi Luke,
>
> Thanks for your patience during this release.
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
> - Checked that the diff in LICENSE-binary (between this and previous RC) is
> correct
> -- For each difference I checked the binary on the scala 2.13 package had
> the version specified in the file
>
> This gets a +1 from me.
>
>
> Best,
>
> On Wed, May 24, 2023 at 6:48 PM Federico Valeri 
> wrote:
>
> > Hi Luke, these are the things I checked:
> >
> > - Source signature, checksum and license file
> > - Build from source with Java 17 and Scala 2.13
> > - Unit and integration test suites
> > - Java app with staging Maven artifacts
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> >
> > On Wed, May 24, 2023 at 6:36 PM Chris Egerton 
> > wrote:
> > >
> > > Hi Luke,
> > >
> > > Thanks again for running this release! Echoing your hopes that this
> will
> > be
> > > the final 3.4.1 release candidate :)
> > >
> > > To verify, I:
> > > - Built from source with Java 11 with both:
> > > - - the 3.4.1-rc2 tag on GitHub
> > > - - the kafka-3.4.1-src.tgz artifact from
> > > https://home.apache.org/~showuon/kafka-3.4.1-rc2/
> > > - Checked signatures and checksums
> > > - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> > > https://home.apache.org/~showuon/kafka-3.4.1-rc2/ with Java 11 and
> > Scala 13
> > > in KRaft mode
> > > - Ran all unit tests
> > > - - No failures this time! 🙌
> > > - Ran all integration tests for Connect and MM2
> > >
> > > +1 (binding)
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, May 24, 2023 at 6:12 AM Luke Chen  wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the 3rd candidate for release of Apache Kafka 3.4.1.
> > > >
> > > > This is a bugfix release with several fixes since the release of
> > 3.4.0. A
> > > > few of the major issues include:
> > > > - core
> > > > KAFKA-14644 
> > Process
> > > > should stop after failure in raft IO thread
> > > > KAFKA-14946 
> KRaft
> > > > controller node shutting down while renouncing leadership
> > > > KAFKA-14887  ZK
> > session
> > > > timeout can cause broker to shutdown
> > > > - client
> > > > KAFKA-14639 
> Kafka
> > > > CooperativeStickyAssignor revokes/assigns partition in one rebalance
> > cycle
> > > > - connect
> > > > KAFKA-12558  MM2
> > may
> > > > not
> > > > sync partition offsets correctly
> > > > KAFKA-14666  MM2
> > should
> > > > translate consumer group offsets behind replication flow
> > > > - stream
> > > > KAFKA-14172  bug:
> > State
> > > > stores lose state when tasks are reassigned under EOS
> > > >
> > > >
> > > > Release notes for the 3.4.1 release:
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by May 31, 2023
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc2/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc2/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.4.1-rc2
> > > >
> > > > * Documentation: (will be updated after released)
> > > > https://kafka.apache.org/34/documentation.html
> > > >
> > > > * Protocol: (will be updated after released)
> > > > https://kafka.apache.org/34/protocol.html
> > > > The most recent build has had test failures. These all appear to be
> > due to
> > > > flakiness, but it would be nice if someone more familiar with the
> > failed
> > > > tests could confirm this. I may update this threa

[jira] [Created] (KAFKA-15020) integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor test is flaky

[Atul Sharma (Jira)]

Atul Sharma created KAFKA-15020:
---

 Summary: 
integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor
 test is flaky
 Key: KAFKA-15020
 URL: https://issues.apache.org/jira/browse/KAFKA-15020
 Project: Kafka
  Issue Type: Test
Reporter: Atul Sharma


Sometimes the test fails with the following log:

{code:java}

Gradle Test Run :core:integrationTest > Gradle Test Executor 175 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor() FAILED
org.opentest4j.AssertionFailedError: Consumed 0 records before timeout instead 
of the expected 2 records
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:135)
at kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1087)
at 
integration.kafka.server.FetchFromFollowerIntegrationTest.$anonfun$testRackAwareRangeAssignor$11(FetchFromFollowerIntegrationTest.scala:216)
at 
integration.kafka.server.FetchFromFollowerIntegrationTest.$anonfun$testRackAwareRangeAssignor$11$adapted(FetchFromFollowerIntegrationTest.scala:215)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at 
integration.kafka.server.FetchFromFollowerIntegrationTest.verifyAssignments$1(FetchFromFollowerIntegrationTest.scala:215)
 at 
integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor(FetchFromFollowerIntegrationTest.scala:244)

{code}





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1869

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 572471 lines...]
[2023-05-25T10:34:17.775Z] 
[2023-05-25T10:34:17.775Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduce(TestInfo) PASSED
[2023-05-25T10:34:17.775Z] 
[2023-05-25T10:34:17.775Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey(TestInfo) STARTED
[2023-05-25T10:34:20.703Z] 
[2023-05-25T10:34:20.703Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault() PASSED
[2023-05-25T10:34:20.703Z] 
[2023-05-25T10:34:20.703Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED
[2023-05-25T10:34:27.834Z] 
[2023-05-25T10:34:27.834Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED
[2023-05-25T10:34:27.834Z] 
[2023-05-25T10:34:27.834Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED
[2023-05-25T10:34:27.931Z] 
[2023-05-25T10:34:27.931Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey(TestInfo) PASSED
[2023-05-25T10:34:27.931Z] 
[2023-05-25T10:34:27.931Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed(TestInfo) STARTED
[2023-05-25T10:34:31.728Z] 
[2023-05-25T10:34:31.728Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 179 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
[2023-05-25T10:34:31.728Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:31.728Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2023-05-25T10:34:33.959Z] 
[2023-05-25T10:34:33.959Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed(TestInfo) PASSED
[2023-05-25T10:34:37.204Z] 
[2023-05-25T10:34:37.204Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-05-25T10:34:37.204Z] 
[2023-05-25T10:34:37.204Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-05-25T10:34:37.204Z] 
[2023-05-25T10:34:37.204Z] See 
https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings
[2023-05-25T10:34:37.204Z] 
[2023-05-25T10:34:37.204Z] BUILD SUCCESSFUL in 3h 10m 25s
[2023-05-25T10:34:37.204Z] 230 actionable tasks: 124 executed, 106 up-to-date
[2023-05-25T10:34:37.204Z] 
[2023-05-25T10:34:37.204Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2023-05-25-07-24-22.html
[2023-05-25T10:34:37.204Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2023-05-25T10:34:38.107Z] Recording test results
[2023-05-25T10:34:42.203Z] 
[2023-05-25T10:34:42.203Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED
[2023-05-25T10:34:50.721Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2023-05-25T10:34:50.722Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-05-25T10:34:56.310Z] 
[2023-05-25T10:34:56.310Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED
[2023-05-25T10:35:00.415Z] 
[2023-05-25T10:35:00.415Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Execu

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Christo Lolov]

Heya Igor!

I don't have any concerns or suggestions for improvements at this stage -
the overall approach makes sense to me!

I would be quite interested in attending a call, but as Divij has pointed
out the 29th of May is a public holiday, so I won't be able to make that
date. If there is another time I will do my best to appear.

Best,
Christo

On Tue, 23 May 2023 at 17:59, Igor Soarez  wrote:

> Hi everyone,
>
> Someone suggested at the recent Kafka Summit that it may be useful
> to have a video call to discuss remaining concerns.
>
> I'm proposing we have a video call Monday 29th May 16:30-17:00 UTC.
>
> If you'd like to join, please reply to the thread or to me directly so
> I can send you a link.
>
> Please also do let me know if you'd like to attend but the proposed
> time does not work for you.
>
> Thanks,
>
> --
> Igor
>
>

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Alexandre Dupriez]

Hi, Igor,

Thanks for the excellent, thorough and very comprehensive KIP.

Although not directly in scope of the KIP, but related to it, I would
have the following question about a potential future work on disk
degradation.

Today, what characterises as a disk failure in Kafka is an I/O
exception surfaced by the JDK libraries. There are other types of
(more or less) soft failures where a disk (or the system behind its
abstraction) remains available, but experiences degradation, typically
in the form of elevated I/O latency. Currently, Kafka is not made
aware of the “health” of a disk. It may be useful to let Kafka know
about the QoS of its disks so that it can take actions which could
improve availability, e.g. via leader movements.

The KIP builds upon the existing concepts of online and offline states
for log directories, and the propagation of a disk failure via the
broker heartbeat and registration relies on the offline(d) directories
list. I wonder if it could make sense to extend the definition of
state of a log directory beyond online/offline to be able to refer to
disk degradation. In which case, the new fields added to the broker
heartbeat and registration requests may be the place where this
alternative state can also be conveyed. Perhaps the changes to the
RPCs could be designed to accommodate this new type of semantic in the
future.

What do you think?

Thanks,
Alexandre

Le mer. 26 avr. 2023 à 14:05, Igor Soarez  a écrit :
>
> Thank you for another review Ziming, much appreciated!
>
> 1. and 2. You are correct, it would be a big and perhaps strange difference.
> Since our last exchange of emails, the proposal has changed and now it
> does follow your suggestion to bump metadata.version.
> The KIP mentions it under "Compatibility, Deprecation, and Migration Plan".
>
> 3. I tried to describe this under "Controller", under the heading
> "Handling replica assignments", but perhaps it could be improved.
> Let me know what you think.
>
> Best,
>
> --
> Igor
>

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Divij, Christo,

Thank you for pointing that out.

Let's aim instead for Monday 5th of June, at the same time – 16:30-17:00 UTC.

Please let me know if this doesn't work either.

Best,

--
Igor

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Christo Lolov]

Heya!

5th of June 16:30 - 17:00 UTC works for me.

Best,
Christo

On Thu, 25 May 2023 at 15:14, Igor Soarez  wrote:

> Hi Divij, Christo,
>
> Thank you for pointing that out.
>
> Let's aim instead for Monday 5th of June, at the same time – 16:30-17:00
> UTC.
>
> Please let me know if this doesn't work either.
>
> Best,
>
> --
> Igor
>
>

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Matthias J. Sax]

Walker,

thanks for the updates. The KIP itself reads fine (of course Victoria 
made good comments about some phrases), but there is a couple of things 
from your latest reply I don't understand, and that I still think need 
some more discussions.


Lukas, asked about in-memory option and `WindowStoreSupplier` and you 
mention "semantic concerns". There should not be any semantic difference 
from the underlying buffer implementation, so I am not sure what you 
mean here (also the relationship to suppress() is unclear to me)? -- I 
am ok to not make it configurable for now. We can always do it via a 
follow up KIP, and keep interface changes limited for now.


Does it really make sense to allow a grace period if the table is 
non-versioned? You also say: "If table is not materialized it will 
materialize it as versioned." -- What history retention time would we 
pick for this case (also asked by Victoria)? Or should we rather not 
support this and force the user to materialize the table explicitly, and 
thus explicitly picking a history retention time? It's tradeoff between 
usability and guiding uses that there will be a significant impact on 
disk usage. There is also compatibility concerns: If the table is not 
explicitly materialized in the old program, we would already need to 
materialize it also in the old program (of course, we would use a 
non-versioned store so far). Thus, if somebody adds a grace period, we 
cannot just switch the store type, as it would be a breaking change, 
potentially required an application re-set, or following the upgrade 
path for versioned state stores, and also changing the program to 
explicitly materialize using a versioned store. Also note, that we might 
not materialize the actual join table, but only an upstream table, and 
use `ValueGetter` to access the upstream data.


To this end, as you already mentioned, history retention of the table 
should be at least grace period. You proposed to include this in a 
follow up KIP, but I am wondering if it's a fundamental requirement and 
thus we should put a check in place right away and reject an invalid 
configuration? (It always easier to lift restriction than to introduce 
them later.) This would also imply that a non-versioned table cannot be 
supported, because it does not have a history retention that is larger 
than grace period, and maybe also answer the requirement about 
materialization: as we already always materialize something on the 
tablet side as non-versioned store right now, it seems difficult to 
migrate the store to a versioned store. Ie, it might be ok to push the 
burden onto the user and say: if you start using grace period, you also 
need to manually switch from non-versioned to versioned KTables. Doing 
stuff automatically under the hood if very complex for this case, we if 
we push the burden onto the user, it might be ok to not complicate this 
KIP significantly.


To summarize the last two paragraphs, I would propose to:
 - don't support non-versioned KTables
 - if grace period is added, users need to explicitly materialize the 
table as version (either directly, or upstream. Upstream only works if 
downstream tables "inherit" versioned semantics -- cf KIP-914)
 - the table's history retention time must be larger than the grace 
period (should be easy to check at runtime, when we build the topology)
 - because switching from non-versioned to version stores is not 
backward compatibly (cf KIP-914), users need to take care of this 
themselves, and this also implies that adding grace period is not a 
backward compatible change (even only if via indirect means)


About dropping late records: wondering if we should never drop a 
stream-side record for a left-join, even if it's late? In general, one 
thing I observed over the years is, that it's easier to keep stuff and 
let users filter explicitly downstream (or make it configurable), 
instead of dropping pro-actively, because users have no good way to 
resurrect record that got already dropped.


For ordering, sounds reasonable to me only start with one 
implementation, and maybe make it configurable as a follow up. However, 
I am wondering if starting with offset order might be the better option 
as it seems to align more with what we do so far? So instead of storing 
record ordered by timestamp, we can just store them ordered by offset, 
and still "poll" from the buffer based on the head records timestamp. Or 
would this complicate the implementation significantly?


I also think it's ok to not "sync" stream-time between the table and the 
stream in this KIP, but we should consider doing this as a follow up 
change (not sure if we would need a KIP or not for a change this this).


About increasing/decreasing grace period: what you describe make sense 
to me. If decreased, the next record would just trigger emitting a lot 
of records, and for increase, the buffer would just need to "fill up" 
again. For reprocessing getting a different result with

[jira] [Created] (KAFKA-15021) KRaft controller increases leader epoch when shrinking ISR

[Jira]

José Armando García Sancio created KAFKA-15021:
--

 Summary: KRaft controller increases leader epoch when shrinking ISR
 Key: KAFKA-15021
 URL: https://issues.apache.org/jira/browse/KAFKA-15021
 Project: Kafka
  Issue Type: Bug
  Components: controller, kraft
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio


When the KRaft controller shrinks the ISR it also forces the leader epoch to 
increase. This is unnecessary and cases all of the follower replica fetches to 
get invalidated.

Here is an example trace of this behavior after replica 8 was shutdown:
{code:java}
kafka-dump-log --cluster-metadata-decoder --files 
__cluster_metadata-0/38589501.log | grep Pd7wMb4lSkKI00--SrWNXw
...
| offset: 38655592 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1],"leader":1}}
| offset: 38655593 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4],"leader":4}}
| offset: 38655594 CreateTime: 1683849857362 keySize: -1 valueSize: 41 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1],"leader":0}}
| offset: 38656159 CreateTime: 1683849974945 keySize: -1 valueSize: 39 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1,8]}}
| offset: 38656256 CreateTime: 1683849994297 keySize: -1 valueSize: 39 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4,8]}}
| offset: 38656299 CreateTime: 1683849997139 keySize: -1 valueSize: 39 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1,8]}}
| offset: 38657003 CreateTime: 1683850157379 keySize: -1 valueSize: 30 
sequence: -1 headerKeys: [] payload: 
{"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","leader":8}}
 {code}
Also, notice how the leader epoch was not increased when the ISR was expanded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1870

[Apache Jenkins Server]

See 

[jira] [Reopened] (KAFKA-8713) [Connect] JsonConverter NULL Values are replaced by default values even in NULLABLE fields

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris reopened KAFKA-8713:


> [Connect] JsonConverter NULL Values are replaced by default values even in 
> NULLABLE fields
> --
>
> Key: KAFKA-8713
> URL: https://issues.apache.org/jira/browse/KAFKA-8713
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.3.0, 2.2.1
>Reporter: Cheng Pan
>Assignee: Mickael Maison
>Priority: Major
>  Labels: needs-kip
> Fix For: 3.5.0
>
>
> Class JsonConverter line: 582
> {code:java}
> private static JsonNode convertToJson(Schema schema, Object logicalValue) 
> {
> if (logicalValue == null) {
> if (schema == null) // Any schema is valid and we don't have a 
> default, so treat this as an optional schema
> return null;
> if (schema.defaultValue() != null)
> return convertToJson(schema, schema.defaultValue());
> if (schema.isOptional())
> return JsonNodeFactory.instance.nullNode();
> throw new DataException("Conversion error: null value for field 
> that is required and has no default value");
> }
> 
> }
> {code}
> h1.Expect:
> Value `null` is valid for an optional filed, even though the filed has a 
> default value.
>  Only when field is required, the converter return default value fallback 
> when value is `null`.
> h1.Actual:
> Always return default value if `null` was given.
> h1. Example:
> I'm not sure if the current behavior is the exactly expected, but at least on 
> MySQL, a table  define as 
> {code:sql}
> create table t1 {
>name varchar(40) not null,
>create_time datetime default '1999-01-01 11:11:11' null,
>update_time datetime default '1999-01-01 11:11:11' null
> }
> {code}
> Just insert a record:
> {code:sql}
> INSERT INTO `t1` (`name`,  `update_time`) VALUES ('kafka', null);
> {code}
> The result is:
> {code:json}
> {
> "name": "kafka",
> "create_time": "1999-01-01 11:11:11",
> "update_time": null
> }
> {code}
> But when I use debezium pull binlog and send the record to Kafka with 
> JsonConverter, the result changed to:
> {code:json}
> {
> "name": "kafka",
> "create_time": "1999-01-01 11:11:11",
> "update_time": "1999-01-01 11:11:11"
> }
> {code}
> For more details, see: https://issues.jboss.org/browse/DBZ-1064



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.5.0 RC0

[Greg Harris]

Hey all,

A contributor just pointed out a small but noticeable flaw in the
implementation of KIP-581
https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
which is planned for this release.
Impact: the feature works for root values in a record, but does not
work for any fields within structs. Fields within structs will
continue to have their previous, backwards-compatible behavior.
The contributor has submitted a bug-fix PR which reports the problem
and does not yet have a merge-able solution, but they are actively
responding and interested in having this fixed:
https://github.com/apache/kafka/pull/13748
The overall fix should be a one-liner + some unit tests. While this is
not a regression, it does make the feature largely useless, as the
majority of use-cases will be for struct fields.

Thanks!
Greg Harris

On Wed, May 24, 2023 at 7:05 PM Ismael Juma  wrote:
>
> I agree the migration should be functional - it wasn't obvious if the
> migration issues are edge cases or not. If they are edge cases, I think
> 3.5.1 would be fine given the preview status.
>
> I understand that a new RC is needed, but that doesn't mean we should let
> everything in. Each change carries some risk. And if we don't agree on the
> bar for the migration work, we may be having the same discussion next week.
> :)
>
> Ismael
>
> On Wed, May 24, 2023, 12:00 PM Josep Prat 
> wrote:
>
> > Hi there,
> > Is the plan described in KIP-833[1] still valid? In there it states that
> > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the path to
> > migrate to Kraft should be somewhat functional (in my opinion). If we don't
> > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion that
> > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> >
> > [1]:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > Best,
> >
> > On Wed, May 24, 2023 at 8:51 PM Ismael Juma  wrote:
> >
> > > Are all these blockers? For example, zk to kraft migration are is still
> > in
> > > preview - can we fix some of these in 3.5.1?
> > >
> > > Ismael
> > >
> > > On Wed, May 24, 2023, 10:22 AM Colin McCabe  wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for putting together this RC. Unfortunately, we've identified
> > > > several blocker issues in this release candidate.
> > > >
> > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > migration.
> > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > migration (dual-write)
> > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > (dual-write) when partitions are changed for topic
> > > > KAFKA-14996: The KRaft controller should properly handle overly large
> > > user
> > > > operations
> > > >
> > > > We are working on PRs for these issues and will get them in soon, we
> > > think!
> > > >
> > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > another
> > > > RC next week.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > Hi David,
> > > > >
> > > > > We're already quite a bit behind schedule. If you think these fixes
> > > > > are really important and can be ready in the next couple of days, I'm
> > > > > open to backport them and build another release candidate. Let me
> > know
> > > > > once you've investigated the severity of KAFKA-15010.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur
> > > > >  wrote:
> > > > >>
> > > > >> Mickael, we have some migration fixes on trunk, is it okay to
> > > > cherry-pick
> > > > >> these to 3.5?
> > > > >>
> > > > >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator
> > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration
> > > > >>
> > > > >> There is another issue KAFKA-15010 that I'm also investigating to
> > > > determine
> > > > >> the impact and likelihood of seeing it in practice. This one may be
> > a
> > > > >> significant migration blocker
> > > > >>
> > > > >> Cheers,
> > > > >> David
> > > > >>
> > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Christo,
> > > > >> >
> > > > >> > Yes this is expected. This happens when nested fields also accept
> > > > >> > optional tagged fields. The tables list all fields, so they may
> > > > >> > include _tagged_fields multiple times.
> > > > >> > Clearly the layout of this page could be improved, if you have
> > ideas
> > > > >> > how to describe the protocol in a better way, feel free to share
> > > them.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Mickael
> > > > >> >
> > > > >> > On Tue, May 23, 2023 at 3:

[jira] [Created] (KAFKA-15022) Support rack aware task assignment in Kafka streams

[Hao Li (Jira)]

Hao Li created KAFKA-15022:
--

 Summary: Support rack aware task assignment in Kafka streams 
 Key: KAFKA-15022
 URL: https://issues.apache.org/jira/browse/KAFKA-15022
 Project: Kafka
  Issue Type: Improvement
Reporter: Hao Li


For KIP-925: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15023) Get rack information for source topic partitions for a task

[Hao Li (Jira)]

Hao Li created KAFKA-15023:
--

 Summary: Get rack information for source topic partitions for a 
task
 Key: KAFKA-15023
 URL: https://issues.apache.org/jira/browse/KAFKA-15023
 Project: Kafka
  Issue Type: Sub-task
Reporter: Hao Li






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15024) Add cost function for task/client

[Hao Li (Jira)]

Hao Li created KAFKA-15024:
--

 Summary: Add cost function for task/client
 Key: KAFKA-15024
 URL: https://issues.apache.org/jira/browse/KAFKA-15024
 Project: Kafka
  Issue Type: Sub-task
Reporter: Hao Li






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15026) Implement min-cost flow balancing tasks for same subtopology

[Hao Li (Jira)]

Hao Li created KAFKA-15026:
--

 Summary: Implement min-cost flow balancing tasks for same 
subtopology
 Key: KAFKA-15026
 URL: https://issues.apache.org/jira/browse/KAFKA-15026
 Project: Kafka
  Issue Type: Sub-task
Reporter: Hao Li






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15025) Implement min-cost flow without balancing tasks for same subtopology

[Hao Li (Jira)]

Hao Li created KAFKA-15025:
--

 Summary: Implement min-cost flow without balancing tasks for same 
subtopology
 Key: KAFKA-15025
 URL: https://issues.apache.org/jira/browse/KAFKA-15025
 Project: Kafka
  Issue Type: Sub-task
Reporter: Hao Li






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15027) Implement rack aware assignment for standby tasks

[Hao Li (Jira)]

Hao Li created KAFKA-15027:
--

 Summary: Implement rack aware assignment for standby tasks
 Key: KAFKA-15027
 URL: https://issues.apache.org/jira/browse/KAFKA-15027
 Project: Kafka
  Issue Type: Sub-task
Reporter: Hao Li






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15028) AddPartitionsToTxnManager metrics

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-15028:
--

 Summary: AddPartitionsToTxnManager metrics
 Key: KAFKA-15028
 URL: https://issues.apache.org/jira/browse/KAFKA-15028
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan


KIP-890 added metrics for the AddPartitionsToTxnManager

VerificationTimeMs – number of milliseconds from adding partition info to the 
manager to the time the response is sent. This will include the round trip to 
the transaction coordinator if it is called. This will also account for 
verifications that fail before the coordinator is called.

VerificationFailureRate – rate of verifications that returned in failure either 
from the AddPartitionsToTxn response or through errors in the manager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Requesting permissions to contribute to Apache Kafka

[Beyene, Mehari]

Hello Team,

I would like to contribute to the KIPs and I am requesting permission as 
documented here: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-GettingStarted

Wiki ID: mehbey
Jira ID: mehbey

Thank you,
Mehari

[jira] [Created] (KAFKA-15029) Make ProducerIdBlock size configurable

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-15029:


 Summary: Make ProducerIdBlock size configurable
 Key: KAFKA-15029
 URL: https://issues.apache.org/jira/browse/KAFKA-15029
 Project: Kafka
  Issue Type: Task
Reporter: Jeff Kim


The producer id block size is currently fixed at 1000. Increasing the size 
allows the pid manager to process more init pid requests before allocating 
more. The current 1000 is quite small and given that clusters can have 
thousands of producers, we should make this configurable to adapt in the 
future. We shoudl also consider updating to a global block size of 5000



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #141

[Apache Jenkins Server]

See 

Re: Requesting permissions to contribute to Apache Kafka

[Luke Chen]

Hi Mehari,

Your accounts are all set.

Thanks.
Luke

On Fri, May 26, 2023 at 7:43 AM Beyene, Mehari 
wrote:

> Hello Team,
>
> I would like to contribute to the KIPs and I am requesting permission as
> documented here:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-GettingStarted
>
> Wiki ID: mehbey
> Jira ID: mehbey
>
> Thank you,
> Mehari
>