[jira] [Resolved] (KAFKA-16357) Kafka Client JAR manifest breaks javac linting

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-16357.
-
Resolution: Duplicate

> Kafka Client JAR manifest breaks javac linting
> --
>
> Key: KAFKA-16357
> URL: https://issues.apache.org/jira/browse/KAFKA-16357
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.7.0
> Environment: Linux, JDK 21 (Docker image eclipse-temurin:21-jdk-jammy)
>Reporter: Jacek Wojciechowski
>Priority: Critical
>
> I upgraded kafka-clients from 3.6.1 to 3.7.0 and discovered that my project 
> is not building anymore.
> The reason is that kafka-clients-3.7.0.jar contains the following entry in 
> its JAR manifest file:
> Class-Path: zstd-jni-1.5.5-6.jar lz4-java-1.8.0.jar snappy-java-1.1.10
>  .5.jar slf4j-api-1.7.36.jar
> I'm using Maven repo to keep my dependencies and those files are not in the 
> same directory as kafka-clients-3.7.0.jar, so the paths in the manifest's 
> Class-Path are not correct. It fails my build because we build with javac 
> with all linting options on, in particular -Xlint:-path. It produces the 
> following warnings coming from javac:
> [WARNING] COMPILATION WARNING : 
> [INFO] -
> [WARNING] [path] bad path element 
> "/home/ci/.m2/repository/org/apache/kafka/kafka-clients/3.7.0/zstd-jni-1.5.5-6.jar":
>  no such file or directory
> [WARNING] [path] bad path element 
> "/home/ci/.m2/repository/org/apache/kafka/kafka-clients/3.7.0/lz4-java-1.8.0.jar":
>  no such file or directory
> [WARNING] [path] bad path element 
> "/home/ci/.m2/repository/org/apache/kafka/kafka-clients/3.7.0/snappy-java-1.1.10.5.jar":
>  no such file or directory
> [WARNING] [path] bad path element 
> "/home/ci/.m2/repository/org/apache/kafka/kafka-clients/3.7.0/slf4j-api-1.7.36.jar":
>  no such file or directory
> Since we have also {{-Werror}} option enabled, it turns warnings into errors 
> and fails our build.
> I think our setup is quite typical: using Maven repo to store dependencies, 
> having linting on and -Werror. Unfortunatelly, it doesn't work with the 
> lastest kafka-clients because of the entries in the manifest's Class-Path. 
> And I think it might affect quite a lot of projects set up in a similar way.
> I don't know what was the reason to add Class-Path entry in the JAR manifest 
> file - but perhaps this effect was not considered.
> It would be great if you removed the Class-Path entry from the JAR manifest 
> file.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

[Matthias J. Sax]

Thanks for the KIP Shashwat. Closing this testing gap is great! It did 
come up a few time already...


One question: why do you propose to `extend MockProcessorContext`?

Given how the actual runtime context classes are setup, it seems that 
the regular context and fixed-key-context are distinct, and thus I 
believe both mock-context classes should be distinct, too?


What I mean is that FixedKeyProcessorContext does not extend 
ProcessorContext. Both classes have a common parent ProcessINGContext 
(note the very similar but different names), but they are "siblings" 
only, so why make the mock processor a parent-child relationship?


It seems better to do

public class MockFixedKeyProcessorContext
  implements FixedKeyProcessorContext,
 RecordCollector.Supplier


Of course, if there is code we can share between both mock-context we 
should so this, but it should not leak into the public API?



-Matthias



On 3/11/24 5:21 PM, Shashwat Pandey wrote:

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1027%3A+Add+MockFixedKeyProcessorContext

This adds MockFixedKeyProcessorContext to the Kafka Streams Test Utils
library.

Regards,
Shashwat Pandey

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Matthias J. Sax]

Yes, it's used inside `TimeWindowedSerializer` and actually also inside 
`TimeWindowDeserializer`.


However, it does IMHO not change that we should remove it from 
`StreamsConfig` because both configs are not intended to be used in Java 
code... If one writes Java code, they should use


  new TimeWindowedSerializer(Serializer)
  new TimeWindowDeserializer(Deserializer, Long)
  new TimeWindowSerdes(Serde, Long)

and thus they don't need either config.

The configs are only needed for command line tool, that create the 
(de)serializer via reflection using the default constructor.


Does this make sense?



The only open question is really, if and where to add them... Strictly 
speaking, we don't need either config as public variable as nobody 
should use them in Java code. To me, it just feels right/better do make 
them public for documentation purpose that these configs exists?


`inner.window.class.serde` has "serde" in it's name, so we could add it 
to `TimeWindowSerdes`? For `window.size.ms`, it's only used by the 
deserialize to maybe add it there? Just some ideas. -- Or we sidestep 
this question and just don't add them; also fine with me.



-Matthias

On 3/11/24 10:53 AM, Lucia Cerchie wrote:

PS-- I was re-reading the PR that originated this discussion and realized
that `window.inner.serde.class` is used here

in KStreams. This goes against removing it, yes?

On Mon, Mar 11, 2024 at 10:40 AM Lucia Cerchie 
wrote:


Sophie, I'll add a paragraph about removing `windowed.inner.serde.class`
to the KIP. I'll also add putting it in the `TimeWindowedSerde` class with
some add'tl guidance on the docs addition.

Also, I double-checked setting window.size.ms on the client and it
doesn't throw an error at all, in response to Matthias's question. Changing
the KIP in response to that.

On Sun, Mar 10, 2024 at 6:04 PM Sophie Blee-Goldman 
wrote:


Thanks for responding Matthias -- you got there first, but I was going to
say exactly the same thing as in your most reply. In other words, I see
the
`windowed.inner.serde.class` as being in the same boat as the `
window.size.ms` config, so whatever we do with one we should do for the
other.

I do agree with removing these from StreamsConfig, but defining them in
ConsumerConfig feels weird as well. There's really no great answer here.

My only concern about adding it to the corresponding
serde/serializer/deserializer class is that it might be difficult for
people to find them. I generally assume that people tend not to look at
the
serde/serializer/deserializer classes/implementations. But maybe in this
case, someone who actually needed these configs would be likely to be
motivated enough to find them by looking at the class? And with sufficient
documentation, it's likely not a problem. So, I'm +1 on putting it into
the
TimeWindowedSerde class

(I would personally stick them into the serde class, rather than the
serializer and/or deserializer, but I could be convinced either way)

On Fri, Mar 1, 2024 at 3:00 PM Matthias J. Sax  wrote:


One more thought after I did some more digging on the related PR.

Should we do the same thing for `windowed.inner.serde.class`?


Both config belong to windowed serdes (which KS provide) but the KS code
itself does never use them (and in fact, disallows to use them and would
throw an error is used). Both are intended for plain consumer use cases
for which the window serdes are used.

The question to me is, should we add them back somewhere else? It does
not really belong into `ConsumerConfig` either, but maybe we could add
them to the corresponding serde or (de)serialize classes?


-Matthias


On 2/21/24 2:41 PM, Matthias J. Sax wrote:

Thanks for the KIP. Sounds like a nice cleanup.


window.size.ms  is not a true KafkaStreams config, and results in an
error when set from a KStreams application


What error?


Given that the configs is used by `TimeWindowedDeserializer` I am
wondering if we should additionally add

public class TimeWindowedDeserializer {

  public static final String WINDOW_SIZE_MS_CONFIG = "

window.size.ms

";

}



-Matthias


On 2/15/24 6:32 AM, Lucia Cerchie wrote:

Hey everyone,

I'd like to discuss KIP-1020
<



https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804

,

which would move to deprecate `window.size.ms` in `StreamsConfig`

since `

window.size.ms` is a client config.

Thanks in advance!

Lucia Cerchie








--

[image: Confluent] 
Lucia Cerchie
Developer Advocate
Follow us: [image: Blog]
[image:
Twitter] [image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

[jira] [Resolved] (KAFKA-16360) Release plan of 3.x kafka releases.

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-16360.
-
Resolution: Invalid

Please don't use Jira to ask questions. Jira tickets are for bug reports and 
features only.

Question should be asked on the user and/or dev mailing lists: 
https://kafka.apache.org/contact

> Release plan of 3.x kafka releases.
> ---
>
> Key: KAFKA-16360
> URL: https://issues.apache.org/jira/browse/KAFKA-16360
> Project: Kafka
>  Issue Type: Improvement
>Reporter: kaushik srinivas
>Priority: Major
>
> KIP 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-ReleaseTimeline]
>  mentions ,
> h2. Kafka 3.7
>  * January 2024
>  * Final release with ZK mode
> But we see in Jira, some tickets are marked for 3.8 release. Does apache 
> continue to make 3.x releases having zookeeper and kraft supported 
> independent of pure kraft 4.x releases ?
> If yes, how many more releases can be expected on 3.x release line ?
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

[Shashwat Pandey]

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1027%3A+Add+MockFixedKeyProcessorContext

This adds MockFixedKeyProcessorContext to the Kafka Streams Test Utils
library.

Regards,
Shashwat Pandey

[jira] [Created] (KAFKA-16366) Refactor KTable source optimization

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-16366:
---

 Summary: Refactor KTable source optimization
 Key: KAFKA-16366
 URL: https://issues.apache.org/jira/browse/KAFKA-16366
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


Kafka Streams DSL offers an optimization to re-use an input topic as table 
changelog, in favor of creating a dedicated changelog topic.

So far, the Processor API did not support any such feature, and thus when the 
DSL compiles down into a Topology, we needed to access topology internal stuff 
to allow for this optimization.

With KIP-813 (merged for AK 3.8), we added `Topology#addReadOnlyStateStore` as 
public API, and thus we should refactor the DSL compilation code, to use this 
public API to build the `Topology` instead of internal APIs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16365) AssignmentsManager mismanages completion notifications

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-16365:
---

 Summary: AssignmentsManager mismanages completion notifications
 Key: KAFKA-16365
 URL: https://issues.apache.org/jira/browse/KAFKA-16365
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez
Assignee: Igor Soarez


When moving replicas between directories in the same broker, future replica 
promotion hinges on acknowledgment from the controller of a change in the 
directory assignment.
 
ReplicaAlterLogDirsThread relies on AssignmentsManager for a completion 
notification of the directory assignment change.
 
In its current form, under certain assignment scheduling, AssignmentsManager 
both miss completion notifications, or prematurely trigger them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16364) MM2 High-Resolution Offset Translation

[Greg Harris (Jira)]

Greg Harris created KAFKA-16364:
---

 Summary: MM2 High-Resolution Offset Translation
 Key: KAFKA-16364
 URL: https://issues.apache.org/jira/browse/KAFKA-16364
 Project: Kafka
  Issue Type: New Feature
  Components: mirrormaker
Reporter: Greg Harris


The current OffsetSyncStore implementation 
[https://github.com/apache/kafka/blob/8b72a2c72f09838fdd2e7416c98d30fe876b4078/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/OffsetSyncStore.java#L57]
 stores a sparse index of offset syncs. This attempts to strike a balanced 
default behavior between offset translation availability, memory usage, and 
throughput on the offset syncs topic.

However, this balanced default behavior is not good enough in all 
circumstances. When precise offset translation is needed away from the end of 
the topic, such as for consumer groups with persistent lag, offset translation 
can be more precise. Users should have a way to configure high-precision offset 
translation, either through additional memory usage or re-reading the offset 
syncs topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #108

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 457202 lines...]
[2024-03-11T21:23:32.885Z] > Task :group-coordinator:testClasses
[2024-03-11T21:23:33.983Z] > Task :core:compileScala
[2024-03-11T21:23:33.983Z] 
[2024-03-11T21:23:33.983Z] > Task :clients:javadoc
[2024-03-11T21:23:33.983Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.7/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2024-03-11T21:23:33.983Z] 2 warnings
[2024-03-11T21:23:33.983Z] 
[2024-03-11T21:23:33.983Z] > Task :clients:javadocJar
[2024-03-11T21:23:35.248Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2024-03-11T21:23:35.248Z] > Task :metadata:compileTestJava
[2024-03-11T21:23:35.248Z] > Task :metadata:testClasses
[2024-03-11T21:23:35.248Z] > Task :clients:srcJar
[2024-03-11T21:23:35.248Z] > Task :clients:testJar
[2024-03-11T21:23:36.347Z] > Task :clients:testSrcJar
[2024-03-11T21:23:36.347Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2024-03-11T21:23:36.347Z] > Task :clients:publishToMavenLocal
[2024-03-11T21:23:36.347Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2024-03-11T21:23:36.347Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2024-03-11T21:23:36.347Z] > Task :connect:api:testClasses UP-TO-DATE
[2024-03-11T21:23:36.347Z] > Task :connect:api:testJar
[2024-03-11T21:23:36.347Z] > Task :connect:api:testSrcJar
[2024-03-11T21:23:36.347Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2024-03-11T21:23:36.347Z] > Task :connect:api:publishToMavenLocal
[2024-03-11T21:23:41.773Z] > Task :streams:javadoc
[2024-03-11T21:23:41.773Z] > Task :streams:javadocJar
[2024-03-11T21:23:41.773Z] > Task :streams:srcJar
[2024-03-11T21:23:41.773Z] > Task :streams:processTestResources UP-TO-DATE
[2024-03-11T21:24:26.447Z] > Task :core:classes
[2024-03-11T21:24:26.447Z] > Task :core:compileTestJava NO-SOURCE
[2024-03-11T21:24:54.773Z] > Task :core:compileTestScala
[2024-03-11T21:25:46.921Z] > Task :core:testClasses
[2024-03-11T21:26:11.527Z] > Task :streams:compileTestJava
[2024-03-11T21:26:30.561Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2024-03-11T21:26:30.563Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2024-03-11T21:27:34.626Z] > Task :streams:testClasses
[2024-03-11T21:27:34.626Z] > Task :streams:testJar
[2024-03-11T21:27:34.626Z] > Task :streams:testSrcJar
[2024-03-11T21:27:34.626Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2024-03-11T21:27:34.626Z] > Task :streams:publishToMavenLocal
[2024-03-11T21:27:34.626Z] 
[2024-03-11T21:27:34.626Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2024-03-11T21:27:34.626Z] 
[2024-03-11T21:27:34.626Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2024-03-11T21:27:34.626Z] 
[2024-03-11T21:27:34.626Z] For more on this, please refer to 
https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
[2024-03-11T21:27:34.626Z] 
[2024-03-11T21:27:34.626Z] BUILD SUCCESSFUL in 4m 43s
[2024-03-11T21:27:34.626Z] 95 actionable tasks: 41 executed, 54 up-to-date
[2024-03-11T21:27:34.626Z] 
[2024-03-11T21:27:34.626Z] Publishing build scan...
[2024-03-11T21:27:34.626Z] https://ge.apache.org/s/dv5olsxhbcpr6
[2024-03-11T21:27:34.626Z] 
[Pipeline] sh
[2024-03-11T21:27:37.601Z] + grep ^version= gradle.properties
[2024-03-11T21:27:37.601Z] + cut -d= -f 2
[Pipeline] dir
[2024-03-11T21:27:38.452Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.7/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2024-03-11T21:27:40.576Z] + mvn clean install -Dgpg.skip
[2024-03-11T21:27:41.674Z] [INFO] Scanning for projects...
[2024-03-11T21:27:42.773Z] [INFO] 

[2024-03-11T21:27:42.773Z] [INFO] Reactor Build Order:
[2024-03-11T21:27:42.773Z] [INFO] 
[2024-03-11T21:27:42.773Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2024-03-11T21:27:42.773Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2024-03-11T21:27:42.773Z] [INFO] 
[2024-03-11T21:27:42.773Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2024-03-11T21:27:42.773Z] [INFO] Building Kafka Streams :: Quickstart 
3.7.1-SNAPSHOT[1/2]
[2024-03-11T21:27:42.773Z] [INFO]   from pom.xml

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #149

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2715

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16363) Storage crashes if dir is unavailable

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-16363:
---

 Summary: Storage crashes if dir is unavailable
 Key: KAFKA-16363
 URL: https://issues.apache.org/jira/browse/KAFKA-16363
 Project: Kafka
  Issue Type: Sub-task
  Components: tools
Affects Versions: 3.7.0
Reporter: Igor Soarez


The storage tool crashes if one of the configured log directories is 
unavailable. 

 
{code:java}
sh-4.4# ./bin/kafka-storage.sh format --ignore-formatted -t $KAFKA_CLUSTER_ID 
-c server.properties
[2024-03-11 17:51:05,391] ERROR Error while reading meta.properties file 
/data/d2/meta.properties 
(org.apache.kafka.metadata.properties.MetaPropertiesEnsemble)
java.nio.file.AccessDeniedException: /data/d2/meta.properties
        at 
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at 
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at 
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at 
java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
        at 
java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
        at java.base/java.nio.file.Files.newInputStream(Files.java:160)
        at 
org.apache.kafka.metadata.properties.PropertiesUtils.readPropertiesFile(PropertiesUtils.java:77)
        at 
org.apache.kafka.metadata.properties.MetaPropertiesEnsemble$Loader.load(MetaPropertiesEnsemble.java:135)
        at kafka.tools.StorageTool$.formatCommand(StorageTool.scala:431)
        at kafka.tools.StorageTool$.main(StorageTool.scala:95)
        at kafka.tools.StorageTool.main(StorageTool.scala)
metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, 
dirs={/data/d1: MetaProperties(version=1, clusterId=RwO2UIkmTBWltwRllP05aA, 
nodeId=101, directoryId=zm7fSw3zso9aR0AtuzsI_A), /data/metadata: 
MetaProperties(version=1, clusterId=RwO2UIkmTBWltwRllP05aA, nodeId=101, 
directoryId=eRO8vOP7ddbpx_W2ZazjLw), /data/d2: ERROR})
I/O error trying to read log directory /data/d2.
 {code}
When configured with multiple directories, Kafka tolerates some of them (but 
not all) being inaccessible, so this tool should be able to handle the same 
scenarios without crashing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Lucia Cerchie]

PS-- I was re-reading the PR that originated this discussion and realized
that `window.inner.serde.class` is used here

in KStreams. This goes against removing it, yes?

On Mon, Mar 11, 2024 at 10:40 AM Lucia Cerchie 
wrote:

> Sophie, I'll add a paragraph about removing `windowed.inner.serde.class`
> to the KIP. I'll also add putting it in the `TimeWindowedSerde` class with
> some add'tl guidance on the docs addition.
>
> Also, I double-checked setting window.size.ms on the client and it
> doesn't throw an error at all, in response to Matthias's question. Changing
> the KIP in response to that.
>
> On Sun, Mar 10, 2024 at 6:04 PM Sophie Blee-Goldman 
> wrote:
>
>> Thanks for responding Matthias -- you got there first, but I was going to
>> say exactly the same thing as in your most reply. In other words, I see
>> the
>> `windowed.inner.serde.class` as being in the same boat as the `
>> window.size.ms` config, so whatever we do with one we should do for the
>> other.
>>
>> I do agree with removing these from StreamsConfig, but defining them in
>> ConsumerConfig feels weird as well. There's really no great answer here.
>>
>> My only concern about adding it to the corresponding
>> serde/serializer/deserializer class is that it might be difficult for
>> people to find them. I generally assume that people tend not to look at
>> the
>> serde/serializer/deserializer classes/implementations. But maybe in this
>> case, someone who actually needed these configs would be likely to be
>> motivated enough to find them by looking at the class? And with sufficient
>> documentation, it's likely not a problem. So, I'm +1 on putting it into
>> the
>> TimeWindowedSerde class
>>
>> (I would personally stick them into the serde class, rather than the
>> serializer and/or deserializer, but I could be convinced either way)
>>
>> On Fri, Mar 1, 2024 at 3:00 PM Matthias J. Sax  wrote:
>>
>> > One more thought after I did some more digging on the related PR.
>> >
>> > Should we do the same thing for `windowed.inner.serde.class`?
>> >
>> >
>> > Both config belong to windowed serdes (which KS provide) but the KS code
>> > itself does never use them (and in fact, disallows to use them and would
>> > throw an error is used). Both are intended for plain consumer use cases
>> > for which the window serdes are used.
>> >
>> > The question to me is, should we add them back somewhere else? It does
>> > not really belong into `ConsumerConfig` either, but maybe we could add
>> > them to the corresponding serde or (de)serialize classes?
>> >
>> >
>> > -Matthias
>> >
>> >
>> > On 2/21/24 2:41 PM, Matthias J. Sax wrote:
>> > > Thanks for the KIP. Sounds like a nice cleanup.
>> > >
>> > >> window.size.ms  is not a true KafkaStreams config, and results in an
>> > >> error when set from a KStreams application
>> > >
>> > > What error?
>> > >
>> > >
>> > > Given that the configs is used by `TimeWindowedDeserializer` I am
>> > > wondering if we should additionally add
>> > >
>> > > public class TimeWindowedDeserializer {
>> > >
>> > >  public static final String WINDOW_SIZE_MS_CONFIG = "
>> window.size.ms
>> > ";
>> > > }
>> > >
>> > >
>> > >
>> > > -Matthias
>> > >
>> > >
>> > > On 2/15/24 6:32 AM, Lucia Cerchie wrote:
>> > >> Hey everyone,
>> > >>
>> > >> I'd like to discuss KIP-1020
>> > >> <
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804
>> > >,
>> > >> which would move to deprecate `window.size.ms` in `StreamsConfig`
>> > since `
>> > >> window.size.ms` is a client config.
>> > >>
>> > >> Thanks in advance!
>> > >>
>> > >> Lucia Cerchie
>> > >>
>> >
>>
>
>
> --
>
> [image: Confluent] 
> Lucia Cerchie
> Developer Advocate
> Follow us: [image: Blog]
> [image:
> Twitter] [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> 
>


-- 

[image: Confluent] 
Lucia Cerchie
Developer Advocate
Follow us: [image: Blog]
[image:
Twitter] [image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Lucia Cerchie]

Sophie, I'll add a paragraph about removing `windowed.inner.serde.class` to
the KIP. I'll also add putting it in the `TimeWindowedSerde` class with
some add'tl guidance on the docs addition.

Also, I double-checked setting window.size.ms on the client and it doesn't
throw an error at all, in response to Matthias's question. Changing the KIP
in response to that.

On Sun, Mar 10, 2024 at 6:04 PM Sophie Blee-Goldman 
wrote:

> Thanks for responding Matthias -- you got there first, but I was going to
> say exactly the same thing as in your most reply. In other words, I see the
> `windowed.inner.serde.class` as being in the same boat as the `
> window.size.ms` config, so whatever we do with one we should do for the
> other.
>
> I do agree with removing these from StreamsConfig, but defining them in
> ConsumerConfig feels weird as well. There's really no great answer here.
>
> My only concern about adding it to the corresponding
> serde/serializer/deserializer class is that it might be difficult for
> people to find them. I generally assume that people tend not to look at the
> serde/serializer/deserializer classes/implementations. But maybe in this
> case, someone who actually needed these configs would be likely to be
> motivated enough to find them by looking at the class? And with sufficient
> documentation, it's likely not a problem. So, I'm +1 on putting it into the
> TimeWindowedSerde class
>
> (I would personally stick them into the serde class, rather than the
> serializer and/or deserializer, but I could be convinced either way)
>
> On Fri, Mar 1, 2024 at 3:00 PM Matthias J. Sax  wrote:
>
> > One more thought after I did some more digging on the related PR.
> >
> > Should we do the same thing for `windowed.inner.serde.class`?
> >
> >
> > Both config belong to windowed serdes (which KS provide) but the KS code
> > itself does never use them (and in fact, disallows to use them and would
> > throw an error is used). Both are intended for plain consumer use cases
> > for which the window serdes are used.
> >
> > The question to me is, should we add them back somewhere else? It does
> > not really belong into `ConsumerConfig` either, but maybe we could add
> > them to the corresponding serde or (de)serialize classes?
> >
> >
> > -Matthias
> >
> >
> > On 2/21/24 2:41 PM, Matthias J. Sax wrote:
> > > Thanks for the KIP. Sounds like a nice cleanup.
> > >
> > >> window.size.ms  is not a true KafkaStreams config, and results in an
> > >> error when set from a KStreams application
> > >
> > > What error?
> > >
> > >
> > > Given that the configs is used by `TimeWindowedDeserializer` I am
> > > wondering if we should additionally add
> > >
> > > public class TimeWindowedDeserializer {
> > >
> > >  public static final String WINDOW_SIZE_MS_CONFIG = "
> window.size.ms
> > ";
> > > }
> > >
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 2/15/24 6:32 AM, Lucia Cerchie wrote:
> > >> Hey everyone,
> > >>
> > >> I'd like to discuss KIP-1020
> > >> <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804
> > >,
> > >> which would move to deprecate `window.size.ms` in `StreamsConfig`
> > since `
> > >> window.size.ms` is a client config.
> > >>
> > >> Thanks in advance!
> > >>
> > >> Lucia Cerchie
> > >>
> >
>


-- 

[image: Confluent] 
Lucia Cerchie
Developer Advocate
Follow us: [image: Blog]
[image:
Twitter] [image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jun

Thanks for the feedback. See my comments below.

On Wed, Mar 6, 2024 at 4:47 PM Jun Rao  wrote:
> 20.1. It seems that the PreferredSuccessors field didn't get fixed. It's
> still there together with PreferredCandidates.
> +{ "name": "PreferredSuccessors", "type": "[]int32", "versions":
> "0",
> +  "about": "A sorted list of preferred successors to start the
> election" },
> +{ "name": "PreferredCandidates", "type": "[]ReplicaInfo",
> "versions": "1+",
> +  "about": "A sorted list of preferred candidates to start the
> election", "fields": [

Notice that the PreferredSuccessors field is only for version 0 while
the PreferredCandidate field is for version 1 or greater. I had to
create a new field because arrays of int32 ([]int32) are not
compatible with arrays of structs because of tagged fields in
sub-structs.

> 37. If we don't support batching in AddVoterResponse, RemoveVoterResponse
> and UpdateVoterResponse, should we combine CurrentLeader and NodeEndpoint
> into a single field?

Yes. I replaced the LeaderIdAndEpoch and NodeEpoint structs into a
single struct that contains the leader id, epoch, host and port.

> 42. We include VoterId and VoterUuid for the receiver in Vote and
> BeginQuorumEpoch requests, but not in EndQuorumEpoch, Fetch and
> FetchSnapshot. Could you explain how they are used?

For the Vote request and BeginQuorumEpoch request the replica
(candidate for Vote and leader for BeginQuorumEpoch) sending the
request needs to make sure that it is sending the request to the
correct node. This is needed for correctness. The most important case
that I wanted to make sure that replicas handle correctly is the
following:
1. Voter set is A, B, C  and the leader is A. The voter A is both the
voter id and voter uuid
2. Assume that A crashes and loses its disk. When it recovers it will
come back as A'. A' means a replica with the same id but with a
different replica uuid.

Replica A' will most likely be accepting connection and handling
requests (e.g. Vote and BeginQuorumEpoch) on the same endpoints as A.
There can be inconsistency in the state, if for example replica B
sends a Vote request to A' but A' handles it as if it was A. This is
the reason the sender sends the remote replica's id and uuid (VoterId
and VoterUuid) in the request. The same analysis applies to
BeginEpochQuorum.

For the Fetch and FetchSnapshot request the closest equivalent would
be leader id and leader epoch. Those RPCs only have leader epochs. You
can argue that they don't need the leader id because a leader epoch
can have at most one leader id. In other words, the leader epoch also
uniquely identifies the leader id if there is one. I am reluctant to
change the Fetch RPC unless it is strictly required because that RPC
is also used for regular topic partition replication. I tried to make
the FetchSnapshot RPC as consistent to the Fetch RPC as possible since
they have similar access patterns.

EndQuorumEpoch is not needed for correctness. It is there for improved
availability; to speedup leadership change when the nodes are
cooperating (controlled shutdown and resignation). The sending replica
(leader) doesn't need to wait for the response or to check that the
RPC was handled correctly.

I'll reread the KIP and update it to better explain the need for
VoteId and VoteUuid in the Vote and BeginQuorumEpoch requests.

Thanks,
-- 
-José

[DISCUSS] Minimum constraint for segment.ms

[Divij Vaidya]

Hey folks

Before I file a KIP to change this in 4.0, I wanted to understand the
historical context for the value of the following setting.

Currently, segment.ms minimum threshold is set to 1ms [1].

Segments are expensive. Every segment uses multiple file descriptors and
it's easy to run out of OS limits when creating a large number of segments.
Large number of segments also delays log loading on startup because of
expensive operations such as iterating through all directories &
conditionally loading all producer state.

I am currently not aware of a reason as to why someone might want to work
with a segment.ms of less than ~10s (number chosen arbitrary that looks
sane)

What was the historical context of setting the minimum threshold to 1ms for
this setting?

[1] https://kafka.apache.org/documentation.html#topicconfigs_segment.ms

--
Divij Vaidya

[jira] [Created] (KAFKA-16362) Fix type-unsafety in KStreamKStreamJoin caused by isLeftSide

[Greg Harris (Jira)]

Greg Harris created KAFKA-16362:
---

 Summary: Fix type-unsafety in KStreamKStreamJoin caused by 
isLeftSide
 Key: KAFKA-16362
 URL: https://issues.apache.org/jira/browse/KAFKA-16362
 Project: Kafka
  Issue Type: Task
  Components: streams
Affects Versions: 3.7.0
Reporter: Greg Harris


The implementation of KStreamKStreamJoin has several places that the compiler 
emits warnings for, that are later suppressed or ignored:
 * LeftOrRightValue.make returns a raw LeftOrRightValue without generic 
arguments, because the generic type arguments depend on the boolean input.
 * Calling LeftOrRightValue includes an unchecked cast before inserting the 
record into the outerJoinStore
 * emitNonJoinedOuterRecords swaps the left and right values, and performs an 
unchecked cast

These seem to be closely related to the isLeftSide variable, which makes the 
class behave differently whether it is present on the left or right side of a 
join.

We should figure out if these warnings can be eliminated by a refactor, perhaps 
into KStreamKstreamJoin.Left and KStreamKStreamJoin.Right, or with some generic 
arguments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16361) Rack aware sticky assignor minQuota violations

[Luke D (Jira)]

Luke D created KAFKA-16361:
--

 Summary: Rack aware sticky assignor minQuota violations
 Key: KAFKA-16361
 URL: https://issues.apache.org/jira/browse/KAFKA-16361
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.6.1, 3.7.0, 3.5.1
Reporter: Luke D


In some low topic replication scenarios the rack aware assignment in the 
StickyAssignor fails to balance consumers to its own expectations and throws an 
IllegalStateException, commonly crashing the application (depending on 
application implementation). While uncommon the error is deterministic, and so 
persists until the replication state changes. 

 

We have observed this in the wild in 3.5.1, and 3.6.1. We have reproduced it 
locally in a test case in 3.6.1 and 3.7.0 (3.5.1 we did not try but likely 
would also be reproducible there) 

 

Here is the error and stack from our test case against 3.7.0
{code:java}
We haven't reached the expected number of members with more than the minQuota 
partitions, but no more partitions to be assigned
java.lang.IllegalStateException: We haven't reached the expected number of 
members with more than the minQuota partitions, but no more partitions to be 
assigned
    at 
org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor$ConstrainedAssignmentBuilder.verifyUnfilledMembers(AbstractStickyAssignor.java:820)
    at 
org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor$ConstrainedAssignmentBuilder.build(AbstractStickyAssignor.java:652)
    at 
org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor.assignPartitions(AbstractStickyAssignor.java:113)
    at 
org.apache.kafka.clients.consumer.internals.AbstractPartitionAssignor.assign(AbstractPartitionAssignor.java:91)
 {code}
Here is a specific test case from 3.7.0 that fails when passed to 
StickyAssignor.assign:
{code:java}
Cluster(id = cluster-id, nodes = [host-3:1 (id: 4 rack: rack-3), host-3:1 (id: 
3 rack: rack-3), host-2:1 (id: 2 rack: rack-2), host-1:1 (id: 1 rack: rack-1)], 
partitions = [Partition(topic = topic_name, partition = 57, leader = 4, 
replicas = [4], isr = [4], offlineReplicas = []), Partition(topic = topic_name, 
partition = 90, leader = 2, replicas = [2], isr = [2], offlineReplicas = []), 
Partition(topic = topic_name, partition = 28, leader = 3, replicas = [3], isr = 
[3], offlineReplicas = []), Partition(topic = topic_name, partition = 53, 
leader = 4, replicas = [4], isr = [4], offlineReplicas = []), Partition(topic = 
topic_name, partition = 86, leader = 2, replicas = [2], isr = [2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 24, leader = 
4, replicas = [4,3,1], isr = [4,3,1], offlineReplicas = []), Partition(topic = 
topic_name, partition = 49, leader = 1, replicas = [1,2], isr = [1,2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 82, leader = 
4, replicas = [4,2], isr = [4,2], offlineReplicas = []), Partition(topic = 
topic_name, partition = 20, leader = 2, replicas = [2,1], isr = [2,1], 
offlineReplicas = []), Partition(topic = topic_name, partition = 45, leader = 
2, replicas = [2], isr = [2], offlineReplicas = []), Partition(topic = 
topic_name, partition = 78, leader = 1, replicas = [1], isr = [1], 
offlineReplicas = []), Partition(topic = topic_name, partition = 16, leader = 
4, replicas = [4], isr = [4], offlineReplicas = []), Partition(topic = 
topic_name, partition = 41, leader = 1, replicas = [1,2], isr = [1,2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 74, leader = 
4, replicas = [4,3,1], isr = [4,3,1], offlineReplicas = []), Partition(topic = 
topic_name, partition = 12, leader = 2, replicas = [2], isr = [2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 37, leader = 
1, replicas = [1], isr = [1], offlineReplicas = []), Partition(topic = 
topic_name, partition = 70, leader = 2, replicas = [2], isr = [2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 8, leader = 4, 
replicas = [4,3,1], isr = [4,3,1], offlineReplicas = []), Partition(topic = 
topic_name, partition = 33, leader = 1, replicas = [1], isr = [1], 
offlineReplicas = []), Partition(topic = topic_name, partition = 66, leader = 
4, replicas = [4], isr = [4], offlineReplicas = []), Partition(topic = 
topic_name, partition = 4, leader = 2, replicas = [2], isr = [2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 29, leader = 
3, replicas = [3,1,2], isr = [3,1,2], offlineReplicas = []), Partition(topic = 
topic_name, partition = 62, leader = 3, replicas = [3,2,1], isr = [3,2,1], 
offlineReplicas = []), Partition(topic = topic_name, partition = 95, leader = 
4, replicas = [4,3,2], isr = [4,3,2], offlineReplicas = []), Partition(topic = 
topic_name, partition = 0, leader = 4, replicas = [4,1,2], isr = [4,1,2], 
offlineReplicas = []), Partition(topic = topic_name, partition = 25, leader 

[jira] [Created] (KAFKA-16360) Release plan of 3.x kafka releases.

[kaushik srinivas (Jira)]

kaushik srinivas created KAFKA-16360:


 Summary: Release plan of 3.x kafka releases.
 Key: KAFKA-16360
 URL: https://issues.apache.org/jira/browse/KAFKA-16360
 Project: Kafka
  Issue Type: Improvement
Reporter: kaushik srinivas


KIP 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-ReleaseTimeline]
 mentions ,
h2. Kafka 3.7
 * January 2024
 * Final release with ZK mode

But we see in Jira, some tickets are marked for 3.8 release. Does apache 
continue to make 3.x releases having zookeeper and kraft supported independent 
of pure kraft 4.x releases ?

If yes, how many more releases can be expected on 3.x release line ?

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin]

Thanks Folks,

Renamed the field to `offsets_status`.

And now that we have 3 binding votes, I will update the KIP status to
`accepted` .

Thanks again for all the valuable feedback.

Ashwin

On Wed, Mar 6, 2024 at 2:42 PM Chris Egerton 
wrote:

> Hi Yash,
>
> Thanks for the follow-up, I like the benefits it's yielded. I too think
> "offsets_status" would be a better name for the response field.
> @Ashwin--thoughts?
>
> Cheers,
>
> Chris
>
>
> On Wed, Mar 6, 2024, 03:08 Ashwin  wrote:
>
> > Thanks Yash,
> >
> > Yes , I think we can use @JsonInclude(JsonInclude.Include.NON_NULL) to
> > exclude “initial_offsets_response” from the create response if offset is
> > not specified.
> >
> > I’ll close the voting this week , if there are no further comments.
> >
> > Thanks for voting, everyone!
> >
> >
> > Ashwin
> >
> > On Tue, Mar 5, 2024 at 11:20 PM Yash Mayya  wrote:
> >
> > > Hi Chris,
> > >
> > > I followed up with Ashwin offline and I believe he wanted to take a
> > closer
> > > look at the `ConnectorInfoWithInitialOffsetsResponse` stuff he
> mentioned
> > in
> > > the previous email and whether or not that'll be required
> (alternatively
> > > using some Jackson JSON tricks). However, that's an implementation
> detail
> > > and shouldn't hold up the KIP. Bikeshedding a little on the
> > > "initial_offsets_response" field - I'm wondering if something like
> > > "offsets_status" might be more appropriate, what do you think? I don't
> > > think the current name is terrible though, so I'm +1 (binding) if
> > everyone
> > > else agrees that it's suitable.
> > >
> > > Thanks,
> > > Yash
> > >
> > > On Tue, Mar 5, 2024 at 9:51 PM Chris Egerton 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Wanted to bump this and see if it looks good enough for a third vote.
> > > Yash,
> > > > any thoughts?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Jan 29, 2024 at 2:55 AM Ashwin  >
> > > > wrote:
> > > >
> > > > > Thanks for reviewing this KIP,  Yash.
> > > > >
> > > > > Could you please elaborate on the cleanup steps? For instance, if
> we
> > > > > > encounter an error after wiping existing offsets but before
> writing
> > > the
> > > > > new
> > > > > > offsets, there's not really any good way to "revert" the wiped
> > > offsets.
> > > > > > It's definitely extremely unlikely that a user would expect the
> > > > previous
> > > > > > offsets for a connector to still be present (by creating a new
> > > > connector
> > > > > > with the same name but without initial offsets for instance)
> after
> > > > such a
> > > > > > failed operation, but it would still be good to call this out
> > > > > explicitly. I
> > > > > > presume that we'd want to wipe the newly written initial offsets
> if
> > > we
> > > > > fail
> > > > > > while writing the connector's config however?
> > > > >
> > > > >
> > > > > Agree - I have clarified the cleanup here -
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors#KIP995:Allowuserstospecifyinitialoffsetswhilecreatingconnectors-ProposedChanges
> > > > > .
> > > > >
> > > > > The `PATCH /connectors/{connector}/offsets` and `DELETE
> > > > > > /connectors/{connector}/offsets` endpoints have two possible
> > success
> > > > > > messages in the response depending on whether or not the
> connector
> > > > plugin
> > > > > > has implemented the `alterOffsets` connector method. Since we're
> > > > > proposing
> > > > > > to utilize the same offset validation during connector creation
> if
> > > > > initial
> > > > > > offsets are specified, I think it would be valuable to surface
> > > similar
> > > > > > information to users here as well
> > > > >
> > > > >
> > > > > Thanks for pointing this out. I have updated the response to
> include
> > a
> > > > new
> > > > > field “initial_offsets_response” which will contain the response
> > based
> > > on
> > > > > whether the connector implements alterOffsets or not. This also
> means
> > > > that
> > > > > if initial_offsets is set in the ConnectorCreate request, we will
> > > return
> > > > a
> > > > > new REST entity (ConnectorInfoWithInitialOffsetsResponse ?) which
> > will
> > > > be a
> > > > > child class of ConnectorInfo.
> > > > >
> > > > > (
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConnectorInfo.java#L28-L28
> > > > > )
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > > > On Wed, Jan 17, 2024 at 4:48 PM Yash Mayya 
> > > wrote:
> > > > >
> > > > > > Hi Ashwin,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > > If Connect runtime encounters an error in any of these steps,
> > > > > > > it will cleanup (if required) and return an error response
> > > > > >
> > > > > > Could you please elaborate on the cleanup steps? For instance, if
> > we
> > > > > > encounter an 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2714

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16359) kafka-clients-3.7.0.jar published to Maven Central is defective

[Jeremy Norris (Jira)]

Jeremy Norris created KAFKA-16359:
-

 Summary: kafka-clients-3.7.0.jar published to Maven Central is 
defective
 Key: KAFKA-16359
 URL: https://issues.apache.org/jira/browse/KAFKA-16359
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.7.0
Reporter: Jeremy Norris


The {{kafka-clients-3.7.0.jar}} that has been published to Maven Central is 
defective: it's {{META-INF/MANIFEST.MF}} bogusly include a {{Class-Path}} 
element:
{code}
Manifest-Version: 1.0
Class-Path: zstd-jni-1.5.5-6.jar lz4-java-1.8.0.jar snappy-java-1.1.10
.5.jar slf4j-api-1.7.36.jar
{code}

This bogus {{Class-Path}} element leads to compiler warnings for projects that 
utilize it as a dependency:
{code}
[WARNING] [path] bad path element 
".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/zstd-jni-1.5.5-6.jar": 
no such file or directory
[WARNING] [path] bad path element 
".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/lz4-java-1.8.0.jar": 
no such file or directory
[WARNING] [path] bad path element 
".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/snappy-java-1.1.10.5.jar":
 no such file or directory
[WARNING] [path] bad path element 
".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/slf4j-api-1.7.36.jar": 
no such file or directory
{code}

Either the {{kafka-clients-3.7.0.jar}} needs to be respun and published without 
the bogus {{Class-Path}} element in it's {{META-INF/MANIFEST.MF}} or a new 
release should be published that corrects this defect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

[Jorge Esteban Quilcate Otoya]

Hi Abhijeet,

Thanks for the KIP! Looks good to me. I just have a minor comments on
naming:

Would it be work to align the config names to existing quota names?
e.g. `remote.log.manager.copy.byte.rate.quota` (or similar) instead of
`remote.log.manager.copy.max.bytes.per.second`?

Same for new components, could we use the same verbs as in the configs:
- RLMCopyQuotaManager
- RLMFetchQuotaManager


On Fri, 8 Mar 2024 at 13:43, Abhijeet Kumar 
wrote:

> Thank you all for your comments. As all the comments in the thread are
> addressed, I am starting a Vote thread for the KIP. Please have a look.
>
> Regards.
>
> On Thu, Mar 7, 2024 at 12:34 PM Luke Chen  wrote:
>
> > Hi Abhijeet,
> >
> > Thanks for the update and the explanation.
> > I had another look, and it LGTM now!
> >
> > Thanks.
> > Luke
> >
> > On Tue, Mar 5, 2024 at 2:50 AM Jun Rao  wrote:
> >
> > > Hi, Abhijeet,
> > >
> > > Thanks for the reply. Sounds good to me.
> > >
> > > Jun
> > >
> > >
> > > On Sat, Mar 2, 2024 at 7:40 PM Abhijeet Kumar <
> > abhijeet.cse@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for pointing it out. It makes sense to me. We can have the
> > > following
> > > > metrics instead. What do you think?
> > > >
> > > >- remote-(fetch|copy)-throttle-time-avg (The average time in ms
> > remote
> > > >fetches/copies was throttled by a broker)
> > > >- remote-(fetch|copy)-throttle-time--max (The maximum time in ms
> > > remote
> > > >fetches/copies was throttled by a broker)
> > > >
> > > > These are similar to fetch-throttle-time-avg and
> > fetch-throttle-time-max
> > > > metrics we have for Kafak Consumers?
> > > > The Avg and Max are computed over the (sliding) window as defined by
> > the
> > > > configuration metrics.sample.window.ms on the server.
> > > >
> > > > (Also, I will update the config and metric names to be consistent)
> > > >
> > > > Regards.
> > > >
> > > > On Thu, Feb 29, 2024 at 2:51 AM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Abhijeet,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > The issue with recording the throttle time as a gauge is that it's
> > > > > transient. If the metric is not read immediately, the recorded
> value
> > > > could
> > > > > be reset to 0. The admin won't realize that throttling has
> happened.
> > > > >
> > > > > For client quotas, the throttle time is tracked as the average
> > > > > throttle-time per user/client-id. This makes the metric less
> > transient.
> > > > >
> > > > > Also, the configs use read/write whereas the metrics use
> fetch/copy.
> > > > Could
> > > > > we make them consistent?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Feb 28, 2024 at 6:49 AM Abhijeet Kumar <
> > > > abhijeet.cse@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Clarified the meaning of the two metrics. Also updated the KIP.
> > > > > >
> > > > > > kafka.log.remote:type=RemoteLogManager,
> > name=RemoteFetchThrottleTime
> > > ->
> > > > > The
> > > > > > duration of time required at a given moment to bring the observed
> > > fetch
> > > > > > rate within the allowed limit, by preventing further reads.
> > > > > > kafka.log.remote:type=RemoteLogManager,
> name=RemoteCopyThrottleTime
> > > ->
> > > > > The
> > > > > > duration of time required at a given moment to bring the observed
> > > > remote
> > > > > > copy rate within the allowed limit, by preventing further copies.
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > > On Wed, Feb 28, 2024 at 12:28 AM Jun Rao
>  > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Abhijeet,
> > > > > > >
> > > > > > > Thanks for the explanation. Makes sense to me now.
> > > > > > >
> > > > > > > Just a minor comment. Could you document the exact meaning of
> the
> > > > > > following
> > > > > > > two metrics? For example, is the time accumulated? If so, is it
> > > from
> > > > > the
> > > > > > > start of the broker or over some window?
> > > > > > >
> > > > > > > kafka.log.remote:type=RemoteLogManager,
> > > name=RemoteFetchThrottleTime
> > > > > > > kafka.log.remote:type=RemoteLogManager,
> > name=RemoteCopyThrottleTime
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Feb 27, 2024 at 1:39 AM Abhijeet Kumar <
> > > > > > abhijeet.cse@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > The existing quota system for consumers is designed to
> throttle
> > > the
> > > > > > > > consumer if it exceeds the allowed fetch rate.
> > > > > > > > The additional quota we want to add works on the broker
> level.
> > If
> > > > the
> > > > > > > > broker-level remote read quota is being
> > > > > > > > exceeded, we prevent additional reads from the remote storage
> > but
> > > > do
> > > > > > not
> > > > > > > > prevent local reads for the consumer.
> > > > > > > > If the consumer has specified other partitions to read, which
> > can
> > > > be
> > > > > > > served

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

[Arpit Goyal]

Hi All,
Just a Reminder, KIP-1026  is open for discussion.
Thanks and Regards
Arpit Goyal
8861094754


On Sat, Mar 9, 2024 at 9:27 AM Arpit Goyal  wrote:

> Hi All,
>
> I have created KIP-1026 for handling producerSnapshot empty scenarios
> when the topic is upgraded from the kafka  < 2.8 version.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1026%3A+Handling+producer+snapshot+when+upgrading+from+%3C+v2.8.0+for+Tiered+Storage
>
> Feedback and suggestions are welcome.
>
> Thanks and Regards
> Arpit Goyal
> 8861094754
>

[jira] [Reopened] (KAFKA-10199) Separate state restoration into separate threads

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna reopened KAFKA-10199:
---

> Separate state restoration into separate threads
> 
>
> Key: KAFKA-10199
> URL: https://issues.apache.org/jira/browse/KAFKA-10199
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Bruno Cadonna
>Priority: Major
>  Labels: new-streams-runtime-should-fix
> Fix For: 3.7.0
>
>
> As part of the restoration optimization effort, we would like to move the 
> restoration process to separate threads such that:
> 1. Stream threads would not be restricted by the main consumer `poll` 
> frequency to keep as part of the group.
> 2. We can allow larger batches of data to be written into the restoration.
> Besides this, we'd also like to fix the known issues that for piggy-backed 
> source topics as changelog topics, the serde exception / extra processing 
> logic would be skipped.
> We would also cleanup the global update tasks as part of this effort to 
> consolidate to the separate restoration threads, and would also gear them up 
> with corresponding monitoring metrics (KIPs in progress).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)