[jira] [Created] (KAFKA-14547) Be able to run kafka KRaft Server in tests without needing to run a storage setup script

[Natan Silnitsky (Jira)]

Natan Silnitsky created KAFKA-14547:
---

 Summary: Be able to run kafka KRaft Server in tests without 
needing to run a storage setup script
 Key: KAFKA-14547
 URL: https://issues.apache.org/jira/browse/KAFKA-14547
 Project: Kafka
  Issue Type: Improvement
  Components: kraft
Affects Versions: 3.3.1
Reporter: Natan Silnitsky


Currently kafka KRaft Server requires running kafka-storage.sh in order to 
start properly.
This makes setup much more cubersome for build tools like bazel to work 
properly.

One way to mitigate this is to configure the paths via kafkaConfig...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-8358) KafkaConsumer.endOffsets should be able to also return end offsets while not ignoring control records

[Natan Silnitsky (JIRA)]

Natan Silnitsky created KAFKA-8358:
--

 Summary: KafkaConsumer.endOffsets should be able to also return 
end offsets while not ignoring control records
 Key: KAFKA-8358
 URL: https://issues.apache.org/jira/browse/KAFKA-8358
 Project: Kafka
  Issue Type: Improvement
Reporter: Natan Silnitsky


We have a use case where we have a wrapper on top of {{kafkaConsumer}} for 
compact logs.
In order to know that a user can get "new" values for a key in the compact log, 
on init, or on rebalance, we need to block until all "old" values were read.

We wanted to use {{KafkaConsumer.endOffsets}} to help us find out where the 
"old" values end.
once all "old" values arrive from {{KafkaConsumer.poll}}, we can release the 
blocking on getting new values.

But it seems that [control 
records|https://github.com/apache/kafka/blob/c09e25fac2aaea61af892ae3e5273679a4bdbc7d/clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java#L128]
 are not received in {{KafkaConsumer.poll }} but are taking into account for 
{{KafkaConsumer.endOffsets }}

So the Feature request is for {{KafkaConsumer.endOffsets}} to have a flag to 
ignore control records, the same way that {{KafkaConsumer.poll }} ignores them.



(From a quick review of the code, it seems that 
{{LeaderEpochFile}}.[assign|https://github.com/apache/kafka/blob/c09e25fac2aaea61af892ae3e5273679a4bdbc7d/core/src/main/scala/kafka/server/epoch/LeaderEpochFileCache.scala#L51]
 can be given the flag isControl from 
[batch.isControlBatch|https://github.com/apache/kafka/blob/c09e25fac2aaea61af892ae3e5273679a4bdbc7d/clients/src/main/java/org/apache/kafka/common/record/RecordBatch.java#L239]

But I'm maybe wrong with my understanding there...)

CC:
[~berman7] [~berman]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)