[DISCUSS] KIP-736: Report the real end to end fetch latency

Hi Kafka, I'd like to open up the discussion on a KIP-736. This is to report the "real" fetch latency by removing the effects of waiting due to replica.fetch.wait.max.ms or fetch.min.bytes in purgatory Please give it a look over and let me know what you think. https://cwiki.apache.org/confluen

[DISCUSS] KIP-736: Report the true end to end fetch latency

Hi All, I'd like to open up the discussion on a KIP-736. This is to report the "real" fetch latency by removing the effects of waiting due to replica.fetch.wait.max.ms or fetch.min.bytes. Please give it a look over and let me know what you think. https://cwiki.apache.org/confluence/display/KAFKA

Re: Ask KIP create permission

Thanks, Matthias. The id is "mingaliu", Thanks! On 2021/04/25 20:40:04, "Matthias J. Sax" wrote: > Thanks for your interest, but I could not find your account. > > "Ming Liu" is not your ID/username but you profile name... I did find 4 > users with thi

Ask KIP create permission

ga...@gmail.com and id is "Ming Liu", can you grant me KIP create permission? The relevant KIP I like to propose is https://issues.apache.org/jira/browse/KAFKA-12713. Thanks! Ming

Re: Propose a KIP to report "REAL" broker/consumer fetch latency?

gt; > Maybe there is something in the current Admin API that is not obvious yet > so I will need to investigate first and will get back to you with my > thoughts/suggestions. > > Thanks for bringing this up > > Cheers > > > > On Sat, Apr 24, 2021 at 1:21 PM Ming Liu

Propose a KIP to report "REAL" broker/consumer fetch latency?

Hi All, I am thinking about to start a KIP to report "REAL" broker/consumer fetch latency. Before that, I like to collect any idea or suggestions. I created https://issues.apache.org/jira/browse/KAFKA-12713. The fetch latency is an important metric to monitor for the cluster performance.

[jira] [Created] (KAFKA-12713) Report broker/consumer fetch latency more accurately

Ming Liu created KAFKA-12713: Summary: Report broker/consumer fetch latency more accurately Key: KAFKA-12713 URL: https://issues.apache.org/jira/browse/KAFKA-12713 Project: Kafka Issue Type: Bug

Can't find any sendfile system call trace from Kafka process?

Hi Kafka dev community, As we know, one major reason that Kafka is fast is because it is using sendfile() for zero copy, as what it described at https://kafka.apache.org/documentation/#producerconfigs, *This combination of pagecache and sendfile means that on a Kafka cluster where the consu

Re: KIP idea: Separate publish request from the subscribe request

get a more > concrete design on how this would work end-to-end for others to review. > > Also I'm wondering if you have tried increasing the num.network.threads and > see if that helps with the large fanout multiplexing issue other than > separating them to different ports? >

KIP idea: Separate publish request from the subscribe request

Hi Kafka community, I like to surface a KIP idea, which is to separate publish request from the subscribe request using different ports. The context: We have some workload with over 5000 subscribers, the latency on publish latency can be as high as 3000 ms second. After investigation, we fou

[jira] [Created] (KAFKA-10398) Intra-broker disk move failed with onPartitionFenced()

Ming Liu created KAFKA-10398: Summary: Intra-broker disk move failed with onPartitionFenced() Key: KAFKA-10398 URL: https://issues.apache.org/jira/browse/KAFKA-10398 Project: Kafka Issue Type

[jira] [Created] (KAFKA-10242) Adding metrics to track the total count of idempotent producers that Broker need to track

Ming Liu created KAFKA-10242: Summary: Adding metrics to track the total count of idempotent producers that Broker need to track Key: KAFKA-10242 URL: https://issues.apache.org/jira/browse/KAFKA-10242

KAFKA-8362: Partition snapshot still exist in cleaner-offset-checkpoint after the intra-disk move

Hi Kafka, In Kafka 2.5, it seems KAFKA-8362 still exists. The problem is for compacted topic, after the disk re-assignment, both disk cleaner-offset-checkpoint will contain that partition snapshot. The previous disk cleaner-offset-checkpoint contains the stale one and should be deleted instead

OfflinePartitionLeaderElection improvement?

Hi Kafka community, I like to propose a small change related to OfflinePartitionLeaderElectionStrategy. In our system, we usually has RF = 3, Min_ISR = 2, unclean.leader.election = false and client usually set the ACK.all when publishing. We have observed that occasionally, when disk become b

Fix DK 8 and Scala 2.11 failure reported by Jenkin?

Hey, I have a simple PR https://github.com/apache/kafka/pull/5989/files, it always failed for JDK 8 and Scala 2.11 in jenkin build. I have been trying to repro with the local build (like ./gradlew -PscalaVersion=2.12 test or set Java_HOME etc), but it always build and test successfully for me lo

Finding reviewers for some fixes for sequenceId overflow

Hi, community, Here is the simple fix for two serious problems about sequenceId overflow (one in client, one at broker). It has been there for a week. Can somebody review it? https://issues.apache.org/jira/browse/KAFKA-7693 https://issues.apache.org/jira/browse/KAFKA-7692 Thanks! Ming

Kafka client Metadata update on demand?

Hey community, It seems Kafka Metadata update only happens at the pre-configured Metadata update interval. During upgrade, when leader changes, the client will fail with NotLeaderForPartitionException until next Metadata update happened. I wonder why we don't have the on-demand Metadata updat

[jira] [Created] (KAFKA-7698) Kafka Broker fail to start: ProducerFencedException thrown from producerstatemanager.scala!checkProducerEpoch

Ming Liu created KAFKA-7698: --- Summary: Kafka Broker fail to start: ProducerFencedException thrown from producerstatemanager.scala!checkProducerEpoch Key: KAFKA-7698 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-7692) updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append

Ming Liu created KAFKA-7692: --- Summary: updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append Key: KAFKA-7692 URL: https://issues.apache.org/jira/browse/KAFKA-7692 Project

[jira] [Created] (KAFKA-7693) "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client

Ming Liu created KAFKA-7693: --- Summary: "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client Key: KAFKA-7693 URL: https://issues.apache.org/jira/browse/KAFKA-7693 Proj