Re: Why would all consumers pull from the same partition?
Oooh, that sounds like our situation. Is there a way to avoid this with kafka configuration? My access to the messages and consumers is limited. From: Chandrajeet Padhy Sent: Monday, February 10, 2020 9:52 AM To: users@kafka.apache.org Subject: RE: Why would all consumers pull from the same partition? [EXTERNAL E-MAIL] Partition is decided based on message record key. If it's same, it will hit the same partition. -Original Message- From: Dylan Martin Sent: Monday, February 10, 2020 12:35 PM To: users@kafka.apache.org Subject: Why would all consumers pull from the same partition? I have a cluster of 20'ish brokers. One topic has 60'ish consumers and 100 partitions, but the consumers all seem to be hitting the same broker, which makes me think they're all hitting the same partition. What would cause that? I assume I've configured something wrong. Thanks! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you. The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Why would all consumers pull from the same partition?
I have a cluster of 20'ish brokers. One topic has 60'ish consumers and 100 partitions, but the consumers all seem to be hitting the same broker, which makes me think they're all hitting the same partition. What would cause that? I assume I've configured something wrong. Thanks! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Fw: Confusingly unbalanced broker
I'll look into those tools, thanks! I was able to turn on the JMX polling and consumer metrics in kafka-manager. I now know which topic & partition is causing the problem. It's basically 80MB of a single partiton on a single topic being hit by 60'odd consumers. Now I need to figure out what that means. Thanks! -Dylan From: Alex Woolford Sent: Saturday, February 8, 2020 10:09 PM To: users@kafka.apache.org Cc: Dylan Martin Subject: Re: Confusingly unbalanced broker [EXTERNAL E-MAIL] That's a very intriguing question, Dylan. Even if the partitions for each of the topics are distributed evenly across the brokers, it's not guaranteed that the *data* will be distributed evenly. By default, the producer will send all the messages in a topic with the same key to the same partition. It's possible you have keyed messages, the cardinality of the key is very low, and a disproportionate portion of the messages are going to a single "hot" partition. One thing you could do, off the top of my head, is to take a peek at the file access events. For example, the following one-liner shows that on this particular node, there are a lot of writes to the `aprs` topic, partition 2: # fatrace --seconds 10 | sort | uniq -c | sort -nr | head 161 java(1928): W /var/lib/kafka/aprs-2/81049867.log 155 java(1928): R /var/lib/kafka/_confluent-metrics-2/31360445.log 148 java(1928): R /var/lib/kafka/conn-0/29833400.log 136 ossec-agentd(1733): R /var/ossec/etc/shared/merged.mg<http://merged.mg> 129 osqueryd(2201): O /etc/passwd 104 java(1928): R /var/lib/kafka/_confluent-monitoring-0/46052008.log 95 osqueryd(2201): RC /etc/passwd 91 osqueryd(2201): RCO /etc/passwd 79 java(1928): R /var/lib/kafka/_confluent-controlcenter-5-4-0-1-MetricsAggregateStore-repartition-2/000414771172.log 64 java(1928): R /var/lib/kafka/_confluent-controlcenter-5-4-0-1-monitoring-message-rekey-store-1/02063409.log I'm running CentOS 7. Here's what I did to install fatrace: wget https://dl.fedoraproject.org/pub/fedora/linux/releases/31/Everything/source/tree/Packages/f/fatrace-0.13-5.fc31.src.rpm rpm -i fatrace-0.13-5.fc31.src.rpm yum install bzip2 tar xvf /root/rpmbuild/SOURCES/fatrace-0.13.tar.bz2 cd fatrace-0.13 make make install You could also poke around in the filesystem, perhaps using `ncdu`, to see which topics/partitions are consuming the disk. For example, `ncdu /var/lib/kafka` shows that partition 0 of my syslog topic is consuming most of the space on this particular broker: --- /var/lib/kafka --- 61.1 GiB [##] /syslog-0 6.4 GiB [# ] /aprs-0 3.7 GiB [ ] /syslog-7 3.7 GiB [ ] /syslog-9 Hopefully, someone with better Kafka-fu can suggest a more native way to understand, at the partition level, what's causing this behavior. HTH, Alex Woolford On Fri, Feb 7, 2020 at 2:38 PM Dylan Martin mailto:dmar...@istreamplanet.com>> wrote: Hi all! I have a cluster of about 20 brokers and one of them is transmitting about 4 times as much data as the others (80mB/sec vs 20mB/sec). It has the roughly the same number of topics & partitions and it's the leader for the same number as all the other brokers. The kafka-manager web tool doesn't say it's doing a particuarly large amount of work. Datadog & iftop both agree that it's sending out 4 times as much traffic as any of the others. It's very consistent, in that it's been this way for weeks. Any advice on how to track down what's going on? Thanks! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Confusingly unbalanced broker
Hi all! I have a cluster of about 20 brokers and one of them is transmitting about 4 times as much data as the others (80mB/sec vs 20mB/sec). It has the roughly the same number of topics & partitions and it's the leader for the same number as all the other brokers. The kafka-manager web tool doesn't say it's doing a particuarly large amount of work. Datadog & iftop both agree that it's sending out 4 times as much traffic as any of the others. It's very consistent, in that it's been this way for weeks. Any advice on how to track down what's going on? Thanks! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Is it possible to know the number of producers for a topic?
I have a situation where I should always have exactly one producer per topic, but under certain rare circumstances I can have more than one, and that's really bad when it happens, so I want to detect it. Can I know how many producer a topic has? Can I identify them? Thanks! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Reduce number of brokers?
I have a cluster with 4 brokers and I want to reduce it to 2 brokers. I cannot re-assign __consumer_offsets because it wants at least 3 brokers. Is there a way to do this? Or am I going to have to trash my cluster and start over? -Dylan (206) 855-9740 - Home (206) 235-8809 - Cell The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Completely clear out kafka brokers & zookeeper?
I have some boxes that I'm using to test kafka configurations (and zookeeper). What's the recommended procedure to clean them out to a clean state so I can re-install kafka and zookeeper without worrying about old data or configuration getting in the way? -Dylan (206) 855-9740 - Home (206) 235-8809 - Cell The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Cancel partition reassignment?
Hi all. I'm trying to cancel a failed partition reassignment. I've heard that this can be done by deleting /admin/reassign_partitions in zookeeper. I've tried and /admin/reassign_partitions won't go away. Does anyone know a way to cancel a partition reassignment? -Dylan (206) 855-9740 - Home (206) 235-8809 - Cell The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
Recovering Under Replicated Partitions
Hi all. I've got a cluster of 3 brokers with around 50 topics. Several topics are under replicated. Everything I've seen says I need to restart the followers to fix that. All my under replicated topics have the same broker as the leader. That makes me think it's a leader problem and not a follower problem. Also, I don't want to disrupt the topics that aren't under replicated while I restart the followers. Is there a way to do the same thing as a restart, but on a topic by topic basis? Is there a way to remove a broker from a topic and then re-add it? Wouldn't that do that same thing as restarting it? Thank you all! -Dylan The information contained in this email message, and any attachment thereto, is confidential and may not be disclosed without the sender's express permission. If you are not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution or copying of this message, or any attachment thereto, in whole or in part, is strictly prohibited. If you have received this message in error, please immediately notify the sender by telephone, fax or email and delete the message and all of its attachments. Thank you.
