Need some help in identifying some important metrics to monitor for streams

Hello All, I had few questions regarding monitoring of kafka streams application and what are some important metrics we should collect in our case. Just a brief overview, we have a single thread application (0.10.1.1) reading from single partition topic and it is working all fine. Then we have sam

Getting thread deadlock in streams 10.2.0 if a partition is re-assigned to a different thread in same instance

Hi, We are getting some deadlock state after re-balance and what we found that it happens after re-balance the partition is assigned to a different thread in same instance. Looks like previous thread is not releasing the rocks db lock. Here are the complete logs spanned across multiple files. ht

'Batch Expired' exception when I publish to a kafka topic using one producer per thread.

Hi, I get 'Batch Expired' exception when I publish to a kafka topic using one producer per thread. However using the same producer for each thread works perfectly fine and I do not get the exception. I do not understand what was causing this 'Batch Expired' exception (with one producer/thread

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

This issue is being addressed in KAFKA-4631. See https://issues.apache.org/jira/browse/KAFKA-4631 and the discussion in the PR https://github.com/apache/kafka/pull/2622 for details. Regards, Rajini On Thu, Mar 2, 2017 at 4:35 AM, Jaikiran Pai wrote: > For future reference - I asked this questi

Re: Getting thread deadlock in streams 10.2.0 if a partition is re-assigned to a different thread in same instance

Hi Sachin, This is because another thread hasn't released the lock. It is generally caused because it is taking a long time to restore or process the incoming records. It might be helpful to take a Thread dump so we can see what each thread is doing. Thanks, Damian On Thu, 2 Mar 2017 at 12:14 Sa

Using Kafka-connect transofrmer

Hi, I am new to Kafka/Kafka-connect. I would like to use Kafka-Connect transformer to get specific fields from my data @ a kafka topic. I was not able to find information/examples/documents about how to use Kafka-Connect transformer. I really appreciate if I can get some info on that! Best rega

Call to consumer.poll(1000) hangs

Hi, http://stackoverflow.com/questions/42551704/call-to-consumerrecordsstring-string-records-consumer-poll1000-hangs-and Thanks , Dhirendra

SASL/PLAIN text

Hi I've got a 3 broker kerberised Kafka 0.10 install running in Cloudera and I'm trying to authenticate with SASL/PLAIN I'm passing kafka_server_jaas.conf into the JVM on each of the brokers. KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username=admin pa

Re: Need some help in identifying some important metrics to monitor for streams

Hi Sachin, The new streams metrics are now documented at https://kafka.apache.org/documentation/#kafka_streams_monitoring . Note that not all of them are turned on by default. We have several benchmarks that run nightly to moni

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

Thank you for pointing me to that JIRA. It indeed is the same issue we discussed in this thread. I'll keep a watch on that JIRA for the code to be merged. -Jaikiran On Thursday 02 March 2017 07:11 PM, Rajini Sivaram wrote: This issue is being addressed in KAFKA-4631. See https://issues.apache

Re: Recommended number of partitions on each broker

We normally run over 1,000 partitions per broker, and I know of a major company with 30+ kafka clusters that averages 1,100 partitions per broker across all clusters. So 300 shouldn't be an issue as long as the throughput per partition isn't too high. Given that disk and cpu are so low, I'd guess

RE: [EXTERNAL] - Call to consumer.poll(1000) hangs

Hi Dhirendra; You should probably subscribe to the topic only once, in the constructor ConsumerLoop, not at every run. Isabelle -Original Message- From: Dhirendra Suman [mailto:dhirendra.su...@globallogic.com.INVALID] Sent: 2 mars 2017 05:13 To: users@kafka.apache.org Subject: [EXTERN

Re: Getting thread deadlock in streams 10.2.0 if a partition is re-assigned to a different thread in same instance

Hi, It makes sense looks like task 0_4 took more than max poll timeout time. However I have difficulty processing following lines: DEBUG 2017-03-01 18:17:42,465 [StreamThread-1]: org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [StreamThread-1] creating new task 0_4 ... DE

Re: Need some help in identifying some important metrics to monitor for streams

Hi, I had checked the monitoring docs, but could not figure out which metrics are important ones. Also mainly I am looking at the average time spent between 2 successive poll requests. Can I say that average time between 2 poll requests is sum of commit + poll + process + punctuate (latency-avg).

What is request.timeout in the consumer used for?

In the consumer, what will trigger the request.timeout? Is it just if broker doesn't respond within that period of time? I'm guessing in a healthy cluster, the primary culprit for triggering this is if one of the steps within the consumer group rebalancing taking a long time of inter-broker commu

Kafka mirror maker issue

Hi I am doing some experiments on kafka mirror maker. I have two Kafka cluster, one is kafka_2.11-0.9.0.1 with 3 nodes as source cluster, another is kafka_2.10-0.9.0.1 with one node as target cluster. First I mirror the message for topic TEST_TOPIC from source cluster to target cluster with co

Kafka partition no migrating to another broker.

I have 5 broker kafka cluster. Replication factor = 3, Number of partition = 12, Min Insync repica (ISR) = 3 First output is when all server are up and running. Second output is when I bring down server id = 4. Another server from ISR take server 4's place as leader of partition, so that's good

how to produce/consume kafka through ssh tunnel

We have kafka setup on staging environment, when debug the consumer we want to directly listen to the kafka on staging environment. I set tunnel but it seems i can't produce or consume from my local machine. I can create topic though. I have no problem to produce/consumer on other machine in stagin

Kafka metrics to Prometheus

Hi Kafka users, I am trying to expose Kafka client metrics to Prometheus via *MetricsReporter*. And it looks like Kafka clients don't expose the *Measurable* objects so that I can only do *KafkaMetric.value()* and use it as Gauge in Prometheus even if the metric could be a Percentile in Kafka clie

understanding consumer rebalance trigger(s)

Im wondering what the parameters are to instantiate a consumer rebalance. I have a topic that turns roughly 50K / minute across 6 partitions. Each is serviced by a separate dockerized consumer. Roughly every 8-12 min this goes into a rebalance that may take up to a minute. When it returns it often

'Batch Expired' exception when publishing to a kafka topic using one producer per thread.

Hi, I get 'Batch Expired' exception when I publish to a kafka topic using one producer per thread. However using the same producer for each thread works perfectly fine and I do not get the exception. I do not understand what was causing this 'Batch Expired' exception with one producer per thr

Re: Kafka partition no migrating to another broker.

Hello Shri, That behavior is by current Apache Kafka design. At topic creation time, for every topic partition, replication factor is converted to a replica set (set of ids of brokers which should replicate the partition), and those per partition replica sets is the metadata the gets stored in Zoo