Hi Sachin,
If you check kafka-run-class.bat you can see that when environment variable
KAFKA_LOG4J_OPTS is not provided, a default log4j configuration under
"tools" will be loaded. So setting the environment variable to something
like
Hi,
As suggested this is how I am starting my stream
D:\kafka_2.10-0.10.1.1>bin\windows\kafka-run-class.bat -Dlog4j.debug
-Dlog4j.configurationFile=file:///D:/kafka_2.10-0.10.1.1/log4js.properties
TestKafkaWindowStream
log4j: Using URL
[file:D:/kafka_2.10-0.10.1.1/config/tools-log4j.properties]
If you are using jmxterm then you are going to connect to a running jvm and
you don't need to set StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG. You
need to connect jmxterm to the MBean server that will be running in the jvm
of your streams app. You'll need to provide an appropriate jmx port for
Hi,
I understood what I need to do.
I think is not clear though regarding
StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG
Say I decide to use jmxterm which is cli based client which I can easily
use where my streams app is running.
With respect to that what value should I assign it to the
Hi Sachin,
You can configure an implementation of org.apache.kafka.common.Metrics.
This is done via StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG
There is a list of jmx reporters here:
https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
I'm sure their are plenty more available on
Hi,
Thanks for sharing the info.
I am reading this document for more understanding:
http://kafka.apache.org/documentation.html#monitoring
Is there any special way I need to start my kafka cluster or streams
application (or configure them) to report these metrics.
I suppose both cluster and
You should check out Kafka Streams Metrics (for upcoming 0.10.2 they are
even more detailed).
There is not a lot of documentation for 0.10.0 or 0.10.1, but it work
the same way as for consumer/producer metric that are documented.
-Matthias
On 1/24/17 10:38 PM, Sachin Mittal wrote:
> Hi All,
>
Hi All,
I am running a kafka streaming application with a simple pipeline of:
source topic -> group -> aggregate by key -> for each > save to a sink.
I source topic gets message at rate of 5000 - 1 messages per second.
During peak load we see the delay reaching to 3 million messages.
So I
