Hi Sachin,
If you check kafka-run-class.bat you can see that when environment variable
KAFKA_LOG4J_OPTS is not provided, a default log4j configuration under
"tools" will be loaded. So setting the environment variable to something
like
"-Dlog4j.configurationFile=file:///D:/kafka_2.10-0.10.1.1/log4
Hi,
As suggested this is how I am starting my stream
D:\kafka_2.10-0.10.1.1>bin\windows\kafka-run-class.bat -Dlog4j.debug
-Dlog4j.configurationFile=file:///D:/kafka_2.10-0.10.1.1/log4js.properties
TestKafkaWindowStream
log4j: Using URL
[file:D:/kafka_2.10-0.10.1.1/config/tools-log4j.properties] fo
If you are using jmxterm then you are going to connect to a running jvm and
you don't need to set StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG. You
need to connect jmxterm to the MBean server that will be running in the jvm
of your streams app. You'll need to provide an appropriate jmx port for it
Hi,
I understood what I need to do.
I think is not clear though regarding
StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG
Say I decide to use jmxterm which is cli based client which I can easily
use where my streams app is running.
With respect to that what value should I assign it to the
METRICS_RE
Hi Sachin,
You can configure an implementation of org.apache.kafka.common.Metrics.
This is done via StreamsConfig.METRICS_REPORTER_CLASSES_CONFIG
There is a list of jmx reporters here:
https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
I'm sure their are plenty more available on gith
Hi,
Thanks for sharing the info.
I am reading this document for more understanding:
http://kafka.apache.org/documentation.html#monitoring
Is there any special way I need to start my kafka cluster or streams
application (or configure them) to report these metrics.
I suppose both cluster and strea
You should check out Kafka Streams Metrics (for upcoming 0.10.2 they are
even more detailed).
There is not a lot of documentation for 0.10.0 or 0.10.1, but it work
the same way as for consumer/producer metric that are documented.
-Matthias
On 1/24/17 10:38 PM, Sachin Mittal wrote:
> Hi All,
> I
Hi All,
I am running a kafka streaming application with a simple pipeline of:
source topic -> group -> aggregate by key -> for each > save to a sink.
I source topic gets message at rate of 5000 - 1 messages per second.
During peak load we see the delay reaching to 3 million messages.
So I nee
