Roger Hoover created SAMZA-503:
----------------------------------
Summary: Lag gauge very slow to update for slow jobs
Key: SAMZA-503
URL: https://issues.apache.org/jira/browse/SAMZA-503
Project: Samza
Issue Type: Bug
Components: metrics
Affects Versions: 0.8.0
Environment: Mac OS X, Oracle Java 7, ProcessJobFactory
Reporter: Roger Hoover
For slow jobs, the
KafkaSystemConsumerMetrics.%s-%s-messages-behind-high-watermark) gauge does not
get updated very often.
Here's my test setup. I created a job that processes a single message and
sleep 5 seconds. In another shell, I have another process loading 1000
messages every second to the input topic.
To reproduce:
* Create a job that processes one message and sleeps for 5 seconds
* Create it's input topic but do not populate it yet
* Start the job
* Load 1000s of messages to it's input topic. You can keep adding messages
with a "wait -n 1 <kafka console producer command>"
What happens:
* Run jconsole to view the JMX metrics
* The %s-%s-messages-behind-high-watermark gauge will stay at 0 for a LONG time
(~10 minutes?) before finally updating.
What should happen:
* The gauge should get updated at a reasonable interval (a least every few
seconds)
I think what's happening is that the BrokerProxy only updates the high
watermark when a consumer is ready for more messages. When the job is so slow,
this rarely happens to the metric doesn't get updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
