On Fri, Jun 05, 2015 at 12:53:00AM -0400, Otis Gospodnetić wrote:
> Hi Joel,
>
> On Thu, Jun 4, 2015 at 8:52 PM, Joel Koshy wrote:
>
> > Hi Otis,
> >
> > Yes this is a limitation in the old consumer. i.e., a number of
> > per-topic/partition mbeans remain even on a rebalance. Those need to
> > b
Hi Joel,
On Thu, Jun 4, 2015 at 8:52 PM, Joel Koshy wrote:
> Hi Otis,
>
> Yes this is a limitation in the old consumer. i.e., a number of
> per-topic/partition mbeans remain even on a rebalance. Those need to
> be de-registered. So if you stop consuming from some partition after a
> rebalance th
Hi Jason,
On Thu, Jun 4, 2015 at 5:55 PM, Jason Rosenberg wrote:
> what code do you use to poll the jmx end-point?
I used https://github.com/sematext/jmxc to get a "dump" of Consumer JMX.
It's a simple jar that you call with java -jar jmxc.jar PID > jmx.dump .
Otis
> On Thu, Jun 4, 2015 at
I just sent out a separate email about the project that I've been working
on, Burrow, to change the way we're monitoring consumer status.
Like Joel said, the mbeans can be a little tricky to work with. Internally
at LinkedIn, we've always depended more on using a tool like the Consumer
Offset Chec
Hi Otis,
Yes this is a limitation in the old consumer. i.e., a number of
per-topic/partition mbeans remain even on a rebalance. Those need to
be de-registered. So if you stop consuming from some partition after a
rebalance that lag mbean currently remain which is why it remains
flat. This is a kn
On Thu, Jun 4, 2015 at 1:55 PM, Otis Gospodnetić wrote:
> Hi,
>
> On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds
> wrote:
>
> > I believe the JMX metrics reflect the consumer PRIOR to committing
> offsets
> > to Kafka / Zookeeper. But when you query from the command line using the
> > kafka tool
what code do you use to poll the jmx end-point?
On Thu, Jun 4, 2015 at 4:55 PM, Otis Gospodnetić wrote:
> Hi,
>
> On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds
> wrote:
>
> > I believe the JMX metrics reflect the consumer PRIOR to committing
> offsets
> > to Kafka / Zookeeper. But when you que
Hi,
On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds wrote:
> I believe the JMX metrics reflect the consumer PRIOR to committing offsets
> to Kafka / Zookeeper. But when you query from the command line using the
> kafka tools, you are just getting the committed offsets.
>
Even if that were the ca
I believe the JMX metrics reflect the consumer PRIOR to committing offsets
to Kafka / Zookeeper. But when you query from the command line using the
kafka tools, you are just getting the committed offsets.
On Thu, Jun 4, 2015 at 1:23 PM, Otis Gospodnetic wrote:
> Hi,
>
> Here's something potentia
Hi,
Here's something potentially useful.
1) Before: https://apps.sematext.com/spm-reports/s/eQ9WhLegW9 - the "flat
Lag situation"
2) I restarted the consumer whose lag is shown in the above graph
3) After restart: https://apps.sematext.com/spm-reports/s/4YGkcUP9ms - NO
lag at all!?
So that 815
Hi Jason,
(note: Kafka 0.8.2. GA)
Got some new info below! Could be a Kafka metrics bug
On Thu, Jun 4, 2015 at 2:11 PM, Jason Rosenberg wrote:
> I assume you are looking at a 'MaxLag' metric, which reports the worst case
> lag over a set of partitions.
No, we're looking at MBeans that lo
I assume you are looking at a 'MaxLag' metric, which reports the worst case
lag over a set of partitions. Are you consuming multiple partitions, and
maybe one of them is stuck?
On Tue, Jun 2, 2015 at 4:00 PM, Otis Gospodnetic wrote:
> Hi,
>
> I've noticed that when we restart our Kafka consumer
Hi,
I've noticed that when we restart our Kafka consumers our consumer lag
metric sometimes looks "weird".
Here's an example: https://apps.sematext.com/spm-reports/s/0Hq5zNb4hH
You can see lag go up around 15:00, when some consumers were restarted.
The "weird" thing is that the lag remains flat!
13 matches
Mail list logo
