I've got a very strange bug doing a long poll using SimpleConsumer on 0.8.1.1.
The test app I've got (attached) uses a thread to do a long poll of a topic,
and then sends single messages to that queue, one at a time, waiting for the
poll to return the message before sending the next one.
The pr
In the output of kafka-topics.sh -describe, it gives the ISR. What I want to
find out is why a particular broker is not in ISR, how far behind it is, and
when it will get back into the ISR. I can't seem to find this information in
the logs, cmdline output, or zookeeper. How can I find out how fa
I'm trying to use the SimpleConsumer to consume a queue, using long polling.
I'm encountering some difficulties when using very long poll times (60s or
more). It seems to be conflicting with the socket timeout.
I thought the socket timeout was the timeout to actually connect to the broker,
then
Hi,
I've been looking at the SimpleConsumer example, and that I've noticed that it
always reads from the leader, and reacts to leader changes by reconnecting to
the new leader. Is it possible to read from a replica in ISR that's not the
leader? If so, how does the consumer get notified the repl
I'm updating the kafka APIs we use to the new standalone ones, but it look like
the new consumer isn't ready yet (the code has got lots of placeholders etc),
and there's only the producer in the Javadoc at
http://kafka.apache.org/082/javadoc/index.html. Is there an ETA on when the new
consumer
Reading on the consumer docs, there's no mention of a relatively simple
consumer that doesn't need groups, coordinators, commits, anything like that -
just read and poll from specified offsets of specific topic partitions - but
automatically deals with leadership changes and connection losses (s
Hi,
We've had an issue on a live system (3 brokers, ~10 topics, some replicated,
some partitioned) where a partition wasn't properly reassigned, causing several
other partitions to go down.
First, this exception happened on broker 1 (we weren't doing anything
particular on the system at the ti
Hi,
I'm having significant problems getting the kafka consumer to operate
deterministically with small message numbers & sizes (this is for local
testing).
I'm controlling the offset manually, and using manual partition/topic
assignment. I've set auto commit off, and set fetch.min.bytes to 1.
Forgot to mention - this is on 0.9. We can't upgrade to 0.10 yet, as we haven't
upgraded our brokers.
-Original Message-----
From: Simon Cooper [mailto:simon.coo...@featurespace.co.uk]
Sent: 22 May 2017 16:05
To: users@kafka.apache.org
Subject: Getting the consumer
auses!
Thanks,
SimonC
-Original Message-
From: Simon Cooper [mailto:simon.coo...@featurespace.co.uk]
Sent: 17 August 2015 12:31
To: users@kafka.apache.org
Subject: Topic partitions randomly failed on live system
Hi,
We've had an issue on a live system (3 brokers, ~10 topics, some
Is there a basic interface in the new client APIs to get the list of topics on
a cluster, and get information on the topics (offsets, sizes, etc), without
having to deal with a producer or consumer? I just want a basic synchronous API
to query the metadata as-is. Does this exist in some form?
T
Hi all,
In the kafka docs, it mentions that MirrorMaker will automatically create
topics on the mirror cluster if the right options are set. Does it
automatically delete topics on the mirror that are deleted on the main cluster
as well?
Thanks,
SimonC
a.
Kind regards,
Stevo Slavic.
On Tue, Jan 5, 2016 at 2:53 PM, Simon Cooper < simon.coo...@featurespace.co.uk>
wrote:
> Hi all,
>
> In the kafka docs, it mentions that MirrorMaker will automatically
> create topics on the mirror cluster if the right options are set. Does
&g
Hi,
We've just updated to the 0.9 client & broker, and we're suddenly seeing a lot
of log spam in the consumers:
2016-02-05T03:31:34,182 | INFO | o.a.k.c.c.i.AbstractCoordinator
[kafkaspout-thread-0] | Response-events:7 | Marking the coordinator 2147483646
dead.
2016-02-05T03:31:34,182 | INFO
Actually, this is incorrect - it looks like the consumer does not receive any
messages until the *first* 'coordinator dead' message is logged!
Is anyone able to offer insight into what's going on here?
Simon
-Original Message-----
From: Simon Cooper [mailto:simon.coo...@fea
g the coordinator dead
Hi Simon,
It may be worth trying the 0.9.0 branch as it includes a number of important
fixes to the new consumer.
Ismael
On Fri, Feb 5, 2016 at 12:33 PM, Simon Cooper <
simon.coo...@featurespace.co.uk> wrote:
> Actually, this is incorrect - it looks like the con
Hi all,
We've got a problem with high CPU usage on a 0.9 client. We've got a monitoring
system that polls kafka topics for metadata (to get the last message offset)
every so often, and this has started using very high CPU continuously. We're
seeing the following being spammed in the logs every
tion by the log cleaner?
Thanks for the help!
Simon Cooper
Thanks, that's answered all my questions!
Simon
-Original Message-
From: Gwen Shapira
Sent: 13 June 2018 02:42
To: Users
Subject: Re: Details of segment deletion
See below:
On Mon, Jun 11, 2018 at 3:36 AM, Simon Cooper <
simon.coo...@featurespace.co.uk> wrote:
> Hi
19 matches
Mail list logo