How to monitor lag when "kafka" is used as offset.storage?

Hi, I wonder how we can monitor lag (difference between consumer offset and log ) when "kafka" is set as offset.storage? because the "kafka-run-class.sh kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper is used as storage manager. best, /Shahab

Re: 0.8.2 producer and single message requests

Yes, this will be a problem if you are providing batching for your REST service on top of Kafka and have to acknowledge to your client only when all the callbacks for individual sends are called. Here is one implementation I have done:

Re: Question regarding to reconnect.backoff.ms

Would kafka dev kindly give us some advice on this? Cheers, Steve On Tue, Sep 1, 2015, 11:20 PM Steve Tian wrote: > Thanks, Rahul! In my environment I need to have reconnect.backoff.ms > longer than OS default tcp timeout so that NetworkClient can give second > node a

Re: KafkaSpout throwing OutOfMemoryError

You need to allocate extra memory for the topology to run. On Wed, Sep 2, 2015 at 11:36 PM, Khalasi, Vipul Kantibhai < vipul.kantibhai.khal...@citi.com> wrote: > Hi , > > > > I am using kafkaspout in my topology ann each kafka topic have 8 > partitions and all topic atleast contains 1GB of data.

Recovery skipped after unclean shutdown

One of the brokers in our cluster had an unclean shutdown and after it was restated I found the following logs. $ grep "clean shutdown" /var/groupon/kafka/kafka-broker.log 02/Sep/2015 16:19:23 - warn::[Kafka Server 1], Proceeding to do an unclean shutdown as all the controlled shutdown attempts

Re: Question regarding to reconnect.backoff.ms

Steve, I don't think there is a better solution at the moment. This is an easy issue to miss in unit testing because generally connections to localhost will be rejected immediately if there isn't anything listening on the port. If you're running in an environment where this happens normally, then

Re: Question regarding to reconnect.backoff.ms

Got it. Thanks a lot Ewen! Cheers, Steve On Thu, Sep 3, 2015, 10:06 AM Ewen Cheslack-Postava wrote: > Steve, > > I don't think there is a better solution at the moment. This is an easy > issue to miss in unit testing because generally connections to localhost > will be

Re: java.lang.NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics

It is a NoClassDefFoundError so the clinit of Metrics class is failing. Please look for surrounding stack traces, there must be a root cause for why the class initialization failed. On Wed, Sep 2, 2015 at 1:05 PM, Vadim Keylis wrote: > I suddenly started getting this

Re: Kafka

Muqtafi, There are corresponding Java or Scala classes. You can use them directly, but beware that they are not considered public interfaces, so there are no promises about compatibility. They could completely change between releases. (The command line tools themselves, however, are considered

Kafka

hello all, I've read that Kafka bundled with admin/system tools that can be accessed using console ( https://cwiki.apache.org/confluence/display/KAFKA/System+Tools). Is there any way to call these functions from Java code? Is there any Java API for admin/system functions? Thank you, Regards,

java.lang.NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics

I suddenly started getting this error using the code that was perfectly working a day before. The library containing yammer metrics exists and part of mvn build. Please help how to resolve the issue or disable yammer.metrics. Thanks in advance Exception in thread "Thread-3"

KafkaSpout throwing OutOfMemoryError

Hi , I am using kafkaspout in my topology ann each kafka topic have 8 partitions and all topic atleast contains 1GB of data. java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at

Delay in Zookeeper Offset updates

Hi All, i have run into a weird issue with my Kafka setup. I see that it takes around 5-6 sec for Zookeeper to update the offset for Kafka topics. I am running "ConsumerOffsetChecker" tool to see that lag and what i found that even when my consumer is not up, it takes 5-6 sec for Zookeeper to

Re: How to monitor lag when "kafka" is used as offset.storage?

We use Burrow . There are rest endpoints you can use to get offsets and manually calculate lag, but if you are focused on alerting, I'd use it's consumer statuses as they are a bit smarter than a simple lag calculation. On Wed, Sep 2, 2015 at 4:08 AM shahab

Re: Kafka loses data after one broker reboot

Hi, Lukas. I don't think so because we lost data with offsets which are larger than exist offsets. But retention should delete offsets less than exist offsets. Hi, Gwen. Thanks a lot. We will sleep on it. On Wed, Sep 2, 2015 at 1:07 AM, Gwen Shapira wrote: > There is