Re: querying messages based on timestamps

Yes, we have plan to add better support for this. Thanks, Jiangjie (Becket) Qin On 6/30/15, 4:23 PM, "Zaiming Shi" wrote: >Hi Jiangjie ! >Does the word 'Currently' imply any plan in introducing timestamp in log >entries? > >Regards >/Zaiming >On Jun 30, 2015 11:08 PM, "Jiangjie Qin" >wrote: >

Re: querying messages based on timestamps

We had similar requirement to re-load the data based on timestamp (range between 1PM to 2PM) etc. We store the relationship between timestamp and largest offset number in Time Series Database using jmxtrans (LogEndOffset JMX bean vs current time.). You can setup polling interval to be 60 minutes

Re: querying messages based on timestamps

Hi Jiangjie ! Does the word 'Currently' imply any plan in introducing timestamp in log entries? Regards /Zaiming On Jun 30, 2015 11:08 PM, "Jiangjie Qin" wrote: > Currently Kafka only have a very coarse solution to find offset by time, > which is based on the segment last modified time. > This i

Re: querying messages based on timestamps

Currently Kafka only have a very coarse solution to find offset by time, which is based on the segment last modified time. This interface is only available in simple consumer. You may issue an OffsetRequest to specify a timestamp. The offset returned will be the first offset of segments whose last

Re: querying messages based on timestamps

We faced similar problem and ended up with implementing variant of golden section search, that reads message using simple consumer and checks the timestamp (timestamps are appended by our producer though, they do not come from any Kafka metadata) till it finds message closest to given date. Adam

Re: Reimaging zookeeper host

Hey Rajasekar, Are you updating zookeeper itself or just the image? Either way, it's probably best to preserve the data if possible. Usually people update zookeeper using a rolling reboot to make sure no data is lost. You just have to make sure you give the rebooted host has enough time to rejoin

querying messages based on timestamps

Is it possible using low level consumer to get kafka messages based on timestamp, say I want to get all messages from last 5 minutes. I don't know what were offsets of partitions 5 minutes back. In low level consumer : when I gave epoch for whichTime , it failed. requestInfo.put(topicAndPartitio

Reimaging zookeeper host

We are running 3-node zookeeper cluster and we need to re-image (re-install os) on zookeeper host. Is it ok to lose zookeeper dataDir during upgrade or should back up zookeeper dataDir and restore when zookeeper comes backup online? Will kafka and consumers work fine if we bring up zookeeper with e

Re: Producer repeatedly locking up

Thanks for responding Gwen. There is sometimes an error in server.log, but it isn't 1 to 1 with the connection resets on the producer (although this is a socket from that box). : [2015-06-30 15:26:09,544] ERROR Closing socket for /10.0.6.244 because of error (kafka.network.Processor) java.io.IOEx

Re: Producer repeatedly locking up

Do you see any errors on the brokers when this happens? On Tue, Jun 30, 2015 at 10:14 AM, Shayne S wrote: > This problem is intermittent, not sure what is causing it. Some days > everything runs non-stop with no issues, some days I get the following. > > Setup: > - Single broker > - Running 0.8.2

Producer repeatedly locking up

This problem is intermittent, not sure what is causing it. Some days everything runs non-stop with no issues, some days I get the following. Setup: - Single broker - Running 0.8.2.1 I'm running a single broker. When the problem is presenting, anywhere from 5,000 to 30,000 messages may be processe

Re: no of threads per async producer

That is right. And to determine whether you need multiple producers or not you can check io-wait-ratio on the producer, if it is close to 0 that means the ioThread is busy sending and hence you need multiple instances. Guozhang On Tue, Jun 30, 2015 at 9:43 AM, Shushant Arora wrote: > In new jav

Re: no of threads per async producer

In new java Producer is there only one ioThread irrespective of no of brokers which send message from inmemory buffer to brokers sequentially? So if user need parallelism we should have multiple producer instances ? On Tue, Jun 30, 2015 at 10:02 PM, Guozhang Wang wrote: > ProducerSendThread and

Re: no of threads per async producer

ProducerSendThread and SyncProducer all come from the old Scala producer, I thought you meant that producer not the new Java producer? Guozhang On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora wrote: > According to code of org.apache.kafka.clients.producer.KafkaProducer. > > It has 1 ioThread >

Re: Subscribe kafka users mailing list

Hey Kashif, to subscribe, send a message to users-subscr...@kafka.apache.org . -Jason On Tue, Jun 30, 2015 at 1:16 AM, Kashif Hussain wrote: > Hi, > I want to subscribe Kafka users mailing list. > > Regards, > Kashif >

Re: no of threads per async producer

According to code of org.apache.kafka.clients.producer.KafkaProducer. It has 1 ioThread private final Thread ioThread; String

Re: no of threads per async producer

One ProducerSendThread is always the case for producer, but inside the thread there is one SyncProducer for each destination broker. I think that blog may mis-understand the design a bit. Guozhang On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora wrote: > According to > https://engineering.gnip.c

Subscribe kafka users mailing list

Hi, I want to subscribe Kafka users mailing list. Regards, Kashif

Re: How to monitor consuming rate and lag?

There is also: http://quantifind.com/KafkaOffsetMonitor/ , but there is no monitoring support out of box. 2015-06-30 13:12 GMT+02:00 noah : > If you are committing offsets to Kafka, try Burrow: > https://github.com/linkedin/Burrow > > On Tue, Jun 30, 2015 at 3:41 AM Shady Xu wrote: > > > Hi all,

Re: How to monitor consuming rate and lag?

If you are committing offsets to Kafka, try Burrow: https://github.com/linkedin/Burrow On Tue, Jun 30, 2015 at 3:41 AM Shady Xu wrote: > Hi all, > > I'm now using https://github.com/airbnb/kafka-statsd-metrics2 to monitor > our Kafka cluster. But there are not metrics about consuming rate and la

Re: no of threads per async producer

According to https://engineering.gnip.com/kafka-async-producer/ There is one ProducerSendThread per broker. Is it changed from 0.8.2 ? to use single send thread irrespective of broker or partitions. On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang wrote: > Hi Shushant, > > The number of Produce

How to monitor consuming rate and lag?

Hi all, I'm now using https://github.com/airbnb/kafka-statsd-metrics2 to monitor our Kafka cluster. But there are not metrics about consuming rate and lag, which are key performance metrics we care about. So how do you guys monitor consuming rate and lag of each consumer group?