Re: Retrieve most-recent-n messages from kafka topic

Thanks Johan, I converted your code to vanilla java with a few small modifications (included below in case anyone wants to use it) and ran it a few times. Seems like it works ok for the quick peek use case, but I wouldn't recommend anyone rely on the accuracy of it since I find, at least in our c

Re: Plz help me

This is a known bug in Kafka 0.7.x (see the discussion in http://apache.markmail.org/thread/c7tdalfketpusqkg

Re: Flexible Producers and Consumers

We do have a console producer that allows you to pipe in log data in string format and a log4j Kafka appender. It would be great if people can contribute other types of adapters. Thanks, Jun On Tue, Jul 16, 2013 at 11:31 AM, Danny Patel wrote: > I've read through many sites regarding using Ka

Re: Monitoring with Riemann

Paul, Thanks for sharing. Jun On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles wrote: > Hi, > > Earlier today I finished work on a library that adds an additional metrics > reporter (https://github.com/pingles/kafka-riemann-reporter). I did it > because although we were writing the CSV metrics we

Re: Duplicate Messages on the Consumer

In 0.7.x, if the messages are compressed, there could be duplicated messages during consumer rebalance. This is because we can only checkpoint consumer offset at the compressed unit boundary. You may want to see if you have unnecessary rebalances (see https://cwiki.apache.org/confluence/display/KAF

Re: Topic corruption from hardware failure (0.7.1)

For #1, one potential cause is that someone was consuming a wrong topic name (e.g., datapoints-wr/ite-34). In 0.7, this will accidentally create a directory datapoints-wr in the kafka log root dir. The problem has been fixed in 0.8 by restricting the topic names to alpha-numeric plus "-" and "_".

Re: Kafka consumer not consuming events

For those partitions that are lagging, do you see fetch requests in the log? Thanks, Jun On Fri, Jul 19, 2013 at 12:30 AM, Nihit Purwar wrote: > Hello Jun, > > Sorry for the delay in getting the logs. > Here are the 3 logs from the 3 servers with trace level as suggested: > > > https://docs.g

Re: Message Encoding in Kafka 0.8

The following code indicates that you want to use "encode" as the key associated with the message. Is that your intend? If so, you need to specify StringEncoder as the "key.serializer.class". Otherwise, use the api that doesn't specify the key. val producerData = new KeyedMessage[String, Membe

Re: 0.8 throwing exception "Failed to find leader" and high-level consumer fails to make progress

Do you see ZK session expiration in the consumer and the broker log (search for Expired)? Also, could you try the 0.8 beta1 release? Thanks, Jun On Tue, Jul 16, 2013 at 11:56 AM, Hargett, Phil < phil.harg...@mirror-image.com> wrote: > Hmm > > This issue continues to emerge occasionally, al

Re: Sample Kafka Producer/Consumer example in Scala

What you did is correct. Could you try running it as a standalone app and not from scala REPL? Thanks, Jun On Tue, Jul 16, 2013 at 9:21 AM, prashant amar wrote: > Hello All > > I have been attempting to build an end-to-end example of producer with > multiple consumers using scala. > I managed

Re: Leader not local for partition error?

Any error/exception in state-change or controller log? Also, could you try the 0.8 beta1 release? Thanks, Jun On Mon, Jul 15, 2013 at 1:36 PM, Maxime Petazzoni wrote: > Hi all, > > I'm not sure if I'm doing something wrong or if I missed a step > somewhere. A little while ago I successfully g

Re: NoBrokersForPartitionException

Casey, I am not exactly sure what's causing this particular issue. However, a bunch of produce side issues got fixed in 0.7.2. Thanks, Jun On Mon, Jul 15, 2013 at 5:11 AM, Sybrandy, Casey < casey.sybra...@six3systems.com> wrote: > Jun, > > Unfortunately, upgrades are slow to occur with respec