Re: Partition reassignment stucked

Are you using the 0.8.0 final release? Any error in controller log? Thanks, Jun On Fri, Dec 6, 2013 at 4:38 PM, Maxime Nay wrote: > Hi, > > We are trying to add a broker to a 10 node cluster. We have 7 different > topics, each of them is divided in 10 partitions, and their replication > facto

Partition reassignment stucked

Hi, We are trying to add a broker to a 10 node cluster. We have 7 different topics, each of them is divided in 10 partitions, and their replication factor is 3. To send traffic to this new node, we tried the kafka-reassign-partitions.sh tool, but for some reason, it doesnt work, and now it seems

Re: Consuming "backwards"?

yes, also you can read backwards on the stream if you do this in the consumer val maoList: Iterable[MessageAndOffset] = for(messageAndOffset <- messageSet if(numMessagesConsumed < maxMessages)) yield messageAndOffset for(messageAndOffset <- maoList.toList.reverse) { this way every read is the la

Re: Consuming "backwards"?

Right. If you're not reading contiguously, you need to remember the ranges that you have/haven't read. As long as you do that, it all works out, I think. A kafka client always has to store the last offset it read. In the simplest "both directions" case where you start with current and read in bo

Re: Consuming "backwards"?

you have to allow the fetchSize to be variable so in your example since the new highwatermark is 12 and the last consumsed message is 10 fetchSize = if (highwatermark - lastConsumedOffset < 3) highwatermark - lastConsumedOffset else 3 the real trick though is missing messages having to keep track

Re: Consuming "backwards"?

Hi, On Fri, Dec 6, 2013 at 6:32 PM, Steven Parkes wrote: > On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic > wrote: > > > but I think the > > problem is that each time we grab we could get some of the same messages > we > > already processed > > Doesn't setting the fetchSize to "how far back we ne

Re: Consuming "backwards"?

On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic wrote: > but I think the > problem is that each time we grab we could get some of the same messages we > already processed Doesn't setting the fetchSize to "how far back we need to grab" handle that?

Re: Is there a way to get the offset of a consumer of a topic?

thank you all for your help and suggestion. The ConsumerOffsetChecker is exactly what I needed! thanks again! On Wed, Dec 4, 2013 at 1:03 PM, Philip O'Toole wrote: > Simple tool I wrote to monitor 0.7 consumers. > > https://github.com/otoolep/stormkafkamon > > > On Wed, Dec 4, 2013 at 12:49 P

Re: Consuming "backwards"?

Hi, We discussed various suggestions people made here (thank you all, really!) and thoughts about Joe's offset suggestion. If we understood the suggestion I don't think that will work (well). If would work in the sense that we can, by asking broker for the latest offset before each consume reque

Re: kafka.common.NotLeaderForPartitionException when 1 broker of 2 is down

Leo, Could you attach the state-change log here? Guozhang On Thu, Dec 5, 2013 at 9:24 PM, Jun Rao wrote: > Any error in the controller and state-change log? Also, 0.8.0 final is just > released. Recommend that you try that version if possible. > > Thanks, > > Jun > > > On Thu, Dec 5, 2013 at

Re: Consuming "backwards"?

I think TTL/expiration to another queue might work, depending on your time window tolerance for "recency". JMS queues have this (to a dead letter queue but you could use it differently) but I'm not sure if Kafka does. If you need a fast (though not as great as Kafka) tradition MQ, there are a few

Re: Consuming "backwards"?

Correct. New data needs to get in first. Old data can get back willed slowly. How quickly old data gets backfilled depends on how fast Consumer is relative to Producer. If Consumer if faster (it better be!) it will always consume the latest N messages, but each time some portion of those N would be

Re: Consuming "backwards"?

This functionality is parallel to the stream being consumed into an indexed store: it serves a different point in the latency/completeness/fault tolerance space. It's actually combined with skipping forward and back, to prioritize new over old data, much as Otis describes, I think. On Dec 6, 2

Re: Consuming "backwards"?

If there is a TTL/expiration hook, that's another possibility as well. Add a hook that when a message times out, it gets put onto another "low priority" queue. Or something along those lines. So the main queue only has recent messages to consume. -Suren On Fri, Dec 6, 2013 at 11:11 AM, Surend

Re: Consuming "backwards"?

Depending on your exact requirements, you may consider priority queues as well. I'm not sure if Kafka has this (without rolling your own) but it is in JMS implementations. I guess you aren't looking for strict LIFO order, since messages are constantly being put on the queue, so "LIFO" is only with

Re: Consumer not able to create message streams

I don't see createMessageStreams in the thread dump though. Are you sure it's stuck there? Thanks, Jun On Fri, Dec 6, 2013 at 1:58 AM, Tarang Dawer wrote: > Hi Jun > please find the consumer thread dump in my previous reply. > > > On Fri, Dec 6, 2013 at 3:27 PM, Tarang Dawer > wrote: > > > F

Re: Consuming "backwards"?

Steven, you might be better off reading the Kafka stream into Cassandra and then doing the reads that way /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Consuming "backwards"?

I've figured I may end up doing something like this for reasons that sound similar to what Otis describes. In my case, I may end up binary searching the log to match msg #s to dates in the actual messages to find a particular time range of interest. So far it's just theory: I haven't gotten to

Re: Consuming "backwards"?

well if your going to-do that we should try to contact the Asgard and see if their time dilation device can run in reverse =8^) #stargate hehehe but in all seriousness I think my last suggestion would work with the fetch offset starting at the end and (fetch offset - fetch size), fetch size and t

Re: Consuming "backwards"?

Take apart the hard disk, and flip the magnets in the motors so it spins in reverse. The Kafka software won't be any the wiser. That should give you exactly what you need, combined with high-performance sequential reads. :-D > On Dec 6, 2013, at 7:43 AM, Joe Stein wrote: > > hmmm, I just rea

Re: Consuming "backwards"?

You got the hamster on the wheel with this one :) So one way to make it work without any changes (or at least maybe very minor changes if at all) would possibly be to use your max offset and fetch size this way M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 to get get last 3: M10 M11 M12 get last 3: M7

Re: Consuming "backwards"?

hmmm, I just realized that wouldn't work actually (starting at the end is fine)... the fetch size being taken in is still going to increment forward ... The KafkaApi would have to change because in readMessageSet it is doing a log.read of the FileMessageSet ... it should be possible though but no

Re: Consuming "backwards"?

The fetch requests are very flexible to-do what you want with them. Take a look at SimpleConsumerShell.scala as a reference You could pass in OffsetRequest.LatestTime (-1) with a fetch size of 3 and then just keep doing that over and over again. I think that will do exactly what you are looking

Re: Consuming "backwards"?

Hi, On Fri, Dec 6, 2013 at 9:38 AM, Tom Brown wrote: > Do you mean you want to start from the most recent data and go backwards to > the oldest data, or that you want to start with old data and consume > forwards? > Forwards is the "normal way". I'm looking for the "abnormal way", of course ;)

Re: Consuming "backwards"?

Do you mean you want to start from the most recent data and go backwards to the oldest data, or that you want to start with old data and consume forwards? If the former, it would be difficult or impossible in 0.7.x, but I think doable in 0.8.x. (They added some sort of message index). If the latte

Consuming "backwards"?

Hi, Does Kafka offer a way to consume messages in batches, but "from the end"? This would be valuable to have in all systems where the most recent data is a lot more important than older data, such as performance metrics, and maybe even logsmaybe also trading/financial data, and such. Any ch

Re: Trade-off between topics and partitions?

Jun's advice is exactly what the FAQ says but maybe it should be rephrased because it is true this question comes up once in a while. Your requirement is: user must receive published messages in order. All messages published to a partition will be stored in order. A consumer reading a partition w

Re: Updated kafka client (producer and consumer)

Produce requests in 0.7 are fire-and-forget since there is no acknowledgements being returned by the broker. 0.8 fixes this by the optional use of required.acks. I dont know if a non-JVM client is relevant for your needs, but librdkafka implements all of your listed requirements for 0.8: 1. Messa

Re: Consumer not able to create message streams

Hi Jun please find the consumer thread dump in my previous reply. On Fri, Dec 6, 2013 at 3:27 PM, Tarang Dawer wrote: > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03 mixed mode): > > "DestroyJavaVM" prio=10 tid=0x7feb18006800 nid=0xcc2 waiting on > condition [0x00

Re: Consumer not able to create message streams

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03 mixed mode): "DestroyJavaVM" prio=10 tid=0x7feb18006800 nid=0xcc2 waiting on condition [0x] java.lang.Thread.State: RUNNABLE "consumerGroup675437781_impetus-d898-1386323237902-426eefe2_watcher_executor" prio=10 ti