Re: New kafka spout

1. only advancing the offset when tuple is successfully ack’d Looks like it commits the offsets for all messages read from queue, not just the successful ones Because kafka only exposes a single commitOffsets() which indeed commits everything consumed through the consumer, the spout is batch ba

Re: New kafka spout

I had the same questions and came to the same conclusions (also just a cursory look over the code). That’s why the storm-kafka implementation uses Kafka’s low-level API — so it can keep track of the offsets and not have to store messages in memory. Our implementation *does* keep a batch of mess

Re: New kafka spout

Yes the messages in memory would be lost. But let's say another instance of the Kafka Spout starts on another worker, it would start a new kafka consumer process and so it could read from the last committed offset. Since when the 1st worker crashed no offset was committed for the messages in memor

Re: Kafka high level consumer in storm

Is there any downside to using Kafka high level consumer as spout? The main downside of the high level consumer is that you won't be able to control exactly when it will request a broker for more data and that it will always commit the latest offset you read from the stream it provides. In a so

Re: [VOTE] Storm Logo Contest - Final Round

#9: 2 points #10: 3 points

New kafka spout

ee to send pull requests, experiences, issues or performance numbers. Kind regards, Mattijs Ugen Netherlands Forensic Institute

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

de, I'll go find someone who would be able to say something on the matter from the NFI's point of view. The license should be compatible with such a step I think, I'll get back to you on the rest of this when I know more. Kind regards, Mattijs Ugen -- Netherlands Forensic Institute

Re: kafka high level consumer - threads guaranteed to read a single partition?

> For the kafka high level consumer, if I create exactly the number of > threads as the number of partitions, is there a guarantee that each > thread will be the only thread that reads from a particular partition? > I'm following this example >