Re: Building API to make Kafka reactive

Dean, Thanks a lot for the link. I am going through the documentation. - Shekar On Wed, Jun 29, 2016 at 9:50 AM, Dean Wampler wrote: > Here's another Reactive API: https://github.com/akka/reactive-kafka > > It was developed by Software Mill and it's now > being inte

sarama and __consumer_offsets rebalancing

Hi there, I'm trying to track down why __consumer_offsets is never balancing and what to do about it. We're using Shopify's sarama client, which explicitly states that it doesn't do rebalancing for internal topics ... ever. It seems like sarama is

rate-limiting on rebalancing, or sync from non-leaders?

Hi there, I'm curious if there's anything on the Kafka roadmap for adding rate-limiting or max-throughput for rebalancing processes. Alternately, if you have RF>2, maybe a setting to instruct followers to sync from other followers? I'm super impressed with how fast and efficient the kafka data r

Re: kafka + autoscaling groups fuckery

Gwen, thanks for the response. 1.1 Your life may be a bit simpler if you have a way of starting a new > broker with the same ID as the old one - this means it will > automatically pick up the old replicas and you won't need to > rebalance. Makes life slightly easier in some cases. > Yeah, this i

Re: My Use case is I want to delete the records instantly after consuming them

Hi All Is the number of consumer component equal to the number of partitions created in cluster ? I have created three partitions in cluster but I am using only two consumer poller to subscribe the records. Some time I have noticed that the messages are polled very late. What should be the good po

Re: My Use case is I want to delete the records instantly after consuming them

Thank you so much Ian Thanks and Regards, Navneet Kumar On Sat, Jul 2, 2016 at 9:45 PM, Ian Wrigley wrote: > That’s really not what Kafka was designed to do. You can set a short log > retention period, which will mean messages are deleted relatively soon after > they were written to Kafka,

Re: My Use case is I want to delete the records instantly after consuming them

That’s really not what Kafka was designed to do. You can set a short log retention period, which will mean messages are deleted relatively soon after they were written to Kafka, but there’s no mechanism for deleting records on consumption. Ian. --- Ian Wrigley Director, Education Services Con

My Use case is I want to delete the records instantly after consuming them

Hi All My Use case is I want to delete the records instantly after consuming them. I am using Kafka 0.90 Thanks and Regards, Navneet Kumar

Re: Brokers are crash due to __consumer_offsets folder are deleted

Hi Peter, The server is not restart. Anyway I will change log.dir, and will monitor it again. Thanks for your help. Best Regards, Ben Peter Davis 于2016年7月2日周六 下午12:15写道： > Dear 黄杰斌: > > I am guessing your operating system is configured to delete your /tmp > directory when you restart the server

Re: Brokers are crash due to __consumer_offsets folder are deleted

/var/log would be a better default. Dave > On Jul 2, 2016, at 07:09, Ismael Juma wrote: > > Hi Peter, > > It's a good question why `log.dir` defaults to `/tmp`. I assume it's to > make it easier for people to get started with Kafka, but unsafe defaults > should be avoided as much as possible in

Re: Brokers are crash due to __consumer_offsets folder are deleted

Hi Peter, It's a good question why `log.dir` defaults to `/tmp`. I assume it's to make it easier for people to get started with Kafka, but unsafe defaults should be avoided as much as possible in my opinion. Ismael On Sat, Jul 2, 2016 at 5:15 AM, Peter Davis wrote: > > Dear Community: why does