Re: Manually update consumer offset stored in Kafka

I didn't find an off-the-shelf tool to do this when I needed to a few weeks ago (which was kind of surprising). I used the kafka-python library (my company's wrappers around it, actually) to pretend I was in the consumer group I wanted to update and called commit with the offsets I wanted. First

Re: Mantain only last message by key

+1. Sounds like what you want is a compacted topic. Check out this section of the docs: https://kafka.apache.org/documentation#compaction Keep in mind that compaction isn't free. Like a garbage collector, it comes with some overhead and some knobs for tuning. On Wed, Oct 19, 2016 at 2:46 PM, Rado

Re: Kafka performance on an ordinary machine

> As to what procedures I'd propose is spinning up a cluster and running some tests on it. This is really good advice. Capacity planning for your Kafka cluster is not particularly straightforward - especially if you intend to run a multi-tenant cluster. (If someone has a general model, please shar

Re: Leader became -1 and no ISR for all topics/patitions

Something similar happened to one of our clusters over the weekend. I'm still not sure how we got into that state. Running 0.9.0.1. Restarting the controller broker got us back on our feet. Slight difference is a few of our offline partitions had a single replica in the ISR list - but it wasn't th

Kafka Connect - Kinesis sink or source?

I have a couple of topics in a Kafka cluster that another team would like to process. They have a strong preference for consuming a Kinesis stream instead of a Kafka topic. This seems like a good fit for a Kafka Connect sink. I didn't see a Kinesis connector in the connector hub

Question about heterogeneous brokers in a cluster

Hi there, I have a couple of Kafka brokers and thinking about adding a few more. The new broker machines would have a lot more storage available to them than the existing brokers. Am I setting myself up for operational headaches by deploying a heterogeneous (in terms of storage capacity) cluster?

Re: Question about heterogeneous brokers in a cluster

ches is if you have different configuration > options > > per broker. Or if you try to assign more partitions to the newer brokers > to > > use more of their disk space. > > > > Let's see if others notice anything I'm missing (again, I've never tried &g