Re: Kafka error - append operation on partition

I went through the logs segment of the topic and i see that the data from the producerID still exists in the topic and the new messages received are well with in the retention period ( 336 hr). Starting offset: 359 offset: 359 position: 0 CreateTime: 1541667838856 isvalid: true keysize: 39 valuesi

Kafka error - append operation on partition

Hi Users, Apologies if this question has been asked before but i have not been able to find the answer. I am seeing this error in the Kafka logs : [ReplicaManager broker=1001] Error processing append operation on partition TABLE-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.Unk

Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3

On 2018/11/08 00:13:39, "Matthias J. Sax" wrote: > That is what I try to figure out. I went over the 0.10.2.2 to 0.11.0.3 > Jiras but found nothing I could point out. There are couple of > SessionStore related tickets, but none of them should have an effect > like this. > > To narrow it down, it

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Yes, I understand we need to specify the required broker.id in server.properties/meta.properties file in-order to retain the broker.id which was failed. We can write custom script to query the zookeeper on launch process to get the broker.id needs to be set. Do we have any standards/script/way de

Re: Remove kafka message from Queue

Check out the kafka-consumer-groups command line tool. You can set the current offsets for any inactive consumer group to whatever value you like using that tool. I think that'll do what you want. On Thu, Nov 15, 2018 at 12:17 PM Parth Gandhi < parth.gan...@excellenceinfonet.com> wrote: > Hi Team

Remove kafka message from Queue

Hi Team, We have implemented kafka and would like to know if there is a way to remove kafka message stuck in a Queue and commit offset using the kafka command line. Or has it to be done using the consumer API. Thanks, Parth Gandhi DevOps Disclaimer The information contained in this communicatio

Re: [VOTE] 2.1.0 RC1

Hi Dong, Thanks for running the release. I built binaries from the source, ran quickstart and tests on Ubuntu (with Java 8 & 9). I noticed two issues: * (minor) the documentation still mentions 2.0, and submitted a minor PR to update it. * the unit tes

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Srinivas, You might want to check out the AWS best practices from this blog post: https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/ Kafka broker ids by default are auto-generated, but you can also specify values (in server.properties file). By h

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

If at all you were hell bent on doing this, you could use zookeeper to find out the health of current brokers along with their broker id. That should help re-spin/start the unhealthy instance with same instance id. On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu wrote: > having all this stored i

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

having all this stored in DB is getting too complicated, especially with instance level storage and not EBS. I am sure there should be easy way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated. On Thu, Nov 15, 2018 at 2:27 AM Eno

Re: kafka.common.StateChangeFailedException: Failed to elect leader for partition XXX under strategy PreferredReplicaPartitionLeaderElectionStrategy

Adding Kafka Controller Log. [2018-11-15 11:19:23,985] ERROR [Controller id=4 epoch=8] Controller 4 epoch 8 failed to change state for partition XYXY-24 from OnlinePartition to OnlinePartition (state.change.logger) On Thu, Nov 15, 2018 at 5:12 PM SenthilKumar K wrote: > Hello Kafka Experts, >

kafka.common.StateChangeFailedException: Failed to elect leader for partition XXX under strategy PreferredReplicaPartitionLeaderElectionStrategy

Hello Kafka Experts, We are facing StateChange Failed Exception on one of the broker. Out of 4 brokers, 3 were running fine and only one broker is throwing state change error. I dont find any error on Zookeeper Logs related to this error. Kafka Version : kafka_2.11-1.1.0 Any input would

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

The general answer depends on what control plane software is taking care of your Kafka deployment. You probably have a layer that launches Kafka instances and monitors their health, right? If so, that layer should take care of the mapping between instances and broker IDs and keep that in a table pe