Re: Official Kafka Disaster Recovery is insufficient - Suggestions needed

Sorry to have misspelled your name Henning. On Mon, Sep 3, 2018, 1:26 PM Ryanne Dolan wrote: > Hanning, > > In mission-critical (and indeed GDPR-related) applications, I've ETL'd > Kafka to a secondary store e.g. HDFS, and built tooling around recovering > state back into Kafka. I've had

Re: Official Kafka Disaster Recovery is insufficient - Suggestions needed

Hanning, In mission-critical (and indeed GDPR-related) applications, I've ETL'd Kafka to a secondary store e.g. HDFS, and built tooling around recovering state back into Kafka. I've had situations where data is accidentally or incorrectly ingested into Kafka, causing downstream systems to process

KAFKA-7093 - Warn Messages in Kafka 1.1.0

Hi I am also observing lot of logs as discussed in KAFKA-7093 . Anyone any thoughs? What does this denote? What does it effect and how to recover from this? Thanks,

After stopping and restarting Kafka Broker ISR doesn't show ISR doesn't show newly started broker

Hi All, I have 3 node cluster running Kafka in container environment. we have RF=3 when we stop any broker we could see updated ISR with running brokers, but when we restart broker i could see zookeeper log that broker is up and connected, however ISR don't reflect that broker with recently

Official Kafka Disaster Recovery is insufficient - Suggestions needed

I am looking for advice on how to handle disasters not covered by the official methods of replication, whether intra-cluster replication (via replication factor and producer acks) or multi-cluster replication (using Confluent Replicator). We are looking into using Kafka not only as a broker

Re: resetting consumer group offset to earliest and to-latest not working

After checking at work, it seems to work fine. The resetting would fail because of other consumers joining the group due to deployment config errors and i would afterwards check that everything is fine using the incorrect command, forgetting the "-execute". Thank you On Sat, 1 Sep 2018 at

Best way to manage topics using Java API

Hi, I'm currently using *kafka_2.11* *1.0.0***and I want to list, create, delete topics directly in from my Java code. When searching on stackoverflow, all examples are using *AdminUtils *(kafka.admin.AdminUtils) which is not really documented. I found in the documentation that I should

Could not find or load main class kafka.Kafka (Eclipse Startup)

Hello, I completed set up according to cwiki instruction - https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup I have also built all the sources and jars using `gradlew jarAll` After that, I created a Scala application configuration in Eclipse to debug. I provided the sources to

Re: Determining broker max buffer size from producer.

Are you asking about the maximum message size that can be sent from a producer? max.message.bytes is used to control that at broker/topic level. You can find more details at the below links. *"The largest record batch size allowed by Kafka. If this is increased and there are consumers older than

Re: After stopping and restarting Kafka Broker ISR doesn't show ISR doesn't show newly started broker

Hi Suresh, Can you please share your startup logs, and broker/zookeeper config with us? Also, which kafka version are you using? Regards, On Mon, 3 Sep 2018 at 10:18, suresh.sar...@gmail.com < suresh.sar...@gmail.com> wrote: > Hi All, > > I have 3 node cluster running Kafka in container

After stopping and restarting Kafka Broker ISR doesn't show ISR doesn't show newly started broker

Hi All, I have 3 node cluster running Kafka in container environment. we have RF=3 when we stop any broker we could see updated ISR with running brokers, but when we restart broker i could see zookeeper log that broker is up and connected, however ISR don't reflect that broker with recently

Determining broker max buffer size from producer.

Greetings, I have spent several hours looking through documentation and historical email chains and have been unable to find a solution to my problem. I have a case where I need to construct a producer and I need to know the broker max buffer size so that I don't get any nasty buffer too big