Reliability against rack failure

2018-08-05 Thread Sanjay Awatramani
Hi, I have done some experiments and gone through kafka documentation, which makes me conclude that there is a small chance of data loss or availability in a rack scenario. Can someone please validate my understanding ? The minimum configuration for a single rack system against single machine

Re: Reliability against rack failure

2018-08-05 Thread Svante Karlsson
3 racks, Replication Factor = 3, min.insync.replicas=2, ack=all 2018-08-05 20:21 GMT+02:00 Sanjay Awatramani : > Hi, > > I have done some experiments and gone through kafka documentation, which > makes me conclude that there is a small chance of data loss or availability > in a rack scenario. Ca

Re: Reliability against rack failure

2018-08-05 Thread Sanjay Awatramani
Thanks for the quick response Svante. I forgot to mention that the deployment I am looking at has 2 racks. We came up with this solution, but for this specific deployment adding a rack is out of question. Is there a way to resolve this with 2 racks ? Regards, Sanjay On 05/08/18, 11:57 PM, "Svante

Re: Reliability against rack failure

2018-08-05 Thread Svante Karlsson
You need 3 racks for your zookeepers anyway. It needs 2 out of three. How have you solved that? Den sön 5 aug. 2018 20:31Sanjay Awatramani skrev: > Thanks for the quick response Svante. > I forgot to mention that the deployment I am looking at has 2 racks. We > came up with this solution, but fo

Re: Kafka stream application load balancing

2018-08-05 Thread Siva Ram
Hi Guozhang, Thanks for the suggestions below. I consider we got past the REBALANCING issue. However, we are running into significant memory usage issue. I will open a separate thread for this. 1) During the punctuate we require to perform certain tasks and it was exceeding the consumer reques

Re: Reliability against rack failure

2018-08-05 Thread Sanjay Awatramani
Hi Svante, I just forgot about ZK. Thanks for setting me on the right track :) Regards, Sanjay On 06/08/18, 12:14 AM, "Svante Karlsson" wrote: You need 3 racks for your zookeepers anyway. It needs 2 out of three. How have you solved that? Den sön 5 aug. 2018 20:31Sanjay Awat