Re: Sizing Cluster

2014-10-22 Thread Pete Wright
On 10/21/14 21:13, István wrote: Hi Pete, Yes you are right, both nodes has all of the data. I was just wondering what is the scenario for losing one node, in production it might not fly. If this is for testing only, you are good. Answering your question, I think retention policy

Sizing Cluster

2014-10-21 Thread Pete Wright
Hi There, I have a question regarding sizing disk for kafka brokers. Let's say I have systems capable of providing 10TB of storage, and they act as Kafka brokers. If I were to deploy two of these nodes, and enable replication in Kafka, would I actually have 10TB available for my

Re: Sizing Cluster

2014-10-21 Thread István
One thing that you have to keep in mind is that moving 10T between nodes takes long time. If you have a node failure and you need to rebuild (resync) the data your system is going to be vulnerable against the second node failure. You could mitigate this with using raid. I think generally speaking

Re: Sizing Cluster

2014-10-21 Thread Pete Wright
Thanks Istvan - I think I understand what you are say here - although I was under the impression that if I ensured each topic was being replicated N+1 times a two node cluster would ensure each node has a copy of the entire contents of the message bus at any given time. I agree with your

Re: Sizing Cluster

2014-10-21 Thread István
Hi Pete, Yes you are right, both nodes has all of the data. I was just wondering what is the scenario for losing one node, in production it might not fly. If this is for testing only, you are good. Answering your question, I think retention policy (log.retention.hours) is for controlling the