> As to what procedures I'd propose is spinning up a cluster and running some tests on it.
This is really good advice. Capacity planning for your Kafka cluster is not particularly straightforward - especially if you intend to run a multi-tenant cluster. (If someone has a general model, please share!) I wouldn't recommend using benchmark numbers for planning. Benchmarks are great for demonstrating what's achievable under optimal conditions. These optimal conditions won't necessarily accurately reflect your use case. It's less stressful to discover a reasonable hardware configuration and number of machines for your cluster now than in production. You can spin up a cluster in EC2 and throw some producer and consumer load at it with the perf tools that ship with Kafka or sangrenel to get a signal. Some questions to consider: How many messages are producers going to send to my cluster? How many producers will be connected to my cluster? How big (in bytes) are messages going to be? How many partitions will I need for my topics? How many replicas will I have for each topic? How many consumers will be reading from the cluster? In addition to the docs Karolis already mentioned, you might also get some mileage out of some of Todd Palino's slide decks. This one is my favorite: http://www.slideshare.net/ToddPalino/putting-kafka-into-overdrive -Kevin On Wed, Nov 9, 2016 at 3:00 AM, Majid Golshadi <[email protected]> wrote: > Thank you. > I have red these articles. Do you have any other documentation? > > On Wed, Nov 9, 2016 at 12:07 PM Karolis Pocius <[email protected]> > wrote: > > > There is no 'best' configuration, it all depends on your use cases, > > expected load, etc. > > > > Documentation is a good place to start > > https://kafka.apache.org/documentation.html#hwandos > > > > There are a few good benchmark articles. They're a bit dated by now, but > > still hold true in most cases: > > * > > > > https://engineering.linkedin.com/kafka/benchmarking-apache- > kafka-2-million-writes-second-three-cheap-machines > > * https://grey-boundary.io/load-testing-apache-kafka-on-aws/ > > > > As to what procedures I'd propose is spinning up a cluster and running > > some tests on it. Sangrenel (used in the second article above) is a > > great tool https://github.com/jamiealquiza/sangrenel for that. > > > > > > On 2016.11.09 09:58, Majid Golshadi wrote: > > > Hello > > > We want to use Kafka in our production environment but we don't have > any > > > information about what's the best server configuration and the best > > > benchmark can be achieve in our production environment (Based on our > > > hardware and VMs) > > > I'm really rookie in this area. > > > For getting to the best configuration for our production environment > what > > > procedures do you propose? > > > > > > suppose I want to run kafka on 3 machines with the following > > configuration: > > > cpu: 2 core > > > Ram: 4 gig > > > HDD. 7500 rpm > > > > > > > >
