I would be using the servers available at my place of work. I dont have access to AWS servers. I would starting off with a small number of nodes in the cluster and then plot a graph with x-axis as the number of servers in the cluster and y-axis as the number of topics with partitions, before the cluster gives up.
I need 1 help here: What parameters should I keep in mind for tuning the JVM, if I have to see best performance ? My machine specs: I have 4 core CPUs with 8GB RAM with 500GB HDD (RAID Striped) Regards, Prabhjot On Tue, Jul 28, 2015 at 10:27 PM, JIEFU GONG <jg...@berkeley.edu> wrote: > I think these would definitely be useful statistics to have and I've tried > to do similar tests! The biggest difference is probably going to be the > hardware specs on whatever cluster you decide to run it on. Maybe > benchmarks performed on different AWS servers would be helpful too, but I'd > like to see these! > > On Mon, Jul 27, 2015 at 10:51 PM, Prabhjot Bharaj <prabhbha...@gmail.com> > wrote: > > > Hi, > > > > I'm looking forward to a benchmark which can explain how many total > number > > of topics and partitions can be created in a cluster of n nodes, given > the > > message size varies between x and y bytes and how does it vary with > varying > > heap sizes and how it affects the system performance. > > > > e.g. the result should look like: t topics with p partitions each can be > > supported in a cluster of n nodes with a heap size of h MB, before the > > cluster sees things like JVM crashes or high mem usage or system slowdown > > etc. > > > > I think such benchmarks must exist so that we can make better decisions > on > > ops side > > If these details dont exist, I'll be doing this test myself on varying > the > > values of parameters described above. I would be happy to share the > numbers > > with the community > > > > Thanks, > > prabcs > > > > > > -- > > Jiefu Gong > University of California, Berkeley | Class of 2017 > B.A Computer Science | College of Letters and Sciences > > jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427 > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
