The bookie throughput is around 60k, so I guess that shouldn't be a problem. Irrespective of the number of topics I try, it seems to cap out at a tput of around 1200. This seems weird. Did you use the java client for your benchmarks ?
On Thu, Apr 19, 2012 at 12:58 PM, Ivan Kelly <[email protected]> wrote: > Hi Aniruddha, > > I can a very similar benchmark against a cluster of 3 and of 6 > machines today, and got performance of about 9k ops/s for both. This > is strange, because I would expect tpt to scale linearly as you add > machines if you have more than 1 topic. This leads me to believe that > the client is saturating, but i didn't have time to test this > hypothesis before I left work today. > > In your case, there must be something up with your setup. You could > try benching an individual bookie, using > bookkeeper-benchmark/bin/benchmark bookie -host <hostname> > > Generally we get between 17k & 20k op/s for this, so if you get less, > the bookies are the problem. > > Regards > Ivan > > > On Thu, Apr 19, 2012 at 12:39:49PM -0700, Aniruddha Laud wrote: > > Finished unacked pubs: tput = 1275.9486132380466 ops/s > > Finished acked pubs: count = 179970, tput = 1275.5778267618311 ops/s, > > numFailed = 0, avg latency = 11, 99.9%ile latency = >= 100 > > > > Is the problem on the client side and not the server? It seems that the > > throughput for unacked publishes and acked ones is almost the same. > > > > > > On Thu, Apr 19, 2012 at 12:34 PM, Aniruddha Laud > > <[email protected]>wrote: > > > > > > > > > > > On Thu, Apr 19, 2012 at 2:51 AM, Ivan Kelly <[email protected]> wrote: > > > > > >> Hi Aniruddha, > > >> > > >> 1269 op/s does seem very low. How are your bookies configured? Does > each > > >> bookie have a dedicated disk for the ledger dir and journal dir? With > > >> this machine machines I would expect much much higher performance. > > >> > > > Yes, there is one disk for the journal and 3 for ledgers. > > > > > >> > > >> I'm actually going to run the hedwig benchmarks here in the next > > >> couple of days. I'll let you know what result I get. > > >> > > > That would be really helpful. Thanks. > > > > > >> > > >> -Ivan > > >> > > >> On Wed, Apr 18, 2012 at 06:28:36PM -0700, Aniruddha Laud wrote: > > >> > I ran the benchmark in hedwig-client with the following command - > > >> > > > >> > java -cp hedwig-client-4.1.0-SNAPSHOT.jar -Dmode=pub -DnTopics=30 > > >> > -DnMsgs=180000 -DmsgSize=50 -DstartTopicLabel=1 > > >> > -Dlog4j.configuration=hedwig_client_log.properties > > >> > -Djava.net.preferIPv4Stack=true > > >> > org.apache.hedwig.client.benchmark.HedwigBenchmark > > >> > hedwig-loadtest-client.conf > > >> > > > >> > The hedwig-loadtest-client.conf contains one default_server_host. I > get > > >> a > > >> > throughput of 1269 ops/s . The setup is 15 hubs with 15 bookies and > an > > >> > ensemble size of 5 and quorum of 3. The number seems low to me. Am I > > >> > missing something? Also, I don't quite understand how to use the > > >> > nPartitions, nParallel options. Also, I have just 1 region. > > >> > > > >> > Regards, > > >> > Aniruddha. > > >> > > > >> > On Tue, Apr 17, 2012 at 7:45 PM, Sijie Guo <[email protected]> > wrote: > > >> > > > >> > > Hello Aniruddha, > > >> > > > > >> > > there is a benchmark for hedwig provided in hedwig-client > package. you > > >> > > could check org.apache.hedwig.client.benchmark package. > > >> > > > > >> > > Thanks, > > >> > > Sijie > > >> > > > > >> > > On Wed, Apr 18, 2012 at 5:29 AM, Aniruddha Laud < > > >> [email protected] > > >> > > >wrote: > > >> > > > > >> > > > Hi, > > >> > > > > > >> > > > Are there any benchmarks for hedwig that you could share? > > >> > > > > > >> > > > Regards, > > >> > > > Aniruddha. > > >> > > > > > >> > > > > >> > > > > > > >
