To clear up the question about 2.9k synchronous adds/s, I'd suggest you turn off the disk cache and run the experiment again. If you're on linux, I think you need to run something like: hdparm -W0 /dev/hda
All the comments here about seeks, rotational latency and such seem to be ignoring the presence of on-disk write caches. In fact, if you could share the model of your disk as others have suggested, that could help our understanding. Also, whether you want to run in production with the disk cache on or off is a matter of preference (or requirement). With it on and no battery-backed controller, there is indeed a slight chance of losing data. -Flavio On 19 Oct 2014, at 05:52, Sijie Guo <[email protected]> wrote: > I really doubt the number 2941 entries/s here. since your test application is > writing entries sequentially, so it means each write request need to be > completed in 0.3ms, which doesn't seem to be reasonable per my knowledge. > Could you point what is your disk characteristics? > > Besides that, I think your test is really biased. > > - 10/100/1000 entries is too small, which most of all entries will be just in > memory for ledger storage. you will never know how's the impact on latency if > ledger storage is doing disk I/O. > - since bookkeeper is targeting a durable journal, all requests are needed to > be synced to disk. the latency of adding entry is around millisecond level. > the way you are testing bookie isn't correct. since the latency of an entry > include network RTTs and disk fsync time, if there is an addEntry operation > encountered network/disk hiccups, you will definitely see lower entries be > sent in one second. > - we encourage people to use asynchronous API to addEntry in a real system > for latency and throughput. > > I would suggest: > > * try using bookkeeper-benchmark/bin/benchmark to benchmark your bookie > cluster using different bookkeeper versions, to see what is the difference. > * if you are going to using you test benchmark, try running your benchmark > using a larger entries, to get your benchmark run enough time. > * in 4.3.0 bookkeeper, try choose one stats provider and set > statsProviderClass in your bookie configuration. so you could get metrics > about your journal operations during benchmarking, so we will know what is > slow in your system. > > > Last one, there are several journal settings introduced in bookkeeper 4.3.0. > You could use different combination to support different characterized > workloads. > http://zookeeper.apache.org/bookkeeper/docs/r4.3.0/bookieConfigParams.html > > In our systems, we usually set journalAdaptiveGroupWrites = true, > journalFlushWhenQueueEmpty = true and journalMaxGroupWaitMSec to a value that > meet your latency SLA. Setting journalFlushWhenQueueEmpty to true, it would > make the number of flushes more sustainable when bursty traffic happened; > setting journalAdaptiveGroupWrites to true would try grouping flushes > together into one fsync, which reduce number of fsyncs to your disk. Choosing > a group commit time is a tradeoff between latency and throughput. > > - Sijie > > > > > > > > On Fri, Oct 17, 2014 at 8:28 AM, sky_cord <[email protected]> wrote: > Thanks for your help. > After adding the two config items into bk_server.conf : > journalFlushWhenQueueEmpty=true > journalMaxGroupWaitMSec=0 > The write speed rise up to 800-900 entries per second in one ledger, length > of each entry is 50 bytes. > But it's still lower than I expected（The sync write speed is 2941 entries/s > based onbookkeepr-4.2.2）. > Could you describe your test result? And Is there any way to improve write > performance ? > Thank you. > > > > At 2014-10-17 20:13:07, "Ivan Kelly" <[email protected]> wrote: > >Ah, I know what this is. Twitter added some changes that makes write > >throughput much higher when there's a high write rate. In this case > >the latency for a single writes goes go up though (as it's doing more > >buffering on the journal). Try setting: > > > >journalFlushWhenQueueEmpty=true > >journalMaxGroupWaitMSec=0 > > > >in bk_server.conf > > > > > >Also, the journal and ledger directories are on the same disk. This > >will have a detrimental affect once the write buffers start flushing. > >We always recommend that you put the journal on its own independent > >disk. > > > >The reads are fast because it doesn't need to hit disk, as there's not > >much data that has been written. > > > >-Ivan > > > >On 17 October 2014 13:32, [email protected] <[email protected]> wrote: > >> The write speed of bookkeper that version less than 4.30 is bigger than > >> 1200 > >> entries per second when i use the same test code > >> > >> ________________________________ > >> [email protected] > >> > >> > >> From: Jiannan Wang > >> Date: 2014-10-17 18:31 > >> To: [email protected] > >> CC: [email protected] > >> Subject: Re: hi, why write speed of bookkeeper-4.3.0 is very slow? > >> Could you share more information: > >> - What's the number of write_num? > >> - What's the read throughput under your test? > >> - Does the 5 bookie servers run in same machine? > >> - Could you check whether StringUtil.div works well? > >> - Could you also try async write API and tell us the output? > >> > >> Regards, > >> Jiannan > >> > >> From: "[email protected]" <[email protected]> > >> Reply-To: "[email protected]" > >> <[email protected]> > >> Date: Friday, October 17, 2014 at 3:38 PM > >> To: bookkeeper-user <[email protected]> > >> Subject: hi, why write speed of bookkeeper-4.3.0 is very slow? > >> > >> hi, > >> I have tried bookkeeper-4.3.0 today. But its write speed is 4 entries > >> per sec in my environment. > >> There are 5 bookkeeper-server-4.3.0 in my env. > >> Is there anything wrong? Test codes as follows: > >> public void syncWriteLedger(long write_num) throws InterruptedException, > >> BKException { > >> byte[] pwd = "foobar".getBytes(); > >> LedgerHandle lh = client.createLedger(3, 2, > >> DigestType.MAC, pwd); > >> > >> long ledgerId = lh.getId(); > >> System.out.println("ledgerId=" + ledgerId); > >> System.out > >> .println("data_len=" > >> + ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" + write_num) > >> .getBytes().length); > >> byte[] data = ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" + > >> write_num).getBytes(); > >> long st = System.currentTimeMillis(); > >> for (int i = 0; i < write_num; i++) { > >> lh.addEntry(data); > >> } > >> long ut = System.currentTimeMillis() - st; > >> lh.close(); > >> double uts = StringUtil.div(ut, 1000, 3); > >> double wspeed = StringUtil.div(write_num, uts, 2); > >> System.out.println("ut=" + ut + " ms, write_num=" + write_num > >> + ", write speed=" + wspeed + "/s"); > >> > >> > >> LedgerHandle lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.MAC, > >> pwd); > >> long lastEntry = lh2.getLastAddConfirmed(); > >> st = System.currentTimeMillis(); > >> Enumeration<LedgerEntry> entries = lh2.readEntries(0, lastEntry); > >> while (entries.hasMoreElements()) { > >> byte[] bytes = entries.nextElement().getEntry(); > >> System.out.println(new String(bytes)); > >> } > >> ut = System.currentTimeMillis() - st; > >> lh2.close(); > >> uts = StringUtil.div(ut, 1000, 3); > >> double rspeed = StringUtil.div(write_num, uts, 2); > >> System.out.println("ut=" + ut + " ms, read_num=" + write_num > >> + ", read speed=" + rspeed + "/s"); > >> > >> } > >> ________________________________ > >> [email protected] > > >
