> Have you tried this with latest shardedpool/WQ model which is already in the > Giant branch ? > IOS will be going with this path in the latest code not with op_wq.
Not yet. I mainly work on firefly now. I think I will try it when I have time and I will give the report. :) > Yes, we also saw encode/decode was consuming lot of cpu times and if I > remember correctly profiler was pointing bufferlist::append in many of such > cases. As right as my glove. :) On 26 September 2014 14:40, Somnath Roy <somnath....@sandisk.com> wrote: > Haomai/Dong > Have you tried this with latest shardedpool/WQ model which is already in the > Giant branch ? > IOS will be going with this path in the latest code not with op_wq. > Yes, we also saw encode/decode was consuming lot of cpu times and if I > remember correctly profiler was pointing bufferlist::append in many of such > cases. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Dong Yuan [mailto:yuandong1...@gmail.com] > Sent: Thursday, September 25, 2014 11:31 PM > To: Haomai Wang > Cc: Sage Weil; ceph-devel@vger.kernel.org; Somnath Roy; Allen Samuels; > dieter.kas...@ts.fujitsu.com; pvonstamw...@us.fujitsu.com; Shu, Xinxin; > Stefan Priebe - Profihost AG; xiaoxi.c...@intel.com; Milosz Tanski; > zhiqiang.w...@intel.com; jianpeng...@intel.com; gd...@mellanox.com; > vuhu...@mellanox.com; Mark Nelson > Subject: Re: Weekly performance meeting > > Some data can support Haomai's points. > >> 1. encode/decode plays remarkable latency, especially in >> ObjectStore::Transaction. I'm urgen in refactor ObjectStore API to >> avoid encode/decode codes. It seemed has be signed in note(- remove >> serialization from ObjectStore::Transaction (ymmv)) > > My environment, single OSD on a single SSD with filestore_blackhole = true. > > With All transaction encode, 10000 4K WriteFull operations by single thread > need about 14.3s. While without transaction encode, the same test can be > finished in about 11.5s. > > Considering the FileStore needs to decode the bufferlist too, encode/decode > cost more than 20% time! > > Oprofile results can validate this problem too: methods used by encode/decode > sometimes take 9 of the top 10. > >> 2. obvious latency for threadpool/workqueue model. Do we consider to >> impl performance optimization workqueue to replace existing critical >> workqueue such as op_wq in OSD.h and op_wq in FileStore.h. Now in my >> AsyncMessenger impl, I will try to use custom and simple workqueue >> impl to improve performance. > > When I analyze the latency of a 4K object WriteFull operation, I put static > probes into codes to measure times used by OpWQ. I test 10000 4K object > WriteFull operations and average the results. > > I found it spends 158us for the OpWQ for each IO, including 30us to enqueue, > 108us in the queue, and 20us to dequeue. It takes more than 20% time of PG > layer (not including msg and os layer) when encode is ignored. > > Maybe a more effective ThreadPool/WorkQueue Model is needed or at least some > improvement for WorkQueues in the IO path to reduce the latency. > > On 26 September 2014 10:27, Haomai Wang <haomaiw...@gmail.com> wrote: >> Thanks for sage! >> >> I'm on the flight at Oct 1. :-( >> >> Now my team is mainly worked on the performance of ceph, we have >> observed these points: >> >> 1. encode/decode plays remarkable latency, especially in >> ObjectStore::Transaction. I'm urgen in refactor ObjectStore API to >> avoid encode/decode codes. It seemed has be signed in note(- remove >> serialization from ObjectStore::Transaction (ymmv)) 2. obvious latency >> for threadpool/workqueue model. Do we consider to impl performance >> optimization workqueue to replace existing critical workqueue such as >> op_wq in OSD.h and op_wq in FileStore.h. Now in my AsyncMessenger >> impl, I will try to use custom and simple workqueue impl to improve >> performance. >> 3. Large lock in client library such as ObjectCacher >> >> >> On Fri, Sep 26, 2014 at 2:27 AM, Sage Weil <sw...@redhat.com> wrote: >>> Hi everyone, >>> >>> A number of people have approached me about how to get more involved >>> with the current work on improving performance and how to better >>> coordinate with other interested parties. A few meetings have taken >>> place offline with good results but only a few interested parties were >>> involved. >>> >>> Ideally, we'd like to move as much of this dicussion into the public >>> forums: ceph-devel@vger.kernel.org and #ceph-devel. That isn't >>> always sufficient, however. I'd like to also set up a regular weekly >>> meeting using google hangouts or bluejeans so that all interested >>> parties can share progress. There are a lot of things we can do >>> during the Hammer cycle to improve things but it will require some >>> coordination of effort. >>> >>> Among other things, we can discuss: >>> >>> - observed performance limitations >>> - high level strategies for addressing them >>> - proposed patch sets and their performance impact >>> - anything else that will move us forward >>> >>> One challenge is timezones: there are developers in the US, China, >>> Europe, and Israel who may want to join. As a starting point, how >>> about next Wednesday, 15:00 UTC? If I didn't do my tz math wrong, >>> that's >>> >>> 8:00 (PDT, California) >>> 15:00 (UTC) >>> 18:00 (IDT, Israel) >>> 23:00 (CST, China) >>> >>> That is surely not the ideal time for everyone but it can hopefully >>> be a starting point. >>> >>> I've also created an etherpad for collecting discussion/agenda items >>> at >>> >>> http://pad.ceph.com/p/performance_weekly >>> >>> Is there interest here? Please let everyone know if you are actively >>> working in this area and/or would like to join, and update the pad >>> above with the topics you would like to discuss. >>> >>> Thanks! >>> sage >> >> >> >> -- >> Best Regards, >> >> Wheat >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majord...@vger.kernel.org More majordomo >> info at http://vger.kernel.org/majordomo-info.html > > > > -- > Dong Yuan > Email:yuandong1...@gmail.com > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly > prohibited. If you have received this communication in error, please notify > the sender by telephone or e-mail (as shown above) immediately and destroy > any and all copies of this message in your possession (whether hard copies or > electronically stored copies). > -- Dong Yuan Email:yuandong1...@gmail.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
