Not having OSDs and KVMs compete against each other is one thing. But there are more reasons to do this
1) not moving the processes and threads between cores that much (better cache utilization) 2) aligning the processes with memory on NUMA systems (that means all modern dual socket systems) - you don’t want your OSD running on CPU1 with memory allocated to CPU2 3) the same goes for other resources like NICs or storage controllers - but that’s less important and not always practical to do 4) you can limit the scheduling domain on linux if you limit the cpuset for your OSDs (I’m not sure how important this is, just best practice) 5) you can easily limit memory or CPU usage, set priority, with much greater granularity than without cgroups 6) if you have HyperThreading enabled you get the most gain when the workloads on the threads are dissimiliar - so to have the higher throughput you have to pin OSD to thread1 and KVM to thread2 on the same core. We’re not doing that because latency and performance of the core can vary depending on what the other thread is doing. But it might be useful to someone. Some workloads exhibit >100% performance gain when everything aligns in a NUMA system, compared to a SMP mode on the same hardware. You likely won’t notice it on light workloads, as the interconnects (QPI) are very fast and there’s a lot of bandwidth, but for stuff like big OLAP databases or other data-manipulation workloads there’s a huge difference. And with CEPH being CPU hungy and memory intensive, we’re seeing some big gains here just by co-locating the memory with the processes…. Jan > On 30 Jun 2015, at 08:12, Ray Sun <xiaoq...@gmail.com> wrote: > > Sound great, any update please let me know. > > Best Regards > -- Ray > > On Tue, Jun 30, 2015 at 1:46 AM, Jan Schermer <j...@schermer.cz > <mailto:j...@schermer.cz>> wrote: > I promised you all our scripts for automatic cgroup assignment - they are in > our production already and I just need to put them on github, stay tuned > tomorrow :-) > > Jan > > >> On 29 Jun 2015, at 19:41, Somnath Roy <somnath....@sandisk.com >> <mailto:somnath....@sandisk.com>> wrote: >> >> Presently, you have to do it by using tool like ‘taskset’ or ‘numactl’… >> >> Thanks & Regards >> Somnath >> >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com >> <mailto:ceph-users-boun...@lists.ceph.com>] On Behalf Of Ray Sun >> Sent: Monday, June 29, 2015 9:19 AM >> To: ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> Subject: [ceph-users] How to use cgroup to bind ceph-osd to a specific cpu >> core? >> >> Cephers, >> I want to bind each of my ceph-osd to a specific cpu core, but I didn't find >> any document to explain that, could any one can provide me some detailed >> information. Thanks. >> >> Currently, my ceph is running like this: >> >> oot 28692 1 0 Jun23 ? 00:37:26 /usr/bin/ceph-mon -i >> seed.econe.com <http://seed.econe.com/> --pid-file >> /var/run/ceph/mon.seed.econe.com.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 40063 1 1 Jun23 ? 02:13:31 /usr/bin/ceph-osd -i 0 >> --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 42096 1 0 Jun23 ? 01:33:42 /usr/bin/ceph-osd -i 1 >> --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 43263 1 0 Jun23 ? 01:22:59 /usr/bin/ceph-osd -i 2 >> --pid-file /var/run/ceph/osd.2.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 44527 1 0 Jun23 ? 01:16:53 /usr/bin/ceph-osd -i 3 >> --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 45863 1 0 Jun23 ? 01:25:18 /usr/bin/ceph-osd -i 4 >> --pid-file /var/run/ceph/osd.4.pid -c /etc/ceph/ceph.conf --cluster ceph >> root 47462 1 0 Jun23 ? 01:20:36 /usr/bin/ceph-osd -i 5 >> --pid-file /var/run/ceph/osd.5.pid -c /etc/ceph/ceph.conf --cluster ceph >> >> Best Regards >> -- Ray >> >> >> PLEASE NOTE: The information contained in this electronic mail message is >> intended only for the use of the designated recipient(s) named above. If the >> reader of this message is not the intended recipient, you are hereby >> notified that you have received this message in error and that any review, >> dissemination, distribution, or copying of this message is strictly >> prohibited. If you have received this communication in error, please notify >> the sender by telephone or e-mail (as shown above) immediately and destroy >> any and all copies of this message in your possession (whether hard copies >> or electronically stored copies). >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
