Any update on this matter? I've been thinking of upgrading from 0.80.7 to 0.80.8 - lucky that I see this thread first...
On Thu, Feb 12, 2015 at 10:39 PM, 杨万元 <yangwanyuan8...@gmail.com> wrote: > thanks very much for your advice . > yes，as you said，disabled the rbd_cache will improve the read request，but > if i disabled rbd_cache， the randwrite request will be worse. so this > method maybe can not solve my problem, is it ? > > In addition , I also test the 0.80.6 and 0.80.7 librbd,they are as good as > 0.80.5 performance , so maybe can sure this problem is cause from 0.80.8 > > > 2015-02-12 19:33 GMT+08:00 Alexandre DERUMIER <aderum...@odiso.com>: > >> >>Hi, >> >>Can you test with disabling rbd_cache ? >> >> >>I remember of a bug detected in giant, not sure it's also the case for >> fireflt >> >> This was this tracker: >> >> http://tracker.ceph.com/issues/9513 >> >> But It has been solved and backported to firefly. >> >> Also, can you test 0.80.6 and 0.80.7 ? >> >> >> >> >> >> >> >> ----- Mail original ----- >> De: "killingwolf" <killingw...@qq.com> >> À: "ceph-users" <ceph-users@lists.ceph.com> >> Envoyé: Jeudi 12 Février 2015 12:16:32 >> Objet: [ceph-users] re： Upgrade 0.80.5 to 0.80.8 --the VM's read >> requestbecome too slow >> >> I have this problems too , Help! >> >> ------------------ 原始邮件 ------------------ >> 发件人: "杨万元";<yangwanyuan8...@gmail.com>; >> 发送时间: 2015年2月12日(星期四) 中午11:14 >> 收件人: "ceph-users@lists.ceph.com"<ceph-users@lists.ceph.com>; >> 主题: [ceph-users] Upgrade 0.80.5 to 0.80.8 --the VM's read requestbecome >> too slow >> >> Hello! >> We use Ceph+Openstack in our private cloud. Recently we upgrade our >> centos6.5 based cluster from Ceph Emperor to Ceph Firefly. >> At first,we use redhat yum repo epel to upgrade, this Ceph's version is >> 0.80.5. First upgrade monitor,then osd,last client. when we complete this >> upgrade, we boot a VM on the cluster,then use fio to test the io >> performance. The io performance is as better as before. Everything is ok! >> Then we upgrade the cluster from 0.80.5 to 0.80.8,when we completed , we >> reboot the VM to load the newest librbd. after that we also use fio to test >> the io performance .then we find the randwrite and write is as good as >> before.but the randread and read is become worse, randwrite's iops from >> 4000-5000 to 300-400 ,and the latency is worse. the write's bw from 400MB/s >> to 115MB/s . then I downgrade the ceph client version from 0.80.8 to >> 0.80.5, then the reslut become normal. >> So I think maybe something cause about librbd. I compare the 0.80.8 >> release notes with 0.80.5 ( >> http://ceph.com/docs/master/release-notes/#v0-80-8-firefly ), I just >> find this change in 0.80.8 is something about read request : librbd: cap >> memory utilization for read requests (Jason Dillaman) . Who can explain >> this？ >> >> >> My ceph cluster is 400osd,5mons : >> ceph -s >> health HEALTH_OK >> monmap e11: 5 mons at {BJ-M1-Cloud71= >> 172.28.2.71:6789/0,BJ-M1-Cloud73=172.28.2.73:6789/0,BJ-M2-Cloud80=172.28.2.80:6789/0,BJ-M2-Cloud81=172.28.2.81:6789/0,BJ-M3-Cloud85=172.28.2.85:6789/0 >> }, election epoch 198, quorum 0,1,2,3,4 >> BJ-M1-Cloud71,BJ-M1-Cloud73,BJ-M2-Cloud80,BJ-M2-Cloud81,BJ-M3-Cloud85 >> osdmap e120157: 400 osds: 400 up, 400 in >> pgmap v26161895: 29288 pgs, 6 pools, 20862 GB data, 3014 kobjects >> 41084 GB used, 323 TB / 363 TB avail >> 29288 active+clean >> client io 52640 kB/s rd, 32419 kB/s wr, 5193 op/s >> >> >> The follwing is my ceph client conf : >> [global] >> auth_service_required = cephx >> filestore_xattr_use_omap = true >> auth_client_required = cephx >> auth_cluster_required = cephx >> mon_host = >> 172.29.204.24,172.29.204.48,172.29.204.55,172.29.204.58,172.29.204.73 >> mon_initial_members = ZR-F5-Cloud24, ZR-F6-Cloud48, ZR-F7-Cloud55, >> ZR-F8-Cloud58, ZR-F9-Cloud73 >> fsid = c01c8e28-304e-47a4-b876-cb93acc2e980 >> mon osd full ratio = .85 >> mon osd nearfull ratio = .75 >> public network = 172.29.204.0/24 >> mon warn on legacy crush tunables = false >> >> [osd] >> osd op threads = 12 >> filestore journal writeahead = true >> filestore merge threshold = 40 >> filestore split multiple = 8 >> >> [client] >> rbd cache = true >> rbd cache writethrough until flush = false >> rbd cache size = 67108864 >> rbd cache max dirty = 50331648 >> rbd cache target dirty = 33554432 >> >> [client.cinder] >> admin socket = /var/run/ceph/rbd-$pid.asok >> >> >> >> My VM is 8core16G,we use fio scripts is : >> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randread -size=60G >> -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 >> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=60G >> -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 >> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=read -size=60G >> -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 >> fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=write -size=60G >> -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 >> >> The following is the io test result >> ceph client verison :0.80.5 >> read: bw= 430MB >> write: bw=420MB >> randread: iops= 4875 latency=65ms >> randwrite: iops=6844 latency=46ms >> >> ceph client verison :0.80.8 >> read: bw= 115MB >> write: bw=480MB >> randread: iops= 381 latency=83ms >> randwrite: iops=4843 latency=68ms >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
