Hi, For anybody who may be interested, here I share a process of locating the reason for ceph cluster performance slow down in our environment.
Internally, we have a cluster with capacity 1.1PB, used 800TB, and raw user data is about 500TB. Each day, 3TB' data is uploaded and 3TB oldest data is lifecycled (we are using s3 object store, and bucket lifecycle is enabled). As time goes by, the cluster becomes some slower, we doubt the xfs fragmentation is the fiend. After some testing, we do find xfs fragmentation slow down filestore's performance, for example, at 15% fragmentation, the performance is 85% of the original, and at 25%, the performance is 74.73% of the original. But the main reason for our cluster's deterioration of performance is not the xfs fragmentation. Initially, our ceph cluster contains only osds with 4TB's disk, as time goes by, we scale out our cluster by adding some new osds with 8TB's disk. And as the new disk's capacity is double times of the old disks, so each new osd's weight is double of the old osd. And new osd has double pgs than old osd, and new osd used double disk space than the old osd. Everything looks good and fine. But even though the new osd has double capacity than the old osd, the new osd's performance is not double than the old osd. After digging into our internal system stats, we find the new added's disk io util is about two times than the old. And from time to time, the new disks' io util rises up to 100%. The new added osds are the performance killer. They slow down the whole cluster's performance. As the reason is found, the solution is very simple. After lower new added osds's weight, the annoying slow request warnings have died away. So the conclusion is: in cluster with different osd's disk size, osd's weight is not only determined by its capacity, we should also have a look at its performance. Best wishes, Yao Zongyou _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
