On 04/10/2015 10:10 AM, Lionel Bouton wrote:
On 04/10/15 15:41, Jeff Epstein wrote:
[...]
This seems highly unlikely. We get very good performance without
ceph. Requisitioning and manupulating block devices through LVM
happens instantaneously. We expect that ceph will be a bit slower by
its distributed nature, but we've seen operations block for up to an
hour, which is clearly behind the pale. Furthermore, as the
performance measure I posted show, read/write speed is not the
bottleneck: ceph is simply/waiting/.
So, does anyone else have any ideas why mkfs (and other operations)
takes so long?
As your use case is pretty unique and clearly not something Ceph was
optimized for, if I were you I'd switch to a single pool with the
appropriate number of pgs based on your pool size (replication) and
the number of OSD you use (you should target 100 pgs/OSD to be in what
seems the sweet spot) and create/delete rbd instead of the whole pool.
You would be in "known territory" and any remaining performance
problem would be easier to debug.
I agree that this is a good suggestion. It took me a little while, but
I've changed the configuration so that we now have only one pool,
containing many rbds, and now all data is spread across all six of our
OSD nodes. However, the performance has not perceptibly improved. We
still have the occasional long (>10 minutes) wait periods during write
operations, and the bottleneck still seems to be ceph, rather than the
hardware: the blocking process (most usually, but not always, mkfs) is
stuck in a wait state ("D" in ps) but no I/O is actually being
performed, so one can surmise that the physical limitations of the disk
medium are not the bottleneck. This is similar to what is being reported
in the thread titled "100% IO Wait with CEPH RBD and RSYNC".
Do you have some idea how I can diagnose this problem?
Best,
Jeff
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
