Op 24 april 2017 om 19:52 schreef Florian Haas <flor...@hastexo.com>:
Hi everyone,
so this will be a long email — it's a summary of several off-list
conversations I've had over the last couple of weeks, but the TL;DR
version is this question:
How can a Ceph cluster maintain near-constant performance
characteristics while supporting a steady intake of a large number of
small objects?
This is probably a very common problem, but we have a bit of a dearth of
truly adequate best practices for it. To clarify, what I'm talking about
is an intake on the order of millions per hour. That might sound like a
lot, but if you consider an intake of 700 objects/s at 20 KiB/object,
that's just 14 MB/s. That's not exactly hammering your cluster — but it
amounts to 2.5 million objects created per hour.
I have seen that the amount of objects at some point becomes a problem.
Eventually you will have scrubs running and especially a deep-scrub will cause
issues.
I have never had the use-case to have a sustained intake of so many
objects/hour, but it is interesting though.
Under those circumstances, two things tend to happen:
(1) There's a predictable decline in insert bandwidth. In other words, a
cluster that may allow inserts at a rate of 2.5M/hr rapidly goes down to
1.8M/hr and then 1.7M/hr ... and by "rapidly" I mean hours, not days. As
I understand it, this is mainly due to the FileStore's propensity to
index whole directories with a readdir() call which is an linear-time
operation.
(2) FileStore's mitigation strategy for this is to proactively split
directories so they never get so large as for readdir() to become a
significant bottleneck. That's fine, but in a cluster with a steadily
growing number of objects, that tends to lead to lots and lots of
directory splits happening simultanously — causing inserts to slow to a
crawl.
For (2) there is a workaround: we can initialize a pool with an expected
number of objects, set a pool max_objects quota, and disable on-demand
splitting altogether by setting a negative filestore merge threshold.
That way, all splitting occurs at pool creation time, and before another
split were to happen, you hit the pool quota. So you never hit that
brick wall causes by the thundering herd of directory splits. Of course,
it also means that when you want to insert yet more objects, you need
another pool — but you can handle that at the application level.
It's actually a bit of a dilemma: we want directory splits to happen
proactively, so that readdir() doesn't slow things down, but then we
also *don't* want them to happen, because while they do, inserts flatline.
(2) will likely be killed off completely by BlueStore, because there are
no more directories, hence nothing to split.
For (1) there really isn't a workaround that I'm aware of for FileStore.
And at least preliminary testing shows that BlueStore clusters suffer
from similar, if not the same, performance degradation (although, to be
fair, I haven't yet seen tests under the above parameters with rocksdb
and WAL on NVMe hardware).
Can you point me to this testing of BlueStore?
For (1) however I understand that there would be a potential solution in
FileStore itself, by throwing away Ceph's own directory indexing and
just rely on flat directory lookups — which should be logarithmic-time
operations in both btrfs and XFS, as both use B-trees for directory
indexing. But I understand that that would be a fairly massive operation
that looks even less attractive to undertake with BlueStore around the
corner.
One suggestion that has been made (credit to Greg) was to do object
packing, i.e. bunch up a lot of discrete data chunks into a single RADOS
object. But in terms of distribution and lookup logic that would have to
be built on top, that seems weird to me (CRUSH on top of CRUSH to find
out which RADOS object a chunk belongs to, or some such?)
So I'm hoping for the likes of Wido and Dan and Mark to have some
alternate suggestions here: what's your take on this? Do you have
suggestions for people with a constant intake of small objects?
I have a bit of similar use-case. A customer needs to store a lot of objects
(4M per TB) and we eventually went for a lot of smal(ler) disks instead of big
disks.
In this case we picked 3TB disks instead of 6 or 8TB so that we have a large
number of OSDs, high number of PGs and thus have less objects per OSD.
You are ingesting ~50GB/h. For how long are you keeping the objects in the
cluster? What is the total TB storage you need? Would it work in this use-case
to have a lot of OSDs on smaller disks?
I think that in this you can partly overcome the problem by simply having more
OSDs.
Wido
Looking forward to hearing your thoughts.
Cheers,
Florian
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com