On 13/12/2018 15:10, Mark Nelson wrote: > Hi Florian, > > On 12/13/18 7:52 AM, Florian Haas wrote: >> On 02/12/2018 19:48, Florian Haas wrote: >>> Hi Mark, >>> >>> just taking the liberty to follow up on this one, as I'd really like to >>> get to the bottom of this. >>> >>> On 28/11/2018 16:53, Florian Haas wrote: >>>> On 28/11/2018 15:52, Mark Nelson wrote: >>>>> Option("bluestore_default_buffered_read", Option::TYPE_BOOL, >>>>> Option::LEVEL_ADVANCED) >>>>> .set_default(true) >>>>> .set_flag(Option::FLAG_RUNTIME) >>>>> .set_description("Cache read results by default (unless hinted >>>>> NOCACHE or WONTNEED)"), >>>>> >>>>> Option("bluestore_default_buffered_write", Option::TYPE_BOOL, >>>>> Option::LEVEL_ADVANCED) >>>>> .set_default(false) >>>>> .set_flag(Option::FLAG_RUNTIME) >>>>> .set_description("Cache writes by default (unless hinted >>>>> NOCACHE or >>>>> WONTNEED)"), >>>>> >>>>> >>>>> This is one area where bluestore is a lot more confusing for users >>>>> that >>>>> filestore was. There was a lot of concern about enabling buffer cache >>>>> on writes by default because there's some associated overhead >>>>> (potentially both during writes and in the mempool thread when >>>>> trimming >>>>> the cache). It might be worth enabling >>>>> bluestore_default_buffered_write >>>>> and see if it helps reads. >>>> So yes this is rather counterintuitive, but I happily gave it a shot >>>> and >>>> the results are... more head-scratching than before. :) >>>> >>>> The output is here: http://paste.openstack.org/show/736324/ >>>> >>>> In summary: >>>> >>>> 1. Write benchmark is in the same ballpark as before (good). >>>> >>>> 2. Read benchmark *without* readahead is *way* better than before >>>> (splendid!) but has a weird dip down to 9K IOPS that I find >>>> inexplicable. Any ideas on that? >>>> >>>> 3. Read benchmark *with* readahead is still abysmal, which I also find >>>> rather odd. What do you think about that one? >>> These two still confuse me. >>> >>> And in addition, I'm curious as to what you think of the approach to >>> configure OSDs with bluestore_cache_kv_ratio = .49, so that rather >>> than using 1%/99%/0% of cache memory for metadata/KV data/objects, the >>> OSDs use 1%/49%/50%. Is this sensible? I assume the default of not using >>> any memory to actually cache object data is there for a reason, but I am >>> struggling to grasp what that reason would be. Particularly since in >>> filestore, we always got in-memory object caching for free, via the page >>> cache. >> Hi Mark, >> >> do you mind if I give this another poke? > > > Sorry, I got super busy with things and totally forgot about this.
No worries at all; that's why I follow up. :) > Weird dips always make me think compaction. Once thing we've seen is > that compaction can force the entire cache to flush, invalidate all of > the indexes/filters, and generally slow everything down. If you still > have the OSD log you can run this tool to get compaction event stats > (and restrict it to certain level compactions if you like): > > https://github.com/ceph/cbt/blob/master/tools/ceph_rocksdb_log_parser.py OK — in case this yields that it *is* indeed compaction that's a potential culprit, what would be the remedy? I see there's a rather opaque catch-all bluestore_rocksdb_options option where we can override the compaction_readahead_size, and then there are some BlueFS log compaction settings, but those don't seem to be very much applicable here. > No idea why readahead would be that much slower. We just saw a case > where large sequential reads were incredibly slow with certain NVMe > drives and LVM that were fixed by a kernel upgrade, but that was a very > specific case. Guess there's not that much to do here but leave it disabled, then? > Regarding meta/kv/data ratios: It's really tough to configure optimal > settings for all situations. Generally for RGW you need more KV cache > and for RBD you need more meta cache, but it's highly variable (ie even > in the RBD case you need enough KV cache to make sure all > indexes/filters are cached, and in the RGW case you still may want to > prioritize hot bluestore onodes). That's why I started writing the > autotuning code. Because the cache is hierarchical, the worst case > situation is that you just end up caching the same onode data twice in > both places (especially if you end up forcing out omap data you need > cached). The best case situation is that you cache the most useful > recent data with as little double caching as possible. That's sort of > the direction I'm trying to head with the autotuner. You mentioned KV cache and meta cache, but I'm afraid that doesn't quite address my question about a non-zero data cache. Does setting a non-zero data cache never make sense? Also, given that this is turning into rather deep black magic by itself, what do you think about a recommendation to keep using filestore until a cache autotuner is actually available? Cheers, Florian _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com