Thanks for the helpful commentary Christian. Cluster is performing much better with 50% more spindles (12 to 18 drives), along with setting scrub sleep to 0.1. Didn't see really any gain from moving from the Samsung 850 Pro journal drives to Intel 3710's, even though dd and other direct tests of the drives yielded much better results. rados bench with 4k requests are still awfully low. I'll figure that problem out next.
I ended up bumping up the number of placement groups from 512 to 1024 which should help a little bit. Basically it'll change the worst case scrub performance such that it is distributed a little more across drives rather than clustered on a single drive for longer. I think the real solution here is to create a secondary SSD pool, pin some radosgw buckets to it and put my thumbnail data on the smaller, faster pool. I'll reserve the spindle based pool for original high res photos, which are only read to create thumbnails when necessary. This should put the majority of my random read IO on SSDs, and thumbnails average 50kb each so it shouldn't be too spendy. I am considering trying the newer samsung sm863 drives as we are read heavy, any potential data loss on this thumbnail pool will not be catastrophic. Third, it seems that I am also running into the known "Lots Of Small Files" performance issue. Looks like performance in my use case will be drastically improved with the upcoming bluestore, though migrating to it sounds painful! On Thu, Feb 4, 2016 at 7:56 PM, Christian Balzer <ch...@gol.com> wrote: > > Hello, > > On Thu, 4 Feb 2016 08:44:25 -0800 Cullen King wrote: > > > Replies in-line: > > > > On Wed, Feb 3, 2016 at 9:54 PM, Christian Balzer > > <c-bal...@fusioncom.co.jp> wrote: > > > > > > > > Hello, > > > > > > On Wed, 3 Feb 2016 17:48:02 -0800 Cullen King wrote: > > > > > > > Hello, > > > > > > > > I've been trying to nail down a nasty performance issue related to > > > > scrubbing. I am mostly using radosgw with a handful of buckets > > > > containing millions of various sized objects. When ceph scrubs, both > > > > regular and deep, radosgw blocks on external requests, and my > > > > cluster has a bunch of requests that have blocked for > 32 seconds. > > > > Frequently OSDs are marked down. > > > > > > > From my own (painful) experiences let me state this: > > > > > > 1. When your cluster runs out of steam during deep-scrubs, drop what > > > you're doing and order more HW (OSDs). > > > Because this is a sign that it would also be in trouble when doing > > > recoveries. > > > > > > > When I've initiated recoveries from working on the hardware the cluster > > hasn't had a problem keeping up. It seems that it only has a problem with > > scrubbing, meaning it feels like the IO pattern is drastically > > different. I would think that with scrubbing I'd see something closer to > > bursty sequential reads, rather than just thrashing the drives with a > > more random IO pattern, especially given our low cluster utilization. > > > It's probably more pronounced when phasing in/out entire OSDs, where it > also has to read the entire (primary) data off it. > > > > > > > > > 2. If you cluster is inconvenienced by even mere scrubs, you're really > > > in trouble. > > > Threaten the penny pincher with bodily violence and have that new HW > > > phased in yesterday. > > > > > > > I am the penny pincher, biz owner, dev and ops guy for > > http://ridewithgps.com :) More hardware isn't an issue, it just feels > > pretty crazy to have this low of performance on a 12 OSD system. Granted, > > that feeling isn't backed by anything concrete! In general, I like to > > understand the problem before I solve it with hardware, though I am > > definitely not averse to it. I already ordered 6 more 4tb drives along > > with the new journal SSDs, anticipating the need. > > > > As you can see from the output of ceph status, we are not space hungry by > > any means. > > > > Well, in Ceph having just one OSD pegged to max will impact (eventually) > everything when they need to read/write primary PGs on it. > > More below. > > > > > > > > > > According to atop, the OSDs being deep scrubbed are reading at only > > > > 5mb/s to 8mb/s, and a scrub of a 6.4gb placement group takes 10-20 > > > > minutes. > > > > > > > > Here's a screenshot of atop from a node: > > > > https://s3.amazonaws.com/rwgps/screenshots/DgSSRyeF.png > > > > > > > This looks familiar. > > > Basically at this point in time the competing read request for all the > > > objects clash with write requests and completely saturate your HD > > > (about 120 IOPS and 85% busy according to your atop screenshot). > > > > > > > In your experience would the scrub operation benefit from a bigger > > readahead? Meaning is it more sequential than random reads? I already > > bumped /sys/block/sd{x}/queue/read_ahead_kb to 512kb. > > > I played with that long time ago (in benchmark scenarios) and didn't see > any noticeable improvement. > Deep-scrub might (fragmentation could hurt it though), regular scrub not so > much. > > > About half of our reads are on objects with an average size of 40kb (map > > thumbnails), and the other half are on photo thumbs with a size between > > 10kb and 150kb. > > > > Noted, see below. > > > After doing a little more researching, I came across this: > > > > > http://tracker.ceph.com/projects/ceph/wiki/Optimize_Newstore_for_massive_small_object_storage > > > > Sounds like I am probably running into issues with lots of random read > > IO, combined with known issues around small files. To give an idea, I > > have about 15 million small map thumbnails stored in my two largest > > buckets, and I am pushing out about 30 requests per second right now > > from those two buckets. > > > This is certainly a factor, but that knowledge of a future improvement > won't help you with your current problem of course. ^_- > > > > > > > > There are ceph configuration options that can mitigate this to some > > > extend and which I don't see in your config, like > > > "osd_scrub_load_threshold" and "osd_scrub_sleep" along with the > > > various IO priority settings. > > > However the points above still stand. > > > > > > > Yes, I have a running series of notes of config options to try out, just > > wanted to touch base with other community members before shooting in the > > dark. > > > osd_scrub_sleep is probably the most effective immediately available > option for you to prevent slow, stalled IO. > At the obvious cost of scrubs taking even longer. > There is of course also the option to disable scrubs entirely until your HW > has been upgraded. > > > > > > > > > XFS defragmentation might help, significantly if your FS is badly > > > fragmented. But again, this is only a temporary band-aid. > > > > > > > First question: is this a reasonable speed for scrubbing, given a > > > > very lightly used cluster? Here's some cluster details: > > > > > > > > deploy@drexler:~$ ceph --version > > > > ceph version 0.94.1-5-g85a68f9 > > > > (85a68f9a8237f7e74f44a1d1fbbd6cb4ac50f8e8) > > > > > > > > > > > > 2x Xeon E5-2630 per node, 64gb of ram per node. > > > > > > > More memory can help by keeping hot objects in the page cache (so the > > > actual disks need not be read and can write at their full IOPS > > > capacity). A lot of memory (and the correct sysctl settings) will also > > > allow for a large SLAB space, keeping all those directory entries and > > > other bits in memory without having to go to disk to get them. > > > > > > You seem to be just fine CPU wise. > > > > > > > I thought about bumping each node up to 128gb of ram as another cheap > > insurance policy. I'll try that after the other changes. I'd like to know > > why so I'll try and change one thing at a time, though I am also just > > eager to have this thing stable. > > > > For me everything was sweet and dandy as long all the really hot objects > did fit in the page cache and the FS bits where all in SLAB (no need to > go to disk for a "ls -R"). > > Past the point it all went to molasses land "quickly". > > > > > > > > > > > > > > deploy@drexler:~$ ceph status > > > > cluster 234c6825-0e2b-4256-a710-71d29f4f023e > > > > health HEALTH_WARN > > > > 118 requests are blocked > 32 sec > > > > monmap e1: 3 mons at {drexler= > > > > 10.0.0.36:6789/0,lucy=10.0.0.38:6789/0,paley=10.0.0.34:6789/0} > > > > election epoch 296, quorum 0,1,2 paley,drexler,lucy > > > > mdsmap e19989: 1/1/1 up {0=lucy=up:active}, 1 up:standby > > > > osdmap e1115: 12 osds: 12 up, 12 in > > > > pgmap v21748062: 1424 pgs, 17 pools, 3185 GB data, 20493 > > > > kobjects 10060 GB used, 34629 GB / 44690 GB avail > > > > 1422 active+clean > > > > 1 active+clean+scrubbing+deep > > > > 1 active+clean+scrubbing > > > > client io 721 kB/s rd, 33398 B/s wr, 53 op/s > > > > > > > You want to avoid having scrubs going on willy-nilly in parallel and at > > > high peek times, even IF your cluster is capable of handling them. > > > > > > Depending on how busy your cluster is and its usage pattern, you may do > > > what I did. > > > Kick off a deep scrub of all OSDs "ceph osd deep-scrub \*" like 01:00 > > > on a Saturday morning. > > > If your cluster is fast enough, it will finish before 07:00 (without > > > killing your client performance) and all regular scrubs will now > > > happen in that time frame as well (given default settings). > > > If your cluster isn't fast enough, see my initial 2 points. ^o^ > > > > > > > The problem is our cluster is the image and upload store for our site > > which is a reasonably busy site international site. We have about 60% of > > our customers in North America, and 30% or so in Europe and Asia. We > > definitely would be better off with more scrubs between 11pm and 7am -8 > > to 0 GMT, though we can't afford to slam the cluster. > > > > I suppose that our cluster is a much more random mix of reads than many > > others using ceph as a RBD store. Operating systems probably have a > > stronger mix of sequential reads, whereas our users are concurrently > > viewing different pages with different images, a more random workload. > > > > It sounds like we have to maintain a cluster storage capacity of less > > than 25% in order to have reasonable performance. I guess this makes > > sense, we have much more random IO needs than storage needs. > > > In your use case (and most others) random IOPS tends to be the bottleneck > long long before either space or sequential bandwidth becomes and issues. > > More spindles, more IOPS. See below. ^o^ > > > > > > > > > > deploy@drexler:~$ ceph osd tree > > > > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > > > > -1 43.67999 root default > > > > -2 14.56000 host paley > > > > 0 3.64000 osd.0 up 1.00000 1.00000 > > > > 3 3.64000 osd.3 up 1.00000 1.00000 > > > > 6 3.64000 osd.6 up 1.00000 1.00000 > > > > 9 3.64000 osd.9 up 1.00000 1.00000 > > > > -3 14.56000 host lucy > > > > 1 3.64000 osd.1 up 1.00000 1.00000 > > > > 4 3.64000 osd.4 up 1.00000 1.00000 > > > > 7 3.64000 osd.7 up 1.00000 1.00000 > > > > 11 3.64000 osd.11 up 1.00000 1.00000 > > > > -4 14.56000 host drexler > > > > 2 3.64000 osd.2 up 1.00000 1.00000 > > > > 5 3.64000 osd.5 up 1.00000 1.00000 > > > > 8 3.64000 osd.8 up 1.00000 1.00000 > > > > 10 3.64000 osd.10 up 1.00000 1.00000 > > > > > > > > > > > > My OSDs are 4tb 7200rpm Hitachi DeskStars, using XFS, with Samsung > > > > 850 Pro journals (very slow, ordered s3700 replacements, but > > > > shouldn't pose problems for reading as far as I understand things). > > > > > > Just to make sure, these are genuine DeskStars? > > > I'm asking both because AFAIK they are out of production and their > > > direct successors, the Toshiba DT drives (can) have a nasty firmware > > > bug that totally ruins their performance (from ~8 hours per week to > > > permanently until power-cycled). > > > > > > > These are original deskstars. Didn't realize they weren't in production, > > I just grabbed 6 more of the Hitachi DeskStar NAS edition 4tb drives, > > which are readily available. I probably should have ordered 6tb drives, > > as I'd end up with better seek times due to them not being fully > > utilized - the data would reside closer to the center of the platters. > > > Ah, Deskstar NAS, yes, these still are in production. > > I'd get more, smaller, faster HDDs instead. > HW cache on your controller can also help (depends on the model/FW if it > is used efficiently in JBOD mode). > > And since your space utilization is small (though of course that can and > will change over time of course), you may very well benefit from going > SSD. > > SSD pools if you think you can fit (economically) a set of your high > access data like the thumbnails on it. > > SSD cache tiers are a bit more dubious when comes to rewards, but that > depends a lot on the hot data set. > Plenty of discussion in here about that. > > Regards, > > Christian > > > > > > > > Regards, > > > > > > Christian > > > > MONs are co-located > > > > with OSD nodes, but the nodes are fairly beefy and has very low load. > > > > Drives are on a expanding backplane, with an LSI SAS3008 controller. > > > > > > > > I have a fairly standard config as well: > > > > > > > > https://gist.github.com/kingcu/aae7373eb62ceb7579da > > > > > > > > I know that I don't have a ton of OSDs, but I'd expect a little > > > > better performance than this. Checkout munin of my three nodes: > > > > > > > > > > > > http://munin.ridewithgps.com/ridewithgps.com/drexler.ridewithgps.com/index.html#disk > > > > > > > > http://munin.ridewithgps.com/ridewithgps.com/paley.ridewithgps.com/index.html#disk > > > > > > > > http://munin.ridewithgps.com/ridewithgps.com/lucy.ridewithgps.com/index.html#disk > > > > > > > > > > > > Any input would be appreciated, before I start trying to > > > > micro-optimize config params, as well as upgrading to Infernalis. > > > > > > > > > > > > Cheers, > > > > > > > > Cullen > > > > > > > > > -- > > > Christian Balzer Network/Systems Engineer > > > ch...@gol.com Global OnLine Japan/Rakuten Communications > > > http://www.gol.com/ > > > > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Rakuten Communications > http://www.gol.com/ >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
