Re: [ceph-users] Performance issues related to scrubbing

2016-02-17 Thread Cullen King
On Wed, Feb 17, 2016 at 12:13 AM, Christian Balzer wrote: > > Hello, > > On Tue, 16 Feb 2016 10:46:32 -0800 Cullen King wrote: > > > Thanks for the helpful commentary Christian. Cluster is performing much > > better with 50% more spindles (12 to 18 drives), along with setting

Re: [ceph-users] Performance issues related to scrubbing

2016-02-17 Thread Christian Balzer
Hello, On Tue, 16 Feb 2016 10:46:32 -0800 Cullen King wrote: > Thanks for the helpful commentary Christian. Cluster is performing much > better with 50% more spindles (12 to 18 drives), along with setting scrub > sleep to 0.1. Didn't see really any gain from moving from the Samsung 850 > Pro

Re: [ceph-users] Performance issues related to scrubbing

2016-02-16 Thread Cullen King
Thanks for the helpful commentary Christian. Cluster is performing much better with 50% more spindles (12 to 18 drives), along with setting scrub sleep to 0.1. Didn't see really any gain from moving from the Samsung 850 Pro journal drives to Intel 3710's, even though dd and other direct tests of

Re: [ceph-users] Performance issues related to scrubbing

2016-02-16 Thread Cullen King
Thanks for the tuning tips Bob, I'll play with them after solidifying some of my other fixes (another 24-48 hours before my migration to 1024 placement groups is finished). Glad you enjoy ridewithgps, shoot me an email if you have any questions/ideas/needs :) On Fri, Feb 5, 2016 at 4:42 PM, Bob

Re: [ceph-users] Performance issues related to scrubbing

2016-02-05 Thread Bob R
Cullen, We operate a cluster with 4 nodes, each has 2xE5-2630, 64gb ram, 10x4tb spinners. We've recently replaced 2xm550 journals with a single p3700 nvme drive per server and didn't see the performance gains we were hoping for. After making the changes below we're now seeing significantly better

Re: [ceph-users] Performance issues related to scrubbing

2016-02-04 Thread Christian Balzer
Hello, On Thu, 4 Feb 2016 08:44:25 -0800 Cullen King wrote: > Replies in-line: > > On Wed, Feb 3, 2016 at 9:54 PM, Christian Balzer > wrote: > > > > > Hello, > > > > On Wed, 3 Feb 2016 17:48:02 -0800 Cullen King wrote: > > > > > Hello, > > > > > > I've been trying

Re: [ceph-users] Performance issues related to scrubbing

2016-02-04 Thread Cullen King
Replies in-line: On Wed, Feb 3, 2016 at 9:54 PM, Christian Balzer wrote: > > Hello, > > On Wed, 3 Feb 2016 17:48:02 -0800 Cullen King wrote: > > > Hello, > > > > I've been trying to nail down a nasty performance issue related to > > scrubbing. I am mostly using radosgw

[ceph-users] Performance issues related to scrubbing

2016-02-03 Thread Cullen King
Hello, I've been trying to nail down a nasty performance issue related to scrubbing. I am mostly using radosgw with a handful of buckets containing millions of various sized objects. When ceph scrubs, both regular and deep, radosgw blocks on external requests, and my cluster has a bunch of

Re: [ceph-users] Performance issues related to scrubbing

2016-02-03 Thread Christian Balzer
Hello, On Wed, 3 Feb 2016 17:48:02 -0800 Cullen King wrote: > Hello, > > I've been trying to nail down a nasty performance issue related to > scrubbing. I am mostly using radosgw with a handful of buckets containing > millions of various sized objects. When ceph scrubs, both regular and >