Hi,
On 20 Aug 2014, at 16:55, German Anders
<gand...@despegar.com<mailto:gand...@despegar.com>> wrote:
Hi Dan,
How are you? I want to know how you disable the indexing on the
/var/lib/ceph OSDs?
# grep ceph /etc/updatedb.conf
PRUNEPATHS = "/afs /media /net /sfs /tmp /udev /var/cache/ccache
/var/spool/cups /var/spool/squid /var/tmp /var/lib/ceph"
Did you disable deep scrub on you OSDs?
No but this can be an issue. If you get many PGs scrubbing at once, performance
will suffer.
There is a new feature in 0.67.10 to sleep between scrubbing “chunks”. I set
the that sleep to 0.1 (and the chunk_max to 5, and the scrub size 1MB). In
0.67.10+1 there are some new options to set the iopriority of the scrubbing
threads. Set that to class = 3, priority = 0 to give the scrubbing thread the
idle priority. You need to use the cfq disk scheduler for io priorities to
work. (cfq will also help if updatedb is causing any problems, since it runs
with ionice -c 3).
I’m pretty sure those features will come in 0.80.6 as well.
Do you have the journals on SSD's or RAMDISK?
Never use RAMDISK.
We currently have the journals on the same spinning disk as the OSD, but the
iops performance is low for the rbd and fs use-cases. (For object store it
should be OK). But for rbd or fs, you really need journals on SSDs or your
cluster will suffer.
We now have SSDs on order to augment our cluster. (The way I justified this is
that our cluster has X TB of storage capacity and Y iops capacity. With disk
journals we will run out of iops capacity well before we run out of storage
capacity. So you can either increase the iops capacity substantially by
decreasing the volume of the cluster by 20% and replacing those disks with SSD
journals, or you can just leave 50% of the disk capacity empty since you can’t
use it anyway).
What's the perf of your cluster? randos bench? fio? I've setup a new cluster
and I want to know what would be the best option scheme to go.
It’s not really meaningful to compare performance of different clusters with
different hardware. Some “constants” I can advise
- with few clients, large write throughput is limited by the clients
bandwidth, as long as you have enough OSDs and the client is striping over many
objects.
- with disk journals, small write latency will be ~30-50ms even when the
cluster is idle. if you have SSD journals, maybe ~10ms.
- count your iops. Each disk OSD can do ~100, and you need to divide by the
number of replicas. With SSDs you can do a bit better than this since the
synchronous writes go to the SSDs not the disks. In my tests with our hardware
I estimate that going from disk to SSD journal will multiply the iops capacity
by around 5x.
I also found that I needed to increase some the journal max write and journal
queue max limits, also the filestore limits, to squeeze the best performance
out of the SSD journals. Try increasing filestore queue max ops/bytes,
filestore queue committing max ops/bytes, and the filestore wbthrottle xfs *
options. (I’m not going to publish exact configs here because I haven’t
finished tuning yet).
Cheers, Dan
Thanks a lot!!
Best regards,
German Anders
On Wednesday 20/08/2014 at 11:51, Dan Van Der Ster wrote:
Hi,
Do you get slow requests during the slowness incidents? What about monitor
elections?
Are your MDSs using a lot of CPU? did you try tuning anything in the MDS (I
think the default config is still conservative, and there are options to cache
more entries, etc…)
What about iostat on the OSDs — are your OSD disks busy reading or writing
during these incidents?
What are you using for OSD journals?
Also check the CPU usage for the mons and osds...
Does your hardware provide enough IOPS for what your users need? (e.g. what is
the op/s from ceph -w)
If disabling deep scrub helps, then it might be that something else is reading
the disks heavily. One thing to check is updatedb — we had to disable it from
indexing /var/lib/ceph on our OSDs.
Best Regards,
Dan
-- Dan van der Ster || Data & Storage Services || CERN IT Department --
On 20 Aug 2014, at 16:39, Hugo Mills
<h.r.mi...@reading.ac.uk<mailto:h.r.mi...@reading.ac.uk>> wrote:
We have a ceph system here, and we're seeing performance regularly
descend into unusability for periods of minutes at a time (or longer).
This appears to be triggered by writing large numbers of small files.
Specifications:
ceph 0.80.5
6 machines running 3 OSDs each (one 4 TB rotational HD per OSD, 2 threads)
2 machines running primary and standby MDS
3 monitors on the same machines as the OSDs
Infiniband to about 8 CephFS clients (headless, in the machine room)
Gigabit ethernet to a further 16 or so CephFS clients (Linux desktop
machines, in the analysis lab)
The cluster stores home directories of the users and a larger area
of scientific data (approx 15 TB) which is being processed and
analysed by the users of the cluster.
We have a relatively small number of concurrent users (typically
4-6 at most), who use GUI tools to examine their data, and then
complex sets of MATLAB scripts to process it, with processing often
being distributed across all the machines using Condor.
It's not unusual to see the analysis scripts write out large
numbers (thousands, possibly tens or hundreds of thousands) of small
files, often from many client machines at once in parallel. When this
happens, the ceph cluster becomes almost completely unresponsive for
tens of seconds (or even for minutes) at a time, until the writes are
flushed through the system. Given the nature of modern GUI desktop
environments (often reading and writing small state files in the
user's home directory), this means that desktop interactiveness and
responsiveness for all the other users of the cluster suffer.
1-minute load on the servers typically peaks at about 8 during
these events (on 4-core machines). Load on the clients also peaks
high, because of the number of processes waiting for a response from
the FS. The MDS shows little sign of stress -- it seems to be entirely
down to the OSDs. ceph -w shows requests blocked for more than 10
seconds, and in bad cases, ceph -s shows up to many hundreds of
requests blocked for more than 32s.
We've had to turn off scrubbing and deep scrubbing completely --
except between 01.00 and 04.00 every night -- because it triggers the
exact same symptoms, even with only 2-3 PGs being scrubbed. If it gets
up to 7 PGs being scrubbed, as it did on Monday, it's completely
unusable.
Is this problem something that's often seen? If so, what are the
best options for mitigation or elimination of the problem? I've found
a few references to issue #6278 [1], but that seems to be referencing
scrub specifically, not ordinary (if possibly pathological) writes.
What are the sorts of things I should be looking at to work out
where the bottleneck(s) are? I'm a bit lost about how to drill down
into the ceph system for identifying performance issues. Is there a
useful guide to tools somewhere?
Is an upgrade to 0.84 likely to be helpful? How "development" are
the development releases, from a stability / dangerous bugs point of
view?
Thanks,
Hugo.
[1] http://tracker.ceph.com/issues/6278
--
Hugo Mills :: IT Services, University of Reading
Specialist Engineer, Research Servers
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
