RE: [ceph-users] Flapping osd / continuously reported as failed

2013-07-25 Thread Mostowiec Dominik
make it a little more light by changing config? -- Regards Dominik -Original Message- From: StudziƄski Krzysztof Sent: Wednesday, July 24, 2013 9:48 AM To: Gregory Farnum; Yehuda Sadeh Cc: ceph-devel@vger.kernel.org; ceph-us...@lists.ceph.com; Mostowiec Dominik Subject: RE: [ceph-users

RE: [ceph-users] Flapping osd / continuously reported as failed

2013-08-16 Thread Mostowiec Dominik
Hi, Thanks for your response. > It's possible, as deep scrub in particular will add a bit of load (it > goes through and compares the object contents). It is possible that the scrubbing blocks access(RW or only W) to bucket index when check .dir... file? When rgw index is very large I guess it

large memory leak on scrubbing

2013-08-16 Thread Mostowiec Dominik
Hi, We noticed some issues on CEPH/S3 cluster, I think it related with scrubbing: large memory leaks. Logs 09.xx: https://www.dropbox.com/s/4z1fzg239j43igs/ceph-osd.4.log_09xx.tar.gz >From 09.30 to 09.44 (14 minutes) osd.4 proces grows up to 28G. I think this is something curious: 2013-08-16 09

RE: [ceph-users] large memory leak on scrubbing

2013-08-19 Thread Mostowiec Dominik
that "ceph osd scrub", or "ceph osd deep-scrub" trigger scrub on osd but not for all PG. It is possible to trigger scrubbing all PG on one osd? -- Regards Dominik -Original Message- From: Sage Weil [mailto:s...@inktank.com] Sent: Saturday, August 17, 2013 5:11 PM T

RE: [ceph-users] large memory leak on scrubbing

2013-08-19 Thread Mostowiec Dominik
Hi, > Is that the only slow request message you see? No. Full log: https://www.dropbox.com/s/i3ep5dcimndwvj1/slow_requests.txt.tar.gz It start from: 2013-08-16 09:43:39.662878 mon.0 10.174.81.132:6788/0 4276384 : [DBG] osd.4 10.174.81.131:6805/31460 reported failed by osd.50 10.174.81.135:6842/26

RE: [ceph-users] Flapping osd / continuously reported as failed

2013-08-19 Thread Mostowiec Dominik
Hi, > Yes, it definitely can as scrubbing takes locks on the PG, which will prevent > reads or writes while the message is being processed (which will involve the > rgw index being scanned). It is possible to tune scrubbing config for eliminate slow requests and marking osd down when large rgw b

bucket count limit

2013-08-22 Thread Mostowiec Dominik
Hi, I think about sharding s3 buckets in CEPH cluster, create bucket-per-XX (256 buckets) or even bucket-per-XXX (4096 buckets) where XXX is sign from object md5 url. Could this be the problem? (performance, or some limits) -- Regards Dominik -- To unsubscribe from this list: send the line "unsu