Are you running tight on RAM? You might be running into http://tracker.ceph.com/issues/22464
Paul 2018-06-28 17:17 GMT+02:00 Bryan Banister <bbanis...@jumptrading.com>: > Hi all, > > > > We started running a EC pool based object store, set up with a 4+2 > configuration, and we seem to be getting an almost constant report of > inconsistent PGs during scrub operations. For example: > > root@rook-tools:/# ceph pg ls inconsistent > > PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES > LOG DISK_LOG STATE STATE_STAMP > VERSION REPORTED UP UP_PRIMARY > ACTING ACTING_PRIMARY LAST_SCRUB > SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP > > 19.26 104 0 0 0 0 436207616 > 1537 1537 active+clean+inconsistent 2018-06-28 15:04:54.054227 > 1811'3137 2079:5075 [206,116,68,31,193,156] 206 > [206,116,68,31,193,156] 206 1811'3137 2018-06-27 > 21:00:17.011611 1811'3137 2018-06-27 21:00:17.011611 > > 19.234 98 0 0 0 0 406847488 > 1581 1581 active+clean+inconsistent 2018-06-28 15:05:18.077003 > 1811'2981 2080:4822 [28,131,229,180,84,68] 28 > [28,131,229,180,84,68] 28 1811'2981 2018-06-28 > 14:09:54.092401 1811'2981 2018-06-28 14:09:54.092401 > > 19.2a8 116 0 0 0 0 486539264 > 1561 1561 active+clean+inconsistent 2018-06-28 15:04:54.073762 > 1811'3161 2079:4825 [177,68,222,13,131,107] 177 > [177,68,222,13,131,107] 177 1811'3161 2018-06-28 > 07:51:21.109587 1811'3161 2018-06-28 07:51:21.109587 > > 19.406 126 0 0 0 0 520233399 > 1557 1557 active+clean+inconsistent 2018-06-28 15:04:57.142651 > 1811'3057 2080:4944 [230,199,128,68,92,11] 230 > [230,199,128,68,92,11] 230 1811'3057 2018-06-27 > 18:36:18.497899 1811'3057 2018-06-27 18:36:18.497899 > > 19.46b 109 0 0 0 0 449840274 > 1558 1558 active+clean+inconsistent 2018-06-28 15:04:54.227970 > 1811'3058 2079:4986 [18,68,130,94,181,225] 18 > [18,68,130,94,181,225] 18 1811'3058 2018-06-27 > 14:32:17.800961 1811'3058 2018-06-27 14:32:17.800961 > > [snip] > > > > We sometimes see that running a deep scrub on the PG resolves the issue > but not all the time. > > > > We have been running the PG repair operation on them (e.g. ceph pg repair > <pg num>), which clears the issue. Is this the correct way to resolve this > issue? > > > > Is this a normal behavior for a Ceph cluster? > > > > If so, why doesn’t it attempt to repair itself automatically? > > > > Thanks for the help understanding Ceph, we are very new to it still! > > -Bryan > > > > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential, or privileged information > and/or personal data. If you are not the intended recipient, you are hereby > notified that any review, dissemination, or copying of this email is > strictly prohibited, and requested to notify the sender immediately and > destroy this email and any attachments. Email transmission cannot be > guaranteed to be secure or error-free. The Company, therefore, does not > make any guarantees as to the completeness or accuracy of this email or any > attachments. This email is for informational purposes only and does not > constitute a recommendation, offer, request, or solicitation of any kind to > buy, sell, subscribe, redeem, or perform any type of transaction of a > financial product. Personal data, as defined by applicable data privacy > laws, contained in this email may be processed by the Company, and any of > its affiliated or related companies, for potential ongoing compliance > and/or business-related purposes. You may have rights regarding your > personal data; for information on exercising these rights or the Company’s > treatment of personal data, please email datareque...@jumptrading.com. > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
