Re: [ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
Robin,
Would you generate the values and keys for the various versions of at
least one of the objects? .dir.default.292886573.13181.12 is a good
example because there are 3 variations for the same object.
If there isn't much activity to .dir.default.64449186.344176, you could
do one osd at a time. Otherwise, stop all 3 OSDs 1322, 990, 655 execute
these for all 3. I suspect you'll need to pipe to "od-cx" to get
printable output.
I created a simple object with ascii omap.
$ ceph-objectstore-tool --data-path ... --pgid 5.3d40
.dir.default.64449186.344176 get-omaphdr
obj_header
$ for i in $(ceph-objectstore-tool --data-path ... --pgid 5.3d40
.dir.default.64449186.344176 list-omap)
do
echo -n "${i}: "
ceph-objectstore-tool --data-path ... .dir.default.292886573.13181.12
get-omap $i
done
key1: val1
key2: val2
key3: val3
David
On 9/8/17 12:18 PM, David Zafman wrote:
Robin,
The only two changesets I can spot in Jewel that I think might be
related are these:
1.
http://tracker.ceph.com/issues/20089
https://github.com/ceph/ceph/pull/15416
This should improve the repair functionality.
2.
http://tracker.ceph.com/issues/19404
https://github.com/ceph/ceph/pull/14204
This pull request fixes an issue that corrupted omaps. It also finds
and repairs them. However, the repair process might resurrect deleted
omaps which would show up as an omap digest error.
This could temporarily cause additional inconsistent PGs. So if this
has NOT been occurring longer than your deep-scrub interval since
upgrading, I'd repair the pgs and monitor going forward to make sure
the problem doesn't recur.
---
You have good example of repair scenarios:
.dir.default.292886573.13181.12 only has a omap_digest_mismatch and
no shard errors. The automatic repair won't be sure which is a good
copy.
In this case we can see that osd 1327 doesn't match the other two. To
assist the repair process to repair the right one. Remove the copy on
osd.1327
Stop osd 1327 and use "ceph-objectstore-tool --data-path .1327
.dir.default.292886573.13181.12 remove"
.dir.default.64449186.344176 has selected_object_info with "od
337cf025" so shards have "omap_digest_mismatch_oi" except for osd 990.
The pg repair code will use osd.990 to fix the other 2 copies without
further handling.
David
On 9/8/17 11:16 AM, Robin H. Johnson wrote:
On Thu, Sep 07, 2017 at 08:24:04PM +, Robin H. Johnson wrote:
pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655]
pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91]
Here is the output of 'rados list-inconsistent-obj' for the PGs:
$ sudo rados list-inconsistent-obj 5.f1c0 |json_pp -json_opt
canonical,pretty
{
"epoch" : 1221254,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.292886573.13181.12",
"nspace" : "",
"snap" : "head",
"version" : 483490
},
"selected_object_info" :
"5:038f1cff:::.dir.default.292886573.13181.12:head(1221843'483490
client.417313345.0:19515832 dirty|omap|data_digest s 0 uv 483490 dd
alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 91,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 631,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x6556c868",
"osd" : 1327,
"size" : 0
}
],
"union_shard_errors" : []
}
]
}
$ sudo rados list-inconsistent-obj 5.3d40 |json_pp -json_opt
canonical,pretty
{
"epoch" : 1210895,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.64449186.344176",
"nspace" : "",
"snap" : "head",
"version" : 1177199
},
"selected_object_info" :
"5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639
osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199
dd od 337cf025 alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0x3242b04e",
"osd" : 655,
"size" : 0
},
{
"data_digest" : "0x",
Re: [ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
Robin,
The only two changesets I can spot in Jewel that I think might be
related are these:
1.
http://tracker.ceph.com/issues/20089
https://github.com/ceph/ceph/pull/15416
This should improve the repair functionality.
2.
http://tracker.ceph.com/issues/19404
https://github.com/ceph/ceph/pull/14204
This pull request fixes an issue that corrupted omaps. It also finds
and repairs them. However, the repair process might resurrect deleted
omaps which would show up as an omap digest error.
This could temporarily cause additional inconsistent PGs. So if this
has NOT been occurring longer than your deep-scrub interval since
upgrading, I'd repair the pgs and monitor going forward to make sure the
problem doesn't recur.
---
You have good example of repair scenarios:
.dir.default.292886573.13181.12 only has a omap_digest_mismatch and no
shard errors. The automatic repair won't be sure which is a good copy.
In this case we can see that osd 1327 doesn't match the other two. To
assist the repair process to repair the right one. Remove the copy on
osd.1327
Stop osd 1327 and use "ceph-objectstore-tool --data-path .1327
.dir.default.292886573.13181.12 remove"
.dir.default.64449186.344176 has selected_object_info with "od 337cf025"
so shards have "omap_digest_mismatch_oi" except for osd 990.
The pg repair code will use osd.990 to fix the other 2 copies without
further handling.
David
On 9/8/17 11:16 AM, Robin H. Johnson wrote:
On Thu, Sep 07, 2017 at 08:24:04PM +, Robin H. Johnson wrote:
pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655]
pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91]
Here is the output of 'rados list-inconsistent-obj' for the PGs:
$ sudo rados list-inconsistent-obj 5.f1c0 |json_pp -json_opt canonical,pretty
{
"epoch" : 1221254,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.292886573.13181.12",
"nspace" : "",
"snap" : "head",
"version" : 483490
},
"selected_object_info" :
"5:038f1cff:::.dir.default.292886573.13181.12:head(1221843'483490 client.417313345.0:19515832
dirty|omap|data_digest s 0 uv 483490 dd alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 91,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 631,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x6556c868",
"osd" : 1327,
"size" : 0
}
],
"union_shard_errors" : []
}
]
}
$ sudo rados list-inconsistent-obj 5.3d40 |json_pp -json_opt canonical,pretty
{
"epoch" : 1210895,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.64449186.344176",
"nspace" : "",
"snap" : "head",
"version" : 1177199
},
"selected_object_info" :
"5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639 osd.1322.0:537914
dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd od 337cf025 alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0x3242b04e",
"osd" : 655,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x337cf025",
"osd" : 990,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0xc90d06a8",
"osd" : 1322,
"size" : 0
}
],
"union_shard_errors" : [
"omap_digest_mismatch_oi"
]
}
]
}
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
On Thu, Sep 07, 2017 at 08:24:04PM +, Robin H. Johnson wrote:
> pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655]
> pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91]
Here is the output of 'rados list-inconsistent-obj' for the PGs:
$ sudo rados list-inconsistent-obj 5.f1c0 |json_pp -json_opt canonical,pretty
{
"epoch" : 1221254,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.292886573.13181.12",
"nspace" : "",
"snap" : "head",
"version" : 483490
},
"selected_object_info" :
"5:038f1cff:::.dir.default.292886573.13181.12:head(1221843'483490
client.417313345.0:19515832 dirty|omap|data_digest s 0 uv 483490 dd
alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 91,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 631,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x6556c868",
"osd" : 1327,
"size" : 0
}
],
"union_shard_errors" : []
}
]
}
$ sudo rados list-inconsistent-obj 5.3d40 |json_pp -json_opt canonical,pretty
{
"epoch" : 1210895,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.64449186.344176",
"nspace" : "",
"snap" : "head",
"version" : 1177199
},
"selected_object_info" :
"5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639
osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd
od 337cf025 alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0x3242b04e",
"osd" : 655,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x337cf025",
"osd" : 990,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0xc90d06a8",
"osd" : 1322,
"size" : 0
}
],
"union_shard_errors" : [
"omap_digest_mismatch_oi"
]
}
]
}
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
Hi, Our clusters were upgraded to v10.2.9, from ~v10.2.7 (actually a local git snapshot that was not quite 10.2.7), and since then, we're seeing a LOT more scrub errors than previously. The digest logging on the scrub errors, in some cases, is also now maddeningly short: it doesn't contain ANY information on what the mismatch was, and many of the errors seem to also be 3-way mismatches in the digest :-(. I'm wondering if other people have seen something similar rises in scrub errors in the upgrade, and/or the lack of digest output. I did hear one anecdotal report that 10.2.9 seemed much more likely to fail out marginal disks. The only two changesets I can spot in Jewel that I think might be related are these: 1. http://tracker.ceph.com/issues/20089 https://github.com/ceph/ceph/pull/15416 2. http://tracker.ceph.com/issues/19404 https://github.com/ceph/ceph/pull/14204 Two example PGs that are inconsistent (chosen because they didn't convey any private information so I didn't have to redact anything except IP): $ sudo ceph health detail |grep -e 5.3d40 -e 5.f1c0 pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655] pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91] $ fgrep 5.3d40 /var/log/ceph/ceph.log 2017-09-07 19:50:16.231523 osd.1322 [REDACTED::8861]:6808/3479303 1736 : cluster [INF] osd.1322 pg 5.3d40 Deep scrub errors, upgrading scrub to deep-scrub 2017-09-07 19:50:16.231862 osd.1322 [REDACTED::8861]:6808/3479303 1737 : cluster [INF] 5.3d40 deep-scrub starts 2017-09-07 19:54:38.631232 osd.1322 [REDACTED::8861]:6808/3479303 1738 : cluster [ERR] 5.3d40 shard 655: soid 5:02bc4def:::.dir.default.64449186.344176:head omap_digest 0x3242b04e != omap_digest 0x337cf025 from auth oi 5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639 osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd od 337cf025 alloc_hint [0 0]) 2017-09-07 19:54:38.631332 osd.1322 [REDACTED::8861]:6808/3479303 1739 : cluster [ERR] 5.3d40 shard 1322: soid 5:02bc4def:::.dir.default.64449186.344176:head omap_digest 0xc90d06a8 != omap_digest 0x3242b04e from shard 655, omap_digest 0xc90d06a8 != omap_digest 0x337cf025 from auth oi 5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639 osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd od 337cf025 alloc_hint [0 0]) 2017-09-07 20:03:54.721681 osd.1322 [REDACTED::8861]:6808/3479303 1740 : cluster [ERR] 5.3d40 deep-scrub 0 missing, 1 inconsistent objects 2017-09-07 20:03:54.721687 osd.1322 [REDACTED::8861]:6808/3479303 1741 : cluster [ERR] 5.3d40 deep-scrub 3 errors $ fgrep 5.f1c0 /var/log/ceph/ceph.log 2017-09-07 11:11:36.773986 osd.631 [REDACTED::8877]:6813/4036028 4234 : cluster [INF] osd.631 pg 5.f1c0 Deep scrub errors, upgrading scrub to deep-scrub 2017-09-07 11:11:36.774127 osd.631 [REDACTED::8877]:6813/4036028 4235 : cluster [INF] 5.f1c0 deep-scrub starts 2017-09-07 11:25:26.231502 osd.631 [REDACTED::8877]:6813/4036028 4236 : cluster [ERR] 5.f1c0 deep-scrub 0 missing, 1 inconsistent objects 2017-09-07 11:25:26.231508 osd.631 [REDACTED::8877]:6813/4036028 4237 : cluster [ERR] 5.f1c0 deep-scrub 1 errors -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
