That's right. But please read the notes carefully to understand if you need to set osd skip data digest = true or osd distrust data digest = true
.. dan On Fri, Jul 20, 2018 at 10:02 AM Glen Baars <g...@onsitecomputers.com.au> wrote: > > I saw that on the release notes. > > Does that mean that the active+clean+inconsistent PGs will be OK? > > Is the data still getting replicated even if inconsistent? > > Kind regards, > Glen Baars > > -----Original Message----- > From: Dan van der Ster <d...@vanderster.com> > Sent: Friday, 20 July 2018 3:57 PM > To: Glen Baars <g...@onsitecomputers.com.au> > Cc: ceph-users <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] 12.2.6 upgrade > > CRC errors are expected in 12.2.7 if you ran 12.2.6 with bluestore. See > https://ceph.com/releases/12-2-7-luminous-released/#upgrading-from-v12-2-6 > > On Fri, Jul 20, 2018 at 8:30 AM Glen Baars <g...@onsitecomputers.com.au> > wrote: > > > > Hello Ceph Users, > > > > > > > > We have upgraded all nodes to 12.2.7 now. We have 90PGs ( ~2000 scrub > > errors ) to fix from the time when we ran 12.2.6. It doesn’t seem to be > > affecting production at this time. > > > > > > > > Below is the log of a PG repair. What is the best way to correct these > > errors? Is there any further information required? > > > > > > > > rados list-inconsistent-obj 1.275 --format=json-pretty > > > > { > > > > "epoch": 38481, > > > > "inconsistents": [] > > > > } > > > > > > > > Is it odd that it doesn’t list any inconsistents? > > > > > > > > Ceph.log entries for this PG. > > > > 2018-07-20 12:13:28.381903 osd.124 osd.124 10.4.35.36:6810/1865422 81 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head data_digest > > 0x1a131dab != data_digest 0x92f2c4c8 from auth oi > > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head(37917'314836 > > client.1079025.0:24453722 dirty|data_digest|omap_digest s 4194304 uv 314836 > > dd 92f2c4c8 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:13:28.381907 osd.124 osd.124 10.4.35.36:6810/1865422 82 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head data_digest > > 0x1a131dab != data_digest 0x92f2c4c8 from auth oi > > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head(37917'314836 > > client.1079025.0:24453722 dirty|data_digest|omap_digest s 4194304 uv 314836 > > dd 92f2c4c8 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:13:28.381909 osd.124 osd.124 10.4.35.36:6810/1865422 83 : > > cluster [ERR] 1.275 soid > > 1:ae423e16:::rbd_data.37c2374b0dc51.000000000004917b:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:15:15.310579 osd.124 osd.124 10.4.35.36:6810/1865422 84 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head data_digest > > 0xdf907335 != data_digest 0x38400b00 from auth oi > > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head(38269'330651 > > client.232404.0:23912666 dirty|data_digest|omap_digest s 4194304 uv 307138 > > dd 38400b00 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:15:15.310582 osd.124 osd.124 10.4.35.36:6810/1865422 85 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head data_digest > > 0xdf907335 != data_digest 0x38400b00 from auth oi > > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head(38269'330651 > > client.232404.0:23912666 dirty|data_digest|omap_digest s 4194304 uv 307138 > > dd 38400b00 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:15:15.310584 osd.124 osd.124 10.4.35.36:6810/1865422 86 : > > cluster [ERR] 1.275 soid > > 1:ae455519:::rbd_data.3844874b0dc51.00000000000293f2:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:16:07.518970 osd.124 osd.124 10.4.35.36:6810/1865422 87 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head data_digest > > 0x6555a7c9 != data_digest 0xbad822f from auth oi > > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head(37917'314879 > > client.1079025.0:24564045 dirty|data_digest|omap_digest s 4194304 uv 314879 > > dd bad822f od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:16:07.518975 osd.124 osd.124 10.4.35.36:6810/1865422 88 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head data_digest > > 0x6555a7c9 != data_digest 0xbad822f from auth oi > > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head(37917'314879 > > client.1079025.0:24564045 dirty|data_digest|omap_digest s 4194304 uv 314879 > > dd bad822f od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:16:07.518977 osd.124 osd.124 10.4.35.36:6810/1865422 89 : > > cluster [ERR] 1.275 soid > > 1:ae470eb2:::rbd_data.37c2374b0dc51.0000000000049a4b:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:16:29.476778 osd.124 osd.124 10.4.35.36:6810/1865422 90 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head data_digest > > 0xa394e845 != data_digest 0xd8aa931c from auth oi > > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head(33683'302224 > > client.1079025.0:22963765 dirty|data_digest|omap_digest s 4194304 uv 302224 > > dd d8aa931c od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:16:29.476783 osd.124 osd.124 10.4.35.36:6810/1865422 91 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head data_digest > > 0xa394e845 != data_digest 0xd8aa931c from auth oi > > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head(33683'302224 > > client.1079025.0:22963765 dirty|data_digest|omap_digest s 4194304 uv 302224 > > dd d8aa931c od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:16:29.476787 osd.124 osd.124 10.4.35.36:6810/1865422 92 : > > cluster [ERR] 1.275 soid > > 1:ae47e410:::rbd_data.37c2374b0dc51.0000000000024b09:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:19:59.498922 osd.124 osd.124 10.4.35.36:6810/1865422 93 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head data_digest > > 0x2008cb1b != data_digest 0x218b7cb4 from auth oi > > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head(37426'306744 > > client.1079025.0:23363742 dirty|data_digest|omap_digest s 4194304 uv 306744 > > dd 218b7cb4 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:19:59.498925 osd.124 osd.124 10.4.35.36:6810/1865422 94 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head data_digest > > 0x2008cb1b != data_digest 0x218b7cb4 from auth oi > > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head(37426'306744 > > client.1079025.0:23363742 dirty|data_digest|omap_digest s 4194304 uv 306744 > > dd 218b7cb4 od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:19:59.498927 osd.124 osd.124 10.4.35.36:6810/1865422 95 : > > cluster [ERR] 1.275 soid > > 1:ae4de127:::rbd_data.37c2374b0dc51.000000000002f6a6:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:20:29.937564 osd.124 osd.124 10.4.35.36:6810/1865422 96 : > > cluster [ERR] 1.275 shard 100: soid > > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head data_digest > > 0x1b42858b != data_digest 0x69a5f3de from auth oi > > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head(38220'328463 > > client.1084539.0:403248048 dirty|data_digest|omap_digest s 4194304 uv > > 308146 dd 69a5f3de od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:20:29.937568 osd.124 osd.124 10.4.35.36:6810/1865422 97 : > > cluster [ERR] 1.275 shard 124: soid > > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head data_digest > > 0x1b42858b != data_digest 0x69a5f3de from auth oi > > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head(38220'328463 > > client.1084539.0:403248048 dirty|data_digest|omap_digest s 4194304 uv > > 308146 dd 69a5f3de od ffffffff alloc_hint [4194304 4194304 0]) > > > > 2018-07-20 12:20:29.937570 osd.124 osd.124 10.4.35.36:6810/1865422 98 : > > cluster [ERR] 1.275 soid > > 1:ae4f1dd8:::rbd_data.7695c59bb0bc2.00000000000005bb:head: failed to pick > > suitable auth object > > > > 2018-07-20 12:21:07.463206 osd.124 osd.124 10.4.35.36:6810/1865422 99 : > > cluster [ERR] 1.275 repair 12 errors, 0 fixed > > > > > > > > Kind regards, > > > > Glen Baars > > > > > > > > From: ceph-users <ceph-users-boun...@lists.ceph.com> On Behalf Of Glen Baars > > Sent: Wednesday, 18 July 2018 10:33 PM > > To: ceph-users@lists.ceph.com > > Subject: [ceph-users] 10.2.6 upgrade > > > > > > > > Hello Ceph Users, > > > > > > > > We installed 12.2.6 on a single node in the cluster ( new node added, 80TB > > moved ) > > > > Disabled scrub/deepscrub once the issues with 12.2.6 were discovered. > > > > > > > > Today we upgrade the one affected node to 12.2.7 today, set osd skip data > > digest = true and re enabled the scrubs. It’s a 500TB all bluestore cluster. > > > > > > > > We are now seeing inconsistent PGs and scrub errors now the scrubbing has > > resumed. > > > > > > > > What is the best way forward? > > > > > > > > Upgrade all nodes to 12.2.7? > > Remove the 12.2.7 node and rebuild? > > > > Kind regards, > > > > Glen Baars > > > > BackOnline Manager > > > > This e-mail is intended solely for the benefit of the addressee(s) and any > > other named recipient. It is confidential and may contain legally > > privileged or confidential information. If you are not the recipient, any > > use, distribution, disclosure or copying of this e-mail is prohibited. The > > confidentiality and legal privilege attached to this communication is not > > waived or lost by reason of the mistaken transmission or delivery to you. > > If you have received this e-mail in error, please notify us immediately. > > > > This e-mail is intended solely for the benefit of the addressee(s) and any > > other named recipient. It is confidential and may contain legally > > privileged or confidential information. If you are not the recipient, any > > use, distribution, disclosure or copying of this e-mail is prohibited. The > > confidentiality and legal privilege attached to this communication is not > > waived or lost by reason of the mistaken transmission or delivery to you. > > If you have received this e-mail in error, please notify us immediately. > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
