[ceph-users] Re: 16.2.8 pacific QE validation status, RC2 available for testing

[Neha Ojha]

Hi Yuri,

rados and upgrade/pacific-p2p look good to go.

On Tue, May 10, 2022 at 5:46 AM Benoît Knecht  wrote:
>
> On Mon, May 09, 2022 at 07:32:59PM +1000, Brad Hubbard wrote:
> > It's the current HEAD of the pacific branch or, alternatively,
> > https://github.com/ceph/ceph-ci/tree/pacific-16.2.8_RC2.
> >
> > $ git branch -r --contains 73636a1b00037ff974bcdc969b009c5ecec626cc
> >  ceph-ci/pacific-16.2.8_RC2
> >  upstream/pacific
>
> Thanks Brad, I didn't realize Yuri was referring to the ceph-ci repository.
>
> I've upgraded one of our clusters from 16.2.7 to 16.2.8-rc2, without any issue
> so far (but much better OMAP performance thanks to
> https://github.com/ceph/ceph/pull/46096).

Hi Ben,

Glad to hear it and thanks for testing the RC!

>
> I did notice that rgw still segfaults when `rgw_use_opa_authz=true` because
> https://github.com/ceph/ceph/pull/46106 didn't make the cut though.

Looks like this PR was opened after we did the first freeze for 16.2.8?
Given that we are almost ready for a release, I will add Casey for his
thoughts on whether we need to include it or not.

Thanks,
Neha

>
> Cheers,
>
> --
> Ben
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] repairing damaged cephfs_metadata pool

[Horvath, Dustin Marshall]

Hi there, newcomer here.

I've been trying to figure out if it's possible to repair or recover cephfs 
after some unfortunate issues a couple of months ago; these couple of nodes 
have been offline most of the time since the incident.

I'm sure the problem is that I lack the ceph expertise to quite sus out where 
the broken bits are. This was a 2-node cluster (I know I know) that had a 
hypervisor primary disk fail, and the entire OS was lost. I reinstalled the 
hypervisor, rejoined it to the cluster (proxmox), rejoined ceph to the other 
node, re-added the OSDs. It came back with quorum problems and some PGs were 
inconsistent and some were lost. Some of that is due to my own fiddling around, 
which possibly exacerbated things. Eventually I had to edit the monmap down to 
1 monitor, which had all kinds of screwy journal issues...it's been a while 
since I've tried resuscitating this, so the details in my memory are fuzzy.

My cluster health isn't awful. Output is basically this:
```
root@pve02:~# ceph -s
  cluster:
id: 8b31840b-5706-4c92-8135-0d6e03976af1
health: HEALTH_ERR
1 filesystem is degraded
1 filesystem is offline
1 mds daemon damaged
noout flag(s) set
16 daemons have recently crashed

  services:
mon: 1 daemons, quorum pve02 (age 3d)
mgr: pve01(active, since 4d)
mds: 0/1 daemons up
osd: 7 osds: 7 up (since 2d), 7 in (since 7w)
 flags noout

  data:
   volumes: 0/1 healthy, 1 recovering; 1 damaged
pools:   5 pools, 576 pgs
objects: 1.51M objects, 4.0 TiB
usage:   8.2 TiB used, 9.1 TiB / 17 TiB avail
pgs: 575 active+clean
 1   active+clean+scrubbing+deep

  io:
client:   241 KiB/s wr, 0 op/s rd, 10 op/s wr
```

I've tried a couple times running down the steps in here 
(https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/), but I 
always hit an error at scan_links, where I get a crash dump of sorts. If I try 
and mark the cephfs as repaired/joinable, MDS daemons will try and replay and 
then fail. The only occurrences of err/ERR in the MDS logs are a line like this:
```
2022-05-07T18:31:26.342-0500 7f22b44d8700  1 mds.0.94  waiting for osdmap 
301772 (which blocklists prior instance)
2022-05-07T18:31:26.346-0500 7f22adccb700 -1 log_channel(cluster) log [ERR] : 
failed to read JournalPointer: -1 ((1) Operation not permitted)
2022-05-07T18:31:26.346-0500 7f22af4ce700  0 mds.0.journaler.pq(ro) error 
getting journal off disk
```

I haven't had much luck on the googles with diagnosing that error; seems 
uncommon. My hope is that the cephfs_data pool is fine. I actually never had 
any inconsistent PG issues on a pool other than the metadata pool, so that's 
the only one that suffered actual acute injury during the hardware 
failure/quorum loss.
If I had more experience with the rados tools, I'd probably be more helpful. I 
have plenty of logs lying about and can perform any diagnoses that might help, 
but I hate to spam too much here right out of the gate.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Erasure-coded PG stuck in the failed_repair state

[Wesley Dillingham]

In my experience:

"No scrub information available for pg 11.2b5
error 2: (2) No such file or directory"

is the output you get from the command when the up or acting osd set has
changed since the last deep-scrub. Have you tried to run a deep scrub (ceph
pg deep-scrub 11.2b5) on the pg and then try "rados list-inconsistent-obj
11.2b5" again. I do recognize that part of the pg repair also performs a
deep-scrub but perhaps the deep-scrub alone will help with your attempt to
run rados list-inconsistent-obj.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Tue, May 10, 2022 at 8:52 AM Robert Appleyard - STFC UKRI <
rob.appley...@stfc.ac.uk> wrote:

> Hi,
>
> We've got an outstanding issue with one of our Ceph clusters here at RAL.
> The cluster is 'Echo', our 40PB cluster. We found an object from an 8+3EC
> RGW pool in the failed_repair state. We aren't sure how the object got into
> this state, but it doesn't appear to be a case of correlated drive failure
> (the rest of the PG is fine). However, the detail of how we got into this
> state isn't our focus, it's how to get the PG back to a clean state.
>
> The object (for our purposes, named OBJNAME) in question is from a RadosGW
> data pool. It presented initially as a PG in the failed_repair state.
> Repeated attempts to get the PG to repair failed. At this point we
> contacted the user who owns the data, and determined that the data in
> question was also stored elsewhere and so we could safely delete the
> object. We did that using radosgw-admin object rm OBJNAME, and confirmed
> that the object is gone with various approaches (radosgw-admin object stat,
> rados ls --pgid PGID | grep OBJNAME).
>
> So far, so good. Except, even after the object was deleted and in spite of
> many instructions to repair, the placement group is still in the state
> active+clean+inconsistent+failed_repair, and the cluster won't go to
> HEALTH_OK. Here's what the log from one of these repair attempts looks like
> (from the log on the primary OSD).
>
> 2022-05-08 16:23:43.898 7f79d3872700  0 log_channel(cluster) log [DBG] :
> 11.2b5 repair starts
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 1899(8) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 1911(7) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 2842(10) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 3256(6) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 3399(5) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 3770(9) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 5206(3) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 shard 6047(4) soid 11:ad45a433:::OBJNAME:head : candidate had an ec
> size mismatch
> 2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 soid 11:ad45a433:::OBJNAME:head : failed to pick suitable object info
> 2022-05-08 19:03:12.690 7f79d3872700 -1 log_channel(cluster) log [ERR] :
> 11.2b5 repair 11 errors, 0 fixed
>
> Looking for inconsistent objects in the PG doesn't report anything odd
> about this object (right now we get this rather odd output, but aren't sure
> that this isn't a red herring).
>
> [root@ceph-adm1 ~]# rados list-inconsistent-obj 11.2b5
> No scrub information available for pg 11.2b5
> error 2: (2) No such file or directory
>
> We don't get this output from this command on any other PG that we've
> tried.
>
> So what next? To reiterate, this isn't about data recovery, it's about
> getting the cluster back to a healthy state. I should also note that this
> issue doesn't seem to be impacting the cluster beyond making that PG show
> as being in a bad state.
>
> Rob Appleyard
>
>
> This email and any attachments are intended solely for the use of the
> named recipients. If you are not the intended recipient you must not use,
> disclose, copy or distribute this email or any of its attachments and
> should notify the sender immediately and delete this email from your
> system. UK Research and Innovation (UKRI) has taken every reasonable
> precaution to minimise risk of this email or any attachments containing
> viruses or malware but the recipient should carry out its own virus and
> malware 

[ceph-users] Erasure-coded PG stuck in the failed_repair state

[Robert Appleyard - STFC UKRI]

Hi,

We've got an outstanding issue with one of our Ceph clusters here at RAL. The 
cluster is 'Echo', our 40PB cluster. We found an object from an 8+3EC RGW pool 
in the failed_repair state. We aren't sure how the object got into this state, 
but it doesn't appear to be a case of correlated drive failure (the rest of the 
PG is fine). However, the detail of how we got into this state isn't our focus, 
it's how to get the PG back to a clean state.

The object (for our purposes, named OBJNAME) in question is from a RadosGW data 
pool. It presented initially as a PG in the failed_repair state. Repeated 
attempts to get the PG to repair failed. At this point we contacted the user 
who owns the data, and determined that the data in question was also stored 
elsewhere and so we could safely delete the object. We did that using 
radosgw-admin object rm OBJNAME, and confirmed that the object is gone with 
various approaches (radosgw-admin object stat, rados ls --pgid PGID | grep 
OBJNAME).

So far, so good. Except, even after the object was deleted and in spite of many 
instructions to repair, the placement group is still in the state 
active+clean+inconsistent+failed_repair, and the cluster won't go to HEALTH_OK. 
Here's what the log from one of these repair attempts looks like (from the log 
on the primary OSD).

2022-05-08 16:23:43.898 7f79d3872700  0 log_channel(cluster) log [DBG] : 11.2b5 
repair starts
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 1899(8) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 1911(7) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 2842(10) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 3256(6) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 3399(5) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 3770(9) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 5206(3) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
shard 6047(4) soid 11:ad45a433:::OBJNAME:head : candidate had an ec size 
mismatch
2022-05-08 16:51:38.807 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
soid 11:ad45a433:::OBJNAME:head : failed to pick suitable object info
2022-05-08 19:03:12.690 7f79d3872700 -1 log_channel(cluster) log [ERR] : 11.2b5 
repair 11 errors, 0 fixed

Looking for inconsistent objects in the PG doesn't report anything odd about 
this object (right now we get this rather odd output, but aren't sure that this 
isn't a red herring).

[root@ceph-adm1 ~]# rados list-inconsistent-obj 11.2b5
No scrub information available for pg 11.2b5
error 2: (2) No such file or directory

We don't get this output from this command on any other PG that we've tried.

So what next? To reiterate, this isn't about data recovery, it's about getting 
the cluster back to a healthy state. I should also note that this issue doesn't 
seem to be impacting the cluster beyond making that PG show as being in a bad 
state.

Rob Appleyard


This email and any attachments are intended solely for the use of the named 
recipients. If you are not the intended recipient you must not use, disclose, 
copy or distribute this email or any of its attachments and should notify the 
sender immediately and delete this email from your system. UK Research and 
Innovation (UKRI) has taken every reasonable precaution to minimise risk of 
this email or any attachments containing viruses or malware but the recipient 
should carry out its own virus and malware checks before opening the 
attachments. UKRI does not accept any liability for any losses or damages which 
the recipient may sustain due to presence of any viruses.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: not so empty bucket

[Ulrich Klein]

Yo, I’m having the same problem and can easily reproduce it.
See 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU/
And similar ones.

The problem still exists in Quincy 17.2.0. But looks like it’s too low priority 
to be fixed.

Ciao, Uli

> On 09. 05 2022, at 22:20, Christopher Durham  wrote:
> 
> 
> I am using pacific 16.2.7 on rocket 8.5 Linux 8.5.
> I have a once heavily used radosgw bucket that is now empty. Let's call it 
> 'oldbucket' awscli now shows that there are no objects in the bucket.
> 
> However, radosgw-admin bucket stats --bucket oldbucket shows num_objects in 
> rgw.main as 39 objects, with about 200 gigs used in the size field.
> 
> radosgw-admin bucket check --bucket oldbucket indeed shows 39 objects in a 
> list.
> Each of these objects are of the form:
> _multipart_originalobjectname.someoriginalid.NN
> radosgw-admin bucket radoslist --bucket oldbucket shows only 9 objects, but 
> thoseobjects are alll included as part of the bucket check command above.
> This particular bucket did have alot of multipart uploads. All multipart 
> uploads have beenaborted with awscli. And of course awscli shows no objects 
> in the bucket
> 
> radosgw-admin bucket list --bucket oldbucket shows all 39 objects, which is 
> weird thatI see object names which once were multipart object parts.
> 
> None of the 39 objects exist in the pool (rados -p $pool get $obj /tmp/$obj 
> all return 1)
> If I list all the index objects for this bucket in the index pool and then do 
> a listomapkeys for each of the index objects,I see only the 39 omapkeys.
> So my question is, what do I need to do to fix this bucket (without delting 
> and recreating it?) Would just doing a rmomapkeyon each of the omap keys in 
> the listomapkeys output solve my problem, and reclaim the 200 Gigs?
> Do I have to rebuild the index somehow (--fix in bucket check above did 
> nothing).
> 
> Thanks for any thoughts/insight.
> -Chris
> 
> 
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-crash user requirements

[Eugen Block]

Hi,

there's a profile "crash" for that. In a lab setup with Nautilus  
thre's one crash client with these caps:


admin:~ # ceph auth get client.crash
[client.crash]
key = 
caps mgr = "allow profile crash"
caps mon = "allow profile crash"

On a Octopus cluster deployed by cephadm each host gets its own crash client:

ses7-host1:~ # ceph auth ls | grep crash
installed auth entries:

client.crash.ses7-host1
caps: [mgr] profile crash
caps: [mon] profile crash
client.crash.ses7-host2
caps: [mgr] profile crash
caps: [mon] profile crash
client.crash.ses7-host3
caps: [mgr] profile crash
caps: [mon] profile crash
client.crash.ses7-host4
caps: [mgr] profile crash
caps: [mon] profile crash

But the capabilities are the same.

Regards,
Eugen

Zitat von Burkhard Linke :


Hi,


I just stumpled over some log messages regarding ceph-crash:

May 10 09:32:19 bigfoot60775 ceph-crash[2756]:  
WARNING:ceph-crash:post  
/var/lib/ceph/crash/2022-05-10T07:10:55.837665Z_7f3b726e-0368-4149-8834-6cafd92fb13f as client.admin failed: b'2022-05-10T09:32:19.099+0200 7f911ad92700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2022-05-10T09:32:19.099+0200 7f911ad92700 -1 AuthRegistry(0x7f911405b178) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2022-05-10T09:32:19.107+0200 7f9119b30700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2022-05-10T09:32:19.107+0200 7f9119b30700 -1 AuthRegistry(0x7f91140609d0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2022-05-10T09:32:19.107+0200 7f9119b307
00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2022-05-10T09:32:19.107+0200 7f9119b30700 -1 AuthRegistry(0x7f9119b2efe0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n[errno 2] RADOS object not found (error connecting to the  

cluster)\n'


This cluster is a manual setup, no cephadm or other stuff. We do not  
deploy the admin key to the OSD nodes unless absolutely necessary...



How can I configure a separate cephx user for ceph-crash? Which  
capabilities are required for that user?



Regards,

Burkhard


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph-crash user requirements

[Burkhard Linke]

Hi,


I just stumpled over some log messages regarding ceph-crash:

May 10 09:32:19 bigfoot60775 ceph-crash[2756]: WARNING:ceph-crash:post 
/var/lib/ceph/crash/2022-05-10T07:10:55.837665Z_7f3b726e-0368-4149-8834-6cafd92fb13f 
as client.admin failed: b'2022-05-10T09:32:19.099+0200 7f911ad92700 -1 
auth: unable to find a keyring on 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: 
(2) No such file or directory\n2022-05-10T09:32:19.099+0200 7f911ad92700 
-1 AuthRegistry(0x7f911405b178) no keyring found at 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, 
disabling cephx\n2022-05-10T09:32:19.107+0200 7f9119b30700 -1 auth: 
unable to find a keyring on 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: 
(2) No such file or directory\n2022-05-10T09:32:19.107+0200 7f9119b30700 
-1 AuthRegistry(0x7f91140609d0) no keyring found at 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, 
disabling cephx\n2022-05-10T09:32:19.107+0200 7f9119b30700 -1 auth: 
unable to find a keyring on 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: 
(2) No such file or directory\n2022-05-10T09:32:19.107+0200 7f9119b30700 
-1 AuthRegistry(0x7f9119b2efe0) no keyring found at 
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, 
disabling cephx\n[errno 2] RADOS object not found (error connecting to 
the cluster)\n'



This cluster is a manual setup, no cephadm or other stuff. We do not 
deploy the admin key to the OSD nodes unless absolutely necessary...



How can I configure a separate cephx user for ceph-crash? Which 
capabilities are required for that user?



Regards,

Burkhard


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Is osd_scrub_auto_repair dangerous?

[Janne Johansson]

Den mån 9 maj 2022 kl 21:04 skrev Vladimir Brik
:
> Hello
> Does osd_scrub_auto_repair default to false because it's
> dangerous? I assume `ceph pg repair` is also dangerous then?
>
> In what kinds of situations do they cause problems?

With filestore, there were less (or no?) checksums, so the cluster
might not always have an idea which replica was correct if they
differ, so auto-repair on those would in theory copy the error around
in the worst case. For bluestore there are checksums, so the chances
of your cluster detecting an error somewhere, while one or more of the
replicas have another undetectable error that still gives a correct
checksum so that this error gets copied around feels somewhat less
likely.


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io