date:20230620

[ceph-users] [question] Put with "tagging" is slowly?

2023-06-20 Thread Louis Koo

2023-06-21T02:48:50.754+ 7f1cd5b84700  1 beast: 0x7f1c4b26e630: 10.x.x.83 - 
xx [21/Jun/2023:02:48:47.653 +] "PUT 
/zhucan/deb/content/vol-26/chap-41/3a917ec7-02b3-4b45-8c0c-be32f4914708.bytes?tagging
 HTTP/1.1" 200 0 - "aws-sdk-java/1.12.299 Linux/3.10.0-1127.el7.x86_64 
OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 vendor/Red_Hat,_Inc. 
cfg/retry-mode/legacy" - latency=3.101041317s

2023-06-21T02:48:53.789+ 7f1cf53c3700  1 beast: 0x7f1c4b26e630: 10.x.x.83 - 
xx [21/Jun/2023:02:48:50.758 +] "PUT 
/zhucan/deb/content/vol-26/chap-41/3a917ec7-02b3-4b45-8c0c-be32f4914708.properties?tagging
 HTTP/1.1" 200 0 - "aws-sdk-java/1.12.299 Linux/3.10.0-1127.el7.x86_64 
OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 vendor/Red_Hat,_Inc. 
cfg/retry-mode/legacy" - latency=3.031040430s 

cost more 3s, why?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

2023-06-20 Thread Boris Behrens

I recreated the site and the problem still persists.

I've upped the logging and saw this for a lot of buckets (i've stopped the
debug log after some seconds).
2023-06-20T23:32:29.365+ 7fcaab7fe700 20 get_system_obj_state:
rctx=0x7fcaab7f9320 obj=dc3.rgw.meta:root:s3bucket-fra2
state=0x7fcba05ac0a0 s->prefetch_data=0
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache get:
name=dc3.rgw.meta+root+s3bucket-fra2 : miss
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache put:
name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x6
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 adding
dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache get:
name=dc3.rgw.meta+root+s3bucket-fra2 : type miss (requested=0x1, cached=0x6)
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache put:
name=dc3.rgw.meta+root+s3bucket-fra2 info.flags=0x1
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 moving
dc3.rgw.meta+root+s3bucket-fra2 to cache LRU end
2023-06-20T23:32:29.365+ 7fcaab7fe700 20 get_system_obj_state:
rctx=0x7fcaab7f9320
obj=dc3.rgw.meta:root:.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
state=0x7fcba43ce0a0 s->prefetch_data=0
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache get:
name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
: miss
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache put:
name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
info.flags=0x16
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 adding
dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
to cache LRU end
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache get:
name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
: type miss (requested=0x13, cached=0x16)
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 cache put:
name=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
info.flags=0x13
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 moving
dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29
to cache LRU end
2023-06-20T23:32:29.365+ 7fcaab7fe700 10 chain_cache_entry:
cache_locator=dc3.rgw.meta+root+.bucket.meta.s3bucket-fra2:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297866866.29

Am Di., 20. Juni 2023 um 19:29 Uhr schrieb Boris :

> Hi Casey,
> already did restart all RGW instances.  Only helped for 2 minutes. We now
> stopped the new site.
>
> I will remove and recreate it later.
> As twi other sites don't have the problem I currently think I made a
> mistake in the process.
>
> Mit freundlichen Grüßen
>  - Boris Behrens
>
> > Am 20.06.2023 um 18:30 schrieb Casey Bodley :
> >
> > hi Boris,
> >
> > we've been investigating reports of excessive polling from metadata
> > sync. i just opened https://tracker.ceph.com/issues/61743 to track
> > this. restarting the secondary zone radosgws should help as a
> > temporary workaround
> >
> >> On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens  wrote:
> >>
> >> Hi,
> >> yesterday I added a new zonegroup and it looks like it seems to cycle
> over
> >> the same requests over and over again.
> >>
> >> In the log of the main zone I see these requests:
> >> 2023-06-20T09:48:37.979+ 7f8941fb3700  1 beast: 0x7f8a602f3700:
> >> fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+] "GET
> >>
> /admin/log?type=metadata=62=e8fc96f1-ae86-4dc1-b432-470b0772fded=100&=b39392eb-75f8-47f0-b4f3-7d3882930b26
> >> HTTP/1.1" 200 44 - - -
> >>
> >> Only thing that changes is the 
> >>
> >> We have two other zonegroups that are configured identical (ceph.conf
> and
> >> period) and these don;t seem to spam the main rgw.
> >>
> >> root@host:~# radosgw-admin sync status
> >>  realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main)
> >>  zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3)
> >>   zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3)
> >>  metadata sync syncing
> >>full sync: 0/64 shards
> >>incremental sync: 64/64 shards
> >>metadata is caught up with master
> >>
> >> root@host:~# radosgw-admin period get
> >> {
> >>"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
> >>"epoch": 92,
> >>"predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a",
> >>"sync_status": [
> >>"",
> >> ...
> >>""
> >>],
> >>"period_map": {
> >>"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
> >>"zonegroups": [
> >>{
> >>"id": "b39392eb-75f8-47f0-b4f3-7d3882930b26",
> >>"name": "dc3",
> >>"api_name": "dc3",
> >>"is_master": "false",
> >>"endpoints": [
> >>],
> >>"hostnames": [
> >>],
> >>"hostnames_s3website": [
> >>],
> >>

[ceph-users] Re: Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

2023-06-20 Thread Adiga, Anantha

Hi Adam,

Thank you for the details. I see that the cephadm on the Ceph cluster is 
different from the host that is being added. I will go thru the ticket and the 
logs. Also the cluster is on Ubuntu Focal and the new host is on Ubuntu Jammy
The utility:
cephadm   16.2.13-1focal   
amd64cephadm utility to bootstrap ceph daemons with systemd and 
containers
cephadm   17.2.5-0ubuntu0.22.04.3amd64  
  cephadm utility to bootstrap ceph daemons with systemd and containers

Thanks again,
Anantha
From: Adam King 
Sent: Tuesday, June 20, 2023 4:25 PM
To: Adiga, Anantha 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Error while adding host : Error EINVAL: Traceback 
(most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in 
_handle_command

There was a cephadm bug that wasn't fixed by the time 17.2.6 came out (I'm 
assuming that's the version being used here, although it may have been present 
in some slightly earlier quincy versions) that caused this misleading error to 
be printed out when adding a host failed. There's a tracker for it here 
https://tracker.ceph.com/issues/59081 that has roughly the same traceback. The 
real issue is likely a connectivity or permission issue from the active mgr 
trying to ssh to the host. In the case I saw from the tracker, it was caused by 
the ssh pub key not being set up on the host. If you check the cephadm cluster 
logs ("ceph log last 50 debug cephadm") after trying to add the host I'm 
guessing you'll see some error like the second set of output in the tracker 
that will hopefully give some more info on why adding the host failed.

On Tue, Jun 20, 2023 at 6:38 PM Adiga, Anantha 
mailto:anantha.ad...@intel.com>> wrote:
Hi,

I am seeing this error  after an offline  was deleted and while adding the host 
again. Thereafter, I have removed the /var/lib/cep  folder and removed the ceph 
quincy image in the offline host. What is the cause of this issue and the 
solution.

root@fl31ca104ja0201:/home/general# cephadm shell
Inferring fsid d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
Using recent ceph image 
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e>
root@fl31ca104ja0201:/#

root@fl31ca104ja0201:/# ceph orch host rm fl31ca104ja0302 --offline --force

Removed offline host 'fl31ca104ja0302'

root@fl31ca104ja0201:/# ceph -s

  cluster:

id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e

health: HEALTH_OK



  services:

mon: 3 daemons, quorum fl31ca104ja0201,fl31ca104ja0202,fl31ca104ja0203 (age 
28h)

mgr: fl31ca104ja0203(active, since 6d), standbys: fl31ca104ja0202, 
fl31ca104ja0201

mds: 1/1 daemons up, 2 standby

osd: 33 osds: 33 up (since 28h), 33 in (since 28h)

rgw: 3 daemons active (3 hosts, 1 zones)



  data:

volumes: 1/1 healthy

pools:   24 pools, 737 pgs

objects: 613.56k objects, 1.9 TiB

usage:   2.9 TiB used, 228 TiB / 231 TiB avail

pgs: 737 active+clean



  io:

client:   161 MiB/s rd, 75 op/s rd, 0 op/s wr


root@fl31ca104ja0201:/# ceph orch host add fl31ca104ja0302 10.45.219.5
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command
return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in 
handle_command
return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 
wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # 
noqa: E731
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host
return self._apply_misc([s], False, Format.plain)
 File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc
raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in 
raise_if_exception
e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 
'addr'

Thank you,
Anantha
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

2023-06-20 Thread Adam King

There was a cephadm bug that wasn't fixed by the time 17.2.6 came out (I'm
assuming that's the version being used here, although it may have been
present in some slightly earlier quincy versions) that caused this
misleading error to be printed out when adding a host failed. There's a
tracker for it here https://tracker.ceph.com/issues/59081 that has roughly
the same traceback. The real issue is likely a connectivity or permission
issue from the active mgr trying to ssh to the host. In the case I saw from
the tracker, it was caused by the ssh pub key not being set up on the host.
If you check the cephadm cluster logs ("ceph log last 50 debug cephadm")
after trying to add the host I'm guessing you'll see some error like the
second set of output in the tracker that will hopefully give some more info
on why adding the host failed.

On Tue, Jun 20, 2023 at 6:38 PM Adiga, Anantha 
wrote:

> Hi,
>
> I am seeing this error  after an offline  was deleted and while adding the
> host again. Thereafter, I have removed the /var/lib/cep  folder and removed
> the ceph quincy image in the offline host. What is the cause of this issue
> and the solution.
>
> root@fl31ca104ja0201:/home/general# cephadm shell
> Inferring fsid d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
> Using recent ceph image
> quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
>  :af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e>
> root@fl31ca104ja0201:/#
>
> root@fl31ca104ja0201:/# ceph orch host rm fl31ca104ja0302 --offline
> --force
>
> Removed offline host 'fl31ca104ja0302'
>
> root@fl31ca104ja0201:/# ceph -s
>
>   cluster:
>
> id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
> health: HEALTH_OK
>
>
>
>   services:
>
> mon: 3 daemons, quorum fl31ca104ja0201,fl31ca104ja0202,fl31ca104ja0203
> (age 28h)
>
> mgr: fl31ca104ja0203(active, since 6d), standbys: fl31ca104ja0202,
> fl31ca104ja0201
>
> mds: 1/1 daemons up, 2 standby
>
> osd: 33 osds: 33 up (since 28h), 33 in (since 28h)
>
> rgw: 3 daemons active (3 hosts, 1 zones)
>
>
>
>   data:
>
> volumes: 1/1 healthy
>
> pools:   24 pools, 737 pgs
>
> objects: 613.56k objects, 1.9 TiB
>
> usage:   2.9 TiB used, 228 TiB / 231 TiB avail
>
> pgs: 737 active+clean
>
>
>
>   io:
>
> client:   161 MiB/s rd, 75 op/s rd, 0 op/s wr
>
>
> root@fl31ca104ja0201:/# ceph orch host add fl31ca104ja0302 10.45.219.5
> Error EINVAL: Traceback (most recent call last):
>   File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command
> return self.handle_command(inbuf, cmd)
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in
> handle_command
> return dispatch[cmd['prefix']].call(self, cmd, inbuf)
>   File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
> return self.func(mgr, **kwargs)
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in
> 
> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args,
> **l_kwargs)  # noqa: E731
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in
> wrapper
> return func(*args, **kwargs)
>   File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host
> return self._apply_misc([s], False, Format.plain)
>  File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in
> _apply_misc
> raise_if_exception(completion)
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in
> raise_if_exception
> e = pickle.loads(c.serialized_exception)
> TypeError: __init__() missing 2 required positional arguments: 'hostname'
> and 'addr'
>
> Thank you,
> Anantha
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

2023-06-20 Thread Adiga, Anantha

Hi,

I am seeing this error  after an offline  was deleted and while adding the host 
again. Thereafter, I have removed the /var/lib/cep  folder and removed the ceph 
quincy image in the offline host. What is the cause of this issue and the 
solution.

root@fl31ca104ja0201:/home/general# cephadm shell
Inferring fsid d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
Using recent ceph image 
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
root@fl31ca104ja0201:/#

root@fl31ca104ja0201:/# ceph orch host rm fl31ca104ja0302 --offline --force

Removed offline host 'fl31ca104ja0302'

root@fl31ca104ja0201:/# ceph -s

  cluster:

id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e

health: HEALTH_OK



  services:

mon: 3 daemons, quorum fl31ca104ja0201,fl31ca104ja0202,fl31ca104ja0203 (age 
28h)

mgr: fl31ca104ja0203(active, since 6d), standbys: fl31ca104ja0202, 
fl31ca104ja0201

mds: 1/1 daemons up, 2 standby

osd: 33 osds: 33 up (since 28h), 33 in (since 28h)

rgw: 3 daemons active (3 hosts, 1 zones)



  data:

volumes: 1/1 healthy

pools:   24 pools, 737 pgs

objects: 613.56k objects, 1.9 TiB

usage:   2.9 TiB used, 228 TiB / 231 TiB avail

pgs: 737 active+clean



  io:

client:   161 MiB/s rd, 75 op/s rd, 0 op/s wr


root@fl31ca104ja0201:/# ceph orch host add fl31ca104ja0302 10.45.219.5
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command
return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in 
handle_command
return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 
wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # 
noqa: E731
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host
return self._apply_misc([s], False, Format.plain)
 File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc
raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in 
raise_if_exception
e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 
'addr'

Thank you,
Anantha
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [rgw multisite] Perpetual behind

2023-06-20 Thread kchheda3

And as per the tracker, the issue was merged to quincy and is available in 
17.2.6 (looking at the release notes), so you might want to upgrade your 
cluster and re run your tests.
Note, the existing issue will not go away post upgrading to 17.2.6, you will 
have to manually sync the buckets that are out of sync !
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSDs cannot join cluster anymore

2023-06-20 Thread Malte Stroem


Hello,

we removed some nodes from our cluster. This worked without problems.

Now, lots of OSDs do not want to join the cluster anymore if we reboot 
one of the still available nodes.


It always runs into timeouts:

--> ceph-volume lvm activate successful for osd ID: XX
monclient(hunting): authenticate timed out after 300

MONs and MGRs are running fine.

Network is working, netcat to the MONs' ports are open.

Setting a higher debug level has no effect even if we add it to the 
ceph.conf file.


The PGs are pretty unhappy, e. g.:

7.143  87771   0 0  00 
3147449022350   0  10081 10081 
 down  2023-06-20T09:16:03.546158+961275'1395646 
961300:9605547  [209,NONE,NONE] 209  [209,NONE,NONE] 
209961231'1395512  2023-06-19T23:46:40.101791+961231'1395512 
 2023-06-19T23:46:40.101791+


PG query wants us to set an OSD lost however I do not want to do this.

OSDs are blocked by OSDs from the removed nodes:

ceph osd blocked-by
osd  num_blocked
152   38
244   41
144   54
...

We added the removed hosts again and tried to start the OSDs on this 
node and they also failed into the timeout mentioned above.


This is a containerized cluster running version 16.2.10.

Replication is 3, some pools use an erasure coded profile.

Best regards,
Malte


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [rgw multisite] Perpetual behind

2023-06-20 Thread kchheda3

Hi Yixin,
we had faced similar issue, and this was the tracker 
https://tracker.ceph.com/issues/57562, that has all the details
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Starting v17.2.5 RGW SSE with default key (likely others) no longer works

2023-06-20 Thread Jayanth Reddy

Thanks, Casey for the response. I'll track the fix there.

Thanks,
Jayanth Reddy
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

2023-06-20 Thread Boris

Hi Casey,
already did restart all RGW instances.  Only helped for 2 minutes. We now 
stopped the new site. 

I will remove and recreate it later. 
As twi other sites don't have the problem I currently think I made a mistake in 
the process. 

Mit freundlichen Grüßen
 - Boris Behrens

> Am 20.06.2023 um 18:30 schrieb Casey Bodley :
> 
> hi Boris,
> 
> we've been investigating reports of excessive polling from metadata
> sync. i just opened https://tracker.ceph.com/issues/61743 to track
> this. restarting the secondary zone radosgws should help as a
> temporary workaround
> 
>> On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens  wrote:
>> 
>> Hi,
>> yesterday I added a new zonegroup and it looks like it seems to cycle over
>> the same requests over and over again.
>> 
>> In the log of the main zone I see these requests:
>> 2023-06-20T09:48:37.979+ 7f8941fb3700  1 beast: 0x7f8a602f3700:
>> fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+] "GET
>> /admin/log?type=metadata=62=e8fc96f1-ae86-4dc1-b432-470b0772fded=100&=b39392eb-75f8-47f0-b4f3-7d3882930b26
>> HTTP/1.1" 200 44 - - -
>> 
>> Only thing that changes is the 
>> 
>> We have two other zonegroups that are configured identical (ceph.conf and
>> period) and these don;t seem to spam the main rgw.
>> 
>> root@host:~# radosgw-admin sync status
>>  realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main)
>>  zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3)
>>   zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3)
>>  metadata sync syncing
>>full sync: 0/64 shards
>>incremental sync: 64/64 shards
>>metadata is caught up with master
>> 
>> root@host:~# radosgw-admin period get
>> {
>>"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
>>"epoch": 92,
>>"predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a",
>>"sync_status": [
>>"",
>> ...
>>""
>>],
>>"period_map": {
>>"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
>>"zonegroups": [
>>{
>>"id": "b39392eb-75f8-47f0-b4f3-7d3882930b26",
>>"name": "dc3",
>>"api_name": "dc3",
>>"is_master": "false",
>>"endpoints": [
>>],
>>"hostnames": [
>>],
>>"hostnames_s3website": [
>>],
>>"master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb",
>>"zones": [
>>{
>>"id": "96f5eca9-425b-4194-a152-86e310e91ddb",
>>"name": "dc3",
>>"endpoints": [
>>],
>>"log_meta": "false",
>>"log_data": "false",
>>"bucket_index_max_shards": 11,
>>"read_only": "false",
>>"tier_type": "",
>>"sync_from_all": "true",
>>"sync_from": [],
>>"redirect_zone": ""
>>}
>>],
>>"placement_targets": [
>>{
>>"name": "default-placement",
>>"tags": [],
>>"storage_classes": [
>>"STANDARD"
>>]
>>}
>>],
>>"default_placement": "default-placement",
>>"realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef",
>>"sync_policy": {
>>"groups": []
>>}
>>},
>> ...
>> 
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>> groÃƒ¼en Saal.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 1 PG stucked in "active+undersized+degraded for long time

2023-06-20 Thread siddhit . renake

Hello All,

Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) 
nautilus (stable)

Issue:
1 PG stucked in "active+undersized+degraded for long time
Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg 
degraded, 1 pg undersized

#ceph pg dump_stuck
PG_STAT STATE   UP  
UP_PRIMARY ACTING  
ACTING_PRIMARY
15.28f0  active+undersized+degraded 
[2147483647,343,355,415,426,640,302,392,78,202,607]343 
[2147483647,343,355,415,426,640,302,392,78,202,607]343

PG Query:
#ceph pg 15.28f0 query

{
"state": "active+undersized+degraded",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 303362,
"up": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting": [
2147483647,
343,
355,
415,
426,
640,
302,
392,
78,
202,
607
],
"acting_recovery_backfill": [
"78(8)",
"202(9)",
"302(6)",
"343(1)",
"355(2)",
"392(7)",
"415(3)",
"426(4)",
"607(10)",
"640(5)"
],
"info": {
"pgid": "15.28f0s1",
"last_update": "303161'598853",
"last_complete": "303161'598853",
"log_tail": "261289'595825",
"last_user_version": 598853,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": [],
"history": {
"epoch_created": 19841,
"epoch_pool_created": 16141,
"last_epoch_started": 303017,
"last_interval_started": 303016,
"last_epoch_clean": 250583,
"last_interval_clean": 250582,
"last_epoch_split": 19841,
"last_epoch_marked_full": 0,
"same_up_since": 303016,
"same_interval_since": 303016,
"same_primary_since": 256311,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439"
},
"stats": {
"version": "303161'598853",
"reported_seq": "3594551",
"reported_epoch": "303362",
"state": "active+undersized+degraded",
"last_fresh": "2023-06-20 19:03:59.135295",
"last_change": "2023-06-20 15:11:12.569114",
"last_active": "2023-06-20 19:03:59.135295",
"last_peered": "2023-06-20 19:03:59.135295",
"last_clean": "2021-04-11 15:21:44.271834",
"last_became_active": "2023-06-20 15:11:12.569114",
"last_became_peered": "2023-06-20 15:11:12.569114",
"last_unstale": "2023-06-20 19:03:59.135295",
"last_undegraded": "2023-06-20 15:11:10.430426",
"last_fullsized": "2023-06-20 15:11:10.430154",
"mapping_epoch": 303016,
"log_start": "261289'595825",
"ondisk_log_start": "261289'595825",
"created": 19841,
"last_epoch_clean": 250583,
"parent": "0.0",
"parent_split_bits": 14,
"last_scrub": "255277'537760",
"last_scrub_stamp": "2021-04-11 03:18:39.164439",
"last_deep_scrub": "255277'537756",
"last_deep_scrub_stamp": "2021-04-10 01:42:16.182528",
"last_clean_scrub_stamp": "2021-04-11 03:18:39.164439",
"log_size": 3028,
"ondisk_log_size": 3028,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 54989065178,
"num_objects": 44800,
"num_object_clones": 0,
"num_object_copies": 492800,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 44800,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 44800,
"num_whiteouts": 0,
"num_read": 201078,
"num_read_kb": 30408632,
"num_write": 219335,
"num_write_kb": 59459782,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 121970,

[ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6

2023-06-20 Thread Austin Axworthy

Hi Pritha,

I have increased the debug logs and pasted the output below. I have 2 users, 
austin and test. Austin is the owner user on the buckets, and I am trying to 
assume the role with the test user. I have also tried to assume the role of 
austin with the same user, but still get the same forbidden response. I am able 
to get the temporary credentials in both cases.

I have put in a request on the tracker for an account on Friday. Just waiting 
to be approved to post there. 

2023-06-20T17:07:53.459+ 7f25335ce700 20 req 14769804432280246561 
0.00406s s3:get_obj get_obj_state: setting s->obj_tag to 
5c7714c3-1d41-484a-9ecb-8c4e534b3b02.14130.12619634195847866073
2023-06-20T17:07:53.459+ 7f25335ce700 15 req 14769804432280246561 
0.00406s s3:get_obj decode_policy Read 
AccessControlPolicyhttp://s3.amazonaws.com/doc/2006-03-01/;>austinaustinhttp://www.w3.org/2001/XMLSchema-instance; 
xsi:type="CanonicalUser">austinaustinFULL_CONTROL
2023-06-20T17:07:53.459+ 7f25335ce700  2 req 14769804432280246561 
0.00406s s3:get_obj init op
2023-06-20T17:07:53.459+ 7f25335ce700 20 req 14769804432280246561 
0.00406s s3:get_obj get_system_obj_state: rctx=0x7f26486cf3f8 
obj=default.rgw.meta:users.uid:austin state=0x7f262c05b810 s->prefetch_data=0
2023-06-20T17:07:53.459+ 7f25335ce700 10 req 14769804432280246561 
0.00406s s3:get_obj cache get: name=default.rgw.meta+users.uid+austin : 
expiry miss
2023-06-20T17:07:53.459+ 7f2532dcd700 10 req 14769804432280246561 
0.00406s s3:get_obj cache put: name=default.rgw.meta+users.uid+austin 
info.flags=0x16
2023-06-20T17:07:53.459+ 7f2532dcd700 10 req 14769804432280246561 
0.00406s s3:get_obj adding default.rgw.meta+users.uid+austin to cache LRU 
end
2023-06-20T17:07:53.459+ 7f2532dcd700 10 req 14769804432280246561 
0.00406s s3:get_obj updating xattr: name=ceph.objclass.version 
bl.length()=42
2023-06-20T17:07:53.459+ 7f2532dcd700 20 req 14769804432280246561 
0.00406s s3:get_obj get_system_obj_state: s->obj_tag was set empty
2023-06-20T17:07:53.459+ 7f2532dcd700 20 req 14769804432280246561 
0.00406s s3:get_obj Read xattr: user.rgw.idtag
2023-06-20T17:07:53.459+ 7f2532dcd700 10 req 14769804432280246561 
0.00406s s3:get_obj cache get: name=default.rgw.meta+users.uid+austin : 
type miss (requested=0x13, cached=0x16)
2023-06-20T17:07:53.459+ 7f2532dcd700 20 req 14769804432280246561 
0.00406s s3:get_obj rados->read ofs=0 len=0
2023-06-20T17:07:53.467+ 7f25325cc700 20 req 14769804432280246561 
0.01217s s3:get_obj rados_obj.operate() r=0 bl.length=417
2023-06-20T17:07:53.467+ 7f25325cc700 10 req 14769804432280246561 
0.01217s s3:get_obj cache put: name=default.rgw.meta+users.uid+austin 
info.flags=0x13
2023-06-20T17:07:53.467+ 7f25325cc700 10 req 14769804432280246561 
0.01217s s3:get_obj moving default.rgw.meta+users.uid+austin to cache LRU 
end
2023-06-20T17:07:53.467+ 7f25325cc700 10 req 14769804432280246561 
0.01217s s3:get_obj updating xattr: name=ceph.objclass.version 
bl.length()=42
2023-06-20T17:07:53.467+ 7f25325cc700  2 req 14769804432280246561 
0.01217s s3:get_obj verifying op mask
2023-06-20T17:07:53.467+ 7f25325cc700 20 req 14769804432280246561 
0.01217s s3:get_obj required_mask= 1 user.op_mask=7
2023-06-20T17:07:53.467+ 7f25325cc700  2 req 14769804432280246561 
0.01217s s3:get_obj verifying op permissions
2023-06-20T17:07:53.467+ 7f25325cc700  1 req 14769804432280246561 
0.01217s op->ERRORHANDLER: err_no=-13 new_err_no=-13
2023-06-20T17:07:53.467+ 7f25325cc700 20 req 14769804432280246561 
0.01217s get_system_obj_state: rctx=0x7f26486cf730 
obj=default.rgw.log:script.postrequest. state=0x7f262c05b810 s->prefetch_data=0
2023-06-20T17:07:53.467+ 7f25325cc700 10 req 14769804432280246561 
0.01217s cache get: name=default.rgw.log++script.postrequest. : hit 
(negative entry)
2023-06-20T17:07:53.467+ 7f25325cc700  2 req 14769804432280246561 
0.01217s s3:get_obj op status=0
2023-06-20T17:07:53.467+ 7f25325cc700  2 req 14769804432280246561 
0.01217s s3:get_obj http status=403
2023-06-20T17:07:53.467+ 7f25325cc700  1 == req done req=0x7f26486d06f0 
op status=0 http_status=403 latency=0.01217s ==
2023-06-20T17:07:53.467+ 7f25325cc700  1 beast: 0x7f26486d06f0: 
192.168.175.200 - test [20/Jun/2023:17:07:53.455 +] "HEAD 
/active-data/test.txt HTTP/1.1" 403 0 - "Boto3/1.9.253 Python/3.8.10 Linux/5.4.

Thanks,
Austin



-Original Message-
From: Pritha Srivastava  
Sent: June 15, 2023 1:02 AM
To: Austin Axworthy 
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to 
Quincy 17.2.6

Hi Austin,

Do you have rgw debug logs that can help debug this?

Can you provide more information, as to which user is trying to assume the role 
- which tenants the user and role belong to?
Can you please open a tracker issue with all this

[ceph-users] Re: Recover OSDs from folder /var/lib/ceph/uuid/removed

2023-06-20 Thread Malte Stroem


Well, things I would do:

- add the keyring to ceph auth

ceph auth add osd.XX osd 'allow *' mon 'allow rwx' -i 
/var/lib/ceph/uuid(osd.XX/keyring


- add OSD to crush

ceph osd crush set osd.XX 1.0 root=default ...

- create systemd service

systemctl enable ceph-u...@osd.xx.service

Is there something I am missing?

Best,
Malte

Am 20.06.23 um 18:04 schrieb Malte Stroem:

Hello,

is it possible to recover an OSD if it was removed?

The systemd service was removed but the block device is still listed under

lsblk

and the config files are still available under

/var/lib/ceph/uuid/removed

It is a containerized cluster. So I think we need to add the cephx 
entries, use ceph-volume, crush, and so on.


Best regards,
Malte

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

2023-06-20 Thread Casey Bodley

hi Boris,

we've been investigating reports of excessive polling from metadata
sync. i just opened https://tracker.ceph.com/issues/61743 to track
this. restarting the secondary zone radosgws should help as a
temporary workaround

On Tue, Jun 20, 2023 at 5:57 AM Boris Behrens  wrote:
>
> Hi,
> yesterday I added a new zonegroup and it looks like it seems to cycle over
> the same requests over and over again.
>
> In the log of the main zone I see these requests:
> 2023-06-20T09:48:37.979+ 7f8941fb3700  1 beast: 0x7f8a602f3700:
> fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+] "GET
> /admin/log?type=metadata=62=e8fc96f1-ae86-4dc1-b432-470b0772fded=100&=b39392eb-75f8-47f0-b4f3-7d3882930b26
> HTTP/1.1" 200 44 - - -
>
> Only thing that changes is the 
>
> We have two other zonegroups that are configured identical (ceph.conf and
> period) and these don;t seem to spam the main rgw.
>
> root@host:~# radosgw-admin sync status
>   realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main)
>   zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3)
>zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3)
>   metadata sync syncing
> full sync: 0/64 shards
> incremental sync: 64/64 shards
> metadata is caught up with master
>
> root@host:~# radosgw-admin period get
> {
> "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
> "epoch": 92,
> "predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a",
> "sync_status": [
> "",
> ...
> ""
> ],
> "period_map": {
> "id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
> "zonegroups": [
> {
> "id": "b39392eb-75f8-47f0-b4f3-7d3882930b26",
> "name": "dc3",
> "api_name": "dc3",
> "is_master": "false",
> "endpoints": [
> ],
> "hostnames": [
> ],
> "hostnames_s3website": [
> ],
> "master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb",
> "zones": [
> {
> "id": "96f5eca9-425b-4194-a152-86e310e91ddb",
> "name": "dc3",
> "endpoints": [
> ],
> "log_meta": "false",
> "log_data": "false",
> "bucket_index_max_shards": 11,
> "read_only": "false",
> "tier_type": "",
> "sync_from_all": "true",
> "sync_from": [],
> "redirect_zone": ""
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": [],
> "storage_classes": [
> "STANDARD"
> ]
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef",
> "sync_policy": {
> "groups": []
> }
> },
> ...
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groÃƒ¼en Saal.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Recover OSDs from folder /var/lib/ceph/uuid/removed

2023-06-20 Thread Malte Stroem


Hello,

is it possible to recover an OSD if it was removed?

The systemd service was removed but the block device is still listed under

lsblk

and the config files are still available under

/var/lib/ceph/uuid/removed

It is a containerized cluster. So I think we need to add the cephx 
entries, use ceph-volume, crush, and so on.


Best regards,
Malte
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osd memory target not work

2023-06-20 Thread Mark Nelson


Hi Farhad,


I wrote the underlying osd memory target code.  OSDs won't always use 
all of the memory if there is nothing driving a need. Primarily the 
driver of memory usage will be the meta and data caches needing more 
memory to keep the hit rates high.  If you perform some reads/writes 
across a large dataset you should see the OSDs start using more memory 
and then start oscillating near the target.



Mark


On 6/20/23 05:16, farhad kh wrote:

  when set osd_memory_target for limitation usage memory for osd disk ,This
value is expected to be set for the OSD container .But with the docker stats
command, this value is not seen Is my perception of this process wrong?
---

[root@opcsdfpsbpp0201 ~]# ceph orch ps | grep osd.12
osd.12  opcsdfpsbpp0201
running (9d) 5m ago   9d1205M1953M  17.2.6
c9a1062f7289  bf27cfe16046
[root@opcsdfpsbpp0201 ~]# docker stats | grep osd
1253766d6a78   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-48
0.05% 2.237GiB / 7.732GiB   28.93%0B / 0B   86.8GB / 562GB
63
2bc012e5c604   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-67
0.20% 727MiB / 7.732GiB 9.18% 0B / 0B   37.5GB / 1.29TB
63
dc0bf068050b   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-62
0.11% 360.5MiB / 7.732GiB   4.55% 0B / 0B   125MB / 1.85GB
63
c5f119a37652   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-55
0.12% 312.5MiB / 7.732GiB   3.95% 0B / 0B   86.6MB / 1.66GB
63
7f0b7b61807d   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-5
0.11% 299.4MiB / 7.732GiB   3.78% 0B / 0B   119MB / 1.6GB
63
dadffc77f7b6   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-40
0.11% 274MiB / 7.732GiB 3.46% 0B / 0B   110MB / 1.5GB
63
e439e58d907e   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-34
0.12% 355.9MiB / 7.732GiB   4.49% 0B / 0B   125MB / 1.78GB
63
5e500e2197d6   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-25
0.11% 273.3MiB / 7.732GiB   3.45% 0B / 0B   128MB / 1.55GB
63
a63709567669   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-19
0.11% 714.6MiB / 7.732GiB   9.03% 0B / 0B   89.8MB / 167GB
63
bf27cfe16046   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-12
0.16% 1.177GiB / 7.732GiB   15.23%0B / 0B   40.8GB / 644GB
63
---
# ceph orch ps | grep osd | grep opcsdfpsbpp0201
osd.5 opcsdfpsbpp0201   running (9d) 6m ago   9d 298M
1953M  17.2.6  c9a1062f7289  7f0b7b61807d
osd.12opcsdfpsbpp0201   running (9d) 6m ago   9d1205M
1953M  17.2.6  c9a1062f7289  bf27cfe16046
osd.19opcsdfpsbpp0201   running (9d) 6m ago   9d 704M
1953M  17.2.6  c9a1062f7289  a63709567669
osd.25opcsdfpsbpp0201   running (9d) 6m ago   9d 273M
1953M  17.2.6  c9a1062f7289  5e500e2197d6
osd.34opcsdfpsbpp0201   running (9d) 6m ago   9d 355M
1953M  17.2.6  c9a1062f7289  e439e58d907e
osd.40opcsdfpsbpp0201   running (9d) 6m ago   9d 273M
1953M  17.2.6  c9a1062f7289  dadffc77f7b6
osd.48opcsdfpsbpp0201   running (4h) 6m ago   9d2290M
1953M  17.2.6  c9a1062f7289  1253766d6a78
osd.55opcsdfpsbpp0201   running (9d) 6m ago   9d 312M
1953M  17.2.6  c9a1062f7289  c5f119a37652
osd.62opcsdfpsbpp0201   running (9d) 6m ago   9d 359M
1953M  17.2.6  c9a1062f7289  dc0bf068050b
osd.67opcsdfpsbpp0201   running (9d) 6m ago   9d 727M
1953M  17.2.6  c9a1062f7289  2bc012e5c604
--

  #ceph config get mgr  osd_memory_target
204800
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Best Regards,
Mark Nelson
Head of R (USA)

Clyso GmbH
p: +49 89 21552391 12
a: Loristraße 8 | 80335 München | Germany
w: https://clyso.com | e: mark.nel...@clyso.com

We are hiring: https://www.clyso.com/jobs/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Pacific bluefs enospc bug with newly created OSDs

2023-06-20 Thread Igor Fedotov


Hi Carsten,

first of all Quincy does have a fix for the issue, see 
https://tracker.ceph.com/issues/53466 (and its Quincy counterpart 
https://tracker.ceph.com/issues/58588)


Could you please share a bit more info on OSD disk layout?

SSD or HDD? Standalone or shared DB volume? I presume the latter... What 
is disk size and current utilization?


Please share ceph-bluestore-tool's bluefs-bdev-sizes command output if 
possible



Generally, given my assumption that DB volume is currently collocated 
and you still want to stay on Pacific, you might want to consider 
redeploying OSDs with a standalone DB volume setup.


Just create large enough (2x of the current DB size seems to be pretty 
conservative estimation for that volume's size) additional LV on top of 
the same physical disk. And put DB there...


Separating DB from main disk would result in much less fragmentation at 
DB volume and hence work around the problem. The cost would be having 
some extra spare space at DB volume unavailable for user data .



Hope this helps,

Igor


On 20/06/2023 10:29, Carsten Grommel wrote:

Hi all,

we are experiencing the “bluefs enospc bug” again after redeploying all OSDs of 
our Pacific Cluster.
I know that our cluster is a bit too utilized at the moment with 87.26 % raw 
usage but still this should not happen afaik.
We never hat this problem with previous ceph versions and right now I am kind 
of out of ideas at how to tackle these crashes.
Compacting the database did not help in the past either.
Redeploy seems to no help in the long run as well. For documentation I used 
these commands to redeploy the osds:

systemctl stop ceph-osd@${OSDNUM}
ceph osd destroy --yes-i-really-mean-it ${OSDNUM}
blkdiscard ${DEVICE}
sgdisk -Z ${DEVICE}
dmsetup remove ${DMDEVICE}
ceph-volume lvm create --osd-id ${OSDNUM} --data ${DEVICE}

Any ideas or possible solutions on this?  I am not yet ready to upgrade our 
clusters to quincy, also I do presume that this bug is still present in quincy 
as well?

Follow our cluster information:

Crash Info:
ceph crash info 2023-06-19T21:23:51.285180Z_ac4105d7-cb09-45c8-a6e3-8a6bb6727b25
{
 "assert_condition": "abort",
 "assert_file": "/build/ceph/src/os/bluestore/BlueFS.cc",
 "assert_func": "int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, 
uint64_t)",
 "assert_line": 2810,
 "assert_msg": "/build/ceph/src/os/bluestore/BlueFS.cc: In function 'int 
BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fd561810100 time 
2023-06-19T23:23:51.261617+0200\n/build/ceph/src/os/bluestore/BlueFS.cc: 2810: ceph_abort_msg(\"bluefs 
enospc\")\n",
 "assert_thread_name": "ceph-osd",
 "backtrace": [
 "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fd56225f730]",
 "gsignal()",
 "abort()",
 "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1a7) [0x557bb3c65762]",
 "(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned 
long)+0x1175) [0x557bb42e7945]",
 "(BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0xa1) 
[0x557bb42e7ad1]",
 "(BlueFS::_flush(BlueFS::FileWriter*, bool, 
std::unique_lock&)+0x2e) [0x557bb42f803e]",
 "(BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) 
[0x557bb431134b]",
 "(rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, 
rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x44) [0x557bb478e602]",
 "(rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned 
long)+0x333) [0x557bb4956feb]",
 "(rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x5d1) 
[0x557bb4955569]",
 "(rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, 
rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x11d) [0x557bb4b142e1]",
 "(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, 
rocksdb::BlockHandle*, bool)+0x7d6) [0x557bb4b140ca]",
 "(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, 
rocksdb::BlockHandle*, bool)+0x48) [0x557bb4b138e0]",
 "(rocksdb::BlockBasedTableBuilder::Flush()+0x9a) [0x557bb4b13890]",
 "(rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice 
const&)+0x192) [0x557bb4b133c8]",
 "(rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, 
rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase*, std::vector >, std::allocator > > >, 
rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector >, 
std::allocator > > > const*, unsigned int, std::__cxx11::basic_string, 
std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, 
bool, rocksdb::InternalStats*,

[ceph-users] osd memory target not work

2023-06-20 Thread farhad kh

 when set osd_memory_target for limitation usage memory for osd disk ,This
value is expected to be set for the OSD container .But with the docker stats
command, this value is not seen Is my perception of this process wrong?
---

[root@opcsdfpsbpp0201 ~]# ceph orch ps | grep osd.12
osd.12  opcsdfpsbpp0201
   running (9d) 5m ago   9d1205M1953M  17.2.6
c9a1062f7289  bf27cfe16046
[root@opcsdfpsbpp0201 ~]# docker stats | grep osd
1253766d6a78   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-48
0.05% 2.237GiB / 7.732GiB   28.93%0B / 0B   86.8GB / 562GB
63
2bc012e5c604   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-67
0.20% 727MiB / 7.732GiB 9.18% 0B / 0B   37.5GB / 1.29TB
63
dc0bf068050b   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-62
0.11% 360.5MiB / 7.732GiB   4.55% 0B / 0B   125MB / 1.85GB
63
c5f119a37652   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-55
0.12% 312.5MiB / 7.732GiB   3.95% 0B / 0B   86.6MB / 1.66GB
63
7f0b7b61807d   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-5
0.11% 299.4MiB / 7.732GiB   3.78% 0B / 0B   119MB / 1.6GB
63
dadffc77f7b6   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-40
0.11% 274MiB / 7.732GiB 3.46% 0B / 0B   110MB / 1.5GB
63
e439e58d907e   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-34
0.12% 355.9MiB / 7.732GiB   4.49% 0B / 0B   125MB / 1.78GB
63
5e500e2197d6   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-25
0.11% 273.3MiB / 7.732GiB   3.45% 0B / 0B   128MB / 1.55GB
63
a63709567669   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-19
0.11% 714.6MiB / 7.732GiB   9.03% 0B / 0B   89.8MB / 167GB
63
bf27cfe16046   ceph-79a2627c-0821-11ee-a494-00505695c58c-osd-12
0.16% 1.177GiB / 7.732GiB   15.23%0B / 0B   40.8GB / 644GB
63
---
# ceph orch ps | grep osd | grep opcsdfpsbpp0201
osd.5 opcsdfpsbpp0201   running (9d) 6m ago   9d 298M
1953M  17.2.6  c9a1062f7289  7f0b7b61807d
osd.12opcsdfpsbpp0201   running (9d) 6m ago   9d1205M
1953M  17.2.6  c9a1062f7289  bf27cfe16046
osd.19opcsdfpsbpp0201   running (9d) 6m ago   9d 704M
1953M  17.2.6  c9a1062f7289  a63709567669
osd.25opcsdfpsbpp0201   running (9d) 6m ago   9d 273M
1953M  17.2.6  c9a1062f7289  5e500e2197d6
osd.34opcsdfpsbpp0201   running (9d) 6m ago   9d 355M
1953M  17.2.6  c9a1062f7289  e439e58d907e
osd.40opcsdfpsbpp0201   running (9d) 6m ago   9d 273M
1953M  17.2.6  c9a1062f7289  dadffc77f7b6
osd.48opcsdfpsbpp0201   running (4h) 6m ago   9d2290M
1953M  17.2.6  c9a1062f7289  1253766d6a78
osd.55opcsdfpsbpp0201   running (9d) 6m ago   9d 312M
1953M  17.2.6  c9a1062f7289  c5f119a37652
osd.62opcsdfpsbpp0201   running (9d) 6m ago   9d 359M
1953M  17.2.6  c9a1062f7289  dc0bf068050b
osd.67opcsdfpsbpp0201   running (9d) 6m ago   9d 727M
1953M  17.2.6  c9a1062f7289  2bc012e5c604
--

 #ceph config get mgr  osd_memory_target
204800
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] radosgw new zonegroup hammers master with metadata sync

2023-06-20 Thread Boris Behrens

Hi,
yesterday I added a new zonegroup and it looks like it seems to cycle over
the same requests over and over again.

In the log of the main zone I see these requests:
2023-06-20T09:48:37.979+ 7f8941fb3700  1 beast: 0x7f8a602f3700:
fd00:2380:0:24::136 - - [2023-06-20T09:48:37.979941+] "GET
/admin/log?type=metadata=62=e8fc96f1-ae86-4dc1-b432-470b0772fded=100&=b39392eb-75f8-47f0-b4f3-7d3882930b26
HTTP/1.1" 200 44 - - -

Only thing that changes is the 

We have two other zonegroups that are configured identical (ceph.conf and
period) and these don;t seem to spam the main rgw.

root@host:~# radosgw-admin sync status
  realm 5d6f2ea4-b84a-459b-bce2-bccac338b3ef (main)
  zonegroup b39392eb-75f8-47f0-b4f3-7d3882930b26 (dc3)
   zone 96f5eca9-425b-4194-a152-86e310e91ddb (dc3)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master

root@host:~# radosgw-admin period get
{
"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
"epoch": 92,
"predecessor_uuid": "5349ac85-3d6d-4088-993f-7a1d4be3835a",
"sync_status": [
"",
...
""
],
"period_map": {
"id": "e8fc96f1-ae86-4dc1-b432-470b0772fded",
"zonegroups": [
{
"id": "b39392eb-75f8-47f0-b4f3-7d3882930b26",
"name": "dc3",
"api_name": "dc3",
"is_master": "false",
"endpoints": [
],
"hostnames": [
],
"hostnames_s3website": [
],
"master_zone": "96f5eca9-425b-4194-a152-86e310e91ddb",
"zones": [
{
"id": "96f5eca9-425b-4194-a152-86e310e91ddb",
"name": "dc3",
"endpoints": [
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "5d6f2ea4-b84a-459b-bce2-bccac338b3ef",
"sync_policy": {
"groups": []
}
},
...

-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groÃƒ¼en Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] X large objects found in pool 'XXX.rgw.buckets.index'

2023-06-20 Thread Gilles Mocellin


Hello,

I still have large OMAP objects since a year.
These objects are probably from an ancient bucket that has been removed.
So I cannot use bilog trim. Depp-scrub dos nothing.

Also, even if I don't have a huge cluster (my Object Storage pools is 
only arounde 10TB), the rgw-orphan-list is too long to run.


So, As I have only 6 Large OMAP objects (just above 20 keys), I 
would like to find and remove the orphan rados objects.

If someone can tell me if I'm right in my assumptions ?

Let's take that log :

2023-06-09T00:51:10.222449+0200 osd.66 (osd.66) 12 : cluster [WRN] Large 
omap object found. Object: 
3:56f3a469:::.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9:head 
PG: 3.9625cf6a (3.2a) Key count: 200304 Size (bytes): 58606170


I deduce that the bucket-id is : 
aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2

I cannot find a bucket name with that ID in the metadata list :

# radosgw-admin metadata list --metadata-key bucket.instance | grep -i 
aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2


returns nothing.

If I list all objects begining with that bucket ID, I find :

gmo_admin@fidcl-mrs4-sto-sds-01:~$ sudo rados -p MRS4.rgw.buckets.index 
ls | grep aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2 | cat

.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.25
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.0
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.15
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.17
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.9
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.26
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.7
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.23
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.2
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.3
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.16
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.13
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.22
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.21
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.4
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.20
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.19
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.11
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.27
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.24
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.28
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.10
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.14
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.18
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.1
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.6
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.8
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.12
.dir.aefd4003-1866-4b16-b1b3-2f308075cd1c.8348421.2.5

Do you think I'm safe to delete them ? If they are all about a 
non-existent bucket...


How can I be sure that an RGW index object and the omap keys it has, are 
not used ?

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OpenStack (cinder) volumes retyping on Ceph back-end

2023-06-20 Thread Andrea Martra


Hello,
thank you for the confirmation.
I reported the problem on the openstack-discuss mailing.

Thanks,
Andrea

Il 20/06/23 10:15, Eugen Block ha scritto:
I can confirm, in a virtual test openstack environment (Wallaby) with 
ceph quincy I did a retype of an attached volume (root disk of a VM). 
Retyping works, the volume is copied to the other back end pool, but 
the IO is still going to the old pool/image although they already have 
been removed. This was visible in the output of "rbd perf image 
iotop". And now I also understand the reconfiguration part, the xml 
definition still contains the old pool/image information so a reboot 
of the VM fails. I'd say this is rather an openstack (cinder) issue 
than a ceph issue. You should report this in the openstack-discuss 
mailing list or create a bug report on launchpad. If you want I can do 
that as well.

I will do some more testing to have more details.

Thanks,
Eugen

Zitat von Eugen Block :


Hi,

I don't quite understand the issue yet, maybe you can clarify.

If I perform a "change volume type" from OpenStack on volumes 
attached to the VMs the system successfully migrates the volume from 
the source pool to the destination pool and at the end of the 
process the volume is visible in the new pool and is removed from 
the old pool.


Are these volumes root disks or just additional volumes? But 
apparently, the retype works.


The problem encountered is that when reconfiguring the VM, to 
specify the new pool associated with the volumes (performed through 
a resize of the VM, I haven't found any other method to change the 
information on the nova/cinder db automatically.


If the retype already works then what is your goal by "reconfiguring 
the VM"? What information is wrong in the DB? This part needs some 
clarification for me. Can you give some examples?


The VM after the retype continues to work perfectly in RW but the 
"new" volume created in the new pool is not used to write data and 
consequently when the VM is shut down all the changes are lost.


Just wondering, did you shut down the VM before retyping the volume? 
I'll try to reproduce this in a test cluster.


Regards,
Eugen

Zitat von andrea.mar...@oscct.it:


Hello,
I configured different back-end storage on OpenStack (Yoga release) 
and using Ceph (ceph version 17.2.4) with different pools (volumes, 
cloud-basic, shared-hosting-os, shared-hosting-homes,...) for RBD 
application.
I created different volume types towards each of the backends and 
everything works perfectly.
If I perform a "change volume type" from OpenStack on volumes 
attached to the VMs the system successfully migrates the volume from 
the source pool to the destination pool and at the end of the 
process the volume is visible in the new pool and is removed from 
the old pool.
The problem encountered is that when reconfiguring the VM, to 
specify the new pool associated with the volumes (performed through 
a resize of the VM, I haven't found any other method to change the 
information on the nova/cinder db automatically. I also did some 
tests shut-off of the VM and modification of the xml through virsh 
edit and startup the VM) the volume presented to the VM is exactly 
the version and content on the retype date of the volume itself. All 
data written and modified after the retype is lost.
The VM after the retype continues to work perfectly in RW but the 
"new" volume created in the new pool is not used to write data and 
consequently when the VM is shut down all the changes are lost.
Do you have any idea how to carry out a check and possibly how to 
proceed in order not to lose the data of the vm of which I have 
retyped the volume?

The data is written somewhere because the VMs work perfectly.
Thank you
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--

Andrea Martra
Senior Manager
+39 393 9048451

Open Technologies Sas
Via Michele Buniva, 26 - 10064 Pinerolo (TO)
Web:http://www.oscct.it

** Open Source Cloud Computing Technologies **
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OpenStack (cinder) volumes retyping on Ceph back-end

2023-06-20 Thread Eugen Block

I can confirm, in a virtual test openstack environment (Wallaby) with  
ceph quincy I did a retype of an attached volume (root disk of a VM).  
Retyping works, the volume is copied to the other back end pool, but  
the IO is still going to the old pool/image although they already have  
been removed. This was visible in the output of "rbd perf image  
iotop". And now I also understand the reconfiguration part, the xml  
definition still contains the old pool/image information so a reboot  
of the VM fails. I'd say this is rather an openstack (cinder) issue  
than a ceph issue. You should report this in the openstack-discuss  
mailing list or create a bug report on launchpad. If you want I can do  
that as well.

I will do some more testing to have more details.

Thanks,
Eugen

Zitat von Eugen Block :


Hi,

I don't quite understand the issue yet, maybe you can clarify.

If I perform a "change volume type" from OpenStack on volumes  
attached to the VMs the system successfully migrates the volume  
from the source pool to the destination pool and at the end of the  
process the volume is visible in the new pool and is removed from  
the old pool.


Are these volumes root disks or just additional volumes? But  
apparently, the retype works.


The problem encountered is that when reconfiguring the VM, to  
specify the new pool associated with the volumes (performed through  
a resize of the VM, I haven't found any other method to change the  
information on the nova/cinder db automatically.


If the retype already works then what is your goal by "reconfiguring  
the VM"? What information is wrong in the DB? This part needs some  
clarification for me. Can you give some examples?


The VM after the retype continues to work perfectly in RW but the  
"new" volume created in the new pool is not used to write data and  
consequently when the VM is shut down all the changes are lost.


Just wondering, did you shut down the VM before retyping the volume?  
I'll try to reproduce this in a test cluster.


Regards,
Eugen

Zitat von andrea.mar...@oscct.it:


Hello,
I configured different back-end storage on OpenStack (Yoga release)  
and using Ceph (ceph version 17.2.4) with different pools (volumes,  
cloud-basic, shared-hosting-os, shared-hosting-homes,...) for RBD  
application.
I created different volume types towards each of the backends and  
everything works perfectly.
If I perform a "change volume type" from OpenStack on volumes  
attached to the VMs the system successfully migrates the volume  
from the source pool to the destination pool and at the end of the  
process the volume is visible in the new pool and is removed from  
the old pool.
The problem encountered is that when reconfiguring the VM, to  
specify the new pool associated with the volumes (performed through  
a resize of the VM, I haven't found any other method to change the  
information on the nova/cinder db automatically. I also did some  
tests shut-off of the VM and modification of the xml through virsh  
edit and startup the VM) the volume presented to the VM is exactly  
the version and content on the retype date of the volume itself.  
All data written and modified after the retype is lost.
The VM after the retype continues to work perfectly in RW but the  
"new" volume created in the new pool is not used to write data and  
consequently when the VM is shut down all the changes are lost.
Do you have any idea how to carry out a check and possibly how to  
proceed in order not to lose the data of the vm of which I have  
retyped the volume?

The data is written somewhere because the VMs work perfectly.
Thank you
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph Pacific bluefs enospc bug with newly created OSDs

2023-06-20 Thread Carsten Grommel

Hi all,

we are experiencing the “bluefs enospc bug” again after redeploying all OSDs of 
our Pacific Cluster.
I know that our cluster is a bit too utilized at the moment with 87.26 % raw 
usage but still this should not happen afaik.
We never hat this problem with previous ceph versions and right now I am kind 
of out of ideas at how to tackle these crashes.
Compacting the database did not help in the past either.
Redeploy seems to no help in the long run as well. For documentation I used 
these commands to redeploy the osds:

systemctl stop ceph-osd@${OSDNUM}
ceph osd destroy --yes-i-really-mean-it ${OSDNUM}
blkdiscard ${DEVICE}
sgdisk -Z ${DEVICE}
dmsetup remove ${DMDEVICE}
ceph-volume lvm create --osd-id ${OSDNUM} --data ${DEVICE}

Any ideas or possible solutions on this?  I am not yet ready to upgrade our 
clusters to quincy, also I do presume that this bug is still present in quincy 
as well?

Follow our cluster information:

Crash Info:
ceph crash info 2023-06-19T21:23:51.285180Z_ac4105d7-cb09-45c8-a6e3-8a6bb6727b25
{
"assert_condition": "abort",
"assert_file": "/build/ceph/src/os/bluestore/BlueFS.cc",
"assert_func": "int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, 
uint64_t)",
"assert_line": 2810,
"assert_msg": "/build/ceph/src/os/bluestore/BlueFS.cc: In function 'int 
BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 
7fd561810100 time 
2023-06-19T23:23:51.261617+0200\n/build/ceph/src/os/bluestore/BlueFS.cc: 2810: 
ceph_abort_msg(\"bluefs enospc\")\n",
"assert_thread_name": "ceph-osd",
"backtrace": [
"/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fd56225f730]",
"gsignal()",
"abort()",
"(ceph::__ceph_abort(char const*, int, char const*, 
std::__cxx11::basic_string, std::allocator > 
const&)+0x1a7) [0x557bb3c65762]",
"(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned 
long)+0x1175) [0x557bb42e7945]",
"(BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0xa1) 
[0x557bb42e7ad1]",
"(BlueFS::_flush(BlueFS::FileWriter*, bool, 
std::unique_lock&)+0x2e) [0x557bb42f803e]",
"(BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) 
[0x557bb431134b]",
"(rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, 
rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x44) [0x557bb478e602]",
"(rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned 
long)+0x333) [0x557bb4956feb]",
"(rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x5d1) 
[0x557bb4955569]",
"(rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, 
rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x11d) [0x557bb4b142e1]",
"(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, 
rocksdb::BlockHandle*, bool)+0x7d6) [0x557bb4b140ca]",
"(rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, 
rocksdb::BlockHandle*, bool)+0x48) [0x557bb4b138e0]",
"(rocksdb::BlockBasedTableBuilder::Flush()+0x9a) [0x557bb4b13890]",
"(rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, 
rocksdb::Slice const&)+0x192) [0x557bb4b133c8]",
"(rocksdb::BuildTable(std::__cxx11::basic_string, std::allocator > const&, rocksdb::Env*, 
rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, 
rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, 
rocksdb::TableCache*, rocksdb::InternalIteratorBase*, 
std::vector >, 
std::allocator > > >, 
rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, 
std::vector >, 
std::allocator > > > const*, unsigned 
int, std::__cxx11::basic_string, 
std::allocator > const&, std::vector >, unsigned long, rocksdb::SnapshotChecker*, 
rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, 
bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, 
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, 
rocksdb::TableProperties*, int, unsigned long, unsigned long, 
rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x773) [0x557bb4a9aa7d]",
   "(rocksdb::DBImpl::WriteLevel0TableForRecovery(int, 
rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x5de) 
[0x557bb4824676]",
"(rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool, bool*)+0x1aa0) 
[0x557bb48232d0]",
   "(rocksdb::DBImpl::Recover(std::vector > const&, bool, bool, bool, 
unsigned long*)+0x158a) [0x557bb4820846]",
"(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, 
std::__cxx11::basic_string, std::allocator > 
const&, std::vector > const&, 
std::vector >*, rocksdb::DB**, bool, 
bool)+0x679) [0x557bb4825b25]",
"(rocksdb::DB::Open(rocksdb::DBOptions const&, 
std::__cxx11::basic_string, std::allocator > 
const&, std::vector > const&, 
std::vector >*, rocksdb::DB**)+0x52) 
[0x557bb4824efa]",
"(RocksDBStore::do_open(std::ostream&, bool, bool, 
std::__cxx11::basic_string, std::allocator

[ceph-users] [question] Put with "tagging" is slowly?

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

[ceph-users] Re: Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

[ceph-users] Re: Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

[ceph-users] Error while adding host : Error EINVAL: Traceback (most recent call last): File /usr/share/ceph/mgr/mgr_module.py, line 1756, in _handle_command

[ceph-users] Re: [rgw multisite] Perpetual behind

[ceph-users] OSDs cannot join cluster anymore

[ceph-users] Re: [rgw multisite] Perpetual behind

[ceph-users] Re: Starting v17.2.5 RGW SSE with default key (likely others) no longer works

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

[ceph-users] 1 PG stucked in "active+undersized+degraded for long time

[ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6

[ceph-users] Re: Recover OSDs from folder /var/lib/ceph/uuid/removed

[ceph-users] Re: radosgw new zonegroup hammers master with metadata sync

[ceph-users] Recover OSDs from folder /var/lib/ceph/uuid/removed

[ceph-users] Re: osd memory target not work

[ceph-users] Re: Ceph Pacific bluefs enospc bug with newly created OSDs

[ceph-users] osd memory target not work

[ceph-users] radosgw new zonegroup hammers master with metadata sync

[ceph-users] X large objects found in pool 'XXX.rgw.buckets.index'

[ceph-users] Re: OpenStack (cinder) volumes retyping on Ceph back-end

[ceph-users] Re: OpenStack (cinder) volumes retyping on Ceph back-end

[ceph-users] Ceph Pacific bluefs enospc bug with newly created OSDs

23 matches

Site Navigation

Mail list logo

Footer information