[ceph-users] Re: mgr services frequently crash on nodes 2,3,4

2023-08-02 Thread Eugen Block
Can you query those config options yourself? storage01:~ # ceph config get mgr mgr/dashboard/standby_behaviour storage01:~ # ceph config get mgr mgr/dashboard/AUDIT_API_ENABLED I'm not sure if those are responsible for the crash though. Zitat von "Adiga, Anantha" : Hi, Mgr service crash freq

[ceph-users] question about OSD onode hits ratio

2023-08-02 Thread Ben
Hi, We have a cluster running for a while. From grafana ceph dashboard, I saw OSD onode hits ratio 92% when cluster was just up and running. After couple month, it says now 70%. This is not a good trend I think. Just wondering what should be done to stop this trend. Many thank, Ben ___

[ceph-users] Re: ref v18.2.0 QE Validation status

2023-08-02 Thread Brad Hubbard
On Thu, Aug 3, 2023 at 8:31 AM Yuri Weinstein wrote: > Updates: > > 1. bookworm distro build support > We will not build bookworm until Debian bug > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 is resolved > > 2. nfs ganesha > fixed (Thanks Guillaume and Kaleb) > > 3. powercycle fail

[ceph-users] Re: ref v18.2.0 QE Validation status

2023-08-02 Thread Yuri Weinstein
Updates: 1. bookworm distro build support We will not build bookworm until Debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 is resolved 2. nfs ganesha fixed (Thanks Guillaume and Kaleb) 3. powercycle failures due to teuthology - fixed (Thanks Brad, Zack, Dan). I am expecting

[ceph-users] Re: ref v18.2.0 QE Validation status

2023-08-02 Thread Laura Flores
The second smoke issue was also fixed: https://tracker.ceph.com/issues/57206 In any case, it is a non-blocker. On Wed, Aug 2, 2023 at 9:56 AM Yuri Weinstein wrote: > https://github.com/ceph/ceph/pull/52710 merged > > smoke: > https://tracker.ceph.com/issues/62227 marked resolved > https://track

[ceph-users] mgr services frequently crash on nodes 2,3,4

2023-08-02 Thread Adiga, Anantha
Hi, Mgr service crash frequently on nodes 2 3 and 4 with the same condition after the 4th node was added. root@zp3110b001a0104:/# ceph crash stat 19 crashes recorded 16 older than 1 days old: 2023-07-29T03:35:32.006309Z_7b622c2b-a2fc-425a-acb8-dc1673b4c189 2023-07-29T03:35:32.055174Z_a2ee1e23-5

[ceph-users] Re: Disk device path changed - cephadm faild to apply osd service

2023-08-02 Thread Kilian Ries
Ok just tried it it works like expected ... just dump the yaml, edit it and apply it again! Regards ;) Von: Eugen Block Gesendet: Mittwoch, 2. August 2023 12:53:20 An: Kilian Ries Cc: ceph-users@ceph.io Betreff: Re: AW: [ceph-users] Re: Disk device path changed

[ceph-users] Re: ceph-volume lvm migrate error

2023-08-02 Thread Roland Giesler
Ouch, I got exited too quickly! On 2023/08/02 21:27, Roland Giesler wrote: # systemctl start ceph-osd@14 And, viola!, it did it. # ls -la /var/lib/ceph/osd/ceph-14/block* lrwxrwxrwx 1 ceph ceph 50 Dec 25  2022 /var/lib/ceph/osd/ceph-14/block -> /dev/mapper/0GVWr9-dQ65-LHcx-y6fD-z7fI-10A9-gVWZ

[ceph-users] Re: ceph-volume lvm migrate error

2023-08-02 Thread Roland Giesler
On 2023/08/02 13:29, Roland Giesler wrote: On 2023/08/02 12:53, Igor Fedotov wrote: Roland, First of all there are no block.db/block.wal symlinks in OSD folder. Which means there are no standalone DB/WAL any more. That is surprising.  So ceph-volume is not able to extract the DB/WAL from a

[ceph-users] Re: RHEL / CephFS / Pacific / SELinux unavoidable "relabel inode" error?

2023-08-02 Thread Gregory Farnum
I don't think we've seen this reported before. SELinux gets a hefty workout from Red Hat with their downstream ODF for OpenShift (Kubernetes), so it certainly works at a basic level. SELinux is a fussy beast though, so if you're eg mounting CephFS across RHEL nodes and invoking SELinux against it,

[ceph-users] Re: Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?

2023-08-02 Thread Eugen Block
It's all covered in the docs [1], one of the points I already mentioned (require-osd-release), you should have bluestore OSDs and converted them to ceph-volume before you can adopt them with cephadm (if you deployed your cluster pre-nautilus). [1] https://docs.ceph.com/en/nautilus/release

[ceph-users] Re: ref v18.2.0 QE Validation status

2023-08-02 Thread Radoslaw Zarzynski
Final ACK for RADOS. On Tue, Aug 1, 2023 at 6:28 PM Laura Flores wrote: > Rados failures are summarized here: > https://tracker.ceph.com/projects/rados/wiki/REEF#Reef-v1820 > > All are known. Will let Radek give the final ack. > > On Tue, Aug 1, 2023 at 9:05 AM Nizamudeen A wrote: > >> dashboar

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

2023-08-02 Thread Greg O'Neill
Hi Konstantin, I dropped the SATA SSDs from /sys/block and rescanned it. Re-running `ceph-bluestore-tool fsck` resulted in the same output (unexpected aio error). Syslog says the drive is not in write-protect mode, however smart says life remaining is at 1%. Drives have approximately 42k power-

[ceph-users] Re: Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?

2023-08-02 Thread Marc
> > from Ceph perspective it's supported to upgrade from N to P, you can > safely skip O. We have done that on several clusters without any > issues. You just need to make sure that your upgrade to N was > complete. How do you verify if the upgrade was complete? __

[ceph-users] RHEL / CephFS / Pacific / SELinux unavoidable "relabel inode" error?

2023-08-02 Thread Harry G Coin
Hi!  No matter what I try, using the latest cephfs on an all ceph-pacific setup, I've not been able to avoid this error message, always similar to this on RHEL family clients: SELinux: inode=1099954719159 on dev=ceph was found to have an invalid context=system_u:object_r:unlabeled_t:s0.  This

[ceph-users] Re: ref v18.2.0 QE Validation status

2023-08-02 Thread Yuri Weinstein
https://github.com/ceph/ceph/pull/52710 merged smoke: https://tracker.ceph.com/issues/62227 marked resolved https://tracker.ceph.com/issues/62228 in progress Only smoke and prowercycle items are remaining from the test approval standpoint. Here is the quote from Neha's status to the clt summariz

[ceph-users] Re: Disk device path changed - cephadm faild to apply osd service

2023-08-02 Thread Anthony D'Atri
This. You can even constrain placement by size or model number. > On Aug 2, 2023, at 6:53 AM, Eugen Block wrote: > > But that could be done easily like this: > > service_type: osd > service_id: ssd-db > service_name: osd.ssd-db > placement: > hosts: > - storage01 > - storage02 > ... > spec:

[ceph-users] Re: Disk device path changed - cephadm faild to apply osd service

2023-08-02 Thread Eugen Block
But that could be done easily like this: service_type: osd service_id: ssd-db service_name: osd.ssd-db placement: hosts: - storage01 - storage02 ... spec: block_db_size: 64G data_devices: rotational: 1 db_devices: rotational: 0 filter_logic: AND objectstore: bluestore Any

[ceph-users] Re: Disk device path changed - cephadm faild to apply osd service

2023-08-02 Thread Kilian Ries
However since it's already broken now - is this the correct way to fix it? ### ceph orch ls --service_name= --export > myservice.yaml -> change my device path to the correct one ceph orch apply -i myservice.yaml [--dry-run] ### Thanks Von: Eugen Block Gesend

[ceph-users] Re: Disk device path changed - cephadm faild to apply osd service

2023-08-02 Thread Kilian Ries
Yes i need specific device paths because all HDD / SSD are the same size / same vendor etc. I group multiple HDDs with an exclusive SSD for cacheing, for example: spec: data_devices: paths: - /dev/sdh - /dev/sdi - /dev/sdj - /dev/sdk - /dev/sdl db_devices:

[ceph-users] Re: ceph-volume lvm migrate error

2023-08-02 Thread Igor Fedotov
Hi Roland, could you please share the content of thd relevant OSD subfolder? Also you might want to run: ceph-bluestore-tool --path --command bluefs-bdev-sizes to make sure DB/WAL are effectively in use. Thanks, Igor On 8/2/2023 12:04 PM, Roland Giesler wrote: I need some help with this p

[ceph-users] Re: Not all Bucket Shards being used

2023-08-02 Thread Christian Kugler
> Thank you for the information, Christian. When you reshard the bucket id is > updated (with most recent versions of ceph, a generation number is > incremented). The first bucket id matches the bucket marker, but after the > first reshard they diverge. This makes a lot of sense and explains wh

[ceph-users] ceph-volume lvm migrate error

2023-08-02 Thread Roland Giesler
I need some help with this please.  The command below gives and error which is not helpful to me. ceph-volume lvm migrate --osd-id 14 --osd-fsid 4de2a617-4452-420d-a99b-9e0cd6b2a99b --from db wal --target NodeC-nvme1/NodeC-nvme-LV-RocksDB1 --> Source device list is empty Unable to migrate to