[ceph-users] Re: Quincy NFS ingress failover
One more question, for John or anyone: It looks like Ganesha NFS wants to use dbus to talk to other services (although I'm not sure how well this works across containers). I just realised that my build (on AlmaLinux 8 'minimal') did not include dbus, and Ceph has not installed it. I have manually installed the default 'dbus' package and its deps, but this doesn't seem to have had any effect. * Does Ganesha use dbus to talk to other services, even in a cephadm containerised build? * Is there a non-destructive way to make Ganesha retry gsh_dbus_pkginit ? * Is the default dbus package sufficient, or does Ceph require specific dbus plugins? Thank you. On 4/09/2023 9:55 am, Thorne Lawler wrote: John, Thanks for getting back to me. I am indeed using cephadm, and I will dig up those configurations. Even if Ceph Quincy is current completely incapable of configuring its own HA failover, I would really like to know what the /imagined/ process would be for detecting a node failure and failing over. Can you elaborate about those changes that need to happen, or point me to the forums (ideally the posts_ where this work is broken down in more detail? It's too late for me to change this; I am 100% committed to using Ceph for HA NFS, no matter what that involves. Thanks. On 31/08/2023 11:18 pm, John Mulligan wrote: On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote: If there isn't any documentation for this yet, can anyone tell me: * How do I inspect/change my NFS/haproxy/keepalived configuration? * What is it supposed to look like? Does someone have a working example? The configuration for haproxy, keepalive, and ganesha are generated. I'm assuming you are using cephadm orchestration. If you want to see what it generated configs contain look in /var/lib/ceph/// under those dirs may be additional subdirectories like etc/ or config/ (it varies from service to service) It's not simple to customize those files directly. It's not impossible but it's probably not worth it (IMO). Also, speaking only for my personal opinion, the current NFS situation is one of OK scale-out but is HA in name only. Fail over is not mature. To make it so, changes need to happen throughout the stack including in nfs-ganesha. I know there are some conversations happening around this topic but I don't know the best place to get involved upstream. I know this probably isn't very satisfactory, but I hope the information helps. ___ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-le...@ceph.io -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. / ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Quincy NFS ingress failover
John, Thanks for getting back to me. I am indeed using cephadm, and I will dig up those configurations. Even if Ceph Quincy is current completely incapable of configuring its own HA failover, I would really like to know what the /imagined/ process would be for detecting a node failure and failing over. Can you elaborate about those changes that need to happen, or point me to the forums (ideally the posts_ where this work is broken down in more detail? It's too late for me to change this; I am 100% committed to using Ceph for HA NFS, no matter what that involves. Thanks. On 31/08/2023 11:18 pm, John Mulligan wrote: On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote: If there isn't any documentation for this yet, can anyone tell me: * How do I inspect/change my NFS/haproxy/keepalived configuration? * What is it supposed to look like? Does someone have a working example? The configuration for haproxy, keepalive, and ganesha are generated. I'm assuming you are using cephadm orchestration. If you want to see what it generated configs contain look in /var/lib/ceph/// under those dirs may be additional subdirectories like etc/ or config/ (it varies from service to service) It's not simple to customize those files directly. It's not impossible but it's probably not worth it (IMO). Also, speaking only for my personal opinion, the current NFS situation is one of OK scale-out but is HA in name only. Fail over is not mature. To make it so, changes need to happen throughout the stack including in nfs-ganesha. I know there are some conversations happening around this topic but I don't know the best place to get involved upstream. I know this probably isn't very satisfactory, but I hope the information helps. ___ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-le...@ceph.io -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. / ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Running trim / discard on an OSD
TL;DR Is there a way to run trim / discard on an OSD? Long story: I have a Proxmox-Ceph cluster with some OSD as storage for VMs. Discard works perfectly in this cluster. For lab and testing purposes I deploy Proxmox-Ceph clusters as Proxmox VMs in this cluster using nested virtualizacion, each with a few disks acting as OSD. Then a few VMs are configured in the nested Proxmox cluster using the nested Ceph as storage. Hope I've explained myself. The problem I'm facing is that the nested Ceph OSD's are not sending the trim/discard command to the upstream Ceph OSDs, so thin provisioning is not kept. Somehow the discard chain is broken at some point. I've tried using this on the nested ceph OSD: ceph config set global bdev_enable_discard true ceph config set global bdev_async_discard true That somewhat helps, but it does not discard all the used space when data is deleted from the VM or the VM is fully removed, it maybe recovers like 20%. This is why I wonder if there is a way to run trim / discard in an OSD. Thanks in advance -- ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
