[ceph-users] Re: Quincy NFS ingress failover

[Thorne Lawler]

Thanks Christoph!

I had just been assuming that the connectivity check was elsewhere, or 
was implicit in some way.


I have certainly not seen any evidence of Quincy trying to move the IP 
address when the node fails.


On 26/09/2023 8:00 pm, Ackermann, Christoph wrote:

Dear list members,,

after upgrading to reef (18.2.0) I spent some time with CephFS, NFS & 
HA(Ingress). I can confirm that Ingress  (count either 1 or 2) works 
well IF only ONE backend server is configured. But this is, of course, 
no HA. ;-)
Two or more backend servers won't work because there isn't ANY  
"*check*" directive used in HAproxy backend config. This means any 
client will stay/stick on the last NFS backend server used.  IMHO in 
haproxy.cfg this should look more like the following:


[..]
backend backend
   mode        tcp
   option tcp-check
   tcp-check connect port 2049
   balance     source
   hash-type   consistent
   server  nfs.nfstest-st.0 10.100.1.111:2049 
 check
   server  nfs.nfstest-st.1 10.100.1.112:2049 
 check



BTW: There are plenty of configuration directives to get a proper 
working NFS backend. Please refer to:


https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#5.2-check

https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#stick-table

https://infohub.delltechnologies.com/l/ecs-with-haproxy-load-balancer-2/nfs-configuration-definitions/


Did (NFS) HA ever work in previous versions? Unfortunately, I had no 
reason to test this in advance. Currently, only a manual installation 
of  HAproxybased ingress controller(s) should help.


And, can we hope for a correction in future versions?

Thanks,
Christoph



Am Mo., 4. Sept. 2023 um 07:43 Uhr schrieb Thorne Lawler 
:


One more question, for John or anyone:

It looks like Ganesha NFS wants to use dbus to talk to other services
(although I'm not sure how well this works across containers).

I just realised that my build (on AlmaLinux 8 'minimal') did not
include
dbus, and Ceph has not installed it.

I have manually installed the default 'dbus' package and its deps,
but
this doesn't seem to have had any effect.

  * Does Ganesha use dbus to talk to other services, even in a cephadm
    containerised build?
  * Is there a non-destructive way to make Ganesha retry
gsh_dbus_pkginit ?
  * Is the default dbus package sufficient, or does Ceph require
    specific dbus plugins?

Thank you.

On 4/09/2023 9:55 am, Thorne Lawler wrote:
> John,
>
> Thanks for getting back to me. I am indeed using cephadm, and I
will
> dig up those configurations.
>
> Even if Ceph Quincy is current completely incapable of
configuring its
> own HA failover, I would really like to know what the /imagined/
> process would be for detecting a node failure and failing over.
>
> Can you elaborate about those changes that need to happen, or
point me
> to the forums (ideally the posts_ where this work is broken down in
> more detail?
>
> It's too late for me to change this; I am 100% committed to
using Ceph
> for HA NFS, no matter what that involves.
>
> Thanks.
>
> On 31/08/2023 11:18 pm, John Mulligan wrote:
>> On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote:
>>> If there isn't any documentation for this yet, can anyone tell me:
>>>
>>>    * How do I inspect/change my NFS/haproxy/keepalived
configuration?
>>>    * What is it supposed to look like? Does someone have a
working
>>> example?
>> The configuration for haproxy, keepalive, and ganesha are
generated.
>> I'm assuming you are using cephadm orchestration. If you want
to see
>> what it
>> generated configs contain look in /var/lib/ceph///
>> under those dirs may be additional subdirectories like etc/ or
>> config/ (it
>> varies from service to service)
>>
>> It's not simple to customize those files directly. It's not
>> impossible but it's
>> probably not worth it (IMO).
>>
>> Also, speaking only for my personal opinion, the current NFS
>> situation is one
>> of OK scale-out but is HA in name only. Fail over is not
mature. To
>> make it
>> so, changes need to happen throughout the stack including in
>> nfs-ganesha. I
>> know there are some conversations happening around this topic
but I
>> don't know
>> the best place to get involved upstream.
>>
>> I know this probably isn't very satisfactory, but I hope the
information
>> helps.
>>
>> ___
>> ceph-users mailing list --ceph-users@ceph.io
>> To unsubscribe send an email toceph-users-le...@ceph.io
-- 


Regards,

Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 

[ceph-users] Re: After upgrading from 17.2.6 to 18.2.0, OSDs are very frequently restarting due to livenessprobe failures

[Igor Fedotov]

Hi Sudhin,

any publicly available cloud storage, e.g. Google drive should work.


Thanks,

Igor

On 26/09/2023 22:52, sbeng...@gmail.com wrote:

Hi Igor,
Please let where can I upload the OSD logs.
Thanks.
Sudhin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: After upgrading from 17.2.6 to 18.2.0, OSDs are very frequently restarting due to livenessprobe failures

[sbengeri]

Hi Igor,
Please let where can I upload the OSD logs.
Thanks.
Sudhin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CEPH zero iops after upgrade to Reef and manual read balancer

[Laura Flores]

Hi Mosharaf,

Thanks for the update. If you can reproduce the issue, it would be most
helpful to us if you provided the output of `ceph -s`, along with a copy of
your osdmap file.

If you have this information, you can update the tracker here:
https://tracker.ceph.com/issues/62836

Thanks,
Laura

On Mon, Sep 25, 2023 at 6:02 AM Mosharaf Hossain <
mosharaf.hoss...@bol-online.com> wrote:

> Greetings Josh,
>
> I executed the command today, and it effectively resolved the issue.
> Within moments, my pools became active, and read/write IOPS started to
> rise.
> Furthermore, the Hypervisor and VMs can now communicate seamlessly with
> the CEPH Cluster.
>
> *Command run:*
> ceph osd  rm-pg-upmap-primary 21.a0
>
>
> *To summarize our findings:*
>
>- Enabling the Ceph read balancer resulted in libvirtd from the
>hypervisor being unable to communicate with the CEPH cluster.
>- During the case, using RBD command images on the pool was readable
>
>
> I'd like to express my gratitude to everyone involved, especially the
> forum contributors.
>
>
>
> Regards
> Mosharaf Hossain
> Manager, Product Development
> IT Division
> BEXIMCO IT
>
>
>
> On Thu, Sep 14, 2023 at 7:58 PM Josh Salomon  wrote:
>
>> Hi Mosharaf - I will check it but I can assure that this error is a CLI
>> error and the command has not impacted the system or the data. I have no
>> clue what happened - I am sure I tested this scenario.
>> The command syntax is
>> ceph osd  rm-pg-upmap-primary 
>> the error you get is because you did not specify the pg id. Run this
>> command in a loop for all pgs in a pool to return the pool to its original
>> state,
>> Regards,
>>
>> Josh
>>
>>
>> On Thu, Sep 14, 2023 at 4:24 PM Mosharaf Hossain <
>> mosharaf.hoss...@bol-online.com> wrote:
>>
>>> Hello Josh
>>> Thank you your for reply to us.
>>>
>>> After giving the command in the cluster I got the following error. We
>>> are concerned about user data. Could you kindly confirm this command will
>>> not affect any user data?
>>>
>>> root@ceph-node1:/# ceph osd rm-pg-upmap-primary
>>> Traceback (most recent call last):   File "/usr/bin/ceph", line 1327, in
>>>  retval = main()   File "/usr/bin/ceph", line 1036, in main
>>> retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs)   File
>>> "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in
>>> run_in_thread raise t.exception   File
>>> "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run
>>> self.retval = self.func(*self.args, **self.kwargs)   File "rados.pyx", line
>>> 551, in rados.Rados.conf_parse_argv   File "rados.pyx", line 314, in
>>> rados.cstr_list   File "rados.pyx", line 308, in rados.cstr
>>> UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4:
>>> surrogates
>>>
>>> Apart from that, do you need any others information?
>>>
>>>
>>> Regards
>>> Mosharaf Hossain
>>> Manager, Product Development
>>> IT Division
>>> Bangladesh Export Import Company Ltd.
>>>
>>> On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon 
>>> wrote:
>>>
 Hi Mosharaf,

 If you undo the read balancing commands (using the command 'ceph
 osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in
 the performance?

 Regards,

 Josh


 On Thu, Sep 14, 2023 at 12:35 AM Laura Flores 
 wrote:

> Hi Mosharaf,
>
> Can you please create a tracker issue and attach a copy of your
> osdmap? Also, please include any other output that characterizes the
> slowdown in client I/O operations you're noticing in your cluster. I can
> take a look once I have that information,
>
> Thanks,
> Laura
>
> On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain <
> mosharaf.hoss...@bol-online.com> wrote:
>
>> Hello Folks
>> We've recently performed an upgrade on our Cephadm cluster,
>> transitioning
>> from Ceph Quiency to Reef. However, following the manual
>> implementation of
>> a read balancer in the Reef cluster, we've experienced a significant
>> slowdown in client I/O operations within the Ceph cluster, affecting
>> both
>> client bandwidth and overall cluster performance.
>>
>> This slowdown has resulted in unresponsiveness across all virtual
>> machines
>> within the cluster, despite the fact that the cluster exclusively
>> utilizes
>> SSD storage."
>>
>> Kindly guide us to move forward.
>>
>>
>>
>> Regards
>> Mosharaf Hossain
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>

-- 

Laura Flores

She/Her/Hers

[ceph-users] Re: replacing storage server host (not drives)

[Konstantin Shalygin]

Hi,

The procedure is simple: get another host and put current disk to new host. 
Setup boot and network's and back to business


k
Sent from my iPhone

> On Sep 26, 2023, at 17:38, Wyll Ingersoll  
> wrote:
> 
> What is the recommended procedure for replacing the host itself without 
> destroying the OSDs or losing data?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] replacing storage server host (not drives)

[Wyll Ingersoll]

We have a storage node that is failing, but the disks themselves are not.  What 
is the recommended procedure for replacing the host itself without destroying 
the OSDs or losing data?

This cluster is running ceph 16.2.11 using ceph orchestrator with docker 
containers on Ubuntu 20.04 (focal).

thank you,
  Wyllys Ingersoll
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] set proxy for ceph installation

[Majid Varzideh]

hi friends
i have deployed my first node in cluster. we dont have direct internet on
my server so i have to set proxy for that.i set it /etc/environment
/etc/profile but i get bellow error
2023-09-26 17:09:38,254 7f04058b4b80 DEBUG

cephadm ['--image', 'quay.io/ceph/ceph:v17', 'pull']
2023-09-26 17:09:38,302 7f04058b4b80 INFO Pulling container image
quay.io/ceph/ceph:v17...
2023-09-26 17:09:42,083 7f04058b4b80 INFO Non-zero exit code 125 from
/usr/bin/podman pull quay.io/ceph/ceph:v17
2023-09-26 17:09:42,083 7f04058b4b80 INFO /usr/bin/podman: stderr Trying to
pull quay.io/ceph/ceph:v17...
2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
time="2023-09-26T17:09:38+03:30" level=warning msg="Failed, retrying in 1s
... (1/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp
34.228.154.221:443: connect: connection refused"
2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
time="2023-09-26T17:09:39+03:30" level=warning msg="Failed, retrying in 1s
... (2/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp
3.220.246.53:443: connect: connection refused"
2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
time="2023-09-26T17:09:40+03:30" level=warning msg="Failed, retrying in 1s
... (3/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp
18.213.60.205:443: connect: connection refused"
2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr Error:
initializing source docker://quay.io/ceph/ceph:v17: pinging container
registry quay.io: Get "https://quay.io/v2/": dial tcp 34.231.182.47:443:
connect: connection refused
2023-09-26 17:09:42,084 7f04058b4b80 ERROR ERROR: Failed command:
/usr/bin/podman pull quay.io/ceph/ceph:v17


would you please help.
thanks,
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Balancer blocked as autoscaler not acting on scaling change

[Anthony D'Atri]

Note that this will adjust override reweight values, which will conflict with 
balancer upmaps.  

> On Sep 26, 2023, at 3:51 AM, c...@elchaka.de wrote:
> 
> Hi an idea is to see what 
> 
> Ceph osd test-reweight-by-utilization
> shows.
> If it looks usefull you can run the above command without "test"
> 
> Hth
> Mehmet
> 
> Am 22. September 2023 11:22:39 MESZ schrieb b...@sanger.ac.uk:
>> Hi Folks,
>> 
>> We are currently running with one nearfull OSD and 15 nearfull pools. The 
>> most full OSD is about 86% full but the average is 58% full. However, the 
>> balancer is skipping a pool on which the autoscaler is trying to complete a 
>> pg_num reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). 
>> However, the autoscaler has been working on this for the last 20 days, it 
>> works through a list of objects that are misplaced but when it gets close to 
>> the end, more objects get added to the list.
>> 
>> This morning I observed the list get down to c. 7,000 objects misplaced with 
>> 2 PGs active+remapped+backfilling, one PG completed the backfilling then the 
>> list shot up to c. 70,000 objects misplaced with 3 PGs 
>> active+remapped+backfilling.
>> 
>> Has anyone come across this behaviour before? If so, what was your 
>> remediation?
>> 
>> Thanks in advance for sharing.
>> Bruno
>> 
>> Cluster details:
>> 3,068 OSDs when all running, c. 60 per storage node
>> OS: Ubuntu 20.04
>> Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
>> 
>> Use case:
>> S3 storage and OpenStack backend, all pools three-way replicated
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephfs health warn

[Ben]

Hi,
see below for details of warnings.
the cluster is running 17.2.5. the warnings have been around for a while.
one concern of mine is num_segments growing over time. clients with
warn of MDS_CLIENT_OLDEST_TID
increase from 18 to 25 as well. The nodes are with kernel
4.19.0-91.82.42.uelc20.x86_64.
It looks like bugs with client library. And rebooting nodes with problem
will fix it for short period of time? Any suggestions from community for
fixing?

Thanks,
Ben


[root@8cd2c0657c77 /]# ceph health detail

HEALTH_WARN 6 hosts fail cephadm check; 2 clients failing to respond to
capability release; 25 clients failing to advance oldest client/flush tid;
3 MDSs report slow requests; 3 MDSs behind on trimming

[WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check

host host15w (192.168.31.33) failed check: Unable to reach remote host
host15w. Process exited with non-zero exit status 1

host host20w (192.168.31.38) failed check: Unable to reach remote host
host20w. Process exited with non-zero exit status 1

host host19w (192.168.31.37) failed check: Unable to reach remote host
host19w. Process exited with non-zero exit status 1

host host17w (192.168.31.35) failed check: Unable to reach remote host
host17w. Process exited with non-zero exit status 1

host host18w (192.168.31.36) failed check: Unable to reach remote host
host18w. Process exited with non-zero exit status 1

host host16w (192.168.31.34) failed check: Unable to reach remote host
host16w. Process exited with non-zero exit status 1

[WRN] MDS_CLIENT_LATE_RELEASE: 2 clients failing to respond to capability
release

mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to
respond to capability release client_id: 460983

mds.code-store.host16w.vucirx(mds.3): Client  failing to respond to
capability release client_id: 460983

[WRN] MDS_CLIENT_OLDEST_TID: 25 clients failing to advance oldest
client/flush tid

mds.code-store.host18w.fdsqff(mds.1): Client k8s-node36 failing to
advance its oldest client/flush tid.  client_id: 460983

mds.code-store.host18w.fdsqff(mds.1): Client  failing to advance its
oldest client/flush tid.  client_id: 460226

mds.code-store.host18w.fdsqff(mds.1): Client k8s-node32 failing to
advance its oldest client/flush tid.  client_id: 239797

mds.code-store.host15w.reolpx(mds.5): Client k8s-node34 failing to
advance its oldest client/flush tid.  client_id: 460226

mds.code-store.host15w.reolpx(mds.5): Client k8s-node32 failing to
advance its oldest client/flush tid.  client_id: 239797

mds.code-store.host15w.reolpx(mds.5): Client  failing to advance its
oldest client/flush tid.  client_id: 460983

mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node34 failing to
advance its oldest client/flush tid.  client_id: 460226

mds.code-store.host18w.rtyvdy(mds.7): Client  failing to advance its
oldest client/flush tid.  client_id: 239797

mds.code-store.host18w.rtyvdy(mds.7): Client k8s-node36 failing to
advance its oldest client/flush tid.  client_id: 460983

mds.code-store.host17w.kcdopb(mds.2): Client  failing to advance its
oldest client/flush tid.  client_id: 239797

mds.code-store.host17w.kcdopb(mds.2): Client  failing to advance its
oldest client/flush tid.  client_id: 460983

mds.code-store.host17w.kcdopb(mds.2): Client k8s-node34 failing to
advance its oldest client/flush tid.  client_id: 460226

mds.code-store.host17w.kcdopb(mds.2): Client k8s-node24 failing to
advance its oldest client/flush tid.  client_id: 12072730

mds.code-store.host20w.bfoftp(mds.4): Client k8s-node32 failing to
advance its oldest client/flush tid.  client_id: 239797

mds.code-store.host20w.bfoftp(mds.4): Client k8s-node36 failing to
advance its oldest client/flush tid.  client_id: 460983

mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node24 failing to
advance its oldest client/flush tid.  client_id: 12072730

mds.code-store.host19w.ywrmiz(mds.6): Client k8s-node34 failing to
advance its oldest client/flush tid.  client_id: 460226

mds.code-store.host19w.ywrmiz(mds.6): Client  failing to advance its
oldest client/flush tid.  client_id: 239797

mds.code-store.host19w.ywrmiz(mds.6): Client  failing to advance its
oldest client/flush tid.  client_id: 460983

mds.code-store.host16w.vucirx(mds.3): Client  failing to advance its
oldest client/flush tid.  client_id: 460983

mds.code-store.host16w.vucirx(mds.3): Client  failing to advance its
oldest client/flush tid.  client_id: 460226

mds.code-store.host16w.vucirx(mds.3): Client  failing to advance its
oldest client/flush tid.  client_id: 239797

mds.code-store.host17w.pdziet(mds.0): Client k8s-node32 failing to
advance its oldest client/flush tid.  client_id: 239797

mds.code-store.host17w.pdziet(mds.0): Client k8s-node34 failing to
advance its oldest client/flush tid.  client_id: 460226

mds.code-store.host17w.pdziet(mds.0): Client k8s-node36 failing to
advance 

[ceph-users] Re: pgs incossistent every day same osd

[Jorge JP]

Hello,

Thankyou.

I think the instructions are:


  1.  Mark osd failed with out
  2.  Waiting for rebalancing the data and wait to OK status
  3.  Mark as down
  4.  Delete osd
  5.  Replace device by new
  6.  Add new osd

Is correct?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pgs incossistent every day same osd

[Janek Bevendorff]

Yes. If you've seen this reoccur multiple times, you can expect it will 
only get worse with time. You should replace the disk soon. Very often 
these disks are also starting to slow down other operations in the 
cluster as the read times increase.



On 26/09/2023 13:17, Jorge JP wrote:

Hello,

First, sorry for my english...

Since a few weeks, I receive every day notifies with HEALTH ERR in my ceph. The 
notifies are related to inconssistent pgs and ever are on same osd.

I ran smartctl test to the disk osd assigned and the result is "passed".

Should replace the disk by other new?

Regards!

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Bauhaus-Universität Weimar
Bauhausstr. 9a, R308
99423 Weimar, Germany

Phone: +49 3643 58 3577
www.webis.de



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] pgs incossistent every day same osd

[Jorge JP]

Hello,

First, sorry for my english...

Since a few weeks, I receive every day notifies with HEALTH ERR in my ceph. The 
notifies are related to inconssistent pgs and ever are on same osd.

I ran smartctl test to the disk osd assigned and the result is "passed".

Should replace the disk by other new?

Regards!

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Quincy NFS ingress failover

[Ackermann, Christoph]

Dear list members,,

after upgrading to reef (18.2.0) I spent some time with CephFS, NFS &
HA(Ingress). I can confirm that Ingress  (count either 1 or 2) works well
IF only ONE backend server is configured. But this is, of course, no HA. ;-)
Two or more backend servers won't work because there isn't ANY  "*check*"
directive used in HAproxy backend config. This means any client will
stay/stick on the last NFS backend server used.  IMHO in haproxy.cfg this
should look more like the following:

[..]
backend backend
   modetcp
   option tcp-check
   tcp-check connect port 2049
   balance source
   hash-type   consistent
   server  nfs.nfstest-st.0  10.100.1.111:2049 check
   server  nfs.nfstest-st.1  10.100.1.112:2049 check


BTW: There are plenty of configuration directives to get a proper working
NFS backend. Please refer to:

https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#5.2-check

https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#stick-table

https://infohub.delltechnologies.com/l/ecs-with-haproxy-load-balancer-2/nfs-configuration-definitions/


Did (NFS) HA ever work in previous versions? Unfortunately, I had no reason
to test this in advance.  Currently, only a manual installation of  HAproxy
based ingress controller(s) should help.

And, can we hope for a correction in future versions?

Thanks,
Christoph



Am Mo., 4. Sept. 2023 um 07:43 Uhr schrieb Thorne Lawler :

> One more question, for John or anyone:
>
> It looks like Ganesha NFS wants to use dbus to talk to other services
> (although I'm not sure how well this works across containers).
>
> I just realised that my build (on AlmaLinux 8 'minimal') did not include
> dbus, and Ceph has not installed it.
>
> I have manually installed the default 'dbus' package and its deps, but
> this doesn't seem to have had any effect.
>
>   * Does Ganesha use dbus to talk to other services, even in a cephadm
> containerised build?
>   * Is there a non-destructive way to make Ganesha retry gsh_dbus_pkginit ?
>   * Is the default dbus package sufficient, or does Ceph require
> specific dbus plugins?
>
> Thank you.
>
> On 4/09/2023 9:55 am, Thorne Lawler wrote:
> > John,
> >
> > Thanks for getting back to me. I am indeed using cephadm, and I will
> > dig up those configurations.
> >
> > Even if Ceph Quincy is current completely incapable of configuring its
> > own HA failover, I would really like to know what the /imagined/
> > process would be for detecting a node failure and failing over.
> >
> > Can you elaborate about those changes that need to happen, or point me
> > to the forums (ideally the posts_ where this work is broken down in
> > more detail?
> >
> > It's too late for me to change this; I am 100% committed to using Ceph
> > for HA NFS, no matter what that involves.
> >
> > Thanks.
> >
> > On 31/08/2023 11:18 pm, John Mulligan wrote:
> >> On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote:
> >>> If there isn't any documentation for this yet, can anyone tell me:
> >>>
> >>>* How do I inspect/change my NFS/haproxy/keepalived configuration?
> >>>* What is it supposed to look like? Does someone have a working
> >>> example?
> >> The configuration for haproxy, keepalive, and ganesha are generated.
> >> I'm assuming you are using cephadm orchestration. If you want to see
> >> what it
> >> generated configs contain look in /var/lib/ceph///
> >> under those dirs may be additional subdirectories like etc/ or
> >> config/ (it
> >> varies from service to service)
> >>
> >> It's not simple to customize those files directly. It's not
> >> impossible but it's
> >> probably not worth it (IMO).
> >>
> >> Also, speaking only for my personal opinion, the current NFS
> >> situation is one
> >> of OK scale-out but is HA in name only. Fail over is not mature. To
> >> make it
> >> so, changes need to happen throughout the stack including in
> >> nfs-ganesha. I
> >> know there are some conversations happening around this topic but I
> >> don't know
> >> the best place to get involved upstream.
> >>
> >> I know this probably isn't very satisfactory, but I hope the information
> >> helps.
> >>
> >> ___
> >> ceph-users mailing list --ceph-users@ceph.io
> >> To unsubscribe send an email toceph-users-le...@ceph.io
> --
>
> Regards,
>
> Thorne Lawler - Senior System Administrator
> *DDNS* | ABN 76 088 607 265
> First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
> P +61 499 449 170
>
> _DDNS
>
> /_*Please note:* The information contained in this email message and any
> attached files may be confidential information, and may also be the
> subject of legal professional privilege. _If you are not the intended
> recipient any use, disclosure or copying of this email is unauthorised.
> _If you received this email in error, please notify Discount Domain Name
> Services Pty Ltd on 03 9815 6868 to report this matter and delete all
> copies of this 

[ceph-users] rbd rados cephfs libs compilation

[Arnaud Morin]

Hey ceph users!

I'd like to compile the different lib-xyz from ceph (rados, rbd,
cephfs) with ENABLE_SHARED=OFF (I want static libraries).

Since few days I am struggling building the whole ceph repo on debian
12.
Is there any way to build only the libraries? I dont need ceph, but only
the client side libraries.

Regards,
Arnaud.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Libcephfs : ceph_readdirplus_r() with ceph_ll_lookup_vino() : ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

[Joseph Fernandes]

CC ceph-user
(Apologies, Forgot last time I replied)

On Tue, Sep 26, 2023 at 1:11 PM Joseph Fernandes 
wrote:

> Hello Venky,
>
> Did you get a chance to look into the updated program? Am I
> missing something?
> I suppose it's something trivial I am missing, As I see these API used in
> NFS Ganesha also and would have been tested there.
>
> https://github.com/nfs-ganesha/nfs-ganesha/blob/2a57b6d53295426247b200cd100ba0741b12aff9/src/FSAL/FSAL_CEPH/export.c#L322
>
> Thanks,
> ~Joe
>
> On Fri, Sep 22, 2023 at 10:52 PM Joseph Fernandes 
> wrote:
>
>> Hello Venky,
>>
>> Nice to hear from you :) Hope you are doing well.
>>
>> I tried as you suggested,
>>
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# mkdir
>> dir1 dir2
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo
>> "Hello Worldls!" > file2
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# echo
>> "Hello Worldls!" > file1
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls
>> dir1  dir2  file1  file2
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root#
>>
>> Create a new snapshot called  "sofs-4-6"
>>
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root#  ceph fs
>> subvolume snapshot ls cephfs 4
>> [
>> {
>> "name": "sofs-4-1"
>> },
>> {
>> "name": "sofs-4-2"
>> },
>> {
>> "name": "sofs-4-3"
>> },
>> {
>> "name": "sofs-4-4"
>> },
>> {
>> "name": "sofs-4-5"
>> },
>> {
>> "name": "sofs-4-6"
>> }
>> ]
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root#
>>
>> And Delete the file and directory from the active filessystem
>>
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# rm -rf *
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# ls
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root# cd .snap
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# ls
>> _sofs-4-6_1099511627778
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap# cd
>> _sofs-4-6_1099511627778/
>> root@ss-joe-01(bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778#
>> ls
>> dir1  dir2  file1  file2
>> root@ss-joe-01
>> (bash):/mnt/cephfs/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778#
>>
>> Now I modified the program and executed it, I am getting the same result.
>>
>>  root@ss-joe-01(bash):/home/hydrauser# ./snapshot_inode_lookup
>>
>> =/volumes/_nogroup/4/user_root/=
>>
>> Path/Name:"/volumes/_nogroup/4/user_root/"
>> Inode Address: 0x7f7c10008f00
>> Inode Number : 1099511628293
>> Snapshot Number  : 18446744073709551614
>> Inode Number : 1099511628293
>> Snapshot Number  : 18446744073709551614
>> . Ino: 1099511628293 SnapId: 18446744073709551614 Address: 0x7f7c10008f00
>> .. Ino: 1099511627778 SnapId: 18446744073709551614 Address: 0x7f7c100086f0
>>
>>
>> =1099511628293:7=
>>
>> Path/Name:"1099511628293:7"
>> Inode Address: 0x7f7c10009710
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710
>> .. Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710
>>
>>
>> =/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778=
>>
>> Path/Name
>>  :"/volumes/_nogroup/4/user_root/.snap/_sofs-4-6_1099511627778"
>> Inode Address: 0x7f7c10009710
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710
>> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640
>> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030
>> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720
>> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0
>> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420
>>
>>
>> =1099511628293:7=
>>
>> Path/Name:"1099511628293:7"
>> Inode Address: 0x7f7c10009710
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> Inode Number : 1099511628293
>> Snapshot Number  : 7
>> . Ino: 1099511628293 SnapId: 7 Address: 0x7f7c10009710
>> .. Ino: 1099511628293 SnapId: 18446744073709551615 Address: 0x55efc15b4640
>> file1 Ino: 1099511628297 SnapId: 7 Address: 0x7f7c1000a030
>> dir1 Ino: 1099511628294 SnapId: 7 Address: 0x7f7c1000a720
>> dir2 Ino: 1099511628295 SnapId: 7 Address: 0x7f7c1000ada0
>> file2 Ino: 1099511628296 SnapId: 7 Address: 0x7f7c1000b420
>>
>> root@ss-joe-01(bash):/home/hydrauser#
>>
>> I have attached the modified program and ceph client logs from this run.
>>
>> 

[ceph-users] Re: Balancer blocked as autoscaler not acting on scaling change

[ceph]

Hi an idea is to see what 

Ceph osd test-reweight-by-utilization
 shows.
If it looks usefull you can run the above command without "test"

Hth
Mehmet

Am 22. September 2023 11:22:39 MESZ schrieb b...@sanger.ac.uk:
>Hi Folks,
>
>We are currently running with one nearfull OSD and 15 nearfull pools. The most 
>full OSD is about 86% full but the average is 58% full. However, the balancer 
>is skipping a pool on which the autoscaler is trying to complete a pg_num 
>reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the 
>autoscaler has been working on this for the last 20 days, it works through a 
>list of objects that are misplaced but when it gets close to the end, more 
>objects get added to the list.
>
>This morning I observed the list get down to c. 7,000 objects misplaced with 2 
>PGs active+remapped+backfilling, one PG completed the backfilling then the 
>list shot up to c. 70,000 objects misplaced with 3 PGs 
>active+remapped+backfilling.
>
>Has anyone come across this behaviour before? If so, what was your remediation?
>
>Thanks in advance for sharing.
>Bruno
>
>Cluster details:
>3,068 OSDs when all running, c. 60 per storage node
>OS: Ubuntu 20.04
>Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive
>
>Use case:
>S3 storage and OpenStack backend, all pools three-way replicated
>___
>ceph-users mailing list -- ceph-users@ceph.io
>To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

[Janek Bevendorff]

I have had defer_client_eviction_on_laggy_osds set to false for a while 
and I haven't had any further warnings so far (obviously), but also all 
the other problems with laggy clients bringing our MDS to a crawl over 
time seem to have gone. So at least on our cluster, the new configurable 
seems to do more harm than good. I can see why it's there, but the 
implementation appears to be rather buggy.


I also set mds_session_blocklist_on_timeout to false, because I had the 
impression that clients where being blocklisted too quickly.



On 21/09/2023 09:24, Janek Bevendorff wrote:


Hi,

I took a snapshot of MDS.0's logs. We have five active MDS in total, 
each one reporting laggy OSDs/clients, but I cannot find anything 
related to that in the log snippet. Anyhow, I uploaded the log for 
your reference with ceph-post-file ID 
79b5138b-61d7-4ba7-b0a9-c6f02f47b881.


This is what ceph status looks like after a couple of days. This is 
not normal:


HEALTH_WARN
55 client(s) laggy due to laggy OSDs
8 clients failing to respond to capability release
1 clients failing to advance oldest client/flush tid
5 MDSs report slow requests

(55 clients are actually "just" 11 unique client IDs, but each MDS 
makes their own report.)


osd mon_osd_laggy_halflife is not configured on our cluster, so it's 
the default of 3600.



Janek


On 20/09/2023 13:17, Dhairya Parmar wrote:

Hi Janek,

The PR venky mentioned makes use of OSD's laggy parameters 
(laggy_interval and
laggy_probability) to find if any OSD is laggy or not. These laggy 
parameters
can reset to 0 if the interval between the last modification done to 
OSDMap and
the time stamp when OSD was marked down exceeds the grace interval 
threshold

which is the value we get by `mon_osd_laggy_halflife * 48` where
mon_osd_laggy_halflife is a configurable value which is by default 
3600 so only
if the interval I talked about exceeds 172800; the laggy parameters 
would reset
to 0. I'd recommend taking a look at what your configured value 
is(using cmd:

ceph config get osd mon_osd_laggy_halflife).

There is also a "hack" to reset the parameters manually(*Not 
recommended, just

for info*): set mon_osd_laggy_weight to 1 using `ceph config set osd
mon_osd_laggy_weight 1` and reboot the OSD(s) which is/are being said 
laggy and

you will see the lagginess go away.


*Dhairya Parmar*

Associate Software Engineer, CephFS

Red Hat Inc. 

dpar...@redhat.com





On Wed, Sep 20, 2023 at 3:25 PM Venky Shankar  
wrote:


Hey Janek,

I took a closer look at various places where the MDS would consider a
client as laggy and it seems like a wide variety of reasons are taken
into consideration and not all of them might be a reason to defer
client
eviction, so the warning is a bit misleading. I'll post a PR for
this. In
the meantime, could you share the debug logs stated in my
previous email?

On Wed, Sep 20, 2023 at 3:07 PM Venky Shankar
 wrote:

> Hi Janek,
>
> On Tue, Sep 19, 2023 at 4:44 PM Janek Bevendorff <
> janek.bevendo...@uni-weimar.de> wrote:
>
>> Hi Venky,
>>
>> As I said: There are no laggy OSDs. The maximum ping I have
for any OSD
>> in ceph osd perf is around 60ms (just a handful, probably
aging disks). The
>> vast majority of OSDs have ping times of less than 1ms. Same
for the host
>> machines, yet I'm still seeing this message. It seems that the
affected
>> hosts are usually the same, but I have absolutely no clue why.
>>
>
> It's possible that you are running into a bug which does not
clear the
> laggy clients list which the MDS sends to monitors via beacons.
Could you
> help us out with debug mds logs (by setting debug_mds=20) for
the active
> mds for around 15-20 seconds and share the logs please? Also
reset the log
> level once done since it can hurt performance.
>
> # ceph config set mds.<> debug_mds 20
>
> and reset via
>
> # ceph config rm mds.<> debug_mds
>
>
>> Janek
>>
>>
>> On 19/09/2023 12:36, Venky Shankar wrote:
>>
>> Hi Janek,
>>
>> On Mon, Sep 18, 2023 at 9:52 PM Janek Bevendorff <
>> janek.bevendo...@uni-weimar.de> wrote:
>>
>>> Thanks! However, I still don't really understand why I am
seeing this.
>>>
>>
>> This is due to a changes that was merged recently in pacific
>>
>> https://github.com/ceph/ceph/pull/52270
>>
>> The MDS would not evict laggy clients if the OSDs report as
laggy. Laggy
>> OSDs can cause cephfs clients to not flush dirty data (during
cap revokes
>> by the MDS) and thereby showing up as laggy and getting
evicted by the MDS.
>> This behaviour was changed and therefore you get warnings that
some client
>> are laggy but they are not evicted since the OSDs are laggy.
>>
>>
>>> The first time