[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket
With cephadm you're able to set these values cluster wide. See the host-management section of the docs. https://docs.ceph.com/en/reef/cephadm/host-management/#os-tuning-profiles On Fri, 19 Apr 2024 at 12:40, Konstantin Shalygin wrote: > Hi, > > > On 19 Apr 2024, at 10:39, Pardhiv Karri wrote: > > > > Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 > > number in lsof output and then after 2 disks deletion again it got an > error > > and checked lsof and which is above 35000. I'm not sure how to handle > it. > > I rebooted the monitor node, but the open files kept growing. > > > > root@ceph-mon01 ~# lsof | wc -l > > 49296 > > root@ceph-mon01 ~# > > This means that is not a Ceph problem. Is a problem in this system at all > > > k > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket
Hi, > On 19 Apr 2024, at 10:39, Pardhiv Karri wrote: > > Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 > number in lsof output and then after 2 disks deletion again it got an error > and checked lsof and which is above 35000. I'm not sure how to handle it. > I rebooted the monitor node, but the open files kept growing. > > root@ceph-mon01 ~# lsof | wc -l > 49296 > root@ceph-mon01 ~# This means that is not a Ceph problem. Is a problem in this system at all k ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket
Hi Konstantin, Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 number in lsof output and then after 2 disks deletion again it got an error and checked lsof and which is above 35000. I'm not sure how to handle it. I rebooted the monitor node, but the open files kept growing. root@ceph-mon01 ~# lsof | wc -l 49296 root@ceph-mon01 ~# Thanks, Pardh On Thu, Apr 18, 2024 at 11:36 PM Konstantin Shalygin wrote: > Hi, > > Your shell seems reached the default file discriptors limit (1024 mostly) > and your cluster maybe more than 1000 OSD > > Try to set command `ulimit -n 10240` before rbd rm task > > > k > Sent from my iPhone > > > On 18 Apr 2024, at 23:50, Pardhiv Karri wrote: > > > > Hi, > > > > Trying to delete images in a Ceph pool is causing errors in one of > > the clusters. I rebooted all the monitor nodes sequentially to see if the > > error went away, but it still persists. What is the best way to fix this? > > The Ceph cluster is in an OK state, with no rebalancing or scrubbing > > happening (I did set the noscrub and deep-noscrub flags) and also no load > > on the cluster, very few IO. > > > > root@ceph-mon01 ~# rbd rm > 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete > > -p compute > > Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > > > ^C > > root@ceph-mon01 ~# > > > > > > Thanks, > > Pardh > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- *Pardhiv Karri* "Rise and Rise again until LAMBS become LIONS" ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket
Hi, Your shell seems reached the default file discriptors limit (1024 mostly) and your cluster maybe more than 1000 OSD Try to set command `ulimit -n 10240` before rbd rm task k Sent from my iPhone > On 18 Apr 2024, at 23:50, Pardhiv Karri wrote: > > Hi, > > Trying to delete images in a Ceph pool is causing errors in one of > the clusters. I rebooted all the monitor nodes sequentially to see if the > error went away, but it still persists. What is the best way to fix this? > The Ceph cluster is in an OK state, with no rebalancing or scrubbing > happening (I did set the noscrub and deep-noscrub flags) and also no load > on the cluster, very few IO. > > root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete > -p compute > Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1 > NetHandler create_socket couldn't create socket (24) Too many open files > Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1 > NetHandler create_socket couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1 > NetHandler create_socket couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1 > NetHandler create_socket couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket > couldn't create socket (24) Too many open files > > ^C > root@ceph-mon01 ~# > > > Thanks, > Pardh > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
