[ceph-users] Re: ceph cluster iops low
Hi, You SSD is a "desktop" SSD, not a "enterprise" SSD, see [1] This mostly was't suitable for Ceph [1] https://yourcmc.ru/wiki/Ceph_performance#CAPACITORS.21 k > On 25 Jan 2023, at 05:35, peter...@raksmart.com wrote: > > Hi Mark, > Thanks for your response, it is help! > Our Ceph cluster use Samsung SSD 870 EVO all backed with NVME drive. 12 SSD > drives to 2 NVMe drives per storage node. Each 4TB SSD backed 283G NVMe lvm > partition as DB. > Now cluster throughput only 300M write, and around 5K IOPS. I could see NVMe > drive utilization over 95% show on ‘iostat’ command. Will NVMe drive be a > bottle neck quickly if we have large of IO in cluster? > I have read the top article about OSD bundle with CPU cores. However I can > only find script called pincpu on the github to automate process to allocate > CPU core with OSDs. It seems not work for me. Do you have any tool or > official instruction that can guide me to test it? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mount ceph using FQDN
Hi, Do you think kernel should care about DNS resolution? k > On 24 Jan 2023, at 19:07, kushagra.gu...@hsc.com wrote: > > Hi team, > > We have a ceph cluster with 3 storage nodes: > 1. storagenode1 - abcd:abcd:abcd::21 > 2. storagenode2 - abcd:abcd:abcd::22 > 3. storagenode3 - abcd:abcd:abcd::23 > > We have a dns server with ip abcd:abcd:abcd::31 which resolves the above ip's > with a single hostname. > The resolution is as follows: > ``` > $TTL 1D > @ IN SOA storage.com root ( >6 ; serial >1D ; refresh >1H ; retry >1W ; expire >3H ); minimum > > INNS master > master INA10.0.1.31 > storagenode IN abcd:abcd:abcd::21 > storagenode IN abcd:abcd:abcd::22 > storagenode IN abcd:abcd:abcd::23 > ``` > > We want to mount the ceph storage on a node using this hostname. > For this we are using the command: > ``` > mount -t ceph [storagenode.storage.com]:6789:/ /backup -o > name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== > ``` > > We are getting the following logs in /var/log/messages: > ``` > Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' > (ret=-3): failed > Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip > 'storagenode.storage.com:6789' > ``` > > We also tried mounting ceph storage by removing the dns server and resolving > the ip as follows: > ``` > abcd:abcd:abcd::21 storagenode1 > ``` > > But we are getting similar results. > > Also kindly note that we are able to perform the mount operation if we use > ips instead of domain name. > > Could you please help us out with how we can mount ceph using FQDN. > > Kindly let me know if any other imformation is required. > > My ceph.conf configuration is as follows: > ``` > [global] > ms bind ipv6 = true > ms bind ipv4 = false > mon initial members = storagenode1,storagenode2,storagenode3 > osd pool default crush rule = -1 > fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe > mon host = > [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] > public network = abcd:abcd:abcd::/64 > cluster network = eff0:eff0:eff0::/64 > > [osd] > osd memory target = 4294967296 > > [client.rgw.storagenode1.rgw0] > host = storagenode1 > keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring > log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log > rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 > rgw thread pool size = 512 > ``` > > Thanks and Regards > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph cluster iops low
Hi Mark, Thanks for your response, it is help! Our Ceph cluster use Samsung SSD 870 EVO all backed with NVME drive. 12 SSD drives to 2 NVMe drives per storage node. Each 4TB SSD backed 283G NVMe lvm partition as DB. Now cluster throughput only 300M write, and around 5K IOPS. I could see NVMe drive utilization over 95% show on ‘iostat’ command. Will NVMe drive be a bottle neck quickly if we have large of IO in cluster? I have read the top article about OSD bundle with CPU cores. However I can only find script called pincpu on the github to automate process to allocate CPU core with OSDs. It seems not work for me. Do you have any tool or official instruction that can guide me to test it? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Problems with autoscaler (overlapping roots) after changing the pool class
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s
ruleset you can change. The fix is the same as for the replicates
pools, assign a ruleset with hdd class and after some data movement
the autoscaler should not complain anymore.
Regards
Eugen
Zitat von Massimo Sgaravatto :
Dear all
I have just changed the crush rule for all the replicated pools in the
following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd
ceph osd pool set crush_rule replicated_hdd
See also this [*] thread
Before applying this change, these pools were all using
the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool
autoscale-status" doesn't report any output and the mgr log complains about
overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow
ID CLASS WEIGHT TYPE NAME
-18hdd 1329.26501 root default~hdd
-17hdd 329.14154 rack Rack11-PianoAlto~hdd
-15hdd54.56085 host ceph-osd-04~hdd
30hdd 5.45609 osd.30
31hdd 5.45609 osd.31
...
...
-1 1329.26501 root default
-7 329.14154 rack Rack11-PianoAlto
-8 54.56085 host ceph-osd-04
30hdd 5.45609 osd.30
31hdd 5.45609 osd.31
...
I have already read about this behavior but I have no clear ideas how to
fix the problem.
I read somewhere that the problem happens when there are rules that force
some pools to only use one class and there are also pools which does not
make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have
some EC pools which are using a profile where the class is not specified.
As far I understand, I can't force these pools to use only the hdd class:
according to the doc I can't change this profile specifying the hdd class
(or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if
you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Octopus mgr doesn't resume after boot
Dear all, I have a two hosts setup, and I recently rebooted a mgr machine without "set noout" and "set norebalance" commands. The "darkside2" machine is the cephadm machine, and "darkside3" is the improperly rebooted mgr. Now the darkside3 machine does not resume ceph configuration: [root@darkside2 ~]# ceph orch host ls HOST ADDR LABELS STATUS darkside2 darkside2 darkside3 172.22.132.189 Offline If I understood the docs correctly, I should [root@darkside2 ~]# ceph orch host add darkside3 But this fails because darkside3 doesn't accept root ssh connnections. I presume this has been discussed before, but I couldn't find the correct thread. Could someone please point me in the right direction? Cordially, Renata. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.11 pacific QE validation status
Looks good to go! On Tue, Jan 24, 2023 at 7:57 AM Yuri Weinstein wrote: > Josh, this is ready for your final review/approval and publishing > > Release notes - https://github.com/ceph/ceph/pull/49839 > > On Tue, Jan 24, 2023 at 4:00 AM Venky Shankar wrote: > > > > On Mon, Jan 23, 2023 at 11:22 PM Yuri Weinstein > wrote: > > > > > > Ilya, Venky > > > > > > rbd, krbd, fs reruns are almost ready, pls review/approve > > > > fs approved. > > > > > > > > On Mon, Jan 23, 2023 at 2:30 AM Ilya Dryomov > wrote: > > > > > > > > On Fri, Jan 20, 2023 at 5:38 PM Yuri Weinstein > wrote: > > > > > > > > > > The overall progress on this release is looking much better and if > we > > > > > can approve it we can plan to publish it early next week. > > > > > > > > > > Still seeking approvals > > > > > > > > > > rados - Neha, Laura > > > > > rook - Sébastien Han > > > > > cephadm - Adam > > > > > dashboard - Ernesto > > > > > rgw - Casey > > > > > rbd - Ilya (full rbd run in progress now) > > > > > krbd - Ilya > > > > > > > > Hi Yuri, > > > > > > > > There are 12 infra-related failures in rbd and a few a krbd. Please > > > > rerun failed and dead jobs in: > > > > > > > > > https://pulpito.ceph.com/yuriw-2023-01-20_16:09:11-rbd-pacific_16.2.11_RC6.6-distro-default-smithi/ > > > > > https://pulpito.ceph.com/yuriw-2023-01-15_16:16:11-krbd-pacific_16.2.11_RC6.6-testing-default-smithi/ > > > > > > > > Thanks, > > > > > > > > Ilya > > > > > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > > -- > > Cheers, > > Venky > > > ___ > Dev mailing list -- d...@ceph.io > To unsubscribe send an email to dev-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.11 pacific QE validation status
Josh, this is ready for your final review/approval and publishing Release notes - https://github.com/ceph/ceph/pull/49839 On Tue, Jan 24, 2023 at 4:00 AM Venky Shankar wrote: > > On Mon, Jan 23, 2023 at 11:22 PM Yuri Weinstein wrote: > > > > Ilya, Venky > > > > rbd, krbd, fs reruns are almost ready, pls review/approve > > fs approved. > > > > > On Mon, Jan 23, 2023 at 2:30 AM Ilya Dryomov wrote: > > > > > > On Fri, Jan 20, 2023 at 5:38 PM Yuri Weinstein > > > wrote: > > > > > > > > The overall progress on this release is looking much better and if we > > > > can approve it we can plan to publish it early next week. > > > > > > > > Still seeking approvals > > > > > > > > rados - Neha, Laura > > > > rook - Sébastien Han > > > > cephadm - Adam > > > > dashboard - Ernesto > > > > rgw - Casey > > > > rbd - Ilya (full rbd run in progress now) > > > > krbd - Ilya > > > > > > Hi Yuri, > > > > > > There are 12 infra-related failures in rbd and a few a krbd. Please > > > rerun failed and dead jobs in: > > > > > > https://pulpito.ceph.com/yuriw-2023-01-20_16:09:11-rbd-pacific_16.2.11_RC6.6-distro-default-smithi/ > > > https://pulpito.ceph.com/yuriw-2023-01-15_16:16:11-krbd-pacific_16.2.11_RC6.6-testing-default-smithi/ > > > > > > Thanks, > > > > > > Ilya > > > > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > -- > Cheers, > Venky > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Mount ceph using FQDN
Hi team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 We have a dns server with ip abcd:abcd:abcd::31 which resolves the above ip's with a single hostname. The resolution is as follows: ``` $TTL 1D @ IN SOA storage.com root ( 6 ; serial 1D ; refresh 1H ; retry 1W ; expire 3H ); minimum INNS master master INA10.0.1.31 storagenode IN abcd:abcd:abcd::21 storagenode IN abcd:abcd:abcd::22 storagenode IN abcd:abcd:abcd::23 ``` We want to mount the ceph storage on a node using this hostname. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip 'storagenode.storage.com:6789' ``` We also tried mounting ceph storage by removing the dns server and resolving the ip as follows: ``` abcd:abcd:abcd::21 storagenode1 ``` But we are getting similar results. Also kindly note that we are able to perform the mount operation if we use ips instead of domain name. Could you please help us out with how we can mount ceph using FQDN. Kindly let me know if any other imformation is required. My ceph.conf configuration is as follows: ``` [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 ``` Thanks and Regards ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Problems with autoscaler (overlapping roots) after changing the pool class
Dear all
I have just changed the crush rule for all the replicated pools in the
following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd
ceph osd pool set crush_rule replicated_hdd
See also this [*] thread
Before applying this change, these pools were all using
the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool
autoscale-status" doesn't report any output and the mgr log complains about
overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow
ID CLASS WEIGHT TYPE NAME
-18hdd 1329.26501 root default~hdd
-17hdd 329.14154 rack Rack11-PianoAlto~hdd
-15hdd54.56085 host ceph-osd-04~hdd
30hdd 5.45609 osd.30
31hdd 5.45609 osd.31
...
...
-1 1329.26501 root default
-7 329.14154 rack Rack11-PianoAlto
-8 54.56085 host ceph-osd-04
30hdd 5.45609 osd.30
31hdd 5.45609 osd.31
...
I have already read about this behavior but I have no clear ideas how to
fix the problem.
I read somewhere that the problem happens when there are rules that force
some pools to only use one class and there are also pools which does not
make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have
some EC pools which are using a profile where the class is not specified.
As far I understand, I can't force these pools to use only the hdd class:
according to the doc I can't change this profile specifying the hdd class
(or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if
you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Integrating openstack/swift to ceph cluster
Hello team,, I have deployed ceph pacific cluster using ceph-ansible running on ubuntu 20.04 which have 3 OSD hosts and 3 mons on each OSD host we have 20 osd . I am integrating swift in the cluster but I fail to find the policy and upload objects in the container . I have deployed rgwloadbalancer on top below is my ceph.conf configuration and rgw logs. Kindly help if I am missing configs in my ceph.conf and advise. [client] rbd_default_features = 1 [client.rgw.ceph-osd1] rgw_dns_name = ceph-osd1 [client.rgw.ceph-osd1.rgw0] host = ceph-osd1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd1.rgw0.log rgw frontends = beast endpoint=10.10.13.13:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd1 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true [client.rgw.ceph-osd2] rgw_dns_name = ceph-osd2 [client.rgw.ceph-osd2.rgw0] host = ceph-osd2 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd2.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd2.rgw0.log rgw frontends = beast endpoint=10.10.13.14:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd2 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true [client.rgw.ceph-osd3] rgw_dns_name = ceph-osd3 [client.rgw.ceph-osd3.rgw0] host = ceph-osd3 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log rgw frontends = beast endpoint=10.10.13.15:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd3 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster network = 10.10.13.0/24 fsid = c8ffac77-6e6e-479a-92be-08f9b46453e7 mon host = [v2:10.10.13.10:3300,v1:10.10.13.10:6789],[v2:10.10.13.11:3300 ,v1:10.10.13.11:6789],[v2:10.10.13.12:3300,v1:10.10.13.12:6789] mon initial members = ceph-mon1,ceph-mon2,ceph-mon3 mon_allow_pool_delete = True mon_max_pg_per_osd = 400 osd pool default crush rule = -1 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public network = 0.0.0.0/0 2023-01-24T15:06:33.035+ 7f0b4fe7f700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:33.035 +] "HEAD / HTTP/1.0" 200 5 - - - latency=0.0s 2023-01-24T15:06:33.855+ 7f0b50e81700 1 == starting new request req=0x7f0d1055c6e0 = 2023-01-24T15:06:33.855+ 7f0b50e81700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:33.855+ 7f0b50e81700 1 == req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.0s == 2023-01-24T15:06:33.855+ 7f0b50e81700 1 beast: 0x7f0d1055c6e0: 10.10.13.13 - anonymous [24/Jan/2023:15:06:33.855 +] "HEAD / HTTP/1.0" 200 0 - - - latency=0.0s 2023-01-24T15:06:33.907+ 7f0b41e63700 1 == starting new request req=0x7f0d1055c6e0 = 2023-01-24T15:06:33.907+ 7f0b42664700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:33.907+ 7f0b42664700 1 == req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.0s == 2023-01-24T15:06:33.907+ 7f0b42664700 1 beast: 0x7f0d1055c6e0: 10.10.13.14 - anonymous [24/Jan/2023:15:06:33.907 +] "HEAD / HTTP/1.0" 200 0 - - - latency=0.0s 2023-01-24T15:06:35.039+ 7f0b37e4f700 1 == starting new request req=0x7f0d1055c6e0 = 2023-01-24T15:06:35.039+ 7f0b37e4f700 1 == req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.0s == 2023-01-24T15:06:35.039+ 7f0b37e4f700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:35.039 +] "HEAD
[ceph-users] OSDs fail to start after stopping them with ceph osd stop command
We encountered the following problems while trying to perform maintenance on a Ceph cluster: The cluster consists of 7 Nodes with 10 OSDs each. There are 4 pools on it: 3 of them are replicated pools with 3/2 size/min_size and one is an erasure coded pool with m=2 and k=5. The following global flags were set: * noout * norebalance * nobackfill * norecover Then, after those flags were set, all OSDs were stopped via the command ceph osd stop, which seems to have caused the issue. After maintenance was done, all OSDs were started again via systemctl. Only about half of the 70 OSDs in total started at first - while the other half started, but got killed after a few seconds with the following log messages: ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff3fcf8d700 -1 osd.51 12161 map says i am stopped by admin. shutting down. ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 osd.51 12161 *** Got signal Interrupt *** ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 osd.51 12161 *** Immediate shutdown (osd_fast_shutdown=true) *** And indeed, when looking into the osd map via ceph osd dump, the remaining OSDs seem to be marked as stopped: osd.50 down out weight 0 up_from 9213 up_thru 9416 down_at 9760 last_clean_interval [9106,9207) [v2:10.0.1.61:6813/6211,v1:10.0.1.61:6818/6211] [v2:10.0.0.61:6814/6211,v1:10.0.0.61:6816/6211] exists,stop 9a2590c4-f50b-4550-bfd1-5aafb543cb59 We were able to restore some of the remaining OSDs via running ceph out osd XX ceph in osd XX and then starting the service again (via systemctl start). This did work for most OSDs, except for the OSDs that are located on one specific host. Some OSDs required several restarts until they did not kill themselves a few seconds after starting. This whole issue seems to be caused by the OSDs being marked as stopped in the OSD map [1]. Apparently this state should get reset when re-starting the OSD again [2], but for some reason this doesn't happen for some of the OSDs. This behavior seems to have been introduced via the following pull request [3]. We have also found the following commit where the logic regarding stop seemed to have been introduced [4]. We were looking into commands that reset the stopped status of the OSD in the OSD map, but did not find any way of forcing this. Since we are out of ideas on how to proceed with the remaining 10 OSDs that cannot get brought up: How does one recover from this situation? It seems like by running ceph osd stop the cluster got in a state that seems irrecoverable with the normal CLI commands available. We even looked into the possibility of manually manipulating the osdmap via the osdmaptool, but there doesn't seem to be a way to edit the start/stopped status and it also seems like a very invasive procedure. There does not seem to be any way we can see of recovering from this, apart from rebuilding all the OSDs - which we refrained from for now. Kind Regards Hanreich Stefan [1] https://github.com/ceph/ceph/blob/63a77b2c5b683cb241f865daec92c046152175b4/src/osd/OSD.cc#L8240 [2] https://github.com/ceph/ceph/blob/63a77b2c5b683cb241f865daec92c046152175b4/src/osd/OSDMap.cc#L2353 [3] https://github.com/ceph/ceph/pull/43664 [4] https://github.com/ceph/ceph/commit/5dbae13ce0f5b0104ab43e0ccfe94f832d0e1268 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services
Hi, you can also use SRV records in DNS to publish the IPs of the MONs. Read https://docs.ceph.com/en/quincy/rados/configuration/mon-lookup-dns/ for more info. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: deploying Ceph using FQDN for MON / MDS Services
Hi, On 24.01.23 15:02, Lokendra Rathour wrote: My /etc/ceph/ceph.conf is as follows: [global] fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] Does this ceph.conf also exist on the hosts that want to mount the filesystem? Then you do not need to specify a MON host or IP when mounting CephFS. Just do mount -t ceph -o name=admin,secret=XXX :/ /backup Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mds crash at cscs
On Thu, Jan 19, 2023 at 9:07 PM Lo Re Giuseppe wrote: > > Dear all, > > We have started to use more intensively cephfs for some wlcg related workload. > We have 3 active mds instances spread on 3 servers, > mds_cache_memory_limit=12G, most of the other configs are default ones. > One of them has crashed this night leaving the log below. > Do you have any hint on what could be the cause and how to avoid it? Not atm. Telemetry reported similar crashes https://tracker.ceph.com/issues/54959 (cephfs) https://tracker.ceph.com/issues/54685 (mgr) BT indicates tcmalloc involvement, but not sure what's going on. > > Regards, > > Giuseppe > > [root@naret-monitor03 ~]# journalctl -u > ceph-63334166-d991-11eb-99de-40a6b72108d0@mds.cephfs.naret-monitor03.lqppte.service > ... > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific > > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fe291e4fce0] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 2: abort() > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 3: /lib64/libstdc++.so.6(+0x987ba) [0x7fe2912567ba] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 4: /lib64/libstdc++.so.6(+0x9653c) [0x7fe29125453c] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 5: /lib64/libstdc++.so.6(+0x95559) [0x7fe291253559] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 6: __gxx_personality_v0() > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 7: /lib64/libgcc_s.so.1(+0x10b03) [0x7fe290c34b03] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 8: _Unwind_Resume() > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 9: /usr/bin/ceph-mds(+0x18c104) [0x5638351e7104] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 10: /lib64/libpthread.so.0(+0x12ce0) [0x7fe291e4fce0] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 11: gsignal() > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 12: abort() > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 13: /lib64/libstdc++.so.6(+0x9009b) [0x7fe29124e09b] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 14: /lib64/libstdc++.so.6(+0x9653c) [0x7fe29125453c] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 15: /lib64/libstdc++.so.6(+0x96597) [0x7fe291254597] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 16: /lib64/libstdc++.so.6(+0x967f8) [0x7fe2912547f8] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 17: /lib64/libtcmalloc.so.4(+0x19fa4) [0x7fe29bae6fa4] > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 18: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, vo> > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 19: (std::shared_ptr > InodeSt> > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 20: (CInode::_decode_base(ceph::buffer::v15_2_0::list::iterator_impl > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 21: (CInode::decode_import(ceph::buffer::v15_2_0::list::iterator_impl > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 22: (Migrator::decode_import_inode(CDentry*, ceph::buffer::v15_2_0::lis> > Jan 19 04:49:40 naret-monitor03 > ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: > 23: (Migrator::decode_import_dir(ceph::buffer::v15_2_0::list::iterator_> > Jan 19 04:49:40 naret-monitor03 >
[ceph-users] deploying Ceph using FQDN for MON / MDS Services
Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.11 pacific QE validation status
On Mon, Jan 23, 2023 at 11:22 PM Yuri Weinstein wrote: > > Ilya, Venky > > rbd, krbd, fs reruns are almost ready, pls review/approve fs approved. > > On Mon, Jan 23, 2023 at 2:30 AM Ilya Dryomov wrote: > > > > On Fri, Jan 20, 2023 at 5:38 PM Yuri Weinstein wrote: > > > > > > The overall progress on this release is looking much better and if we > > > can approve it we can plan to publish it early next week. > > > > > > Still seeking approvals > > > > > > rados - Neha, Laura > > > rook - Sébastien Han > > > cephadm - Adam > > > dashboard - Ernesto > > > rgw - Casey > > > rbd - Ilya (full rbd run in progress now) > > > krbd - Ilya > > > > Hi Yuri, > > > > There are 12 infra-related failures in rbd and a few a krbd. Please > > rerun failed and dead jobs in: > > > > https://pulpito.ceph.com/yuriw-2023-01-20_16:09:11-rbd-pacific_16.2.11_RC6.6-distro-default-smithi/ > > https://pulpito.ceph.com/yuriw-2023-01-15_16:16:11-krbd-pacific_16.2.11_RC6.6-testing-default-smithi/ > > > > Thanks, > > > > Ilya > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.11 pacific QE validation status
On Mon, Jan 23, 2023 at 6:51 PM Yuri Weinstein wrote: > > Ilya, Venky > > rbd, krbd, fs reruns are almost ready, pls review/approve rbd and krbd approved. Thanks, Ilya ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
