Re: [ceph-users] Hammer broke after adding 3rd osd server
A quick update on the case. I think i've isolated the problem. I've spent a while checking the osd servers for differences in configuration. I've noticed two distinctions. The first one being the sysctl.conf tuning options for ipoib, which were not present on the new server. The second one is the cron.daily schedule for the mlocate. The mlocate was starting at 6:25 in the morning and this is the reason why it was initially overlooked as the problems tend to start after 7:20am. The old osd servers had mlocate service disabled completely. Last night I've adjusted the sysctl.conf options and disabled the mlocate and so far so good. The cluster did not fall to bits in the morning, which is a great sign. My guess is that the mlocate service was utilising the osd disk(s) which was causing the slow requests. Many thanks for all your help Andrei - Original Message - > From: "andrei" > To: "Wido den Hollander" > Cc: "ceph-users" > Sent: Thursday, 28 April, 2016 15:23:06 > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server > Hello guys, > > Done a bit of digging over the last few days and collected a bunch of logs > from > the osd servers including ops in flight. Will be digging through the data > later > on today. > > I've also done a bunch of network connectivity tests and can verify that I did > not find any evidence of network issues. The ping and hping (tcp) tests were > running over the past few days and did not show any errors or packet drops or > similar issues. The network interface stats reflect that as well. I've ran the > network tests between all osds servers and cloud host servers. The network > interfaces are configured to use the same mtu of 65520 (ipoib interface). > > What I did notice today is once again, the problem tend to start between 7:20 > and 7:40 in the morning, produce a tons of slow requests between the old two > osd servers and the new osd server. The slow requests go away for about 20-30 > mins and return and keep returning after about 20-30 minutes. During the slow > requests the ceph-osd processes go nuts on the new osd server only. They > consume all cpu and the server load goes to 300+. A few hours into the slow > requests cycle i've stopped the ceph-osd processes on one of the old osd > servers, but the problem does not go away. The only thing that help the > cluster > is a full reboot of the new osd server. After the reboot the slow requests do > not come back until the next morning. > > If anyone has an idea what else I could try, please let me know. > > Andrei > > - Original Message ----- >> From: "Wido den Hollander" >> To: "andrei" >> Cc: "ceph-users" >> Sent: Tuesday, 26 April, 2016 22:18:37 >> Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server > >>> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky : >>> >>> >>> Hi Wido, >>> >>> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s >>> infiniband switch where the osd servers and hosts are connected to. There >>> are >>> no default gates on the storage network. The IB is used only for ceph; >>> everything else goes over the ethernet. >>> >>> I've checked the stats on the IB interfaces of osd servers and there are no >>> errors. The ipoib interface has a very small number of dropped packets >>> (0.0003%). >>> >>> What kind of network tests would you suggest that I run? What do you mean >>> by "I >>> would suggest that you check if the network towards clients is also OK."? By >>> clients do you mean the host servers? >>> >> >> With clients I mean you verify if the hosts talking to the Ceph cluster can >> reach each machine running OSDs. >> >> In my case there was packet loss from certain clients which caused the >> issues to >> occur. >> >> Wido >> >>> Many thanks >>> >>> Andrei >>> >>> - Original Message - >>> > From: "Wido den Hollander" >>> > To: "ceph-users" , "Andrei Mikhailovsky" >>> > >>> > Sent: Tuesday, 26 April, 2016 21:17:59 >>> > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server >>> >>> >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky >>> >> : >>> >> >>> >> >>> >> Hello everyone, >>> >> >>> >> I've recently performed a
Re: [ceph-users] Hammer broke after adding 3rd osd server
Hello guys, Done a bit of digging over the last few days and collected a bunch of logs from the osd servers including ops in flight. Will be digging through the data later on today. I've also done a bunch of network connectivity tests and can verify that I did not find any evidence of network issues. The ping and hping (tcp) tests were running over the past few days and did not show any errors or packet drops or similar issues. The network interface stats reflect that as well. I've ran the network tests between all osds servers and cloud host servers. The network interfaces are configured to use the same mtu of 65520 (ipoib interface). What I did notice today is once again, the problem tend to start between 7:20 and 7:40 in the morning, produce a tons of slow requests between the old two osd servers and the new osd server. The slow requests go away for about 20-30 mins and return and keep returning after about 20-30 minutes. During the slow requests the ceph-osd processes go nuts on the new osd server only. They consume all cpu and the server load goes to 300+. A few hours into the slow requests cycle i've stopped the ceph-osd processes on one of the old osd servers, but the problem does not go away. The only thing that help the cluster is a full reboot of the new osd server. After the reboot the slow requests do not come back until the next morning. If anyone has an idea what else I could try, please let me know. Andrei - Original Message - > From: "Wido den Hollander" > To: "andrei" > Cc: "ceph-users" > Sent: Tuesday, 26 April, 2016 22:18:37 > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server >> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky : >> >> >> Hi Wido, >> >> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s >> infiniband switch where the osd servers and hosts are connected to. There are >> no default gates on the storage network. The IB is used only for ceph; >> everything else goes over the ethernet. >> >> I've checked the stats on the IB interfaces of osd servers and there are no >> errors. The ipoib interface has a very small number of dropped packets >> (0.0003%). >> >> What kind of network tests would you suggest that I run? What do you mean by >> "I >> would suggest that you check if the network towards clients is also OK."? By >> clients do you mean the host servers? >> > > With clients I mean you verify if the hosts talking to the Ceph cluster can > reach each machine running OSDs. > > In my case there was packet loss from certain clients which caused the issues > to > occur. > > Wido > >> Many thanks >> >> Andrei >> >> - Original Message - >> > From: "Wido den Hollander" >> > To: "ceph-users" , "Andrei Mikhailovsky" >> > >> > Sent: Tuesday, 26 April, 2016 21:17:59 >> > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server >> >> >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky : >> >> >> >> >> >> Hello everyone, >> >> >> >> I've recently performed a hardware upgrade on our small two osd server >> >> ceph >> >> cluster, which seems to have broke the ceph cluster. We are using ceph for >> >> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with >> >> latest updates and kernel 4.4.6 from ubuntu repo. >> >> >> >> Previous hardware: >> >> >> >> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ >> >> 2Ghz each >> >> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using >> >> IPoIB. >> >> >> >> The following things were upgraded: >> >> >> >> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We >> >> now have >> >> 5 osds per single ssd. >> >> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ >> >> 2.6Ghz >> >> 3. Upgraded ram on osd servers to become 64gb >> >> 4. Installed additional osd disk to have 10 osds per server. >> >> >> >> After adding the third osd server and finishing the initial sync, the >> >> cluster >> >> worked okay for 1-2 days. No issues were noticed. On a third day my >> >> monitoring >> >> system started reporting a bunch of issues from the ceph cluster as well >> >> as >> >> from ou
Re: [ceph-users] Hammer broke after adding 3rd osd server
Hi Andrei, are you using Jumbo Frames? My experience, I had a driver issues where one NIC wouldn't accept the MTU set for the interface and the cluster ran into a very similar behavior as you are describing. After I have set the MTU for all NICs and servers to the working value of my troubling NIC, everything went back to normal. Regards, Alwin On 04/26/2016 05:18 PM, Wido den Hollander wrote: > >> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky : >> >> >> Hi Wido, >> >> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s >> infiniband switch where the osd servers and hosts are connected to. There >> are no default gates on the storage network. The IB is used only for ceph; >> everything else goes over the ethernet. >> >> I've checked the stats on the IB interfaces of osd servers and there are no >> errors. The ipoib interface has a very small number of dropped packets >> (0.0003%). >> >> What kind of network tests would you suggest that I run? What do you mean by >> "I would suggest that you check if the network towards clients is also OK."? >> By clients do you mean the host servers? >> > > With clients I mean you verify if the hosts talking to the Ceph cluster can > reach each machine running OSDs. > > In my case there was packet loss from certain clients which caused the issues > to occur. > > Wido > >> Many thanks >> >> Andrei >> >> ----- Original Message ----- >>> From: "Wido den Hollander" >>> To: "ceph-users" , "Andrei Mikhailovsky" >>> >>> Sent: Tuesday, 26 April, 2016 21:17:59 >>> Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server >> >>>> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky : >>>> >>>> >>>> Hello everyone, >>>> >>>> I've recently performed a hardware upgrade on our small two osd server ceph >>>> cluster, which seems to have broke the ceph cluster. We are using ceph for >>>> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with >>>> latest updates and kernel 4.4.6 from ubuntu repo. >>>> >>>> Previous hardware: >>>> >>>> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ >>>> 2Ghz each >>>> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using >>>> IPoIB. >>>> >>>> The following things were upgraded: >>>> >>>> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We >>>> now have >>>> 5 osds per single ssd. >>>> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ >>>> 2.6Ghz >>>> 3. Upgraded ram on osd servers to become 64gb >>>> 4. Installed additional osd disk to have 10 osds per server. >>>> >>>> After adding the third osd server and finishing the initial sync, the >>>> cluster >>>> worked okay for 1-2 days. No issues were noticed. On a third day my >>>> monitoring >>>> system started reporting a bunch of issues from the ceph cluster as well as >>>> from our virtual machines. This tend to happen between 7:20am and 7:40am >>>> and >>>> lasts for about 2-3 hours before things become normal again. I've checked >>>> the >>>> osd servers and there is nothing that I could find in cron or otherwise >>>> that >>>> starts around 7:20am. >>>> >>>> The problem is as follows: the new osd server's load goes to 400+ with >>>> ceph-osd >>>> processes consuming all cpu resources. The ceph -w shows a high number of >>>> slow >>>> requests which relate to osds belonging to the new osd server. The log >>>> files >>>> show the following: >>>> >>>> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster >>>> [WRN] >>>> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: >>>> osd_op(client.140476549.0:13203438 >>>> rbd_data.2c9de71520eedd1.0621 >>>> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write >>>> 2572288~4096] >>>> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting >>>> for >>>> subops from 22 >>>> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3
Re: [ceph-users] Hammer broke after adding 3rd osd server
> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky : > > > Hi Wido, > > Thanks for your reply. We have a very simple ceph network. A single 40gbit/s > infiniband switch where the osd servers and hosts are connected to. There are > no default gates on the storage network. The IB is used only for ceph; > everything else goes over the ethernet. > > I've checked the stats on the IB interfaces of osd servers and there are no > errors. The ipoib interface has a very small number of dropped packets > (0.0003%). > > What kind of network tests would you suggest that I run? What do you mean by > "I would suggest that you check if the network towards clients is also OK."? > By clients do you mean the host servers? > With clients I mean you verify if the hosts talking to the Ceph cluster can reach each machine running OSDs. In my case there was packet loss from certain clients which caused the issues to occur. Wido > Many thanks > > Andrei > > - Original Message - > > From: "Wido den Hollander" > > To: "ceph-users" , "Andrei Mikhailovsky" > > > > Sent: Tuesday, 26 April, 2016 21:17:59 > > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server > > >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky : > >> > >> > >> Hello everyone, > >> > >> I've recently performed a hardware upgrade on our small two osd server ceph > >> cluster, which seems to have broke the ceph cluster. We are using ceph for > >> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with > >> latest updates and kernel 4.4.6 from ubuntu repo. > >> > >> Previous hardware: > >> > >> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ > >> 2Ghz each > >> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using > >> IPoIB. > >> > >> The following things were upgraded: > >> > >> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We > >> now have > >> 5 osds per single ssd. > >> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ > >> 2.6Ghz > >> 3. Upgraded ram on osd servers to become 64gb > >> 4. Installed additional osd disk to have 10 osds per server. > >> > >> After adding the third osd server and finishing the initial sync, the > >> cluster > >> worked okay for 1-2 days. No issues were noticed. On a third day my > >> monitoring > >> system started reporting a bunch of issues from the ceph cluster as well as > >> from our virtual machines. This tend to happen between 7:20am and 7:40am > >> and > >> lasts for about 2-3 hours before things become normal again. I've checked > >> the > >> osd servers and there is nothing that I could find in cron or otherwise > >> that > >> starts around 7:20am. > >> > >> The problem is as follows: the new osd server's load goes to 400+ with > >> ceph-osd > >> processes consuming all cpu resources. The ceph -w shows a high number of > >> slow > >> requests which relate to osds belonging to the new osd server. The log > >> files > >> show the following: > >> > >> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster > >> [WRN] > >> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: > >> osd_op(client.140476549.0:13203438 > >> rbd_data.2c9de71520eedd1.0621 > >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write > >> 2572288~4096] > >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting > >> for > >> subops from 22 > >> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster > >> [WRN] > >> slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: > >> osd_op(client.140476549.0:13203439 > >> rbd_data.2c9de71520eedd1.0621 > >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write > >> 1101824~8192] > >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting > >> for > >> rw locks > >> > >> > >> > >> There are practically every osd involved in the slow requests and they > >> tend to > >> be between the old two osd servers and the new one. There were no issues > >> a
Re: [ceph-users] Hammer broke after adding 3rd osd server
Hi Wido, Thanks for your reply. We have a very simple ceph network. A single 40gbit/s infiniband switch where the osd servers and hosts are connected to. There are no default gates on the storage network. The IB is used only for ceph; everything else goes over the ethernet. I've checked the stats on the IB interfaces of osd servers and there are no errors. The ipoib interface has a very small number of dropped packets (0.0003%). What kind of network tests would you suggest that I run? What do you mean by "I would suggest that you check if the network towards clients is also OK."? By clients do you mean the host servers? Many thanks Andrei - Original Message - > From: "Wido den Hollander" > To: "ceph-users" , "Andrei Mikhailovsky" > > Sent: Tuesday, 26 April, 2016 21:17:59 > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky : >> >> >> Hello everyone, >> >> I've recently performed a hardware upgrade on our small two osd server ceph >> cluster, which seems to have broke the ceph cluster. We are using ceph for >> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with >> latest updates and kernel 4.4.6 from ubuntu repo. >> >> Previous hardware: >> >> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz >> each >> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using IPoIB. >> >> The following things were upgraded: >> >> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now >> have >> 5 osds per single ssd. >> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ >> 2.6Ghz >> 3. Upgraded ram on osd servers to become 64gb >> 4. Installed additional osd disk to have 10 osds per server. >> >> After adding the third osd server and finishing the initial sync, the cluster >> worked okay for 1-2 days. No issues were noticed. On a third day my >> monitoring >> system started reporting a bunch of issues from the ceph cluster as well as >> from our virtual machines. This tend to happen between 7:20am and 7:40am and >> lasts for about 2-3 hours before things become normal again. I've checked the >> osd servers and there is nothing that I could find in cron or otherwise that >> starts around 7:20am. >> >> The problem is as follows: the new osd server's load goes to 400+ with >> ceph-osd >> processes consuming all cpu resources. The ceph -w shows a high number of >> slow >> requests which relate to osds belonging to the new osd server. The log files >> show the following: >> >> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN] >> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: >> osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621 >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write >> 2572288~4096] >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for >> subops from 22 >> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN] >> slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: >> osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621 >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write >> 1101824~8192] >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for >> rw locks >> >> >> >> There are practically every osd involved in the slow requests and they tend >> to >> be between the old two osd servers and the new one. There were no issues as >> far >> as I can see between the old two servers. >> >> The first thing i've checked is the networking. No issue was identified from >> running ping -i .1 as well as using hping3 for the tcp >> connection >> checks. The network tests were running for over a week and not a single >> packet >> was lost. The slow requests took place while the network tests were running. >> >> I've also checked the osd and ssd disks and I was not able to identify >> anything >> problematic. >> >> Stopping all osds on the new server causes no issues between the old two osd >> servers. I've left the new server disconnected for a few days and had no >> issues >> with the cluster. >> >> I am a bit lost on what else to try and how to debug the issue. Could someone >> please
Re: [ceph-users] Hammer broke after adding 3rd osd server
> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky : > > > Hello everyone, > > I've recently performed a hardware upgrade on our small two osd server ceph > cluster, which seems to have broke the ceph cluster. We are using ceph for > cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with > latest updates and kernel 4.4.6 from ubuntu repo. > > Previous hardware: > > 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz > each and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using > IPoIB. > > The following things were upgraded: > > 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now > have 5 osds per single ssd. > 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ > 2.6Ghz > 3. Upgraded ram on osd servers to become 64gb > 4. Installed additional osd disk to have 10 osds per server. > > After adding the third osd server and finishing the initial sync, the cluster > worked okay for 1-2 days. No issues were noticed. On a third day my > monitoring system started reporting a bunch of issues from the ceph cluster > as well as from our virtual machines. This tend to happen between 7:20am and > 7:40am and lasts for about 2-3 hours before things become normal again. I've > checked the osd servers and there is nothing that I could find in cron or > otherwise that starts around 7:20am. > > The problem is as follows: the new osd server's load goes to 400+ with > ceph-osd processes consuming all cpu resources. The ceph -w shows a high > number of slow requests which relate to osds belonging to the new osd server. > The log files show the following: > > 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN] > slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: > osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621 > [stat,set-alloc-hint object_size 4194304 write_size 4194304,write > 2572288~4096] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) > currently waiting for subops from 22 > 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN] > slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: > osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621 > [stat,set-alloc-hint object_size 4194304 write_size 4194304,write > 1101824~8192] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) > currently waiting for rw locks > > > > There are practically every osd involved in the slow requests and they tend > to be between the old two osd servers and the new one. There were no issues > as far as I can see between the old two servers. > > The first thing i've checked is the networking. No issue was identified from > running ping -i .1 as well as using hping3 for the tcp > connection checks. The network tests were running for over a week and not a > single packet was lost. The slow requests took place while the network tests > were running. > > I've also checked the osd and ssd disks and I was not able to identify > anything problematic. > > Stopping all osds on the new server causes no issues between the old two osd > servers. I've left the new server disconnected for a few days and had no > issues with the cluster. > > I am a bit lost on what else to try and how to debug the issue. Could someone > please help me? > I would still say this is a network issue. "currently waiting for rw locks" is usually a network problem. I found this out myself a few weeks ago: http://blog.widodh.nl/2016/01/slow-requests-with-ceph-waiting-for-rw-locks/ The problem there was a wrong gateway on some machines. In that situation the OSDs could talk just fine, but they had problems with sending traffic back to the clients which lead to buffers filling up. I would suggest that you check if the network towards clients is also OK. Wido > Many thanks > > Andrei > > > > > > > > > > > > > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Hammer broke after adding 3rd osd server
Hello everyone, I've recently performed a hardware upgrade on our small two osd server ceph cluster, which seems to have broke the ceph cluster. We are using ceph for cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with latest updates and kernel 4.4.6 from ubuntu repo. Previous hardware: 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz each and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using IPoIB. The following things were upgraded: 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now have 5 osds per single ssd. 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 2.6Ghz 3. Upgraded ram on osd servers to become 64gb 4. Installed additional osd disk to have 10 osds per server. After adding the third osd server and finishing the initial sync, the cluster worked okay for 1-2 days. No issues were noticed. On a third day my monitoring system started reporting a bunch of issues from the ceph cluster as well as from our virtual machines. This tend to happen between 7:20am and 7:40am and lasts for about 2-3 hours before things become normal again. I've checked the osd servers and there is nothing that I could find in cron or otherwise that starts around 7:20am. The problem is as follows: the new osd server's load goes to 400+ with ceph-osd processes consuming all cpu resources. The ceph -w shows a high number of slow requests which relate to osds belonging to the new osd server. The log files show the following: 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN] slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~4096] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for subops from 22 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN] slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~8192] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for rw locks There are practically every osd involved in the slow requests and they tend to be between the old two osd servers and the new one. There were no issues as far as I can see between the old two servers. The first thing i've checked is the networking. No issue was identified from running ping -i .1 as well as using hping3 for the tcp connection checks. The network tests were running for over a week and not a single packet was lost. The slow requests took place while the network tests were running. I've also checked the osd and ssd disks and I was not able to identify anything problematic. Stopping all osds on the new server causes no issues between the old two osd servers. I've left the new server disconnected for a few days and had no issues with the cluster. I am a bit lost on what else to try and how to debug the issue. Could someone please help me? Many thanks Andrei ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
