[ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). This weekend we'be experienced a huge outage from our customers vms (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's started to slow request/block PG's on pool PRIVATE ( replica size 1 ) basically all PG's blocked where just one OSD in the acting set, but all customers on the other pool got their vms almost freezed. while trying to do basic troubleshooting like doing noout and then bringing down the OSD that slowed/blocked the most, inmediatelly another OSD slowed/locked iops on pgs from the same PRIVATE pool, so we rolled back that change and started to move data around with the same logic (reweighting down those OSD) with exactly the same result. So, me made a decition, we decided to delete the pool where all PGS where slowed/locked allways despite the osd. Not even 10 secconds passes after the pool deletion, where not only there were no more degraded PGs, bit also ALL slow iops dissapeared for ever, and performance from hundreds of vms came to normal immediately. I must say that i was kinda scared to see that happen, bascally because there was only ONE POOL's PGS always slowed, but performance hit the another pool, so ... did not the PGS that exists on one pool are not shared by the other ? If my assertion is true, why OSD's locking iops from one pool's pg slowed down all other pgs from other pools ? again, i just deleted a pool that has almost no traffic, because its pgs were locked and affected pgs on another pool, and as soon as that happened, the whole cluster came back to normal (and of course, HEALTH_OK and no slow transaction whatsoever) please, someone help me understand the gap where i miss something, since this , as long as my ceph knowledge is concerned, makes no sense. PS: i have found someone that , looks like went through the same here: https://forum.proxmox.com/threads/ceph-osd-failure-causing-proxmox-node-to-crash.20781/ but i still dont understand what happened. hoping to get the help from the community. -- Alejandrito. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). This weekend we'be experienced a huge outage from our customers vms (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's started to slow request/block PG's on pool PRIVATE ( replica size 1 ) basically all PG's blocked where just one OSD in the acting set, but all customers on the other pool got their vms almost freezed. while trying to do basic troubleshooting like doing noout and then bringing down the OSD that slowed/blocked the most, inmediatelly another OSD slowed/locked iops on pgs from the same PRIVATE pool, so we rolled back that change and started to move data around with the same logic (reweighting down those OSD) with exactly the same result. So, me made a decition, we decided to delete the pool where all PGS where slowed/locked allways despite the osd. Not even 10 secconds passes after the pool deletion, where not only there were no more degraded PGs, bit also ALL slow iops dissapeared for ever, and performance from hundreds of vms came to normal immediately. I must say that i was kinda scared to see that happen, bascally because there was only ONE POOL's PGS always slowed, but performance hit the another pool, so ... did not the PGS that exists on one pool are not shared by the other ? If my assertion is true, why OSD's locking iops from one pool's pg slowed down all other pgs from other pools ? again, i just deleted a pool that has almost no traffic, because its pgs were locked and affected pgs on another pool, and as soon as that happened, the whole cluster came back to normal (and of course, HEALTH_OK and no slow transaction whatsoever) please, someone help me understand the gap where i miss something, since this , as long as my ceph knowledge is concerned, makes no sense. PS: i have found someone that , looks like went through the same here: https://forum.proxmox.com/threads/ceph-osd-failure-causing-proxmox-node-to-crash.20781/ but i still dont understand what happened. hoping to get the help from the community. -- Alejandrito ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Some facts: The OSDs use a lot of gossip protocols to distribute information. The OSDs limit how many client messages they let in to the system at a time. The OSDs do not distinguish between client ops for different pools (the blocking happens before they have any idea what the target is). So, yes: if you have a non-functional pool and clients keep trying to access it, those requests can fill up the OSD memory queues and block access to other pools as it cascades across the system. On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario wrote: > Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). > This weekend we'be experienced a huge outage from our customers vms > (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's > started to slow request/block PG's on pool PRIVATE ( replica size 1 ) > basically all PG's blocked where just one OSD in the acting set, but > all customers on the other pool got their vms almost freezed. > > while trying to do basic troubleshooting like doing noout and then > bringing down the OSD that slowed/blocked the most, inmediatelly > another OSD slowed/locked iops on pgs from the same PRIVATE pool, so > we rolled back that change and started to move data around with the > same logic (reweighting down those OSD) with exactly the same result. > > So, me made a decition, we decided to delete the pool where all PGS > where slowed/locked allways despite the osd. > > Not even 10 secconds passes after the pool deletion, where not only > there were no more degraded PGs, bit also ALL slow iops dissapeared > for ever, and performance from hundreds of vms came to normal > immediately. > > I must say that i was kinda scared to see that happen, bascally > because there was only ONE POOL's PGS always slowed, but performance > hit the another pool, so ... did not the PGS that exists on one pool > are not shared by the other ? > If my assertion is true, why OSD's locking iops from one pool's pg > slowed down all other pgs from other pools ? > > again, i just deleted a pool that has almost no traffic, because its > pgs were locked and affected pgs on another pool, and as soon as that > happened, the whole cluster came back to normal (and of course, > HEALTH_OK and no slow transaction whatsoever) > > please, someone help me understand the gap where i miss something, > since this , as long as my ceph knowledge is concerned, makes no > sense. > > PS: i have found someone that , looks like went through the same here: > > https://forum.proxmox.com/threads/ceph-osd-failure-causing-proxmox-node-to-crash.20781/ > but i still dont understand what happened. > > hoping to get the help from the community. > > -- > Alejandrito. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Gregory, thanks for the response, what you've said is by far, the most enlightneen thing i know about ceph in a long time. What brings even greater doubt, which is, this "non-functional" pool, was only 1.5GB large, vs 50-150GB on the other effected pools, the tiny pool was still being used, and just because that pool was blovking requests, the whole cluster was unresponsive. So , what do you mean by "non-functional" pool ? how a pool can become non-functional ? and what asures me that tomorrow (just becaue i deleted the 1.5GB pool to fix the whole problem) another pool doesnt becomes non-functional ? Ceph Bug ? Another Bug ? Something than can be avoided ? On Tue, Mar 7, 2017 at 2:11 PM, Gregory Farnum wrote: > Some facts: > The OSDs use a lot of gossip protocols to distribute information. > The OSDs limit how many client messages they let in to the system at a > time. > The OSDs do not distinguish between client ops for different pools (the > blocking happens before they have any idea what the target is). > > So, yes: if you have a non-functional pool and clients keep trying to > access it, those requests can fill up the OSD memory queues and block > access to other pools as it cascades across the system. > > On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario > wrote: > >> Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). >> This weekend we'be experienced a huge outage from our customers vms >> (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's >> started to slow request/block PG's on pool PRIVATE ( replica size 1 ) >> basically all PG's blocked where just one OSD in the acting set, but >> all customers on the other pool got their vms almost freezed. >> >> while trying to do basic troubleshooting like doing noout and then >> bringing down the OSD that slowed/blocked the most, inmediatelly >> another OSD slowed/locked iops on pgs from the same PRIVATE pool, so >> we rolled back that change and started to move data around with the >> same logic (reweighting down those OSD) with exactly the same result. >> >> So, me made a decition, we decided to delete the pool where all PGS >> where slowed/locked allways despite the osd. >> >> Not even 10 secconds passes after the pool deletion, where not only >> there were no more degraded PGs, bit also ALL slow iops dissapeared >> for ever, and performance from hundreds of vms came to normal >> immediately. >> >> I must say that i was kinda scared to see that happen, bascally >> because there was only ONE POOL's PGS always slowed, but performance >> hit the another pool, so ... did not the PGS that exists on one pool >> are not shared by the other ? >> If my assertion is true, why OSD's locking iops from one pool's pg >> slowed down all other pgs from other pools ? >> >> again, i just deleted a pool that has almost no traffic, because its >> pgs were locked and affected pgs on another pool, and as soon as that >> happened, the whole cluster came back to normal (and of course, >> HEALTH_OK and no slow transaction whatsoever) >> >> please, someone help me understand the gap where i miss something, >> since this , as long as my ceph knowledge is concerned, makes no >> sense. >> >> PS: i have found someone that , looks like went through the same here: >> https://forum.proxmox.com/threads/ceph-osd-failure- >> causing-proxmox-node-to-crash.20781/ >> but i still dont understand what happened. >> >> hoping to get the help from the community. >> >> -- >> Alejandrito. >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > -- *Alejandro Comisario* *CTO | NUBELIU* E-mail: alejandro@nubeliu.comCell: +54 9 11 3770 1857 _ www.nubeliu.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Any thoughts ? On Tue, Mar 7, 2017 at 3:17 PM, Alejandro Comisario wrote: > Gregory, thanks for the response, what you've said is by far, the most > enlightneen thing i know about ceph in a long time. > > What brings even greater doubt, which is, this "non-functional" pool, was > only 1.5GB large, vs 50-150GB on the other effected pools, the tiny pool > was still being used, and just because that pool was blovking requests, the > whole cluster was unresponsive. > > So , what do you mean by "non-functional" pool ? how a pool can become > non-functional ? and what asures me that tomorrow (just becaue i deleted > the 1.5GB pool to fix the whole problem) another pool doesnt becomes > non-functional ? > > Ceph Bug ? > Another Bug ? > Something than can be avoided ? > > > On Tue, Mar 7, 2017 at 2:11 PM, Gregory Farnum wrote: > >> Some facts: >> The OSDs use a lot of gossip protocols to distribute information. >> The OSDs limit how many client messages they let in to the system at a >> time. >> The OSDs do not distinguish between client ops for different pools (the >> blocking happens before they have any idea what the target is). >> >> So, yes: if you have a non-functional pool and clients keep trying to >> access it, those requests can fill up the OSD memory queues and block >> access to other pools as it cascades across the system. >> >> On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario >> wrote: >> >>> Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). >>> This weekend we'be experienced a huge outage from our customers vms >>> (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's >>> started to slow request/block PG's on pool PRIVATE ( replica size 1 ) >>> basically all PG's blocked where just one OSD in the acting set, but >>> all customers on the other pool got their vms almost freezed. >>> >>> while trying to do basic troubleshooting like doing noout and then >>> bringing down the OSD that slowed/blocked the most, inmediatelly >>> another OSD slowed/locked iops on pgs from the same PRIVATE pool, so >>> we rolled back that change and started to move data around with the >>> same logic (reweighting down those OSD) with exactly the same result. >>> >>> So, me made a decition, we decided to delete the pool where all PGS >>> where slowed/locked allways despite the osd. >>> >>> Not even 10 secconds passes after the pool deletion, where not only >>> there were no more degraded PGs, bit also ALL slow iops dissapeared >>> for ever, and performance from hundreds of vms came to normal >>> immediately. >>> >>> I must say that i was kinda scared to see that happen, bascally >>> because there was only ONE POOL's PGS always slowed, but performance >>> hit the another pool, so ... did not the PGS that exists on one pool >>> are not shared by the other ? >>> If my assertion is true, why OSD's locking iops from one pool's pg >>> slowed down all other pgs from other pools ? >>> >>> again, i just deleted a pool that has almost no traffic, because its >>> pgs were locked and affected pgs on another pool, and as soon as that >>> happened, the whole cluster came back to normal (and of course, >>> HEALTH_OK and no slow transaction whatsoever) >>> >>> please, someone help me understand the gap where i miss something, >>> since this , as long as my ceph knowledge is concerned, makes no >>> sense. >>> >>> PS: i have found someone that , looks like went through the same here: >>> https://forum.proxmox.com/threads/ceph-osd-failure-causing- >>> proxmox-node-to-crash.20781/ >>> but i still dont understand what happened. >>> >>> hoping to get the help from the community. >>> >>> -- >>> Alejandrito. >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > > > -- > *Alejandro Comisario* > *CTO | NUBELIU* > E-mail: alejandro@nubeliu.comCell: +54 9 11 3770 1857 > _ > www.nubeliu.com > -- *Alejandro Comisario* *CTO | NUBELIU* E-mail: alejandro@nubeliu.comCell: +54 9 11 3770 1857 _ www.nubeliu.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
On Tue, Mar 7, 2017 at 10:18 AM Alejandro Comisario wrote: > Gregory, thanks for the response, what you've said is by far, the most > enlightneen thing i know about ceph in a long time. > > What brings even greater doubt, which is, this "non-functional" pool, was > only 1.5GB large, vs 50-150GB on the other effected pools, the tiny pool > was still being used, and just because that pool was blovking requests, the > whole cluster was unresponsive. > > So , what do you mean by "non-functional" pool ? how a pool can become > non-functional ? and what asures me that tomorrow (just becaue i deleted > the 1.5GB pool to fix the whole problem) another pool doesnt becomes > non-functional ? > Well, you said there were a bunch of slow requests. That can happen any number of ways, if you're overloading the OSDs or something. When there are slow requests, those ops take up OSD memory and throttle, and so they don't let in new messages until the old ones are serviced. This can cascade across a cluster -- because everything is interconnected, clients and OSDs end up with all their requests targeted at the slow OSDs which aren't letting in new IO quickly enough. It's one of the weaknesses of the standard deployment patterns, but it usually doesn't come up unless something else has gone pretty wrong first. As for what actually went wrong here, you haven't provided near enough information and probably can't now that the pool has been deleted. *shrug* -Greg > Ceph Bug ? > Another Bug ? > Something than can be avoided ? > > > On Tue, Mar 7, 2017 at 2:11 PM, Gregory Farnum wrote: > > Some facts: > The OSDs use a lot of gossip protocols to distribute information. > The OSDs limit how many client messages they let in to the system at a > time. > The OSDs do not distinguish between client ops for different pools (the > blocking happens before they have any idea what the target is). > > So, yes: if you have a non-functional pool and clients keep trying to > access it, those requests can fill up the OSD memory queues and block > access to other pools as it cascades across the system. > > On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario > wrote: > > Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). > This weekend we'be experienced a huge outage from our customers vms > (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's > started to slow request/block PG's on pool PRIVATE ( replica size 1 ) > basically all PG's blocked where just one OSD in the acting set, but > all customers on the other pool got their vms almost freezed. > > while trying to do basic troubleshooting like doing noout and then > bringing down the OSD that slowed/blocked the most, inmediatelly > another OSD slowed/locked iops on pgs from the same PRIVATE pool, so > we rolled back that change and started to move data around with the > same logic (reweighting down those OSD) with exactly the same result. > > So, me made a decition, we decided to delete the pool where all PGS > where slowed/locked allways despite the osd. > > Not even 10 secconds passes after the pool deletion, where not only > there were no more degraded PGs, bit also ALL slow iops dissapeared > for ever, and performance from hundreds of vms came to normal > immediately. > > I must say that i was kinda scared to see that happen, bascally > because there was only ONE POOL's PGS always slowed, but performance > hit the another pool, so ... did not the PGS that exists on one pool > are not shared by the other ? > If my assertion is true, why OSD's locking iops from one pool's pg > slowed down all other pgs from other pools ? > > again, i just deleted a pool that has almost no traffic, because its > pgs were locked and affected pgs on another pool, and as soon as that > happened, the whole cluster came back to normal (and of course, > HEALTH_OK and no slow transaction whatsoever) > > please, someone help me understand the gap where i miss something, > since this , as long as my ceph knowledge is concerned, makes no > sense. > > PS: i have found someone that , looks like went through the same here: > > https://forum.proxmox.com/threads/ceph-osd-failure-causing-proxmox-node-to-crash.20781/ > but i still dont understand what happened. > > hoping to get the help from the community. > > -- > Alejandrito. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > *Alejandro Comisario* > *CTO | NUBELIU* > E-mail: alejandro@nubeliu.comCell: +54 9 11 3770 1857 > _ > www.nubeliu.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Greg, thanks for the reply. True that i cant provide enough information to know what happened since the pool is gone. But based on your experience, can i please take some of your time, and give me the TOP 5 fo what could happen / would be the reason to happen what hapened to that pool (or any pool) that makes Ceph (maybe hapened specifically in Hammer ) to behave like that ? Information that i think will be of value, is that the cluster was 5 nodes large, running "0.94.6-1trusty" i added two nodes running the latest "0.94.9-1trusty" and replication into those new disks never ended, since i saw WEIRD errors on the new OSDs, so i thought that packages needed to be the same, so i "apt-get upgraded" the 5 old nodes without restrting nothing, so rebalancing started to happen without errors (WEIRD). after these two nodes reached 100% of the disks weight, the cluster worked perfectly for about two weeks, till this happened. After the resolution from my first email, everything has been working perfect. thanks for the responses. On Fri, Mar 10, 2017 at 4:23 PM, Gregory Farnum wrote: > > > On Tue, Mar 7, 2017 at 10:18 AM Alejandro Comisario > wrote: > >> Gregory, thanks for the response, what you've said is by far, the most >> enlightneen thing i know about ceph in a long time. >> >> What brings even greater doubt, which is, this "non-functional" pool, was >> only 1.5GB large, vs 50-150GB on the other effected pools, the tiny pool >> was still being used, and just because that pool was blovking requests, the >> whole cluster was unresponsive. >> >> So , what do you mean by "non-functional" pool ? how a pool can become >> non-functional ? and what asures me that tomorrow (just becaue i deleted >> the 1.5GB pool to fix the whole problem) another pool doesnt becomes >> non-functional ? >> > > Well, you said there were a bunch of slow requests. That can happen any > number of ways, if you're overloading the OSDs or something. > When there are slow requests, those ops take up OSD memory and throttle, > and so they don't let in new messages until the old ones are serviced. This > can cascade across a cluster -- because everything is interconnected, > clients and OSDs end up with all their requests targeted at the slow OSDs > which aren't letting in new IO quickly enough. It's one of the weaknesses > of the standard deployment patterns, but it usually doesn't come up unless > something else has gone pretty wrong first. > As for what actually went wrong here, you haven't provided near enough > information and probably can't now that the pool has been deleted. *shrug* > -Greg > > > > >> Ceph Bug ? >> Another Bug ? >> Something than can be avoided ? >> >> >> On Tue, Mar 7, 2017 at 2:11 PM, Gregory Farnum >> wrote: >> >> Some facts: >> The OSDs use a lot of gossip protocols to distribute information. >> The OSDs limit how many client messages they let in to the system at a >> time. >> The OSDs do not distinguish between client ops for different pools (the >> blocking happens before they have any idea what the target is). >> >> So, yes: if you have a non-functional pool and clients keep trying to >> access it, those requests can fill up the OSD memory queues and block >> access to other pools as it cascades across the system. >> >> On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario >> wrote: >> >> Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). >> This weekend we'be experienced a huge outage from our customers vms >> (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's >> started to slow request/block PG's on pool PRIVATE ( replica size 1 ) >> basically all PG's blocked where just one OSD in the acting set, but >> all customers on the other pool got their vms almost freezed. >> >> while trying to do basic troubleshooting like doing noout and then >> bringing down the OSD that slowed/blocked the most, inmediatelly >> another OSD slowed/locked iops on pgs from the same PRIVATE pool, so >> we rolled back that change and started to move data around with the >> same logic (reweighting down those OSD) with exactly the same result. >> >> So, me made a decition, we decided to delete the pool where all PGS >> where slowed/locked allways despite the osd. >> >> Not even 10 secconds passes after the pool deletion, where not only >> there were no more degraded PGs, bit also ALL slow iops dissapeared >> for ever, and performance from hundreds of vms came to normal >> immediately. >> >> I must say that i was kinda scared to see that happen, bascally >> because there was only ONE POOL's PGS always slowed, but performance >> hit the another pool, so ... did not the PGS that exists on one pool >> are not shared by the other ? >> If my assertion is true, why OSD's locking iops from one pool's pg >> slowed down all other pgs from other pools ? >> >> again, i just deleted a pool that has almost no traffic, because its >> pgs were locked and affected pgs on another pool, and as soon as that >> happened, the
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
any thoughts ? On Tue, Mar 14, 2017 at 10:22 PM, Alejandro Comisario wrote: > Greg, thanks for the reply. > True that i cant provide enough information to know what happened since > the pool is gone. > > But based on your experience, can i please take some of your time, and > give me the TOP 5 fo what could happen / would be the reason to happen what > hapened to that pool (or any pool) that makes Ceph (maybe hapened > specifically in Hammer ) to behave like that ? > > Information that i think will be of value, is that the cluster was 5 nodes > large, running "0.94.6-1trusty" i added two nodes running the latest > "0.94.9-1trusty" and replication into those new disks never ended, since i > saw WEIRD errors on the new OSDs, so i thought that packages needed to be > the same, so i "apt-get upgraded" the 5 old nodes without restrting > nothing, so rebalancing started to happen without errors (WEIRD). > > after these two nodes reached 100% of the disks weight, the cluster worked > perfectly for about two weeks, till this happened. > After the resolution from my first email, everything has been working > perfect. > > thanks for the responses. > > > On Fri, Mar 10, 2017 at 4:23 PM, Gregory Farnum > wrote: > >> >> >> On Tue, Mar 7, 2017 at 10:18 AM Alejandro Comisario < >> alejan...@nubeliu.com> wrote: >> >>> Gregory, thanks for the response, what you've said is by far, the most >>> enlightneen thing i know about ceph in a long time. >>> >>> What brings even greater doubt, which is, this "non-functional" pool, >>> was only 1.5GB large, vs 50-150GB on the other effected pools, the tiny >>> pool was still being used, and just because that pool was blovking >>> requests, the whole cluster was unresponsive. >>> >>> So , what do you mean by "non-functional" pool ? how a pool can become >>> non-functional ? and what asures me that tomorrow (just becaue i deleted >>> the 1.5GB pool to fix the whole problem) another pool doesnt becomes >>> non-functional ? >>> >> >> Well, you said there were a bunch of slow requests. That can happen any >> number of ways, if you're overloading the OSDs or something. >> When there are slow requests, those ops take up OSD memory and throttle, >> and so they don't let in new messages until the old ones are serviced. This >> can cascade across a cluster -- because everything is interconnected, >> clients and OSDs end up with all their requests targeted at the slow OSDs >> which aren't letting in new IO quickly enough. It's one of the weaknesses >> of the standard deployment patterns, but it usually doesn't come up unless >> something else has gone pretty wrong first. >> As for what actually went wrong here, you haven't provided near enough >> information and probably can't now that the pool has been deleted. *shrug* >> -Greg >> >> >> >> >>> Ceph Bug ? >>> Another Bug ? >>> Something than can be avoided ? >>> >>> >>> On Tue, Mar 7, 2017 at 2:11 PM, Gregory Farnum >>> wrote: >>> >>> Some facts: >>> The OSDs use a lot of gossip protocols to distribute information. >>> The OSDs limit how many client messages they let in to the system at a >>> time. >>> The OSDs do not distinguish between client ops for different pools (the >>> blocking happens before they have any idea what the target is). >>> >>> So, yes: if you have a non-functional pool and clients keep trying to >>> access it, those requests can fill up the OSD memory queues and block >>> access to other pools as it cascades across the system. >>> >>> On Sun, Mar 5, 2017 at 6:22 PM Alejandro Comisario < >>> alejan...@nubeliu.com> wrote: >>> >>> Hi, we have a 7 nodes ubuntu ceph hammer pool (78 OSD to be exact). >>> This weekend we'be experienced a huge outage from our customers vms >>> (located on pool CUSTOMERS, replica size 3 ) when lots of OSD's >>> started to slow request/block PG's on pool PRIVATE ( replica size 1 ) >>> basically all PG's blocked where just one OSD in the acting set, but >>> all customers on the other pool got their vms almost freezed. >>> >>> while trying to do basic troubleshooting like doing noout and then >>> bringing down the OSD that slowed/blocked the most, inmediatelly >>> another OSD slowed/locked iops on pgs from the same PRIVATE pool, so >>> we rolled back that change and started to move data around with the >>> same logic (reweighting down those OSD) with exactly the same result. >>> >>> So, me made a decition, we decided to delete the pool where all PGS >>> where slowed/locked allways despite the osd. >>> >>> Not even 10 secconds passes after the pool deletion, where not only >>> there were no more degraded PGs, bit also ALL slow iops dissapeared >>> for ever, and performance from hundreds of vms came to normal >>> immediately. >>> >>> I must say that i was kinda scared to see that happen, bascally >>> because there was only ONE POOL's PGS always slowed, but performance >>> hit the another pool, so ... did not the PGS that exists on one pool >>> are not shared by the other ? >>> If my assertion is true
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
I think Greg (who appears to be a ceph committer) basically said he was interested in looking at it, if only you had the pool that failed this way. Why not try to reproduce it, and make a log of your procedure so he can reproduce it too? What caused the slow requests... copy on write from snapshots? A bad disk? exclusive-lock with 2 clients writing at the same time maybe? I'd be interested in a solution too... like why can't idle disks (non-full disk queue) mean that the osd op or whatever queue can still fill with requests not related to the blocked pg/objects? I would love for ceph to handle this better. I suspect some issues I have are related to this (slow requests on one VM can freeze others [likely blame the osd], even requiring kill -9 [likely blame client librbd]). On 03/22/17 16:18, Alejandro Comisario wrote: > any thoughts ? > > On Tue, Mar 14, 2017 at 10:22 PM, Alejandro Comisario > mailto:alejan...@nubeliu.com>> wrote: > > Greg, thanks for the reply. > True that i cant provide enough information to know what happened > since the pool is gone. > > But based on your experience, can i please take some of your time, > and give me the TOP 5 fo what could happen / would be the reason > to happen what hapened to that pool (or any pool) that makes Ceph > (maybe hapened specifically in Hammer ) to behave like that ? > > Information that i think will be of value, is that the cluster was > 5 nodes large, running "0.94.6-1trusty" i added two nodes running > the latest "0.94.9-1trusty" and replication into those new disks > never ended, since i saw WEIRD errors on the new OSDs, so i > thought that packages needed to be the same, so i "apt-get > upgraded" the 5 old nodes without restrting nothing, so > rebalancing started to happen without errors (WEIRD). > > after these two nodes reached 100% of the disks weight, the > cluster worked perfectly for about two weeks, till this happened. > After the resolution from my first email, everything has been > working perfect. > > thanks for the responses. > > > On Fri, Mar 10, 2017 at 4:23 PM, Gregory Farnum > mailto:gfar...@redhat.com>> wrote: > > > > On Tue, Mar 7, 2017 at 10:18 AM Alejandro Comisario > mailto:alejan...@nubeliu.com>> wrote: > > Gregory, thanks for the response, what you've said is by > far, the most enlightneen thing i know about ceph in a > long time. > > What brings even greater doubt, which is, this > "non-functional" pool, was only 1.5GB large, vs 50-150GB > on the other effected pools, the tiny pool was still being > used, and just because that pool was blovking requests, > the whole cluster was unresponsive. > > So , what do you mean by "non-functional" pool ? how a > pool can become non-functional ? and what asures me that > tomorrow (just becaue i deleted the 1.5GB pool to fix the > whole problem) another pool doesnt becomes non-functional ? > > > Well, you said there were a bunch of slow requests. That can > happen any number of ways, if you're overloading the OSDs or > something. > When there are slow requests, those ops take up OSD memory and > throttle, and so they don't let in new messages until the old > ones are serviced. This can cascade across a cluster -- > because everything is interconnected, clients and OSDs end up > with all their requests targeted at the slow OSDs which aren't > letting in new IO quickly enough. It's one of the weaknesses > of the standard deployment patterns, but it usually doesn't > come up unless something else has gone pretty wrong first. > As for what actually went wrong here, you haven't provided > near enough information and probably can't now that the pool > has been deleted. *shrug* > -Greg > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] can a OSD affect performance from pool X when blocking/slow requests PGs from pool Y ?
Deffinitelly in our case OSD were not the guilty ones, since all osd that where blocking requests allways from the same pool, worked flawlesly (and still do) after we deleted the pool where we always saw the blocked PG's. Since the pool was accesed by just one client, and had almost no ops to it, i really dont know how to reproduce the issue, but surely scares me to happen ever again, and most, taking into consideration that blocked iops on a OSD could cascade through all the cluster and block all other pools. It was technically hard to explain to the management that one 1.5GB pool locked almost 250 vms from different TB size pools, and most of it, not having a root cause (meaning, why only that pool generated blocked iops) hope to hear some more technical insights or someone else that went through the same. best. On Thu, Mar 23, 2017 at 5:47 AM, Peter Maloney < peter.malo...@brockmann-consult.de> wrote: > I think Greg (who appears to be a ceph committer) basically said he was > interested in looking at it, if only you had the pool that failed this way. > > Why not try to reproduce it, and make a log of your procedure so he can > reproduce it too? What caused the slow requests... copy on write from > snapshots? A bad disk? exclusive-lock with 2 clients writing at the same > time maybe? > > I'd be interested in a solution too... like why can't idle disks (non-full > disk queue) mean that the osd op or whatever queue can still fill with > requests not related to the blocked pg/objects? I would love for ceph to > handle this better. I suspect some issues I have are related to this (slow > requests on one VM can freeze others [likely blame the osd], even requiring > kill -9 [likely blame client librbd]). > > On 03/22/17 16:18, Alejandro Comisario wrote: > > any thoughts ? > > On Tue, Mar 14, 2017 at 10:22 PM, Alejandro Comisario < > alejan...@nubeliu.com> wrote: > >> Greg, thanks for the reply. >> True that i cant provide enough information to know what happened since >> the pool is gone. >> >> But based on your experience, can i please take some of your time, and >> give me the TOP 5 fo what could happen / would be the reason to happen what >> hapened to that pool (or any pool) that makes Ceph (maybe hapened >> specifically in Hammer ) to behave like that ? >> >> Information that i think will be of value, is that the cluster was 5 >> nodes large, running "0.94.6-1trusty" i added two nodes running the latest >> "0.94.9-1trusty" and replication into those new disks never ended, since i >> saw WEIRD errors on the new OSDs, so i thought that packages needed to be >> the same, so i "apt-get upgraded" the 5 old nodes without restrting >> nothing, so rebalancing started to happen without errors (WEIRD). >> >> after these two nodes reached 100% of the disks weight, the cluster >> worked perfectly for about two weeks, till this happened. >> After the resolution from my first email, everything has been working >> perfect. >> >> thanks for the responses. >> >> >> On Fri, Mar 10, 2017 at 4:23 PM, Gregory Farnum >> wrote: >> >>> >>> >>> On Tue, Mar 7, 2017 at 10:18 AM Alejandro Comisario < >>> alejan...@nubeliu.com> wrote: >>> Gregory, thanks for the response, what you've said is by far, the most enlightneen thing i know about ceph in a long time. What brings even greater doubt, which is, this "non-functional" pool, was only 1.5GB large, vs 50-150GB on the other effected pools, the tiny pool was still being used, and just because that pool was blovking requests, the whole cluster was unresponsive. So , what do you mean by "non-functional" pool ? how a pool can become non-functional ? and what asures me that tomorrow (just becaue i deleted the 1.5GB pool to fix the whole problem) another pool doesnt becomes non-functional ? >>> >>> Well, you said there were a bunch of slow requests. That can happen any >>> number of ways, if you're overloading the OSDs or something. >>> When there are slow requests, those ops take up OSD memory and throttle, >>> and so they don't let in new messages until the old ones are serviced. This >>> can cascade across a cluster -- because everything is interconnected, >>> clients and OSDs end up with all their requests targeted at the slow OSDs >>> which aren't letting in new IO quickly enough. It's one of the weaknesses >>> of the standard deployment patterns, but it usually doesn't come up unless >>> something else has gone pretty wrong first. >>> As for what actually went wrong here, you haven't provided near enough >>> information and probably can't now that the pool has been deleted. *shrug* >>> -Greg >>> >> > -- *Alejandro Comisario* *CTO | NUBELIU* E-mail: alejandro@nubeliu.comCell: +54 9 11 3770 1857 _ www.nubeliu.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
