Re: rbd map command hangs for 15 minutes during system start up
So far basic things are working fine, and my hang test is at 78 passes and still going good. I'll let you know if any problems crop up with it. On Mon, Dec 31, 2012 at 10:22 AM, Alex Elder wrote: > On 12/26/2012 03:36 PM, Alex Elder wrote: >> On 12/26/2012 11:45 AM, Nick Bartos wrote: >>> Here's a log with a hang on the updated branch: >>> >>> https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log >> >> OK, new naming scheme. Please try: wip-nick-1 > > Now that we've got this resolved, I've created an updated > "stable" branch with ceph-related bug fixes, based on the > latest 3.5 stable branch, 3.5.7. It contains a bunch of > other bug fixes that what you had been working with did > not have. > > I'm starting my own testing with this branch now. But it > would be great if you'd give it a try as well, since I > know you're a "real" user of this code base. > > It's available as branch "linux-3.5.7-ceph" on the > ceph-client git repository. Thanks a lot. > > -Alex > >> >> I added another simple fix, but then collapsed three commits >> into one, and added one more (somewhat unrelated). >> >> I've done simple testing with this and will subject it to >> more rigorous testing shortly. I wanted to make it available >> to you quickly though. >> >> -Alex >> >>> >>> On Thu, Dec 20, 2012 at 1:59 PM, Alex Elder wrote: On 12/20/2012 11:48 AM, Nick Bartos wrote: > Unfortunately, we still have a hang: > > https://gist.github.com/4347052/download The saga continues, and each time we get a little more information. Please try branch: "wip-nick-newerest" Thank you. -Alex > On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: >> On 12/19/2012 03:25 PM, Alex Elder wrote: >>> On 12/18/2012 12:05 PM, Nick Bartos wrote: I've added the output of "ps -ef" in addition to triggering a trace when a hang is detected. Not much is generally running at that point, but you can have a look: https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt >>> >>> This helped a lot. I updated the bug with a little more info. >>> >>> http://tracker.newdream.net/issues/3519 >>> >>> I also think I have now found something that could explain what you >>> are seeing, and am developing a fix. I'll provide you an update >>> as soon as I have tested what I come up with, almost certainly >>> this afternoon. >> >> Nick, I have a new branch for you to try with a new fix in place. >> As you might have predicted, it's named "wip-nick-newest". >> >> Please give it a try to see if it resolved the hang you've >> been seeing and let me know how it goes. If it continues >> to hang, please provide the logs as you have before, it's >> been very helpful. >> >> Thanks a lot. >> >> -Alex >>> >>> -Alex >>> Is it possible that there is some sort of deadlock going on? We are doing the rbd maps (and subsequent filesystem mounts) on the same systems which are running the ceph-osd and ceph-mon processes. To get around the 'sync' deadlock problem, we are using a patch from Sage which ignores system wide sync's on filesystems mounted with the 'mand' option (and we mount the underlying osd filesystems with 'mand'). However I am wondering if there is potential for other types of deadlocks in this environment. Also, we recently saw an rbd hang in a much older version, running kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. It's possible that this issue was around for some time, just the recent patches made it happen more often (and thus more reproducible) for us. On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: > On 12/17/2012 11:12 AM, Nick Bartos wrote: >> Here's a log with the rbd debugging enabled: >> >> https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log >> >> On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder >> wrote: >>> On 12/14/2012 10:53 AM, Nick Bartos wrote: Yes I was only enabling debugging for libceph. I'm adding debugging for rbd as well. I'll do a repro later today when a test cluster opens up. >>> >>> Excellent, thank you. -Alex > > I looked through these debugging messages. Looking only at the > rbd debugging, what I see seems to indicate that rbd is idle at > the
Re: rbd map command hangs for 15 minutes during system start up
On 12/26/2012 03:36 PM, Alex Elder wrote: > On 12/26/2012 11:45 AM, Nick Bartos wrote: >> Here's a log with a hang on the updated branch: >> >> https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log > > OK, new naming scheme. Please try: wip-nick-1 Now that we've got this resolved, I've created an updated "stable" branch with ceph-related bug fixes, based on the latest 3.5 stable branch, 3.5.7. It contains a bunch of other bug fixes that what you had been working with did not have. I'm starting my own testing with this branch now. But it would be great if you'd give it a try as well, since I know you're a "real" user of this code base. It's available as branch "linux-3.5.7-ceph" on the ceph-client git repository. Thanks a lot. -Alex > > I added another simple fix, but then collapsed three commits > into one, and added one more (somewhat unrelated). > > I've done simple testing with this and will subject it to > more rigorous testing shortly. I wanted to make it available > to you quickly though. > > -Alex > >> >> On Thu, Dec 20, 2012 at 1:59 PM, Alex Elder wrote: >>> On 12/20/2012 11:48 AM, Nick Bartos wrote: Unfortunately, we still have a hang: https://gist.github.com/4347052/download >>> >>> The saga continues, and each time we get a little more >>> information. Please try branch: "wip-nick-newerest" >>> >>> Thank you. >>> >>> -Alex >>> >>> On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: > On 12/19/2012 03:25 PM, Alex Elder wrote: >> On 12/18/2012 12:05 PM, Nick Bartos wrote: >>> I've added the output of "ps -ef" in addition to triggering a trace >>> when a hang is detected. Not much is generally running at that point, >>> but you can have a look: >>> >>> https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt >> >> This helped a lot. I updated the bug with a little more info. >> >> http://tracker.newdream.net/issues/3519 >> >> I also think I have now found something that could explain what you >> are seeing, and am developing a fix. I'll provide you an update >> as soon as I have tested what I come up with, almost certainly >> this afternoon. > > Nick, I have a new branch for you to try with a new fix in place. > As you might have predicted, it's named "wip-nick-newest". > > Please give it a try to see if it resolved the hang you've > been seeing and let me know how it goes. If it continues > to hang, please provide the logs as you have before, it's > been very helpful. > > Thanks a lot. > > -Alex >> >> -Alex >> >>> Is it possible that there is some sort of deadlock going on? We are >>> doing the rbd maps (and subsequent filesystem mounts) on the same >>> systems which are running the ceph-osd and ceph-mon processes. To get >>> around the 'sync' deadlock problem, we are using a patch from Sage >>> which ignores system wide sync's on filesystems mounted with the >>> 'mand' option (and we mount the underlying osd filesystems with >>> 'mand'). However I am wondering if there is potential for other types >>> of deadlocks in this environment. >>> >>> Also, we recently saw an rbd hang in a much older version, running >>> kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. >>> It's possible that this issue was around for some time, just the >>> recent patches made it happen more often (and thus more reproducible) >>> for us. >>> >>> >>> On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: On 12/17/2012 11:12 AM, Nick Bartos wrote: > Here's a log with the rbd debugging enabled: > > https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log > > On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder > wrote: >> On 12/14/2012 10:53 AM, Nick Bartos wrote: >>> Yes I was only enabling debugging for libceph. I'm adding debugging >>> for rbd as well. I'll do a repro later today when a test cluster >>> opens up. >> >> Excellent, thank you. -Alex I looked through these debugging messages. Looking only at the rbd debugging, what I see seems to indicate that rbd is idle at the point the "hang" seems to start. This suggests that the hang is not due to rbd itself, but rather whatever it is that might be responsible for using the rbd image once it has been mapped. Is that possible? I don't know what process you have that is mapping the rbd image, and what is sup
Re: rbd map command hangs for 15 minutes during system start up
On 12/27/2012 12:43 PM, Sage Weil wrote: > On Thu, 27 Dec 2012, Nick Bartos wrote: >> I have some exciting news. After 215 test runs, no hung processes >> were detected. I think we may actually have it this time. Thanks for >> all your hard work! This is great news Nick, and I really appreciate your help testing so we could get rid of this ugly thing. > Sweet! I think it was the new branch naming scheme that did it. Obviously. I'm going to blow away all the wip-nick-new* branches. -Alex > > sage > >> >> -Nick >> >> On Wed, Dec 26, 2012 at 1:36 PM, Alex Elder wrote: >>> On 12/26/2012 11:45 AM, Nick Bartos wrote: Here's a log with a hang on the updated branch: https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log >>> >>> OK, new naming scheme. Please try: wip-nick-1 >>> >>> I added another simple fix, but then collapsed three commits >>> into one, and added one more (somewhat unrelated). >>> >>> I've done simple testing with this and will subject it to >>> more rigorous testing shortly. I wanted to make it available >>> to you quickly though. >>> >>> -Alex >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On Thu, 27 Dec 2012, Nick Bartos wrote: > I have some exciting news. After 215 test runs, no hung processes > were detected. I think we may actually have it this time. Thanks for > all your hard work! Sweet! I think it was the new branch naming scheme that did it. sage > > -Nick > > On Wed, Dec 26, 2012 at 1:36 PM, Alex Elder wrote: > > On 12/26/2012 11:45 AM, Nick Bartos wrote: > >> Here's a log with a hang on the updated branch: > >> > >> https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log > > > > OK, new naming scheme. Please try: wip-nick-1 > > > > I added another simple fix, but then collapsed three commits > > into one, and added one more (somewhat unrelated). > > > > I've done simple testing with this and will subject it to > > more rigorous testing shortly. I wanted to make it available > > to you quickly though. > > > > -Alex > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
I have some exciting news. After 215 test runs, no hung processes were detected. I think we may actually have it this time. Thanks for all your hard work! -Nick On Wed, Dec 26, 2012 at 1:36 PM, Alex Elder wrote: > On 12/26/2012 11:45 AM, Nick Bartos wrote: >> Here's a log with a hang on the updated branch: >> >> https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log > > OK, new naming scheme. Please try: wip-nick-1 > > I added another simple fix, but then collapsed three commits > into one, and added one more (somewhat unrelated). > > I've done simple testing with this and will subject it to > more rigorous testing shortly. I wanted to make it available > to you quickly though. > > -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/26/2012 11:45 AM, Nick Bartos wrote: > Here's a log with a hang on the updated branch: > > https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log OK, new naming scheme. Please try: wip-nick-1 I added another simple fix, but then collapsed three commits into one, and added one more (somewhat unrelated). I've done simple testing with this and will subject it to more rigorous testing shortly. I wanted to make it available to you quickly though. -Alex > > On Thu, Dec 20, 2012 at 1:59 PM, Alex Elder wrote: >> On 12/20/2012 11:48 AM, Nick Bartos wrote: >>> Unfortunately, we still have a hang: >>> >>> https://gist.github.com/4347052/download >> >> The saga continues, and each time we get a little more >> information. Please try branch: "wip-nick-newerest" >> >> Thank you. >> >> -Alex >> >> >>> On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: On 12/19/2012 03:25 PM, Alex Elder wrote: > On 12/18/2012 12:05 PM, Nick Bartos wrote: >> I've added the output of "ps -ef" in addition to triggering a trace >> when a hang is detected. Not much is generally running at that point, >> but you can have a look: >> >> https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt > > This helped a lot. I updated the bug with a little more info. > > http://tracker.newdream.net/issues/3519 > > I also think I have now found something that could explain what you > are seeing, and am developing a fix. I'll provide you an update > as soon as I have tested what I come up with, almost certainly > this afternoon. Nick, I have a new branch for you to try with a new fix in place. As you might have predicted, it's named "wip-nick-newest". Please give it a try to see if it resolved the hang you've been seeing and let me know how it goes. If it continues to hang, please provide the logs as you have before, it's been very helpful. Thanks a lot. -Alex > > -Alex > >> Is it possible that there is some sort of deadlock going on? We are >> doing the rbd maps (and subsequent filesystem mounts) on the same >> systems which are running the ceph-osd and ceph-mon processes. To get >> around the 'sync' deadlock problem, we are using a patch from Sage >> which ignores system wide sync's on filesystems mounted with the >> 'mand' option (and we mount the underlying osd filesystems with >> 'mand'). However I am wondering if there is potential for other types >> of deadlocks in this environment. >> >> Also, we recently saw an rbd hang in a much older version, running >> kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. >> It's possible that this issue was around for some time, just the >> recent patches made it happen more often (and thus more reproducible) >> for us. >> >> >> On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: >>> On 12/17/2012 11:12 AM, Nick Bartos wrote: Here's a log with the rbd debugging enabled: https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: > On 12/14/2012 10:53 AM, Nick Bartos wrote: >> Yes I was only enabling debugging for libceph. I'm adding debugging >> for rbd as well. I'll do a repro later today when a test cluster >> opens up. > > Excellent, thank you. -Alex >>> >>> I looked through these debugging messages. Looking only at the >>> rbd debugging, what I see seems to indicate that rbd is idle at >>> the point the "hang" seems to start. This suggests that the hang >>> is not due to rbd itself, but rather whatever it is that might >>> be responsible for using the rbd image once it has been mapped. >>> >>> Is that possible? I don't know what process you have that is >>> mapping the rbd image, and what is supposed to be the next thing >>> it does. (I realize this may not make a lot of sense, given >>> a patch in rdb seems to have caused the hang to begin occurring.) >>> >>> Also note that the debugging information available (i.e., the >>> lines in the code that can output debugging information) may >>> well be incomplete. So if you don't find anything it may be >>> necessary to provide you with another update which might include >>> more debugging. >>> >>> Anyway, could you provide a little more context about what >>> is going on sort of *around* rbd when activity seems to stop? >>> >>> Thanks a lot. >>> >>>
Re: rbd map command hangs for 15 minutes during system start up
Here's a log with a hang on the updated branch: https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log On Thu, Dec 20, 2012 at 1:59 PM, Alex Elder wrote: > On 12/20/2012 11:48 AM, Nick Bartos wrote: >> Unfortunately, we still have a hang: >> >> https://gist.github.com/4347052/download > > The saga continues, and each time we get a little more > information. Please try branch: "wip-nick-newerest" > > Thank you. > > -Alex > > >> On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: >>> On 12/19/2012 03:25 PM, Alex Elder wrote: On 12/18/2012 12:05 PM, Nick Bartos wrote: > I've added the output of "ps -ef" in addition to triggering a trace > when a hang is detected. Not much is generally running at that point, > but you can have a look: > > https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt This helped a lot. I updated the bug with a little more info. http://tracker.newdream.net/issues/3519 I also think I have now found something that could explain what you are seeing, and am developing a fix. I'll provide you an update as soon as I have tested what I come up with, almost certainly this afternoon. >>> >>> Nick, I have a new branch for you to try with a new fix in place. >>> As you might have predicted, it's named "wip-nick-newest". >>> >>> Please give it a try to see if it resolved the hang you've >>> been seeing and let me know how it goes. If it continues >>> to hang, please provide the logs as you have before, it's >>> been very helpful. >>> >>> Thanks a lot. >>> >>> -Alex -Alex > Is it possible that there is some sort of deadlock going on? We are > doing the rbd maps (and subsequent filesystem mounts) on the same > systems which are running the ceph-osd and ceph-mon processes. To get > around the 'sync' deadlock problem, we are using a patch from Sage > which ignores system wide sync's on filesystems mounted with the > 'mand' option (and we mount the underlying osd filesystems with > 'mand'). However I am wondering if there is potential for other types > of deadlocks in this environment. > > Also, we recently saw an rbd hang in a much older version, running > kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. > It's possible that this issue was around for some time, just the > recent patches made it happen more often (and thus more reproducible) > for us. > > > On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: >> On 12/17/2012 11:12 AM, Nick Bartos wrote: >>> Here's a log with the rbd debugging enabled: >>> >>> https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log >>> >>> On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: On 12/14/2012 10:53 AM, Nick Bartos wrote: > Yes I was only enabling debugging for libceph. I'm adding debugging > for rbd as well. I'll do a repro later today when a test cluster > opens up. Excellent, thank you. -Alex >> >> I looked through these debugging messages. Looking only at the >> rbd debugging, what I see seems to indicate that rbd is idle at >> the point the "hang" seems to start. This suggests that the hang >> is not due to rbd itself, but rather whatever it is that might >> be responsible for using the rbd image once it has been mapped. >> >> Is that possible? I don't know what process you have that is >> mapping the rbd image, and what is supposed to be the next thing >> it does. (I realize this may not make a lot of sense, given >> a patch in rdb seems to have caused the hang to begin occurring.) >> >> Also note that the debugging information available (i.e., the >> lines in the code that can output debugging information) may >> well be incomplete. So if you don't find anything it may be >> necessary to provide you with another update which might include >> more debugging. >> >> Anyway, could you provide a little more context about what >> is going on sort of *around* rbd when activity seems to stop? >> >> Thanks a lot. >> >> -Alex >>> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/26/2012 11:45 AM, Nick Bartos wrote: > Here's a log with a hang on the updated branch: > > https://gist.github.com/raw/4381750/772476e1bae1e6366347a223f34aa6c440b92765/rdb-hang-1356543132.log I'm starting to look this over. Thanks a lot for supplying it. Sorry we still haven't nailed the problem. -Alex > > On Thu, Dec 20, 2012 at 1:59 PM, Alex Elder wrote: >> On 12/20/2012 11:48 AM, Nick Bartos wrote: >>> Unfortunately, we still have a hang: >>> >>> https://gist.github.com/4347052/download >> >> The saga continues, and each time we get a little more >> information. Please try branch: "wip-nick-newerest" >> >> Thank you. >> >> -Alex >> >> >>> On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: On 12/19/2012 03:25 PM, Alex Elder wrote: > On 12/18/2012 12:05 PM, Nick Bartos wrote: >> I've added the output of "ps -ef" in addition to triggering a trace >> when a hang is detected. Not much is generally running at that point, >> but you can have a look: >> >> https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt > > This helped a lot. I updated the bug with a little more info. > > http://tracker.newdream.net/issues/3519 > > I also think I have now found something that could explain what you > are seeing, and am developing a fix. I'll provide you an update > as soon as I have tested what I come up with, almost certainly > this afternoon. Nick, I have a new branch for you to try with a new fix in place. As you might have predicted, it's named "wip-nick-newest". Please give it a try to see if it resolved the hang you've been seeing and let me know how it goes. If it continues to hang, please provide the logs as you have before, it's been very helpful. Thanks a lot. -Alex > > -Alex > >> Is it possible that there is some sort of deadlock going on? We are >> doing the rbd maps (and subsequent filesystem mounts) on the same >> systems which are running the ceph-osd and ceph-mon processes. To get >> around the 'sync' deadlock problem, we are using a patch from Sage >> which ignores system wide sync's on filesystems mounted with the >> 'mand' option (and we mount the underlying osd filesystems with >> 'mand'). However I am wondering if there is potential for other types >> of deadlocks in this environment. >> >> Also, we recently saw an rbd hang in a much older version, running >> kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. >> It's possible that this issue was around for some time, just the >> recent patches made it happen more often (and thus more reproducible) >> for us. >> >> >> On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: >>> On 12/17/2012 11:12 AM, Nick Bartos wrote: Here's a log with the rbd debugging enabled: https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: > On 12/14/2012 10:53 AM, Nick Bartos wrote: >> Yes I was only enabling debugging for libceph. I'm adding debugging >> for rbd as well. I'll do a repro later today when a test cluster >> opens up. > > Excellent, thank you. -Alex >>> >>> I looked through these debugging messages. Looking only at the >>> rbd debugging, what I see seems to indicate that rbd is idle at >>> the point the "hang" seems to start. This suggests that the hang >>> is not due to rbd itself, but rather whatever it is that might >>> be responsible for using the rbd image once it has been mapped. >>> >>> Is that possible? I don't know what process you have that is >>> mapping the rbd image, and what is supposed to be the next thing >>> it does. (I realize this may not make a lot of sense, given >>> a patch in rdb seems to have caused the hang to begin occurring.) >>> >>> Also note that the debugging information available (i.e., the >>> lines in the code that can output debugging information) may >>> well be incomplete. So if you don't find anything it may be >>> necessary to provide you with another update which might include >>> more debugging. >>> >>> Anyway, could you provide a little more context about what >>> is going on sort of *around* rbd when activity seems to stop? >>> >>> Thanks a lot. >>> >>> -Alex > >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel
Re: rbd map command hangs for 15 minutes during system start up
On 12/20/2012 11:48 AM, Nick Bartos wrote: > Unfortunately, we still have a hang: > > https://gist.github.com/4347052/download The saga continues, and each time we get a little more information. Please try branch: "wip-nick-newerest" Thank you. -Alex > On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: >> On 12/19/2012 03:25 PM, Alex Elder wrote: >>> On 12/18/2012 12:05 PM, Nick Bartos wrote: I've added the output of "ps -ef" in addition to triggering a trace when a hang is detected. Not much is generally running at that point, but you can have a look: https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt >>> >>> This helped a lot. I updated the bug with a little more info. >>> >>> http://tracker.newdream.net/issues/3519 >>> >>> I also think I have now found something that could explain what you >>> are seeing, and am developing a fix. I'll provide you an update >>> as soon as I have tested what I come up with, almost certainly >>> this afternoon. >> >> Nick, I have a new branch for you to try with a new fix in place. >> As you might have predicted, it's named "wip-nick-newest". >> >> Please give it a try to see if it resolved the hang you've >> been seeing and let me know how it goes. If it continues >> to hang, please provide the logs as you have before, it's >> been very helpful. >> >> Thanks a lot. >> >> -Alex >>> >>> -Alex >>> Is it possible that there is some sort of deadlock going on? We are doing the rbd maps (and subsequent filesystem mounts) on the same systems which are running the ceph-osd and ceph-mon processes. To get around the 'sync' deadlock problem, we are using a patch from Sage which ignores system wide sync's on filesystems mounted with the 'mand' option (and we mount the underlying osd filesystems with 'mand'). However I am wondering if there is potential for other types of deadlocks in this environment. Also, we recently saw an rbd hang in a much older version, running kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. It's possible that this issue was around for some time, just the recent patches made it happen more often (and thus more reproducible) for us. On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: > On 12/17/2012 11:12 AM, Nick Bartos wrote: >> Here's a log with the rbd debugging enabled: >> >> https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log >> >> On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: >>> On 12/14/2012 10:53 AM, Nick Bartos wrote: Yes I was only enabling debugging for libceph. I'm adding debugging for rbd as well. I'll do a repro later today when a test cluster opens up. >>> >>> Excellent, thank you. -Alex > > I looked through these debugging messages. Looking only at the > rbd debugging, what I see seems to indicate that rbd is idle at > the point the "hang" seems to start. This suggests that the hang > is not due to rbd itself, but rather whatever it is that might > be responsible for using the rbd image once it has been mapped. > > Is that possible? I don't know what process you have that is > mapping the rbd image, and what is supposed to be the next thing > it does. (I realize this may not make a lot of sense, given > a patch in rdb seems to have caused the hang to begin occurring.) > > Also note that the debugging information available (i.e., the > lines in the code that can output debugging information) may > well be incomplete. So if you don't find anything it may be > necessary to provide you with another update which might include > more debugging. > > Anyway, could you provide a little more context about what > is going on sort of *around* rbd when activity seems to stop? > > Thanks a lot. > > -Alex >>> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
Unfortunately, we still have a hang: https://gist.github.com/4347052/download On Wed, Dec 19, 2012 at 2:42 PM, Alex Elder wrote: > On 12/19/2012 03:25 PM, Alex Elder wrote: >> On 12/18/2012 12:05 PM, Nick Bartos wrote: >>> I've added the output of "ps -ef" in addition to triggering a trace >>> when a hang is detected. Not much is generally running at that point, >>> but you can have a look: >>> >>> https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt >> >> This helped a lot. I updated the bug with a little more info. >> >> http://tracker.newdream.net/issues/3519 >> >> I also think I have now found something that could explain what you >> are seeing, and am developing a fix. I'll provide you an update >> as soon as I have tested what I come up with, almost certainly >> this afternoon. > > Nick, I have a new branch for you to try with a new fix in place. > As you might have predicted, it's named "wip-nick-newest". > > Please give it a try to see if it resolved the hang you've > been seeing and let me know how it goes. If it continues > to hang, please provide the logs as you have before, it's > been very helpful. > > Thanks a lot. > > -Alex >> >> -Alex >> >>> Is it possible that there is some sort of deadlock going on? We are >>> doing the rbd maps (and subsequent filesystem mounts) on the same >>> systems which are running the ceph-osd and ceph-mon processes. To get >>> around the 'sync' deadlock problem, we are using a patch from Sage >>> which ignores system wide sync's on filesystems mounted with the >>> 'mand' option (and we mount the underlying osd filesystems with >>> 'mand'). However I am wondering if there is potential for other types >>> of deadlocks in this environment. >>> >>> Also, we recently saw an rbd hang in a much older version, running >>> kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. >>> It's possible that this issue was around for some time, just the >>> recent patches made it happen more often (and thus more reproducible) >>> for us. >>> >>> >>> On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: On 12/17/2012 11:12 AM, Nick Bartos wrote: > Here's a log with the rbd debugging enabled: > > https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log > > On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: >> On 12/14/2012 10:53 AM, Nick Bartos wrote: >>> Yes I was only enabling debugging for libceph. I'm adding debugging >>> for rbd as well. I'll do a repro later today when a test cluster >>> opens up. >> >> Excellent, thank you. -Alex I looked through these debugging messages. Looking only at the rbd debugging, what I see seems to indicate that rbd is idle at the point the "hang" seems to start. This suggests that the hang is not due to rbd itself, but rather whatever it is that might be responsible for using the rbd image once it has been mapped. Is that possible? I don't know what process you have that is mapping the rbd image, and what is supposed to be the next thing it does. (I realize this may not make a lot of sense, given a patch in rdb seems to have caused the hang to begin occurring.) Also note that the debugging information available (i.e., the lines in the code that can output debugging information) may well be incomplete. So if you don't find anything it may be necessary to provide you with another update which might include more debugging. Anyway, could you provide a little more context about what is going on sort of *around* rbd when activity seems to stop? Thanks a lot. -Alex >> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/19/2012 03:25 PM, Alex Elder wrote: > On 12/18/2012 12:05 PM, Nick Bartos wrote: >> I've added the output of "ps -ef" in addition to triggering a trace >> when a hang is detected. Not much is generally running at that point, >> but you can have a look: >> >> https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt > > This helped a lot. I updated the bug with a little more info. > > http://tracker.newdream.net/issues/3519 > > I also think I have now found something that could explain what you > are seeing, and am developing a fix. I'll provide you an update > as soon as I have tested what I come up with, almost certainly > this afternoon. Nick, I have a new branch for you to try with a new fix in place. As you might have predicted, it's named "wip-nick-newest". Please give it a try to see if it resolved the hang you've been seeing and let me know how it goes. If it continues to hang, please provide the logs as you have before, it's been very helpful. Thanks a lot. -Alex > > -Alex > >> Is it possible that there is some sort of deadlock going on? We are >> doing the rbd maps (and subsequent filesystem mounts) on the same >> systems which are running the ceph-osd and ceph-mon processes. To get >> around the 'sync' deadlock problem, we are using a patch from Sage >> which ignores system wide sync's on filesystems mounted with the >> 'mand' option (and we mount the underlying osd filesystems with >> 'mand'). However I am wondering if there is potential for other types >> of deadlocks in this environment. >> >> Also, we recently saw an rbd hang in a much older version, running >> kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. >> It's possible that this issue was around for some time, just the >> recent patches made it happen more often (and thus more reproducible) >> for us. >> >> >> On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: >>> On 12/17/2012 11:12 AM, Nick Bartos wrote: Here's a log with the rbd debugging enabled: https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: > On 12/14/2012 10:53 AM, Nick Bartos wrote: >> Yes I was only enabling debugging for libceph. I'm adding debugging >> for rbd as well. I'll do a repro later today when a test cluster >> opens up. > > Excellent, thank you. -Alex >>> >>> I looked through these debugging messages. Looking only at the >>> rbd debugging, what I see seems to indicate that rbd is idle at >>> the point the "hang" seems to start. This suggests that the hang >>> is not due to rbd itself, but rather whatever it is that might >>> be responsible for using the rbd image once it has been mapped. >>> >>> Is that possible? I don't know what process you have that is >>> mapping the rbd image, and what is supposed to be the next thing >>> it does. (I realize this may not make a lot of sense, given >>> a patch in rdb seems to have caused the hang to begin occurring.) >>> >>> Also note that the debugging information available (i.e., the >>> lines in the code that can output debugging information) may >>> well be incomplete. So if you don't find anything it may be >>> necessary to provide you with another update which might include >>> more debugging. >>> >>> Anyway, could you provide a little more context about what >>> is going on sort of *around* rbd when activity seems to stop? >>> >>> Thanks a lot. >>> >>> -Alex > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/18/2012 12:05 PM, Nick Bartos wrote: > I've added the output of "ps -ef" in addition to triggering a trace > when a hang is detected. Not much is generally running at that point, > but you can have a look: > > https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt This helped a lot. I updated the bug with a little more info. http://tracker.newdream.net/issues/3519 I also think I have now found something that could explain what you are seeing, and am developing a fix. I'll provide you an update as soon as I have tested what I come up with, almost certainly this afternoon. -Alex > Is it possible that there is some sort of deadlock going on? We are > doing the rbd maps (and subsequent filesystem mounts) on the same > systems which are running the ceph-osd and ceph-mon processes. To get > around the 'sync' deadlock problem, we are using a patch from Sage > which ignores system wide sync's on filesystems mounted with the > 'mand' option (and we mount the underlying osd filesystems with > 'mand'). However I am wondering if there is potential for other types > of deadlocks in this environment. > > Also, we recently saw an rbd hang in a much older version, running > kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. > It's possible that this issue was around for some time, just the > recent patches made it happen more often (and thus more reproducible) > for us. > > > On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: >> On 12/17/2012 11:12 AM, Nick Bartos wrote: >>> Here's a log with the rbd debugging enabled: >>> >>> https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log >>> >>> On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: On 12/14/2012 10:53 AM, Nick Bartos wrote: > Yes I was only enabling debugging for libceph. I'm adding debugging > for rbd as well. I'll do a repro later today when a test cluster > opens up. Excellent, thank you. -Alex >> >> I looked through these debugging messages. Looking only at the >> rbd debugging, what I see seems to indicate that rbd is idle at >> the point the "hang" seems to start. This suggests that the hang >> is not due to rbd itself, but rather whatever it is that might >> be responsible for using the rbd image once it has been mapped. >> >> Is that possible? I don't know what process you have that is >> mapping the rbd image, and what is supposed to be the next thing >> it does. (I realize this may not make a lot of sense, given >> a patch in rdb seems to have caused the hang to begin occurring.) >> >> Also note that the debugging information available (i.e., the >> lines in the code that can output debugging information) may >> well be incomplete. So if you don't find anything it may be >> necessary to provide you with another update which might include >> more debugging. >> >> Anyway, could you provide a little more context about what >> is going on sort of *around* rbd when activity seems to stop? >> >> Thanks a lot. >> >> -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
I've added the output of "ps -ef" in addition to triggering a trace when a hang is detected. Not much is generally running at that point, but you can have a look: https://gist.github.com/raw/4330223/2f131ee312ee43cb3d8c307a9bf2f454a7edfe57/rbd-hang-1355851498.txt Is it possible that there is some sort of deadlock going on? We are doing the rbd maps (and subsequent filesystem mounts) on the same systems which are running the ceph-osd and ceph-mon processes. To get around the 'sync' deadlock problem, we are using a patch from Sage which ignores system wide sync's on filesystems mounted with the 'mand' option (and we mount the underlying osd filesystems with 'mand'). However I am wondering if there is potential for other types of deadlocks in this environment. Also, we recently saw an rbd hang in a much older version, running kernel 3.5.3 with only the sync hack patch, along side ceph 0.48.1. It's possible that this issue was around for some time, just the recent patches made it happen more often (and thus more reproducible) for us. On Tue, Dec 18, 2012 at 8:09 AM, Alex Elder wrote: > On 12/17/2012 11:12 AM, Nick Bartos wrote: >> Here's a log with the rbd debugging enabled: >> >> https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log >> >> On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: >>> On 12/14/2012 10:53 AM, Nick Bartos wrote: Yes I was only enabling debugging for libceph. I'm adding debugging for rbd as well. I'll do a repro later today when a test cluster opens up. >>> >>> Excellent, thank you. -Alex > > I looked through these debugging messages. Looking only at the > rbd debugging, what I see seems to indicate that rbd is idle at > the point the "hang" seems to start. This suggests that the hang > is not due to rbd itself, but rather whatever it is that might > be responsible for using the rbd image once it has been mapped. > > Is that possible? I don't know what process you have that is > mapping the rbd image, and what is supposed to be the next thing > it does. (I realize this may not make a lot of sense, given > a patch in rdb seems to have caused the hang to begin occurring.) > > Also note that the debugging information available (i.e., the > lines in the code that can output debugging information) may > well be incomplete. So if you don't find anything it may be > necessary to provide you with another update which might include > more debugging. > > Anyway, could you provide a little more context about what > is going on sort of *around* rbd when activity seems to stop? > > Thanks a lot. > > -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/17/2012 11:12 AM, Nick Bartos wrote: > Here's a log with the rbd debugging enabled: > > https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log > > On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: >> On 12/14/2012 10:53 AM, Nick Bartos wrote: >>> Yes I was only enabling debugging for libceph. I'm adding debugging >>> for rbd as well. I'll do a repro later today when a test cluster >>> opens up. >> >> Excellent, thank you. -Alex I looked through these debugging messages. Looking only at the rbd debugging, what I see seems to indicate that rbd is idle at the point the "hang" seems to start. This suggests that the hang is not due to rbd itself, but rather whatever it is that might be responsible for using the rbd image once it has been mapped. Is that possible? I don't know what process you have that is mapping the rbd image, and what is supposed to be the next thing it does. (I realize this may not make a lot of sense, given a patch in rdb seems to have caused the hang to begin occurring.) Also note that the debugging information available (i.e., the lines in the code that can output debugging information) may well be incomplete. So if you don't find anything it may be necessary to provide you with another update which might include more debugging. Anyway, could you provide a little more context about what is going on sort of *around* rbd when activity seems to stop? Thanks a lot. -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
Here's a log with the rbd debugging enabled: https://gist.github.com/raw/4319962/d9690fd92c169198efc5eecabf275ef1808929d2/rbd-hang-test-1355763470.log On Fri, Dec 14, 2012 at 10:03 AM, Alex Elder wrote: > On 12/14/2012 10:53 AM, Nick Bartos wrote: >> Yes I was only enabling debugging for libceph. I'm adding debugging >> for rbd as well. I'll do a repro later today when a test cluster >> opens up. > > Excellent, thank you. -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/14/2012 10:53 AM, Nick Bartos wrote: > Yes I was only enabling debugging for libceph. I'm adding debugging > for rbd as well. I'll do a repro later today when a test cluster > opens up. Excellent, thank you. -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
The kernel is 3.5.7 with the following patches applied (and in the order specified below): 001-libceph_eliminate_connection_state_DEAD_13_days_ago.patch 002-libceph_kill_bad_proto_ceph_connection_op_13_days_ago.patch 003-libceph_rename_socket_callbacks_13_days_ago.patch 004-libceph_rename_kvec_reset_and_kvec_add_functions_13_days_ago.patch 005-libceph_embed_ceph_messenger_structure_in_ceph_client_13_days_ago.patch 006-libceph_start_separating_connection_flags_from_state_13_days_ago.patch 007-libceph_start_tracking_connection_socket_state_13_days_ago.patch 008-libceph_provide_osd_number_when_creating_osd_13_days_ago.patch 009-libceph_set_CLOSED_state_bit_in_con_init_13_days_ago.patch 010-libceph_embed_ceph_connection_structure_in_mon_client_13_days_ago.patch 011-libceph_drop_connection_refcounting_for_mon_client_13_days_ago.patch 012-libceph_init_monitor_connection_when_opening_13_days_ago.patch 013-libceph_fully_initialize_connection_in_con_init_13_days_ago.patch 014-libceph_tweak_ceph_alloc_msg_13_days_ago.patch 015-libceph_have_messages_point_to_their_connection_13_days_ago.patch 016-libceph_have_messages_take_a_connection_reference_13_days_ago.patch 017-libceph_make_ceph_con_revoke_a_msg_operation_13_days_ago.patch 018-libceph_make_ceph_con_revoke_message_a_msg_op_13_days_ago.patch 019-libceph_fix_overflow_in___decode_pool_names_13_days_ago.patch 020-libceph_fix_overflow_in_osdmap_decode_13_days_ago.patch 021-libceph_fix_overflow_in_osdmap_apply_incremental_13_days_ago.patch 022-libceph_transition_socket_state_prior_to_actual_connect_13_days_ago.patch 023-libceph_fix_NULL_dereference_in_reset_connection_13_days_ago.patch 024-libceph_use_con_get_put_methods_13_days_ago.patch 025-libceph_drop_ceph_con_get_put_helpers_and_nref_member_13_days_ago.patch 026-libceph_encapsulate_out_message_data_setup_13_days_ago.patch 027-libceph_encapsulate_advancing_msg_page_13_days_ago.patch 028-libceph_don_t_mark_footer_complete_before_it_is_13_days_ago.patch 029-libceph_move_init_bio__functions_up_13_days_ago.patch 030-libceph_move_init_of_bio_iter_13_days_ago.patch 031-libceph_don_t_use_bio_iter_as_a_flag_13_days_ago.patch 032-libceph_SOCK_CLOSED_is_a_flag_not_a_state_13_days_ago.patch 033-libceph_don_t_change_socket_state_on_sock_event_13_days_ago.patch 034-libceph_just_set_SOCK_CLOSED_when_state_changes_13_days_ago.patch 035-libceph_don_t_touch_con_state_in_con_close_socket_13_days_ago.patch 036-libceph_clear_CONNECTING_in_ceph_con_close_13_days_ago.patch 037-libceph_clear_NEGOTIATING_when_done_13_days_ago.patch 038-libceph_define_and_use_an_explicit_CONNECTED_state_13_days_ago.patch 039-libceph_separate_banner_and_connect_writes_13_days_ago.patch 040-libceph_distinguish_two_phases_of_connect_sequence_13_days_ago.patch 041-libceph_small_changes_to_messenger.c_13_days_ago.patch 042-libceph_add_some_fine_ASCII_art_13_days_ago.patch 043-libceph_set_peer_name_on_con_open_not_init_13_days_ago.patch 044-libceph_initialize_mon_client_con_only_once_13_days_ago.patch 045-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED_13_days_ago.patch 046-libceph_initialize_msgpool_message_types_13_days_ago.patch 047-libceph_prevent_the_race_of_incoming_work_during_teardown_13_days_ago.patch 048-libceph_report_socket_read_write_error_message_13_days_ago.patch 049-libceph_fix_mutex_coverage_for_ceph_con_close_13_days_ago.patch 050-libceph_resubmit_linger_ops_when_pg_mapping_changes_12_days_ago.patch 051-libceph_re_initialize_bio_iter_on_start_of_message_receive_28_hours_ago.patch 052-libceph_protect_ceph_con_open_with_mutex_28_hours_ago.patch 053-libceph_reset_connection_retry_on_successfully_negotiation_28_hours_ago.patch 054-libceph_fix_fault_locking_close_socket_on_lossy_fault_28_hours_ago.patch 055-libceph_move_msgr_clear_standby_under_con_mutex_protection_28_hours_ago.patch 056-libceph_move_ceph_con_send_closed_check_under_the_con_mutex_28_hours_ago.patch 057-libceph_drop_gratuitous_socket_close_calls_in_con_work_28_hours_ago.patch 058-libceph_close_socket_directly_from_ceph_con_close_28_hours_ago.patch 059-libceph_drop_unnecessary_CLOSED_check_in_socket_state_change_callback_28_hours_ago.patch 060-libceph_replace_connection_state_bits_with_states_28_hours_ago.patch 061-libceph_clean_up_con_flags_28_hours_ago.patch 062-libceph_clear_all_flags_on_con_close_28_hours_ago.patch 063-libceph_fix_handling_of_immediate_socket_connect_failure_28_hours_ago.patch 064-libceph_revoke_mon_client_messages_on_session_restart_28_hours_ago.patch 065-libceph_verify_state_after_retaking_con_lock_after_dispatch_28_hours_ago.patch 066-libceph_avoid_dropping_con_mutex_before_fault_28_hours_ago.patch 067-libceph_change_ceph_con_in_msg_alloc_convention_to_be_less_weird_28_hours_ago.patch 068-libceph_recheck_con_state_after_allocating_incoming_message_28_hours_ago.patch 069-libceph_fix_crypto_key_null_deref_memory_leak_28_hours_ago.patch 070-libceph_delay_debugfs_initialization_until_we_learn_global_id_28_hours_ago.patch 071-libceph_avoid_truncation_due_t
Re: rbd map command hangs for 15 minutes during system start up
On 12/13/2012 01:00 PM, Nick Bartos wrote: > Here's another log with the kernel debugging enabled: > https://gist.github.com/raw/4278697/1c9e41d275e614783fbbdee8ca5842680f46c249/rbd-hang-1355424455.log > > Note that it hung on the 2nd try. Just to make sure I'm working with the right code base, can you confirm that you're using a kernel built with the equivalent of what's now in the "wip-nick-newer" branch (commit id 1728893)? Also, looking at this log I don't think I see any rbd debug output. Does that make sense to you? How are you activating debugging to get these messages? If it includes something like: echo module libceph +p > /sys/kernel/debug/dynamic_debug/control it might be that you need to also do: echo module rbd +p > /sys/kernel/debug/dynamic_debug/control This information would be helpful in providing some more context about what rbd is doing that's leading to the various messaging activity I seen in this log. Please send me a log with that info if you are able to produce one. Thanks a lot. -Alex -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 12/13/2012 01:00 PM, Nick Bartos wrote: > Here's another log with the kernel debugging enabled: > https://gist.github.com/raw/4278697/1c9e41d275e614783fbbdee8ca5842680f46c249/rbd-hang-1355424455.log > > Note that it hung on the 2nd try. OK, thanks for the info. We'll keep looking. -Alex > > On Wed, Dec 12, 2012 at 4:57 PM, Nick Bartos wrote: >> Using wip-nick-newer, the problem still presented itself after 4 >> successful runs (so it may be a fluke, but it got slightly further >> than before). The log is here: >> https://gist.github.com/raw/4273114/9085ed00d5bdd5ebab9a94b48f4a562d1fbac431/rbd-hang-1355359129.log >> >> Unfortunately I forgot to enable libceph debugging, I'll do that in a >> bit and get you another log later. >> >> >> On Tue, Dec 11, 2012 at 11:44 AM, Alex Elder wrote: >>> On 12/11/2012 12:01 PM, Alex Elder wrote: On 12/11/2012 11:26 AM, Nick Bartos wrote: > Thanks! I'm creating a build with the new patches now. I'll let you > know how testing goes. FYI, I've been testing with these changes and have *not* been hitting the kinds of problems I'd been previously. However those problems were different from yours, so I'm offering no promises... But there's a chance it'll be more helpful than I thought. I am preparing yet another branch for you, this time adding all the rest of the commits you started with, just in case this does improve things. >>> >>> This new branch is ready. Feel free to try it out, and again >>> let me know how it works for you. >>> >>> The branch is "wip-nick-newer" >>> >>> -Alex >>> >>> Please keep me informed how your testing goes. -Alex > On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: >> On 12/02/2012 10:43 PM, Alex Elder wrote: >>> On 12/01/2012 11:34 PM, Nick Bartos wrote: Unfortunately the hangs happen with the new set of patches. Here's some debug info: https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >>> >>> Well I'm sorry to hear that but I'm glad to have the new info. >>> >>> In retrospect running the new patches *without* the one that >>> seems to cause the hang (#50) were good validation that they >>> didn't lead to any new problems. >>> >>> I'll look at this some more in the morning, and I think I'll >>> confer with Sage whenever he's available for ideas on how to >>> proceed. >> >> Over the course of last week I have been finding and fixing a >> few problems in rbd, the osd client, and the messenger in the >> Linux kernel code. I've added a handful of new patches to the >> end of the ones I gave you last time. >> >> At this point I don't expect these changes directly affect >> the hangs you have been seeing, but a couple of these are >> real problems you could (also) hit, and I'd like to avoid >> that. >> >> I haven't done rigorous testing on this but I believe >> the changes are correct (and Sage has looked at them >> and says they look OK to him). >> >> The new version is available in the branch "wip-nick-new" >> in the ceph-client git repository. If you reproduce your >> hang with this updated code (or do not), please let me know. >> >> Thanks. >> >> -Alex >> >> >>> -Alex >>> >>> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: > On 11/29/2012 02:37 PM, Alex Elder wrote: >> On 11/22/2012 12:04 PM, Nick Bartos wrote: >>> Here are the ceph log messages (including the libceph kernel debug >>> stuff you asked for) from a node boot with the rbd command hung for >>> a >>> couple of minutes: > > I'm sorry, but I did something stupid... > > Yes, the branch I gave you includes these fixes. However > it does *not* include the commit that was giving you trouble > to begin with. > > So... > > I have updated that same branch (wip-nick) to contain: > - Linux 3.5.5 > - Plus the first *50* (not 49) patches you listed > - Plus the ones I added before. > > The new commit id for that branch begins with be3198d6. > > I'm really sorry for this mistake. Please try this new > branch and report back what you find. > > -Alex > > >> Nick, I have put together a branch that includes two fixes >> that might be helpful. I don't expect these fixes will >> necessarily *fix* what you're seeing, but one of them >> pulls a big hunk of processing out of the picture and >> might help elimina
Re: rbd map command hangs for 15 minutes during system start up
Here's another log with the kernel debugging enabled: https://gist.github.com/raw/4278697/1c9e41d275e614783fbbdee8ca5842680f46c249/rbd-hang-1355424455.log Note that it hung on the 2nd try. On Wed, Dec 12, 2012 at 4:57 PM, Nick Bartos wrote: > Using wip-nick-newer, the problem still presented itself after 4 > successful runs (so it may be a fluke, but it got slightly further > than before). The log is here: > https://gist.github.com/raw/4273114/9085ed00d5bdd5ebab9a94b48f4a562d1fbac431/rbd-hang-1355359129.log > > Unfortunately I forgot to enable libceph debugging, I'll do that in a > bit and get you another log later. > > > On Tue, Dec 11, 2012 at 11:44 AM, Alex Elder wrote: >> On 12/11/2012 12:01 PM, Alex Elder wrote: >>> On 12/11/2012 11:26 AM, Nick Bartos wrote: Thanks! I'm creating a build with the new patches now. I'll let you know how testing goes. >>> >>> FYI, I've been testing with these changes and have *not* been >>> hitting the kinds of problems I'd been previously. However >>> those problems were different from yours, so I'm offering no >>> promises... But there's a chance it'll be more helpful than >>> I thought. >>> >>> I am preparing yet another branch for you, this time adding >>> all the rest of the commits you started with, just in case >>> this does improve things. >> >> This new branch is ready. Feel free to try it out, and again >> let me know how it works for you. >> >> The branch is "wip-nick-newer" >> >> -Alex >> >> >>> Please keep me informed how your testing goes. >>> >>> -Alex >>> >>> On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: > On 12/02/2012 10:43 PM, Alex Elder wrote: >> On 12/01/2012 11:34 PM, Nick Bartos wrote: >>> Unfortunately the hangs happen with the new set of patches. Here's >>> some debug info: >>> >>> https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >>> >> >> Well I'm sorry to hear that but I'm glad to have the new info. >> >> In retrospect running the new patches *without* the one that >> seems to cause the hang (#50) were good validation that they >> didn't lead to any new problems. >> >> I'll look at this some more in the morning, and I think I'll >> confer with Sage whenever he's available for ideas on how to >> proceed. > > Over the course of last week I have been finding and fixing a > few problems in rbd, the osd client, and the messenger in the > Linux kernel code. I've added a handful of new patches to the > end of the ones I gave you last time. > > At this point I don't expect these changes directly affect > the hangs you have been seeing, but a couple of these are > real problems you could (also) hit, and I'd like to avoid > that. > > I haven't done rigorous testing on this but I believe > the changes are correct (and Sage has looked at them > and says they look OK to him). > > The new version is available in the branch "wip-nick-new" > in the ceph-client git repository. If you reproduce your > hang with this updated code (or do not), please let me know. > > Thanks. > > -Alex > > >> -Alex >> >> >>> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: On 11/29/2012 02:37 PM, Alex Elder wrote: > On 11/22/2012 12:04 PM, Nick Bartos wrote: >> Here are the ceph log messages (including the libceph kernel debug >> stuff you asked for) from a node boot with the rbd command hung for a >> couple of minutes: I'm sorry, but I did something stupid... Yes, the branch I gave you includes these fixes. However it does *not* include the commit that was giving you trouble to begin with. So... I have updated that same branch (wip-nick) to contain: - Linux 3.5.5 - Plus the first *50* (not 49) patches you listed - Plus the ones I added before. The new commit id for that branch begins with be3198d6. I'm really sorry for this mistake. Please try this new branch and report back what you find. -Alex > Nick, I have put together a branch that includes two fixes > that might be helpful. I don't expect these fixes will > necessarily *fix* what you're seeing, but one of them > pulls a big hunk of processing out of the picture and > might help eliminate some potential causes. I had to > pull in several other patches as prerequisites in order > to get those fixes to apply cleanly. > > Would you be able to give it a try, and let us know what > res
Re: rbd map command hangs for 15 minutes during system start up
Using wip-nick-newer, the problem still presented itself after 4 successful runs (so it may be a fluke, but it got slightly further than before). The log is here: https://gist.github.com/raw/4273114/9085ed00d5bdd5ebab9a94b48f4a562d1fbac431/rbd-hang-1355359129.log Unfortunately I forgot to enable libceph debugging, I'll do that in a bit and get you another log later. On Tue, Dec 11, 2012 at 11:44 AM, Alex Elder wrote: > On 12/11/2012 12:01 PM, Alex Elder wrote: >> On 12/11/2012 11:26 AM, Nick Bartos wrote: >>> Thanks! I'm creating a build with the new patches now. I'll let you >>> know how testing goes. >> >> FYI, I've been testing with these changes and have *not* been >> hitting the kinds of problems I'd been previously. However >> those problems were different from yours, so I'm offering no >> promises... But there's a chance it'll be more helpful than >> I thought. >> >> I am preparing yet another branch for you, this time adding >> all the rest of the commits you started with, just in case >> this does improve things. > > This new branch is ready. Feel free to try it out, and again > let me know how it works for you. > > The branch is "wip-nick-newer" > > -Alex > > >> Please keep me informed how your testing goes. >> >> -Alex >> >> >>> On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: On 12/02/2012 10:43 PM, Alex Elder wrote: > On 12/01/2012 11:34 PM, Nick Bartos wrote: >> Unfortunately the hangs happen with the new set of patches. Here's >> some debug info: >> >> https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >> > > Well I'm sorry to hear that but I'm glad to have the new info. > > In retrospect running the new patches *without* the one that > seems to cause the hang (#50) were good validation that they > didn't lead to any new problems. > > I'll look at this some more in the morning, and I think I'll > confer with Sage whenever he's available for ideas on how to > proceed. Over the course of last week I have been finding and fixing a few problems in rbd, the osd client, and the messenger in the Linux kernel code. I've added a handful of new patches to the end of the ones I gave you last time. At this point I don't expect these changes directly affect the hangs you have been seeing, but a couple of these are real problems you could (also) hit, and I'd like to avoid that. I haven't done rigorous testing on this but I believe the changes are correct (and Sage has looked at them and says they look OK to him). The new version is available in the branch "wip-nick-new" in the ceph-client git repository. If you reproduce your hang with this updated code (or do not), please let me know. Thanks. -Alex > -Alex > > >> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: >>> On 11/29/2012 02:37 PM, Alex Elder wrote: On 11/22/2012 12:04 PM, Nick Bartos wrote: > Here are the ceph log messages (including the libceph kernel debug > stuff you asked for) from a node boot with the rbd command hung for a > couple of minutes: >>> >>> I'm sorry, but I did something stupid... >>> >>> Yes, the branch I gave you includes these fixes. However >>> it does *not* include the commit that was giving you trouble >>> to begin with. >>> >>> So... >>> >>> I have updated that same branch (wip-nick) to contain: >>> - Linux 3.5.5 >>> - Plus the first *50* (not 49) patches you listed >>> - Plus the ones I added before. >>> >>> The new commit id for that branch begins with be3198d6. >>> >>> I'm really sorry for this mistake. Please try this new >>> branch and report back what you find. >>> >>> -Alex >>> >>> Nick, I have put together a branch that includes two fixes that might be helpful. I don't expect these fixes will necessarily *fix* what you're seeing, but one of them pulls a big hunk of processing out of the picture and might help eliminate some potential causes. I had to pull in several other patches as prerequisites in order to get those fixes to apply cleanly. Would you be able to give it a try, and let us know what results you get? The branch contains: - Linux 3.5.5 - Plus the first 49 patches you listed - Plus four patches, which are prerequisites... libceph: define ceph_extract_encoded_string() rbd: define some new format constants rbd: define rbd_dev_image_id() rbd: kill create_snap sysfs entry
Re: rbd map command hangs for 15 minutes during system start up
On 12/11/2012 12:01 PM, Alex Elder wrote: > On 12/11/2012 11:26 AM, Nick Bartos wrote: >> Thanks! I'm creating a build with the new patches now. I'll let you >> know how testing goes. > > FYI, I've been testing with these changes and have *not* been > hitting the kinds of problems I'd been previously. However > those problems were different from yours, so I'm offering no > promises... But there's a chance it'll be more helpful than > I thought. > > I am preparing yet another branch for you, this time adding > all the rest of the commits you started with, just in case > this does improve things. This new branch is ready. Feel free to try it out, and again let me know how it works for you. The branch is "wip-nick-newer" -Alex > Please keep me informed how your testing goes. > > -Alex > > >> On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: >>> On 12/02/2012 10:43 PM, Alex Elder wrote: On 12/01/2012 11:34 PM, Nick Bartos wrote: > Unfortunately the hangs happen with the new set of patches. Here's > some debug info: > > https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt > Well I'm sorry to hear that but I'm glad to have the new info. In retrospect running the new patches *without* the one that seems to cause the hang (#50) were good validation that they didn't lead to any new problems. I'll look at this some more in the morning, and I think I'll confer with Sage whenever he's available for ideas on how to proceed. >>> >>> Over the course of last week I have been finding and fixing a >>> few problems in rbd, the osd client, and the messenger in the >>> Linux kernel code. I've added a handful of new patches to the >>> end of the ones I gave you last time. >>> >>> At this point I don't expect these changes directly affect >>> the hangs you have been seeing, but a couple of these are >>> real problems you could (also) hit, and I'd like to avoid >>> that. >>> >>> I haven't done rigorous testing on this but I believe >>> the changes are correct (and Sage has looked at them >>> and says they look OK to him). >>> >>> The new version is available in the branch "wip-nick-new" >>> in the ceph-client git repository. If you reproduce your >>> hang with this updated code (or do not), please let me know. >>> >>> Thanks. >>> >>> -Alex >>> >>> -Alex > On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: >> On 11/29/2012 02:37 PM, Alex Elder wrote: >>> On 11/22/2012 12:04 PM, Nick Bartos wrote: Here are the ceph log messages (including the libceph kernel debug stuff you asked for) from a node boot with the rbd command hung for a couple of minutes: >> >> I'm sorry, but I did something stupid... >> >> Yes, the branch I gave you includes these fixes. However >> it does *not* include the commit that was giving you trouble >> to begin with. >> >> So... >> >> I have updated that same branch (wip-nick) to contain: >> - Linux 3.5.5 >> - Plus the first *50* (not 49) patches you listed >> - Plus the ones I added before. >> >> The new commit id for that branch begins with be3198d6. >> >> I'm really sorry for this mistake. Please try this new >> branch and report back what you find. >> >> -Alex >> >> >>> Nick, I have put together a branch that includes two fixes >>> that might be helpful. I don't expect these fixes will >>> necessarily *fix* what you're seeing, but one of them >>> pulls a big hunk of processing out of the picture and >>> might help eliminate some potential causes. I had to >>> pull in several other patches as prerequisites in order >>> to get those fixes to apply cleanly. >>> >>> Would you be able to give it a try, and let us know what >>> results you get? The branch contains: >>> - Linux 3.5.5 >>> - Plus the first 49 patches you listed >>> - Plus four patches, which are prerequisites... >>> libceph: define ceph_extract_encoded_string() >>> rbd: define some new format constants >>> rbd: define rbd_dev_image_id() >>> rbd: kill create_snap sysfs entry >>> - ...for these two bug fixes: >>> libceph: remove 'osdtimeout' option >>> ceph: don't reference req after put >>> >>> The branch is available in the ceph-client git repository >>> under the name "wip-nick" and has commit id dd9323aa. >>> https://github.com/ceph/ceph-client/tree/wip-nick >>> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt >>> >>> This full debug output is very helpful. Please supply >>> that again as wel
Re: rbd map command hangs for 15 minutes during system start up
On 12/11/2012 11:26 AM, Nick Bartos wrote: > Thanks! I'm creating a build with the new patches now. I'll let you > know how testing goes. FYI, I've been testing with these changes and have *not* been hitting the kinds of problems I'd been previously. However those problems were different from yours, so I'm offering no promises... But there's a chance it'll be more helpful than I thought. I am preparing yet another branch for you, this time adding all the rest of the commits you started with, just in case this does improve things. Please keep me informed how your testing goes. -Alex > On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: >> On 12/02/2012 10:43 PM, Alex Elder wrote: >>> On 12/01/2012 11:34 PM, Nick Bartos wrote: Unfortunately the hangs happen with the new set of patches. Here's some debug info: https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >>> >>> Well I'm sorry to hear that but I'm glad to have the new info. >>> >>> In retrospect running the new patches *without* the one that >>> seems to cause the hang (#50) were good validation that they >>> didn't lead to any new problems. >>> >>> I'll look at this some more in the morning, and I think I'll >>> confer with Sage whenever he's available for ideas on how to >>> proceed. >> >> Over the course of last week I have been finding and fixing a >> few problems in rbd, the osd client, and the messenger in the >> Linux kernel code. I've added a handful of new patches to the >> end of the ones I gave you last time. >> >> At this point I don't expect these changes directly affect >> the hangs you have been seeing, but a couple of these are >> real problems you could (also) hit, and I'd like to avoid >> that. >> >> I haven't done rigorous testing on this but I believe >> the changes are correct (and Sage has looked at them >> and says they look OK to him). >> >> The new version is available in the branch "wip-nick-new" >> in the ceph-client git repository. If you reproduce your >> hang with this updated code (or do not), please let me know. >> >> Thanks. >> >> -Alex >> >> >>> -Alex >>> >>> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: > On 11/29/2012 02:37 PM, Alex Elder wrote: >> On 11/22/2012 12:04 PM, Nick Bartos wrote: >>> Here are the ceph log messages (including the libceph kernel debug >>> stuff you asked for) from a node boot with the rbd command hung for a >>> couple of minutes: > > I'm sorry, but I did something stupid... > > Yes, the branch I gave you includes these fixes. However > it does *not* include the commit that was giving you trouble > to begin with. > > So... > > I have updated that same branch (wip-nick) to contain: > - Linux 3.5.5 > - Plus the first *50* (not 49) patches you listed > - Plus the ones I added before. > > The new commit id for that branch begins with be3198d6. > > I'm really sorry for this mistake. Please try this new > branch and report back what you find. > > -Alex > > >> Nick, I have put together a branch that includes two fixes >> that might be helpful. I don't expect these fixes will >> necessarily *fix* what you're seeing, but one of them >> pulls a big hunk of processing out of the picture and >> might help eliminate some potential causes. I had to >> pull in several other patches as prerequisites in order >> to get those fixes to apply cleanly. >> >> Would you be able to give it a try, and let us know what >> results you get? The branch contains: >> - Linux 3.5.5 >> - Plus the first 49 patches you listed >> - Plus four patches, which are prerequisites... >> libceph: define ceph_extract_encoded_string() >> rbd: define some new format constants >> rbd: define rbd_dev_image_id() >> rbd: kill create_snap sysfs entry >> - ...for these two bug fixes: >> libceph: remove 'osdtimeout' option >> ceph: don't reference req after put >> >> The branch is available in the ceph-client git repository >> under the name "wip-nick" and has commit id dd9323aa. >> https://github.com/ceph/ceph-client/tree/wip-nick >> >>> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt >>> >> >> This full debug output is very helpful. Please supply >> that again as well. >> >> Thanks. >> >>-Alex >> >>> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos >>> wrote: It's very easy to reproduce now with my automated install script, the most I've seen it succeed with that patch is 2 in a row, and hanging on the 3rd, although it hang
Re: rbd map command hangs for 15 minutes during system start up
Thanks! I'm creating a build with the new patches now. I'll let you know how testing goes. On Mon, Dec 10, 2012 at 1:57 PM, Alex Elder wrote: > On 12/02/2012 10:43 PM, Alex Elder wrote: >> On 12/01/2012 11:34 PM, Nick Bartos wrote: >>> Unfortunately the hangs happen with the new set of patches. Here's >>> some debug info: >>> >>> https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >>> >> >> Well I'm sorry to hear that but I'm glad to have the new info. >> >> In retrospect running the new patches *without* the one that >> seems to cause the hang (#50) were good validation that they >> didn't lead to any new problems. >> >> I'll look at this some more in the morning, and I think I'll >> confer with Sage whenever he's available for ideas on how to >> proceed. > > Over the course of last week I have been finding and fixing a > few problems in rbd, the osd client, and the messenger in the > Linux kernel code. I've added a handful of new patches to the > end of the ones I gave you last time. > > At this point I don't expect these changes directly affect > the hangs you have been seeing, but a couple of these are > real problems you could (also) hit, and I'd like to avoid > that. > > I haven't done rigorous testing on this but I believe > the changes are correct (and Sage has looked at them > and says they look OK to him). > > The new version is available in the branch "wip-nick-new" > in the ceph-client git repository. If you reproduce your > hang with this updated code (or do not), please let me know. > > Thanks. > > -Alex > > >> -Alex >> >> >>> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: On 11/29/2012 02:37 PM, Alex Elder wrote: > On 11/22/2012 12:04 PM, Nick Bartos wrote: >> Here are the ceph log messages (including the libceph kernel debug >> stuff you asked for) from a node boot with the rbd command hung for a >> couple of minutes: I'm sorry, but I did something stupid... Yes, the branch I gave you includes these fixes. However it does *not* include the commit that was giving you trouble to begin with. So... I have updated that same branch (wip-nick) to contain: - Linux 3.5.5 - Plus the first *50* (not 49) patches you listed - Plus the ones I added before. The new commit id for that branch begins with be3198d6. I'm really sorry for this mistake. Please try this new branch and report back what you find. -Alex > Nick, I have put together a branch that includes two fixes > that might be helpful. I don't expect these fixes will > necessarily *fix* what you're seeing, but one of them > pulls a big hunk of processing out of the picture and > might help eliminate some potential causes. I had to > pull in several other patches as prerequisites in order > to get those fixes to apply cleanly. > > Would you be able to give it a try, and let us know what > results you get? The branch contains: > - Linux 3.5.5 > - Plus the first 49 patches you listed > - Plus four patches, which are prerequisites... > libceph: define ceph_extract_encoded_string() > rbd: define some new format constants > rbd: define rbd_dev_image_id() > rbd: kill create_snap sysfs entry > - ...for these two bug fixes: > libceph: remove 'osdtimeout' option > ceph: don't reference req after put > > The branch is available in the ceph-client git repository > under the name "wip-nick" and has commit id dd9323aa. > https://github.com/ceph/ceph-client/tree/wip-nick > >> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt >> > > This full debug output is very helpful. Please supply > that again as well. > > Thanks. > >-Alex > >> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos >> wrote: >>> It's very easy to reproduce now with my automated install script, the >>> most I've seen it succeed with that patch is 2 in a row, and hanging >>> on the 3rd, although it hangs on most builds. So it shouldn't take >>> much to get it to do it again. I'll try and get to that tomorrow, >>> when I'm a bit more rested and my brain is working better. >>> >>> Yes during this the OSDs are probably all syncing up. All the osd >>> and >>> mon daemons have started by the time the rdb commands are ran, >>> though. >>> >>> On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: On Wed, 21 Nov 2012, Nick Bartos wrote: > FYI the build which included all 3.5 backports except patch #50 is > still going strong after 21 builds. Okay, that one at least makes some sen
Re: rbd map command hangs for 15 minutes during system start up
On 12/02/2012 10:43 PM, Alex Elder wrote: > On 12/01/2012 11:34 PM, Nick Bartos wrote: >> Unfortunately the hangs happen with the new set of patches. Here's >> some debug info: >> >> https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt >> > > Well I'm sorry to hear that but I'm glad to have the new info. > > In retrospect running the new patches *without* the one that > seems to cause the hang (#50) were good validation that they > didn't lead to any new problems. > > I'll look at this some more in the morning, and I think I'll > confer with Sage whenever he's available for ideas on how to > proceed. Over the course of last week I have been finding and fixing a few problems in rbd, the osd client, and the messenger in the Linux kernel code. I've added a handful of new patches to the end of the ones I gave you last time. At this point I don't expect these changes directly affect the hangs you have been seeing, but a couple of these are real problems you could (also) hit, and I'd like to avoid that. I haven't done rigorous testing on this but I believe the changes are correct (and Sage has looked at them and says they look OK to him). The new version is available in the branch "wip-nick-new" in the ceph-client git repository. If you reproduce your hang with this updated code (or do not), please let me know. Thanks. -Alex > -Alex > > >> On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: >>> On 11/29/2012 02:37 PM, Alex Elder wrote: On 11/22/2012 12:04 PM, Nick Bartos wrote: > Here are the ceph log messages (including the libceph kernel debug > stuff you asked for) from a node boot with the rbd command hung for a > couple of minutes: >>> >>> I'm sorry, but I did something stupid... >>> >>> Yes, the branch I gave you includes these fixes. However >>> it does *not* include the commit that was giving you trouble >>> to begin with. >>> >>> So... >>> >>> I have updated that same branch (wip-nick) to contain: >>> - Linux 3.5.5 >>> - Plus the first *50* (not 49) patches you listed >>> - Plus the ones I added before. >>> >>> The new commit id for that branch begins with be3198d6. >>> >>> I'm really sorry for this mistake. Please try this new >>> branch and report back what you find. >>> >>> -Alex >>> >>> Nick, I have put together a branch that includes two fixes that might be helpful. I don't expect these fixes will necessarily *fix* what you're seeing, but one of them pulls a big hunk of processing out of the picture and might help eliminate some potential causes. I had to pull in several other patches as prerequisites in order to get those fixes to apply cleanly. Would you be able to give it a try, and let us know what results you get? The branch contains: - Linux 3.5.5 - Plus the first 49 patches you listed - Plus four patches, which are prerequisites... libceph: define ceph_extract_encoded_string() rbd: define some new format constants rbd: define rbd_dev_image_id() rbd: kill create_snap sysfs entry - ...for these two bug fixes: libceph: remove 'osdtimeout' option ceph: don't reference req after put The branch is available in the ceph-client git repository under the name "wip-nick" and has commit id dd9323aa. https://github.com/ceph/ceph-client/tree/wip-nick > https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt > This full debug output is very helpful. Please supply that again as well. Thanks. -Alex > On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos > wrote: >> It's very easy to reproduce now with my automated install script, the >> most I've seen it succeed with that patch is 2 in a row, and hanging >> on the 3rd, although it hangs on most builds. So it shouldn't take >> much to get it to do it again. I'll try and get to that tomorrow, >> when I'm a bit more rested and my brain is working better. >> >> Yes during this the OSDs are probably all syncing up. All the osd >> and >> mon daemons have started by the time the rdb commands are ran, >> though. >> >> On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: >>> On Wed, 21 Nov 2012, Nick Bartos wrote: FYI the build which included all 3.5 backports except patch #50 is still going strong after 21 builds. >>> >>> Okay, that one at least makes some sense. I've opened >>> >>> http://tracker.newdream.net/issues/3519 >>> >>> How easy is this to reproduce? If it is something you can >>> trigger with >>> debugging enabled ('echo module libceph +p > >>> /sys/kernel/debug/dynamic_debug/control') th
Re: rbd map command hangs for 15 minutes during system start up
On 12/01/2012 11:34 PM, Nick Bartos wrote: Unfortunately the hangs happen with the new set of patches. Here's some debug info: https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt Well I'm sorry to hear that but I'm glad to have the new info. In retrospect running the new patches *without* the one that seems to cause the hang (#50) were good validation that they didn't lead to any new problems. I'll look at this some more in the morning, and I think I'll confer with Sage whenever he's available for ideas on how to proceed. -Alex On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: On 11/29/2012 02:37 PM, Alex Elder wrote: On 11/22/2012 12:04 PM, Nick Bartos wrote: Here are the ceph log messages (including the libceph kernel debug stuff you asked for) from a node boot with the rbd command hung for a couple of minutes: I'm sorry, but I did something stupid... Yes, the branch I gave you includes these fixes. However it does *not* include the commit that was giving you trouble to begin with. So... I have updated that same branch (wip-nick) to contain: - Linux 3.5.5 - Plus the first *50* (not 49) patches you listed - Plus the ones I added before. The new commit id for that branch begins with be3198d6. I'm really sorry for this mistake. Please try this new branch and report back what you find. -Alex Nick, I have put together a branch that includes two fixes that might be helpful. I don't expect these fixes will necessarily *fix* what you're seeing, but one of them pulls a big hunk of processing out of the picture and might help eliminate some potential causes. I had to pull in several other patches as prerequisites in order to get those fixes to apply cleanly. Would you be able to give it a try, and let us know what results you get? The branch contains: - Linux 3.5.5 - Plus the first 49 patches you listed - Plus four patches, which are prerequisites... libceph: define ceph_extract_encoded_string() rbd: define some new format constants rbd: define rbd_dev_image_id() rbd: kill create_snap sysfs entry - ...for these two bug fixes: libceph: remove 'osdtimeout' option ceph: don't reference req after put The branch is available in the ceph-client git repository under the name "wip-nick" and has commit id dd9323aa. https://github.com/ceph/ceph-client/tree/wip-nick https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt This full debug output is very helpful. Please supply that again as well. Thanks. -Alex On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: It's very easy to reproduce now with my automated install script, the most I've seen it succeed with that patch is 2 in a row, and hanging on the 3rd, although it hangs on most builds. So it shouldn't take much to get it to do it again. I'll try and get to that tomorrow, when I'm a bit more rested and my brain is working better. Yes during this the OSDs are probably all syncing up. All the osd and mon daemons have started by the time the rdb commands are ran, though. On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: On Wed, 21 Nov 2012, Nick Bartos wrote: FYI the build which included all 3.5 backports except patch #50 is still going strong after 21 builds. Okay, that one at least makes some sense. I've opened http://tracker.newdream.net/issues/3519 How easy is this to reproduce? If it is something you can trigger with debugging enabled ('echo module libceph +p > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. I'm guessing that during this startup time the OSDs are still in the process of starting? Alex, I bet that a test that does a lot of map/unmap stuff in a loop while thrashing OSDs could hit this. Thanks! sage On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: With 8 successful installs already done, I'm reasonably confident that it's patch #50. I'm making another build which applies all patches from the 3.5 backport branch, excluding that specific one. I'll let you know if that turns up any unexpected failures. What will the potential fall out be for removing that specific patch? On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: It's really looking like it's the libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When patches 1-50 (listed below) are applied to 3.5.7, the hang is present. So far I have gone through 4 successful installs with no hang with only 1-49 applied. I'm still leaving my test run to make sure it's not a fluke, but since previously it hangs within the first couple of builds, it really looks like this is where the problem originated. 1-libceph_eliminate_connection_state_DEAD.patch 2-libceph_kill_bad_proto_ceph_connection_op.patch 3-libceph_rename_socket_callbacks.patch 4-libceph_rename_kvec_reset_an
Re: rbd map command hangs for 15 minutes during system start up
Unfortunately the hangs happen with the new set of patches. Here's some debug info: https://gist.github.com/raw/4187123/90194ce172130244a9c1c968ed185eee7282d809/gistfile1.txt On Fri, Nov 30, 2012 at 3:22 PM, Alex Elder wrote: > On 11/29/2012 02:37 PM, Alex Elder wrote: >> On 11/22/2012 12:04 PM, Nick Bartos wrote: >>> Here are the ceph log messages (including the libceph kernel debug >>> stuff you asked for) from a node boot with the rbd command hung for a >>> couple of minutes: > > I'm sorry, but I did something stupid... > > Yes, the branch I gave you includes these fixes. However > it does *not* include the commit that was giving you trouble > to begin with. > > So... > > I have updated that same branch (wip-nick) to contain: > - Linux 3.5.5 > - Plus the first *50* (not 49) patches you listed > - Plus the ones I added before. > > The new commit id for that branch begins with be3198d6. > > I'm really sorry for this mistake. Please try this new > branch and report back what you find. > > -Alex > > >> Nick, I have put together a branch that includes two fixes >> that might be helpful. I don't expect these fixes will >> necessarily *fix* what you're seeing, but one of them >> pulls a big hunk of processing out of the picture and >> might help eliminate some potential causes. I had to >> pull in several other patches as prerequisites in order >> to get those fixes to apply cleanly. >> >> Would you be able to give it a try, and let us know what >> results you get? The branch contains: >> - Linux 3.5.5 >> - Plus the first 49 patches you listed >> - Plus four patches, which are prerequisites... >> libceph: define ceph_extract_encoded_string() >> rbd: define some new format constants >> rbd: define rbd_dev_image_id() >> rbd: kill create_snap sysfs entry >> - ...for these two bug fixes: >> libceph: remove 'osdtimeout' option >> ceph: don't reference req after put >> >> The branch is available in the ceph-client git repository >> under the name "wip-nick" and has commit id dd9323aa. >> https://github.com/ceph/ceph-client/tree/wip-nick >> >>> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt >> >> This full debug output is very helpful. Please supply >> that again as well. >> >> Thanks. >> >> -Alex >> >>> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: It's very easy to reproduce now with my automated install script, the most I've seen it succeed with that patch is 2 in a row, and hanging on the 3rd, although it hangs on most builds. So it shouldn't take much to get it to do it again. I'll try and get to that tomorrow, when I'm a bit more rested and my brain is working better. Yes during this the OSDs are probably all syncing up. All the osd and mon daemons have started by the time the rdb commands are ran, though. On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: > On Wed, 21 Nov 2012, Nick Bartos wrote: >> FYI the build which included all 3.5 backports except patch #50 is >> still going strong after 21 builds. > > Okay, that one at least makes some sense. I've opened > > http://tracker.newdream.net/issues/3519 > > How easy is this to reproduce? If it is something you can trigger with > debugging enabled ('echo module libceph +p > > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. > > I'm guessing that during this startup time the OSDs are still in the > process of starting? > > Alex, I bet that a test that does a lot of map/unmap stuff in a loop while > thrashing OSDs could hit this. > > Thanks! > sage > > >> >> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos >> wrote: >>> With 8 successful installs already done, I'm reasonably confident that >>> it's patch #50. I'm making another build which applies all patches >>> from the 3.5 backport branch, excluding that specific one. I'll let >>> you know if that turns up any unexpected failures. >>> >>> What will the potential fall out be for removing that specific patch? >>> >>> >>> On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos >>> wrote: It's really looking like it's the libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When patches 1-50 (listed below) are applied to 3.5.7, the hang is present. So far I have gone through 4 successful installs with no hang with only 1-49 applied. I'm still leaving my test run to make sure it's not a fluke, but since previously it hangs within the first couple of builds, it really looks like this is where the problem originated. 1-libceph_eliminate_connection_state_DEAD.patch 2-libceph_kill_bad_proto_ceph_connection_op.patch 3-libceph_rename_so
Re: rbd map command hangs for 15 minutes during system start up
On 11/29/2012 02:37 PM, Alex Elder wrote: > On 11/22/2012 12:04 PM, Nick Bartos wrote: >> Here are the ceph log messages (including the libceph kernel debug >> stuff you asked for) from a node boot with the rbd command hung for a >> couple of minutes: I'm sorry, but I did something stupid... Yes, the branch I gave you includes these fixes. However it does *not* include the commit that was giving you trouble to begin with. So... I have updated that same branch (wip-nick) to contain: - Linux 3.5.5 - Plus the first *50* (not 49) patches you listed - Plus the ones I added before. The new commit id for that branch begins with be3198d6. I'm really sorry for this mistake. Please try this new branch and report back what you find. -Alex > Nick, I have put together a branch that includes two fixes > that might be helpful. I don't expect these fixes will > necessarily *fix* what you're seeing, but one of them > pulls a big hunk of processing out of the picture and > might help eliminate some potential causes. I had to > pull in several other patches as prerequisites in order > to get those fixes to apply cleanly. > > Would you be able to give it a try, and let us know what > results you get? The branch contains: > - Linux 3.5.5 > - Plus the first 49 patches you listed > - Plus four patches, which are prerequisites... > libceph: define ceph_extract_encoded_string() > rbd: define some new format constants > rbd: define rbd_dev_image_id() > rbd: kill create_snap sysfs entry > - ...for these two bug fixes: > libceph: remove 'osdtimeout' option > ceph: don't reference req after put > > The branch is available in the ceph-client git repository > under the name "wip-nick" and has commit id dd9323aa. > https://github.com/ceph/ceph-client/tree/wip-nick > >> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt > > This full debug output is very helpful. Please supply > that again as well. > > Thanks. > > -Alex > >> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: >>> It's very easy to reproduce now with my automated install script, the >>> most I've seen it succeed with that patch is 2 in a row, and hanging >>> on the 3rd, although it hangs on most builds. So it shouldn't take >>> much to get it to do it again. I'll try and get to that tomorrow, >>> when I'm a bit more rested and my brain is working better. >>> >>> Yes during this the OSDs are probably all syncing up. All the osd and >>> mon daemons have started by the time the rdb commands are ran, though. >>> >>> On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: On Wed, 21 Nov 2012, Nick Bartos wrote: > FYI the build which included all 3.5 backports except patch #50 is > still going strong after 21 builds. Okay, that one at least makes some sense. I've opened http://tracker.newdream.net/issues/3519 How easy is this to reproduce? If it is something you can trigger with debugging enabled ('echo module libceph +p > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. I'm guessing that during this startup time the OSDs are still in the process of starting? Alex, I bet that a test that does a lot of map/unmap stuff in a loop while thrashing OSDs could hit this. Thanks! sage > > On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: >> With 8 successful installs already done, I'm reasonably confident that >> it's patch #50. I'm making another build which applies all patches >> from the 3.5 backport branch, excluding that specific one. I'll let >> you know if that turns up any unexpected failures. >> >> What will the potential fall out be for removing that specific patch? >> >> >> On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos >> wrote: >>> It's really looking like it's the >>> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >>> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >>> So far I have gone through 4 successful installs with no hang with >>> only 1-49 applied. I'm still leaving my test run to make sure it's >>> not a fluke, but since previously it hangs within the first couple of >>> builds, it really looks like this is where the problem originated. >>> >>> 1-libceph_eliminate_connection_state_DEAD.patch >>> 2-libceph_kill_bad_proto_ceph_connection_op.patch >>> 3-libceph_rename_socket_callbacks.patch >>> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >>> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch >>> 6-libceph_start_separating_connection_flags_from_state.patch >>> 7-libceph_start_tracking_connection_socket_state.patch >>> 8-libceph_provide_osd_number_when_creating_osd.patch >>> 9-libc
Re: rbd map command hangs for 15 minutes during system start up
On Fri, 30 Nov 2012, Alex Elder wrote: > On 11/30/2012 12:49 PM, Nick Bartos wrote: > > My initial tests using a 3.5.7 kernel with the 55 patches from > > wip-nick are going well. So far I've gone through 8 installs without > > an incident, I'll leave it run for a bit longer to see if it crops up > > again. > > This is great news! Now I wonder which of the two fixes took > care of the problem... > > > Can I get a branch with these patches integrated into all of the > > backported patches to 3.5.x? I'd like to get this into our main > > testing branch, which is currently running 3.5.7 with the patches from > > wip-3.5 excluding the > > libceph_resubmit_linger_ops_when_pg_mapping_changes patch. > > I will put together a new branch that includes the remainder > of those patches for you shortly. > > > Note that we had a case of a rbd map hang with our main testing > > branch, but I don't have a script that can reproduce that yet. It was > > after the cluster was all up and working, and we were doing a rolling > > reboot (cycling through each node). > > If you are able to reproduce this please let us know. It sounds to me like it might be the same problem. If we're lucky, those 2 patches will resolve this as well. (says the optimist!) sage > > -Alex > > > > > > > On Thu, Nov 29, 2012 at 12:37 PM, Alex Elder wrote: > >> On 11/22/2012 12:04 PM, Nick Bartos wrote: > >>> Here are the ceph log messages (including the libceph kernel debug > >>> stuff you asked for) from a node boot with the rbd command hung for a > >>> couple of minutes: > >> > >> Nick, I have put together a branch that includes two fixes > >> that might be helpful. I don't expect these fixes will > >> necessarily *fix* what you're seeing, but one of them > >> pulls a big hunk of processing out of the picture and > >> might help eliminate some potential causes. I had to > >> pull in several other patches as prerequisites in order > >> to get those fixes to apply cleanly. > >> > >> Would you be able to give it a try, and let us know what > >> results you get? The branch contains: > >> - Linux 3.5.5 > >> - Plus the first 49 patches you listed > >> - Plus four patches, which are prerequisites... > >> libceph: define ceph_extract_encoded_string() > >> rbd: define some new format constants > >> rbd: define rbd_dev_image_id() > >> rbd: kill create_snap sysfs entry > >> - ...for these two bug fixes: > >> libceph: remove 'osdtimeout' option > >> ceph: don't reference req after put > >> > >> The branch is available in the ceph-client git repository > >> under the name "wip-nick" and has commit id dd9323aa. > >> https://github.com/ceph/ceph-client/tree/wip-nick > >> > >>> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt > >> > >> This full debug output is very helpful. Please supply > >> that again as well. > >> > >> Thanks. > >> > >> -Alex > >> > >>> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: > It's very easy to reproduce now with my automated install script, the > most I've seen it succeed with that patch is 2 in a row, and hanging > on the 3rd, although it hangs on most builds. So it shouldn't take > much to get it to do it again. I'll try and get to that tomorrow, > when I'm a bit more rested and my brain is working better. > > Yes during this the OSDs are probably all syncing up. All the osd and > mon daemons have started by the time the rdb commands are ran, though. > > On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: > > On Wed, 21 Nov 2012, Nick Bartos wrote: > >> FYI the build which included all 3.5 backports except patch #50 is > >> still going strong after 21 builds. > > > > Okay, that one at least makes some sense. I've opened > > > > http://tracker.newdream.net/issues/3519 > > > > How easy is this to reproduce? If it is something you can trigger with > > debugging enabled ('echo module libceph +p > > > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. > > > > I'm guessing that during this startup time the OSDs are still in the > > process of starting? > > > > Alex, I bet that a test that does a lot of map/unmap stuff in a loop > > while > > thrashing OSDs could hit this. > > > > Thanks! > > sage > > > > > >> > >> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos > >> wrote: > >>> With 8 successful installs already done, I'm reasonably confident that > >>> it's patch #50. I'm making another build which applies all patches > >>> from the 3.5 backport branch, excluding that specific one. I'll let > >>> you know if that turns up any unexpected failures. > >>> > >>> What will the potential fall out be for removing that specific patch? > >>> > >>> > >
Re: rbd map command hangs for 15 minutes during system start up
On 11/30/2012 12:49 PM, Nick Bartos wrote: > My initial tests using a 3.5.7 kernel with the 55 patches from > wip-nick are going well. So far I've gone through 8 installs without > an incident, I'll leave it run for a bit longer to see if it crops up > again. This is great news! Now I wonder which of the two fixes took care of the problem... > Can I get a branch with these patches integrated into all of the > backported patches to 3.5.x? I'd like to get this into our main > testing branch, which is currently running 3.5.7 with the patches from > wip-3.5 excluding the > libceph_resubmit_linger_ops_when_pg_mapping_changes patch. I will put together a new branch that includes the remainder of those patches for you shortly. > Note that we had a case of a rbd map hang with our main testing > branch, but I don't have a script that can reproduce that yet. It was > after the cluster was all up and working, and we were doing a rolling > reboot (cycling through each node). If you are able to reproduce this please let us know. -Alex > > > On Thu, Nov 29, 2012 at 12:37 PM, Alex Elder wrote: >> On 11/22/2012 12:04 PM, Nick Bartos wrote: >>> Here are the ceph log messages (including the libceph kernel debug >>> stuff you asked for) from a node boot with the rbd command hung for a >>> couple of minutes: >> >> Nick, I have put together a branch that includes two fixes >> that might be helpful. I don't expect these fixes will >> necessarily *fix* what you're seeing, but one of them >> pulls a big hunk of processing out of the picture and >> might help eliminate some potential causes. I had to >> pull in several other patches as prerequisites in order >> to get those fixes to apply cleanly. >> >> Would you be able to give it a try, and let us know what >> results you get? The branch contains: >> - Linux 3.5.5 >> - Plus the first 49 patches you listed >> - Plus four patches, which are prerequisites... >> libceph: define ceph_extract_encoded_string() >> rbd: define some new format constants >> rbd: define rbd_dev_image_id() >> rbd: kill create_snap sysfs entry >> - ...for these two bug fixes: >> libceph: remove 'osdtimeout' option >> ceph: don't reference req after put >> >> The branch is available in the ceph-client git repository >> under the name "wip-nick" and has commit id dd9323aa. >> https://github.com/ceph/ceph-client/tree/wip-nick >> >>> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt >> >> This full debug output is very helpful. Please supply >> that again as well. >> >> Thanks. >> >> -Alex >> >>> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: It's very easy to reproduce now with my automated install script, the most I've seen it succeed with that patch is 2 in a row, and hanging on the 3rd, although it hangs on most builds. So it shouldn't take much to get it to do it again. I'll try and get to that tomorrow, when I'm a bit more rested and my brain is working better. Yes during this the OSDs are probably all syncing up. All the osd and mon daemons have started by the time the rdb commands are ran, though. On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: > On Wed, 21 Nov 2012, Nick Bartos wrote: >> FYI the build which included all 3.5 backports except patch #50 is >> still going strong after 21 builds. > > Okay, that one at least makes some sense. I've opened > > http://tracker.newdream.net/issues/3519 > > How easy is this to reproduce? If it is something you can trigger with > debugging enabled ('echo module libceph +p > > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. > > I'm guessing that during this startup time the OSDs are still in the > process of starting? > > Alex, I bet that a test that does a lot of map/unmap stuff in a loop while > thrashing OSDs could hit this. > > Thanks! > sage > > >> >> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos >> wrote: >>> With 8 successful installs already done, I'm reasonably confident that >>> it's patch #50. I'm making another build which applies all patches >>> from the 3.5 backport branch, excluding that specific one. I'll let >>> you know if that turns up any unexpected failures. >>> >>> What will the potential fall out be for removing that specific patch? >>> >>> >>> On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos >>> wrote: It's really looking like it's the libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When patches 1-50 (listed below) are applied to 3.5.7, the hang is present. So far I have gone through 4 successful installs with no hang with only 1-49 applied. I'm still leaving my test run to make sure it's
Re: rbd map command hangs for 15 minutes during system start up
My initial tests using a 3.5.7 kernel with the 55 patches from wip-nick are going well. So far I've gone through 8 installs without an incident, I'll leave it run for a bit longer to see if it crops up again. Can I get a branch with these patches integrated into all of the backported patches to 3.5.x? I'd like to get this into our main testing branch, which is currently running 3.5.7 with the patches from wip-3.5 excluding the libceph_resubmit_linger_ops_when_pg_mapping_changes patch. Note that we had a case of a rbd map hang with our main testing branch, but I don't have a script that can reproduce that yet. It was after the cluster was all up and working, and we were doing a rolling reboot (cycling through each node). On Thu, Nov 29, 2012 at 12:37 PM, Alex Elder wrote: > On 11/22/2012 12:04 PM, Nick Bartos wrote: >> Here are the ceph log messages (including the libceph kernel debug >> stuff you asked for) from a node boot with the rbd command hung for a >> couple of minutes: > > Nick, I have put together a branch that includes two fixes > that might be helpful. I don't expect these fixes will > necessarily *fix* what you're seeing, but one of them > pulls a big hunk of processing out of the picture and > might help eliminate some potential causes. I had to > pull in several other patches as prerequisites in order > to get those fixes to apply cleanly. > > Would you be able to give it a try, and let us know what > results you get? The branch contains: > - Linux 3.5.5 > - Plus the first 49 patches you listed > - Plus four patches, which are prerequisites... > libceph: define ceph_extract_encoded_string() > rbd: define some new format constants > rbd: define rbd_dev_image_id() > rbd: kill create_snap sysfs entry > - ...for these two bug fixes: > libceph: remove 'osdtimeout' option > ceph: don't reference req after put > > The branch is available in the ceph-client git repository > under the name "wip-nick" and has commit id dd9323aa. > https://github.com/ceph/ceph-client/tree/wip-nick > >> https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt > > This full debug output is very helpful. Please supply > that again as well. > > Thanks. > > -Alex > >> On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: >>> It's very easy to reproduce now with my automated install script, the >>> most I've seen it succeed with that patch is 2 in a row, and hanging >>> on the 3rd, although it hangs on most builds. So it shouldn't take >>> much to get it to do it again. I'll try and get to that tomorrow, >>> when I'm a bit more rested and my brain is working better. >>> >>> Yes during this the OSDs are probably all syncing up. All the osd and >>> mon daemons have started by the time the rdb commands are ran, though. >>> >>> On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: On Wed, 21 Nov 2012, Nick Bartos wrote: > FYI the build which included all 3.5 backports except patch #50 is > still going strong after 21 builds. Okay, that one at least makes some sense. I've opened http://tracker.newdream.net/issues/3519 How easy is this to reproduce? If it is something you can trigger with debugging enabled ('echo module libceph +p > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. I'm guessing that during this startup time the OSDs are still in the process of starting? Alex, I bet that a test that does a lot of map/unmap stuff in a loop while thrashing OSDs could hit this. Thanks! sage > > On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: >> With 8 successful installs already done, I'm reasonably confident that >> it's patch #50. I'm making another build which applies all patches >> from the 3.5 backport branch, excluding that specific one. I'll let >> you know if that turns up any unexpected failures. >> >> What will the potential fall out be for removing that specific patch? >> >> >> On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos >> wrote: >>> It's really looking like it's the >>> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >>> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >>> So far I have gone through 4 successful installs with no hang with >>> only 1-49 applied. I'm still leaving my test run to make sure it's >>> not a fluke, but since previously it hangs within the first couple of >>> builds, it really looks like this is where the problem originated. >>> >>> 1-libceph_eliminate_connection_state_DEAD.patch >>> 2-libceph_kill_bad_proto_ceph_connection_op.patch >>> 3-libceph_rename_socket_callbacks.patch >>> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >>> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patc
Re: rbd map command hangs for 15 minutes during system start up
On 11/22/2012 12:04 PM, Nick Bartos wrote: > Here are the ceph log messages (including the libceph kernel debug > stuff you asked for) from a node boot with the rbd command hung for a > couple of minutes: Nick, I have put together a branch that includes two fixes that might be helpful. I don't expect these fixes will necessarily *fix* what you're seeing, but one of them pulls a big hunk of processing out of the picture and might help eliminate some potential causes. I had to pull in several other patches as prerequisites in order to get those fixes to apply cleanly. Would you be able to give it a try, and let us know what results you get? The branch contains: - Linux 3.5.5 - Plus the first 49 patches you listed - Plus four patches, which are prerequisites... libceph: define ceph_extract_encoded_string() rbd: define some new format constants rbd: define rbd_dev_image_id() rbd: kill create_snap sysfs entry - ...for these two bug fixes: libceph: remove 'osdtimeout' option ceph: don't reference req after put The branch is available in the ceph-client git repository under the name "wip-nick" and has commit id dd9323aa. https://github.com/ceph/ceph-client/tree/wip-nick > https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt This full debug output is very helpful. Please supply that again as well. Thanks. -Alex > On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: >> It's very easy to reproduce now with my automated install script, the >> most I've seen it succeed with that patch is 2 in a row, and hanging >> on the 3rd, although it hangs on most builds. So it shouldn't take >> much to get it to do it again. I'll try and get to that tomorrow, >> when I'm a bit more rested and my brain is working better. >> >> Yes during this the OSDs are probably all syncing up. All the osd and >> mon daemons have started by the time the rdb commands are ran, though. >> >> On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: >>> On Wed, 21 Nov 2012, Nick Bartos wrote: FYI the build which included all 3.5 backports except patch #50 is still going strong after 21 builds. >>> >>> Okay, that one at least makes some sense. I've opened >>> >>> http://tracker.newdream.net/issues/3519 >>> >>> How easy is this to reproduce? If it is something you can trigger with >>> debugging enabled ('echo module libceph +p > >>> /sys/kernel/debug/dynamic_debug/control') that would help tremendously. >>> >>> I'm guessing that during this startup time the OSDs are still in the >>> process of starting? >>> >>> Alex, I bet that a test that does a lot of map/unmap stuff in a loop while >>> thrashing OSDs could hit this. >>> >>> Thanks! >>> sage >>> >>> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: > With 8 successful installs already done, I'm reasonably confident that > it's patch #50. I'm making another build which applies all patches > from the 3.5 backport branch, excluding that specific one. I'll let > you know if that turns up any unexpected failures. > > What will the potential fall out be for removing that specific patch? > > > On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: >> It's really looking like it's the >> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >> So far I have gone through 4 successful installs with no hang with >> only 1-49 applied. I'm still leaving my test run to make sure it's >> not a fluke, but since previously it hangs within the first couple of >> builds, it really looks like this is where the problem originated. >> >> 1-libceph_eliminate_connection_state_DEAD.patch >> 2-libceph_kill_bad_proto_ceph_connection_op.patch >> 3-libceph_rename_socket_callbacks.patch >> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch >> 6-libceph_start_separating_connection_flags_from_state.patch >> 7-libceph_start_tracking_connection_socket_state.patch >> 8-libceph_provide_osd_number_when_creating_osd.patch >> 9-libceph_set_CLOSED_state_bit_in_con_init.patch >> 10-libceph_embed_ceph_connection_structure_in_mon_client.patch >> 11-libceph_drop_connection_refcounting_for_mon_client.patch >> 12-libceph_init_monitor_connection_when_opening.patch >> 13-libceph_fully_initialize_connection_in_con_init.patch >> 14-libceph_tweak_ceph_alloc_msg.patch >> 15-libceph_have_messages_point_to_their_connection.patch >> 16-libceph_have_messages_take_a_connection_reference.patch >> 17-libceph_make_ceph_con_revoke_a_msg_operation.patch >> 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch >> 19-libceph_fix_overflow_in___decode_pool_names.patch >> 20-libceph_fix_overflow_in_osdmap_decode.patch
Re: rbd map command hangs for 15 minutes during system start up
FYI the build which included all 3.5 backports except patch #50 is still going strong after 21 builds. On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: > With 8 successful installs already done, I'm reasonably confident that > it's patch #50. I'm making another build which applies all patches > from the 3.5 backport branch, excluding that specific one. I'll let > you know if that turns up any unexpected failures. > > What will the potential fall out be for removing that specific patch? > > > On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: >> It's really looking like it's the >> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >> So far I have gone through 4 successful installs with no hang with >> only 1-49 applied. I'm still leaving my test run to make sure it's >> not a fluke, but since previously it hangs within the first couple of >> builds, it really looks like this is where the problem originated. >> >> 1-libceph_eliminate_connection_state_DEAD.patch >> 2-libceph_kill_bad_proto_ceph_connection_op.patch >> 3-libceph_rename_socket_callbacks.patch >> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch >> 6-libceph_start_separating_connection_flags_from_state.patch >> 7-libceph_start_tracking_connection_socket_state.patch >> 8-libceph_provide_osd_number_when_creating_osd.patch >> 9-libceph_set_CLOSED_state_bit_in_con_init.patch >> 10-libceph_embed_ceph_connection_structure_in_mon_client.patch >> 11-libceph_drop_connection_refcounting_for_mon_client.patch >> 12-libceph_init_monitor_connection_when_opening.patch >> 13-libceph_fully_initialize_connection_in_con_init.patch >> 14-libceph_tweak_ceph_alloc_msg.patch >> 15-libceph_have_messages_point_to_their_connection.patch >> 16-libceph_have_messages_take_a_connection_reference.patch >> 17-libceph_make_ceph_con_revoke_a_msg_operation.patch >> 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch >> 19-libceph_fix_overflow_in___decode_pool_names.patch >> 20-libceph_fix_overflow_in_osdmap_decode.patch >> 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch >> 22-libceph_transition_socket_state_prior_to_actual_connect.patch >> 23-libceph_fix_NULL_dereference_in_reset_connection.patch >> 24-libceph_use_con_get_put_methods.patch >> 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch >> 26-libceph_encapsulate_out_message_data_setup.patch >> 27-libceph_encapsulate_advancing_msg_page.patch >> 28-libceph_don_t_mark_footer_complete_before_it_is.patch >> 29-libceph_move_init_bio__functions_up.patch >> 30-libceph_move_init_of_bio_iter.patch >> 31-libceph_don_t_use_bio_iter_as_a_flag.patch >> 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch >> 33-libceph_don_t_change_socket_state_on_sock_event.patch >> 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch >> 35-libceph_don_t_touch_con_state_in_con_close_socket.patch >> 36-libceph_clear_CONNECTING_in_ceph_con_close.patch >> 37-libceph_clear_NEGOTIATING_when_done.patch >> 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch >> 39-libceph_separate_banner_and_connect_writes.patch >> 40-libceph_distinguish_two_phases_of_connect_sequence.patch >> 41-libceph_small_changes_to_messenger.c.patch >> 42-libceph_add_some_fine_ASCII_art.patch >> 43-libceph_set_peer_name_on_con_open_not_init.patch >> 44-libceph_initialize_mon_client_con_only_once.patch >> 45-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED.patch >> 46-libceph_initialize_msgpool_message_types.patch >> 47-libceph_prevent_the_race_of_incoming_work_during_teardown.patch >> 48-libceph_report_socket_read_write_error_message.patch >> 49-libceph_fix_mutex_coverage_for_ceph_con_close.patch >> 50-libceph_resubmit_linger_ops_when_pg_mapping_changes.patch >> >> >> On Wed, Nov 21, 2012 at 8:50 AM, Sage Weil wrote: >>> Thanks for hunting this down. I'm very curious what the culprit is... >>> >>> sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On Wed, 21 Nov 2012, Nick Bartos wrote: > FYI the build which included all 3.5 backports except patch #50 is > still going strong after 21 builds. Okay, that one at least makes some sense. I've opened http://tracker.newdream.net/issues/3519 How easy is this to reproduce? If it is something you can trigger with debugging enabled ('echo module libceph +p > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. I'm guessing that during this startup time the OSDs are still in the process of starting? Alex, I bet that a test that does a lot of map/unmap stuff in a loop while thrashing OSDs could hit this. Thanks! sage > > On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: > > With 8 successful installs already done, I'm reasonably confident that > > it's patch #50. I'm making another build which applies all patches > > from the 3.5 backport branch, excluding that specific one. I'll let > > you know if that turns up any unexpected failures. > > > > What will the potential fall out be for removing that specific patch? > > > > > > On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: > >> It's really looking like it's the > >> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When > >> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. > >> So far I have gone through 4 successful installs with no hang with > >> only 1-49 applied. I'm still leaving my test run to make sure it's > >> not a fluke, but since previously it hangs within the first couple of > >> builds, it really looks like this is where the problem originated. > >> > >> 1-libceph_eliminate_connection_state_DEAD.patch > >> 2-libceph_kill_bad_proto_ceph_connection_op.patch > >> 3-libceph_rename_socket_callbacks.patch > >> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch > >> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch > >> 6-libceph_start_separating_connection_flags_from_state.patch > >> 7-libceph_start_tracking_connection_socket_state.patch > >> 8-libceph_provide_osd_number_when_creating_osd.patch > >> 9-libceph_set_CLOSED_state_bit_in_con_init.patch > >> 10-libceph_embed_ceph_connection_structure_in_mon_client.patch > >> 11-libceph_drop_connection_refcounting_for_mon_client.patch > >> 12-libceph_init_monitor_connection_when_opening.patch > >> 13-libceph_fully_initialize_connection_in_con_init.patch > >> 14-libceph_tweak_ceph_alloc_msg.patch > >> 15-libceph_have_messages_point_to_their_connection.patch > >> 16-libceph_have_messages_take_a_connection_reference.patch > >> 17-libceph_make_ceph_con_revoke_a_msg_operation.patch > >> 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch > >> 19-libceph_fix_overflow_in___decode_pool_names.patch > >> 20-libceph_fix_overflow_in_osdmap_decode.patch > >> 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch > >> 22-libceph_transition_socket_state_prior_to_actual_connect.patch > >> 23-libceph_fix_NULL_dereference_in_reset_connection.patch > >> 24-libceph_use_con_get_put_methods.patch > >> 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch > >> 26-libceph_encapsulate_out_message_data_setup.patch > >> 27-libceph_encapsulate_advancing_msg_page.patch > >> 28-libceph_don_t_mark_footer_complete_before_it_is.patch > >> 29-libceph_move_init_bio__functions_up.patch > >> 30-libceph_move_init_of_bio_iter.patch > >> 31-libceph_don_t_use_bio_iter_as_a_flag.patch > >> 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch > >> 33-libceph_don_t_change_socket_state_on_sock_event.patch > >> 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch > >> 35-libceph_don_t_touch_con_state_in_con_close_socket.patch > >> 36-libceph_clear_CONNECTING_in_ceph_con_close.patch > >> 37-libceph_clear_NEGOTIATING_when_done.patch > >> 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch > >> 39-libceph_separate_banner_and_connect_writes.patch > >> 40-libceph_distinguish_two_phases_of_connect_sequence.patch > >> 41-libceph_small_changes_to_messenger.c.patch > >> 42-libceph_add_some_fine_ASCII_art.patch > >> 43-libceph_set_peer_name_on_con_open_not_init.patch > >> 44-libceph_initialize_mon_client_con_only_once.patch > >> 45-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED.patch > >> 46-libceph_initialize_msgpool_message_types.patch > >> 47-libceph_prevent_the_race_of_incoming_work_during_teardown.patch > >> 48-libceph_report_socket_read_write_error_message.patch > >> 49-libceph_fix_mutex_coverage_for_ceph_con_close.patch > >> 50-libceph_resubmit_linger_ops_when_pg_mapping_changes.patch > >> > >> > >> On Wed, Nov 21, 2012 at 8:50 AM, Sage Weil wrote: > >>> Thanks for hunting this down. I'm very curious what the culprit is... > >>> > >>> sage > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
It's very easy to reproduce now with my automated install script, the most I've seen it succeed with that patch is 2 in a row, and hanging on the 3rd, although it hangs on most builds. So it shouldn't take much to get it to do it again. I'll try and get to that tomorrow, when I'm a bit more rested and my brain is working better. Yes during this the OSDs are probably all syncing up. All the osd and mon daemons have started by the time the rdb commands are ran, though. On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: > On Wed, 21 Nov 2012, Nick Bartos wrote: >> FYI the build which included all 3.5 backports except patch #50 is >> still going strong after 21 builds. > > Okay, that one at least makes some sense. I've opened > > http://tracker.newdream.net/issues/3519 > > How easy is this to reproduce? If it is something you can trigger with > debugging enabled ('echo module libceph +p > > /sys/kernel/debug/dynamic_debug/control') that would help tremendously. > > I'm guessing that during this startup time the OSDs are still in the > process of starting? > > Alex, I bet that a test that does a lot of map/unmap stuff in a loop while > thrashing OSDs could hit this. > > Thanks! > sage > > >> >> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: >> > With 8 successful installs already done, I'm reasonably confident that >> > it's patch #50. I'm making another build which applies all patches >> > from the 3.5 backport branch, excluding that specific one. I'll let >> > you know if that turns up any unexpected failures. >> > >> > What will the potential fall out be for removing that specific patch? >> > >> > >> > On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: >> >> It's really looking like it's the >> >> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >> >> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >> >> So far I have gone through 4 successful installs with no hang with >> >> only 1-49 applied. I'm still leaving my test run to make sure it's >> >> not a fluke, but since previously it hangs within the first couple of >> >> builds, it really looks like this is where the problem originated. >> >> >> >> 1-libceph_eliminate_connection_state_DEAD.patch >> >> 2-libceph_kill_bad_proto_ceph_connection_op.patch >> >> 3-libceph_rename_socket_callbacks.patch >> >> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >> >> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch >> >> 6-libceph_start_separating_connection_flags_from_state.patch >> >> 7-libceph_start_tracking_connection_socket_state.patch >> >> 8-libceph_provide_osd_number_when_creating_osd.patch >> >> 9-libceph_set_CLOSED_state_bit_in_con_init.patch >> >> 10-libceph_embed_ceph_connection_structure_in_mon_client.patch >> >> 11-libceph_drop_connection_refcounting_for_mon_client.patch >> >> 12-libceph_init_monitor_connection_when_opening.patch >> >> 13-libceph_fully_initialize_connection_in_con_init.patch >> >> 14-libceph_tweak_ceph_alloc_msg.patch >> >> 15-libceph_have_messages_point_to_their_connection.patch >> >> 16-libceph_have_messages_take_a_connection_reference.patch >> >> 17-libceph_make_ceph_con_revoke_a_msg_operation.patch >> >> 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch >> >> 19-libceph_fix_overflow_in___decode_pool_names.patch >> >> 20-libceph_fix_overflow_in_osdmap_decode.patch >> >> 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch >> >> 22-libceph_transition_socket_state_prior_to_actual_connect.patch >> >> 23-libceph_fix_NULL_dereference_in_reset_connection.patch >> >> 24-libceph_use_con_get_put_methods.patch >> >> 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch >> >> 26-libceph_encapsulate_out_message_data_setup.patch >> >> 27-libceph_encapsulate_advancing_msg_page.patch >> >> 28-libceph_don_t_mark_footer_complete_before_it_is.patch >> >> 29-libceph_move_init_bio__functions_up.patch >> >> 30-libceph_move_init_of_bio_iter.patch >> >> 31-libceph_don_t_use_bio_iter_as_a_flag.patch >> >> 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch >> >> 33-libceph_don_t_change_socket_state_on_sock_event.patch >> >> 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch >> >> 35-libceph_don_t_touch_con_state_in_con_close_socket.patch >> >> 36-libceph_clear_CONNECTING_in_ceph_con_close.patch >> >> 37-libceph_clear_NEGOTIATING_when_done.patch >> >> 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch >> >> 39-libceph_separate_banner_and_connect_writes.patch >> >> 40-libceph_distinguish_two_phases_of_connect_sequence.patch >> >> 41-libceph_small_changes_to_messenger.c.patch >> >> 42-libceph_add_some_fine_ASCII_art.patch >> >> 43-libceph_set_peer_name_on_con_open_not_init.patch >> >> 44-libceph_initialize_mon_client_con_only_once.patch >> >> 45-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED.patch >> >> 46-libceph_initialize_msgpool_message_types.patch >> >> 47-libceph_prevent_the_race_of_incoming_work_during_teardown.patch >> >> 48-l
Re: rbd map command hangs for 15 minutes during system start up
Here are the ceph log messages (including the libceph kernel debug stuff you asked for) from a node boot with the rbd command hung for a couple of minutes: https://raw.github.com/gist/4132395/7cb5f0150179b012429c6e57749120dd88616cce/gistfile1.txt On Wed, Nov 21, 2012 at 9:49 PM, Nick Bartos wrote: > It's very easy to reproduce now with my automated install script, the > most I've seen it succeed with that patch is 2 in a row, and hanging > on the 3rd, although it hangs on most builds. So it shouldn't take > much to get it to do it again. I'll try and get to that tomorrow, > when I'm a bit more rested and my brain is working better. > > Yes during this the OSDs are probably all syncing up. All the osd and > mon daemons have started by the time the rdb commands are ran, though. > > On Wed, Nov 21, 2012 at 8:47 PM, Sage Weil wrote: >> On Wed, 21 Nov 2012, Nick Bartos wrote: >>> FYI the build which included all 3.5 backports except patch #50 is >>> still going strong after 21 builds. >> >> Okay, that one at least makes some sense. I've opened >> >> http://tracker.newdream.net/issues/3519 >> >> How easy is this to reproduce? If it is something you can trigger with >> debugging enabled ('echo module libceph +p > >> /sys/kernel/debug/dynamic_debug/control') that would help tremendously. >> >> I'm guessing that during this startup time the OSDs are still in the >> process of starting? >> >> Alex, I bet that a test that does a lot of map/unmap stuff in a loop while >> thrashing OSDs could hit this. >> >> Thanks! >> sage >> >> >>> >>> On Wed, Nov 21, 2012 at 9:34 AM, Nick Bartos wrote: >>> > With 8 successful installs already done, I'm reasonably confident that >>> > it's patch #50. I'm making another build which applies all patches >>> > from the 3.5 backport branch, excluding that specific one. I'll let >>> > you know if that turns up any unexpected failures. >>> > >>> > What will the potential fall out be for removing that specific patch? >>> > >>> > >>> > On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: >>> >> It's really looking like it's the >>> >> libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When >>> >> patches 1-50 (listed below) are applied to 3.5.7, the hang is present. >>> >> So far I have gone through 4 successful installs with no hang with >>> >> only 1-49 applied. I'm still leaving my test run to make sure it's >>> >> not a fluke, but since previously it hangs within the first couple of >>> >> builds, it really looks like this is where the problem originated. >>> >> >>> >> 1-libceph_eliminate_connection_state_DEAD.patch >>> >> 2-libceph_kill_bad_proto_ceph_connection_op.patch >>> >> 3-libceph_rename_socket_callbacks.patch >>> >> 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch >>> >> 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch >>> >> 6-libceph_start_separating_connection_flags_from_state.patch >>> >> 7-libceph_start_tracking_connection_socket_state.patch >>> >> 8-libceph_provide_osd_number_when_creating_osd.patch >>> >> 9-libceph_set_CLOSED_state_bit_in_con_init.patch >>> >> 10-libceph_embed_ceph_connection_structure_in_mon_client.patch >>> >> 11-libceph_drop_connection_refcounting_for_mon_client.patch >>> >> 12-libceph_init_monitor_connection_when_opening.patch >>> >> 13-libceph_fully_initialize_connection_in_con_init.patch >>> >> 14-libceph_tweak_ceph_alloc_msg.patch >>> >> 15-libceph_have_messages_point_to_their_connection.patch >>> >> 16-libceph_have_messages_take_a_connection_reference.patch >>> >> 17-libceph_make_ceph_con_revoke_a_msg_operation.patch >>> >> 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch >>> >> 19-libceph_fix_overflow_in___decode_pool_names.patch >>> >> 20-libceph_fix_overflow_in_osdmap_decode.patch >>> >> 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch >>> >> 22-libceph_transition_socket_state_prior_to_actual_connect.patch >>> >> 23-libceph_fix_NULL_dereference_in_reset_connection.patch >>> >> 24-libceph_use_con_get_put_methods.patch >>> >> 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch >>> >> 26-libceph_encapsulate_out_message_data_setup.patch >>> >> 27-libceph_encapsulate_advancing_msg_page.patch >>> >> 28-libceph_don_t_mark_footer_complete_before_it_is.patch >>> >> 29-libceph_move_init_bio__functions_up.patch >>> >> 30-libceph_move_init_of_bio_iter.patch >>> >> 31-libceph_don_t_use_bio_iter_as_a_flag.patch >>> >> 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch >>> >> 33-libceph_don_t_change_socket_state_on_sock_event.patch >>> >> 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch >>> >> 35-libceph_don_t_touch_con_state_in_con_close_socket.patch >>> >> 36-libceph_clear_CONNECTING_in_ceph_con_close.patch >>> >> 37-libceph_clear_NEGOTIATING_when_done.patch >>> >> 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch >>> >> 39-libceph_separate_banner_and_connect_writes.patch >>> >> 40-libceph_distinguish_two_phases_of_connect_sequence.patch >>> >> 41-libceph_small_c
Re: rbd map command hangs for 15 minutes during system start up
With 8 successful installs already done, I'm reasonably confident that it's patch #50. I'm making another build which applies all patches from the 3.5 backport branch, excluding that specific one. I'll let you know if that turns up any unexpected failures. What will the potential fall out be for removing that specific patch? On Wed, Nov 21, 2012 at 9:02 AM, Nick Bartos wrote: > It's really looking like it's the > libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When > patches 1-50 (listed below) are applied to 3.5.7, the hang is present. > So far I have gone through 4 successful installs with no hang with > only 1-49 applied. I'm still leaving my test run to make sure it's > not a fluke, but since previously it hangs within the first couple of > builds, it really looks like this is where the problem originated. > > 1-libceph_eliminate_connection_state_DEAD.patch > 2-libceph_kill_bad_proto_ceph_connection_op.patch > 3-libceph_rename_socket_callbacks.patch > 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch > 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch > 6-libceph_start_separating_connection_flags_from_state.patch > 7-libceph_start_tracking_connection_socket_state.patch > 8-libceph_provide_osd_number_when_creating_osd.patch > 9-libceph_set_CLOSED_state_bit_in_con_init.patch > 10-libceph_embed_ceph_connection_structure_in_mon_client.patch > 11-libceph_drop_connection_refcounting_for_mon_client.patch > 12-libceph_init_monitor_connection_when_opening.patch > 13-libceph_fully_initialize_connection_in_con_init.patch > 14-libceph_tweak_ceph_alloc_msg.patch > 15-libceph_have_messages_point_to_their_connection.patch > 16-libceph_have_messages_take_a_connection_reference.patch > 17-libceph_make_ceph_con_revoke_a_msg_operation.patch > 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch > 19-libceph_fix_overflow_in___decode_pool_names.patch > 20-libceph_fix_overflow_in_osdmap_decode.patch > 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch > 22-libceph_transition_socket_state_prior_to_actual_connect.patch > 23-libceph_fix_NULL_dereference_in_reset_connection.patch > 24-libceph_use_con_get_put_methods.patch > 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch > 26-libceph_encapsulate_out_message_data_setup.patch > 27-libceph_encapsulate_advancing_msg_page.patch > 28-libceph_don_t_mark_footer_complete_before_it_is.patch > 29-libceph_move_init_bio__functions_up.patch > 30-libceph_move_init_of_bio_iter.patch > 31-libceph_don_t_use_bio_iter_as_a_flag.patch > 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch > 33-libceph_don_t_change_socket_state_on_sock_event.patch > 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch > 35-libceph_don_t_touch_con_state_in_con_close_socket.patch > 36-libceph_clear_CONNECTING_in_ceph_con_close.patch > 37-libceph_clear_NEGOTIATING_when_done.patch > 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch > 39-libceph_separate_banner_and_connect_writes.patch > 40-libceph_distinguish_two_phases_of_connect_sequence.patch > 41-libceph_small_changes_to_messenger.c.patch > 42-libceph_add_some_fine_ASCII_art.patch > 43-libceph_set_peer_name_on_con_open_not_init.patch > 44-libceph_initialize_mon_client_con_only_once.patch > 45-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED.patch > 46-libceph_initialize_msgpool_message_types.patch > 47-libceph_prevent_the_race_of_incoming_work_during_teardown.patch > 48-libceph_report_socket_read_write_error_message.patch > 49-libceph_fix_mutex_coverage_for_ceph_con_close.patch > 50-libceph_resubmit_linger_ops_when_pg_mapping_changes.patch > > > On Wed, Nov 21, 2012 at 8:50 AM, Sage Weil wrote: >> Thanks for hunting this down. I'm very curious what the culprit is... >> >> sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
It's really looking like it's the libceph_resubmit_linger_ops_when_pg_mapping_changes commit. When patches 1-50 (listed below) are applied to 3.5.7, the hang is present. So far I have gone through 4 successful installs with no hang with only 1-49 applied. I'm still leaving my test run to make sure it's not a fluke, but since previously it hangs within the first couple of builds, it really looks like this is where the problem originated. 1-libceph_eliminate_connection_state_DEAD.patch 2-libceph_kill_bad_proto_ceph_connection_op.patch 3-libceph_rename_socket_callbacks.patch 4-libceph_rename_kvec_reset_and_kvec_add_functions.patch 5-libceph_embed_ceph_messenger_structure_in_ceph_client.patch 6-libceph_start_separating_connection_flags_from_state.patch 7-libceph_start_tracking_connection_socket_state.patch 8-libceph_provide_osd_number_when_creating_osd.patch 9-libceph_set_CLOSED_state_bit_in_con_init.patch 10-libceph_embed_ceph_connection_structure_in_mon_client.patch 11-libceph_drop_connection_refcounting_for_mon_client.patch 12-libceph_init_monitor_connection_when_opening.patch 13-libceph_fully_initialize_connection_in_con_init.patch 14-libceph_tweak_ceph_alloc_msg.patch 15-libceph_have_messages_point_to_their_connection.patch 16-libceph_have_messages_take_a_connection_reference.patch 17-libceph_make_ceph_con_revoke_a_msg_operation.patch 18-libceph_make_ceph_con_revoke_message_a_msg_op.patch 19-libceph_fix_overflow_in___decode_pool_names.patch 20-libceph_fix_overflow_in_osdmap_decode.patch 21-libceph_fix_overflow_in_osdmap_apply_incremental.patch 22-libceph_transition_socket_state_prior_to_actual_connect.patch 23-libceph_fix_NULL_dereference_in_reset_connection.patch 24-libceph_use_con_get_put_methods.patch 25-libceph_drop_ceph_con_get_put_helpers_and_nref_member.patch 26-libceph_encapsulate_out_message_data_setup.patch 27-libceph_encapsulate_advancing_msg_page.patch 28-libceph_don_t_mark_footer_complete_before_it_is.patch 29-libceph_move_init_bio__functions_up.patch 30-libceph_move_init_of_bio_iter.patch 31-libceph_don_t_use_bio_iter_as_a_flag.patch 32-libceph_SOCK_CLOSED_is_a_flag_not_a_state.patch 33-libceph_don_t_change_socket_state_on_sock_event.patch 34-libceph_just_set_SOCK_CLOSED_when_state_changes.patch 35-libceph_don_t_touch_con_state_in_con_close_socket.patch 36-libceph_clear_CONNECTING_in_ceph_con_close.patch 37-libceph_clear_NEGOTIATING_when_done.patch 38-libceph_define_and_use_an_explicit_CONNECTED_state.patch 39-libceph_separate_banner_and_connect_writes.patch 40-libceph_distinguish_two_phases_of_connect_sequence.patch 41-libceph_small_changes_to_messenger.c.patch 42-libceph_add_some_fine_ASCII_art.patch 43-libceph_set_peer_name_on_con_open_not_init.patch 44-libceph_initialize_mon_client_con_only_once.patch 45-libceph_allow_sock_transition_from_CONNECTING_to_CLOSED.patch 46-libceph_initialize_msgpool_message_types.patch 47-libceph_prevent_the_race_of_incoming_work_during_teardown.patch 48-libceph_report_socket_read_write_error_message.patch 49-libceph_fix_mutex_coverage_for_ceph_con_close.patch 50-libceph_resubmit_linger_ops_when_pg_mapping_changes.patch On Wed, Nov 21, 2012 at 8:50 AM, Sage Weil wrote: > Thanks for hunting this down. I'm very curious what the culprit is... > > sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On Tue, 20 Nov 2012, Nick Bartos wrote: > Since I now have a decent script which can reproduce this, I decided > to re-test with the same 3.5.7 kernel, but just not applying the > patches from the wip-3.5 branch. With the patches, I can only go 2 > builds before I run into a hang. Without the patches, I have gone 9 > consecutive builds (and still going) without seeing the hang. So it > seems like a reasonable assumption that the problem was introduced in > one of those patches. > > We started seeing the problem before applying all the 3.5 patches, so > it seems like one of these is the culprit: > > 1-libceph-encapsulate-out-message-data-setup.patch > 2-libceph-dont-mark-footer-complete-before-it-is.patch > 3-libceph-move-init-of-bio_iter.patch > 4-libceph-dont-use-bio_iter-as-a-flag.patch > 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch > 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch > 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch > 8-libceph-protect-ceph_con_open-with-mutex.patch > 9-libceph-reset-connection-retry-on-successfully-negotiation.patch > 10-rbd-only-reset-capacity-when-pointing-to-head.patch > 11-rbd-set-image-size-when-header-is-updated.patch > 12-libceph-fix-crypto-key-null-deref-memory-leak.patch > 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch > 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch > 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch > 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch > 17-libceph-check-for-invalid-mapping.patch > 18-ceph-propagate-layout-error-on-osd-request-creation.patch > 19-rbd-BUG-on-invalid-layout.patch > 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch > 21-ceph-avoid-32-bit-page-index-overflow.patch > 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch > > I'll start doing some other builds to try and narrow down the patch > introducing the problem more specifically. Thanks for hunting this down. I'm very curious what the culprit is... sage > > > On Tue, Nov 20, 2012 at 1:53 PM, Nick Bartos wrote: > > I reproduced the problem and got several sysrq states captured. > > During this run, the monitor running on the host complained a few > > times about the clocks being off, but all messages were for under 0.55 > > seconds. > > > > Here are the kernel logs. Note that there are several traces, I > > thought multiple during the incident may help: > > https://raw.github.com/gist/4121395/a6dda7552ed8a45725ee5d632fe3ba38703f8cfc/gistfile1.txt > > > > > > On Mon, Nov 19, 2012 at 3:34 PM, Gregory Farnum wrote: > >> Hmm, yep ? that param is actually only used for the warning; I guess > >> we forgot what it actually covers. :( > >> > >> Have your monitor clocks been off by more than 5 seconds at any point? > >> > >> On Mon, Nov 19, 2012 at 3:04 PM, Nick Bartos wrote: > >>> Making 'mon clock drift allowed' very small (0.1) does not > >>> reliably reproduce the hang. I started looking at the code for 0.48.2 > >>> and it looks like this is only used in Paxos::warn_on_future_time, > >>> which only handles the warning, nothing else. > >>> > >>> > >>> On Fri, Nov 16, 2012 at 2:21 PM, Sage Weil wrote: > On Fri, 16 Nov 2012, Nick Bartos wrote: > > Should I be lowering the clock drift allowed, or the lease interval to > > help reproduce it? > > clock drift allowed. > > > > > > > On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: > > > You can safely set the clock drift allowed as high as 500ms. The real > > > limitation is that it needs to be well under the lease interval, > > > which is > > > currently 5 seconds by default. > > > > > > You might be able to reproduce more easily by lowering the > > > threshold... > > > > > > sage > > > > > > > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > > > > > >> How far off do the clocks need to be before there is a problem? It > > >> would seem to be hard to ensure a very large cluster has all of it's > > >> nodes synchronized within 50ms (which seems to be the default for > > >> "mon > > >> clock drift allowed"). Does the mon clock drift allowed parameter > > >> change anything other than the log messages? Are there any other > > >> tuning options that may help, assuming that this is the issue and > > >> it's > > >> not feasible to get the clocks more than 500ms in sync between all > > >> nodes? > > >> > > >> I'm trying to get a good way of reproducing this and get a trace on > > >> the ceph processes to see what they're waiting on. I'll let you know > > >> when I have more info. > > >> > > >> > > >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: > > >> > I just realized I was mixing up this thread with the other deadlock > > >> > thread. > > >> > > > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: > > >> >> Turns
Re: rbd map command hangs for 15 minutes during system start up
Since I now have a decent script which can reproduce this, I decided to re-test with the same 3.5.7 kernel, but just not applying the patches from the wip-3.5 branch. With the patches, I can only go 2 builds before I run into a hang. Without the patches, I have gone 9 consecutive builds (and still going) without seeing the hang. So it seems like a reasonable assumption that the problem was introduced in one of those patches. We started seeing the problem before applying all the 3.5 patches, so it seems like one of these is the culprit: 1-libceph-encapsulate-out-message-data-setup.patch 2-libceph-dont-mark-footer-complete-before-it-is.patch 3-libceph-move-init-of-bio_iter.patch 4-libceph-dont-use-bio_iter-as-a-flag.patch 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch 8-libceph-protect-ceph_con_open-with-mutex.patch 9-libceph-reset-connection-retry-on-successfully-negotiation.patch 10-rbd-only-reset-capacity-when-pointing-to-head.patch 11-rbd-set-image-size-when-header-is-updated.patch 12-libceph-fix-crypto-key-null-deref-memory-leak.patch 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch 17-libceph-check-for-invalid-mapping.patch 18-ceph-propagate-layout-error-on-osd-request-creation.patch 19-rbd-BUG-on-invalid-layout.patch 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch 21-ceph-avoid-32-bit-page-index-overflow.patch 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch I'll start doing some other builds to try and narrow down the patch introducing the problem more specifically. On Tue, Nov 20, 2012 at 1:53 PM, Nick Bartos wrote: > I reproduced the problem and got several sysrq states captured. > During this run, the monitor running on the host complained a few > times about the clocks being off, but all messages were for under 0.55 > seconds. > > Here are the kernel logs. Note that there are several traces, I > thought multiple during the incident may help: > https://raw.github.com/gist/4121395/a6dda7552ed8a45725ee5d632fe3ba38703f8cfc/gistfile1.txt > > > On Mon, Nov 19, 2012 at 3:34 PM, Gregory Farnum wrote: >> Hmm, yep — that param is actually only used for the warning; I guess >> we forgot what it actually covers. :( >> >> Have your monitor clocks been off by more than 5 seconds at any point? >> >> On Mon, Nov 19, 2012 at 3:04 PM, Nick Bartos wrote: >>> Making 'mon clock drift allowed' very small (0.1) does not >>> reliably reproduce the hang. I started looking at the code for 0.48.2 >>> and it looks like this is only used in Paxos::warn_on_future_time, >>> which only handles the warning, nothing else. >>> >>> >>> On Fri, Nov 16, 2012 at 2:21 PM, Sage Weil wrote: On Fri, 16 Nov 2012, Nick Bartos wrote: > Should I be lowering the clock drift allowed, or the lease interval to > help reproduce it? clock drift allowed. > > On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: > > You can safely set the clock drift allowed as high as 500ms. The real > > limitation is that it needs to be well under the lease interval, which > > is > > currently 5 seconds by default. > > > > You might be able to reproduce more easily by lowering the threshold... > > > > sage > > > > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > > > >> How far off do the clocks need to be before there is a problem? It > >> would seem to be hard to ensure a very large cluster has all of it's > >> nodes synchronized within 50ms (which seems to be the default for "mon > >> clock drift allowed"). Does the mon clock drift allowed parameter > >> change anything other than the log messages? Are there any other > >> tuning options that may help, assuming that this is the issue and it's > >> not feasible to get the clocks more than 500ms in sync between all > >> nodes? > >> > >> I'm trying to get a good way of reproducing this and get a trace on > >> the ceph processes to see what they're waiting on. I'll let you know > >> when I have more info. > >> > >> > >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: > >> > I just realized I was mixing up this thread with the other deadlock > >> > thread. > >> > > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: > >> >> Turns out we're having the 'rbd map' hang on startup again, after we > >> >> started using the wip-3.5 patch set. How critical is the > >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I > >> >> removed before which seemed to get rid of the problem (although I'm > >> >> not completely sure if it completely got rid of it, at least s
Re: rbd map command hangs for 15 minutes during system start up
I reproduced the problem and got several sysrq states captured. During this run, the monitor running on the host complained a few times about the clocks being off, but all messages were for under 0.55 seconds. Here are the kernel logs. Note that there are several traces, I thought multiple during the incident may help: https://raw.github.com/gist/4121395/a6dda7552ed8a45725ee5d632fe3ba38703f8cfc/gistfile1.txt On Mon, Nov 19, 2012 at 3:34 PM, Gregory Farnum wrote: > Hmm, yep — that param is actually only used for the warning; I guess > we forgot what it actually covers. :( > > Have your monitor clocks been off by more than 5 seconds at any point? > > On Mon, Nov 19, 2012 at 3:04 PM, Nick Bartos wrote: >> Making 'mon clock drift allowed' very small (0.1) does not >> reliably reproduce the hang. I started looking at the code for 0.48.2 >> and it looks like this is only used in Paxos::warn_on_future_time, >> which only handles the warning, nothing else. >> >> >> On Fri, Nov 16, 2012 at 2:21 PM, Sage Weil wrote: >>> On Fri, 16 Nov 2012, Nick Bartos wrote: Should I be lowering the clock drift allowed, or the lease interval to help reproduce it? >>> >>> clock drift allowed. >>> >>> >>> On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: > You can safely set the clock drift allowed as high as 500ms. The real > limitation is that it needs to be well under the lease interval, which is > currently 5 seconds by default. > > You might be able to reproduce more easily by lowering the threshold... > > sage > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > >> How far off do the clocks need to be before there is a problem? It >> would seem to be hard to ensure a very large cluster has all of it's >> nodes synchronized within 50ms (which seems to be the default for "mon >> clock drift allowed"). Does the mon clock drift allowed parameter >> change anything other than the log messages? Are there any other >> tuning options that may help, assuming that this is the issue and it's >> not feasible to get the clocks more than 500ms in sync between all >> nodes? >> >> I'm trying to get a good way of reproducing this and get a trace on >> the ceph processes to see what they're waiting on. I'll let you know >> when I have more info. >> >> >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: >> > I just realized I was mixing up this thread with the other deadlock >> > thread. >> > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: >> >> Turns out we're having the 'rbd map' hang on startup again, after we >> >> started using the wip-3.5 patch set. How critical is the >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >> >> removed before which seemed to get rid of the problem (although I'm >> >> not completely sure if it completely got rid of it, at least seemed >> >> to >> >> happen much less often). >> >> >> >> It seems like we only started having this issue after we started >> >> patching the 3.5 ceph client (we started patching to try and get rid >> >> of a kernel oops, which the patches seem to have fixed). >> > >> > Right. That patch fixes a real bug. It also seems pretty unlikely >> > that >> > this patch is related to the startup hang. The original log showed >> > clock >> > drift on the monitor that could very easily cause this sort of hang. >> > Can >> > you confirm that that isn't the case with this recent instance of the >> > problem? And/or attach a log? >> > >> > Thanks- >> > sage >> > >> > >> >> >> >> >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >> >> >> Sorry I guess this e-mail got missed. I believe those patches >> >> >> came >> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 >> >> >> and >> >> >> this backport for now until we can figure out what's wrong with >> >> >> 3.6. >> >> >> >> >> >> I typically ignore the wip branches just due to the naming when >> >> >> I'm >> >> >> looking for updates. Where should I typically look for updates >> >> >> that >> >> >> aren't in released kernels? Also, is there anything else in the >> >> >> wip* >> >> >> branches that you think we may find particularly useful? >> >> > >> >> > You were looking in the right place. The problem was we weren't >> >> > super >> >> > organized with our stable patches, and changed our minds about >> >> > what to >> >> > send upstream. These are 'wip' in the sense that they were in >> >> > preparation >> >> > for going u
Re: rbd map command hangs for 15 minutes during system start up
Hmm, yep — that param is actually only used for the warning; I guess we forgot what it actually covers. :( Have your monitor clocks been off by more than 5 seconds at any point? On Mon, Nov 19, 2012 at 3:04 PM, Nick Bartos wrote: > Making 'mon clock drift allowed' very small (0.1) does not > reliably reproduce the hang. I started looking at the code for 0.48.2 > and it looks like this is only used in Paxos::warn_on_future_time, > which only handles the warning, nothing else. > > > On Fri, Nov 16, 2012 at 2:21 PM, Sage Weil wrote: >> On Fri, 16 Nov 2012, Nick Bartos wrote: >>> Should I be lowering the clock drift allowed, or the lease interval to >>> help reproduce it? >> >> clock drift allowed. >> >> >> >>> >>> On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: >>> > You can safely set the clock drift allowed as high as 500ms. The real >>> > limitation is that it needs to be well under the lease interval, which is >>> > currently 5 seconds by default. >>> > >>> > You might be able to reproduce more easily by lowering the threshold... >>> > >>> > sage >>> > >>> > >>> > On Fri, 16 Nov 2012, Nick Bartos wrote: >>> > >>> >> How far off do the clocks need to be before there is a problem? It >>> >> would seem to be hard to ensure a very large cluster has all of it's >>> >> nodes synchronized within 50ms (which seems to be the default for "mon >>> >> clock drift allowed"). Does the mon clock drift allowed parameter >>> >> change anything other than the log messages? Are there any other >>> >> tuning options that may help, assuming that this is the issue and it's >>> >> not feasible to get the clocks more than 500ms in sync between all >>> >> nodes? >>> >> >>> >> I'm trying to get a good way of reproducing this and get a trace on >>> >> the ceph processes to see what they're waiting on. I'll let you know >>> >> when I have more info. >>> >> >>> >> >>> >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: >>> >> > I just realized I was mixing up this thread with the other deadlock >>> >> > thread. >>> >> > >>> >> > On Fri, 16 Nov 2012, Nick Bartos wrote: >>> >> >> Turns out we're having the 'rbd map' hang on startup again, after we >>> >> >> started using the wip-3.5 patch set. How critical is the >>> >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >>> >> >> removed before which seemed to get rid of the problem (although I'm >>> >> >> not completely sure if it completely got rid of it, at least seemed to >>> >> >> happen much less often). >>> >> >> >>> >> >> It seems like we only started having this issue after we started >>> >> >> patching the 3.5 ceph client (we started patching to try and get rid >>> >> >> of a kernel oops, which the patches seem to have fixed). >>> >> > >>> >> > Right. That patch fixes a real bug. It also seems pretty unlikely >>> >> > that >>> >> > this patch is related to the startup hang. The original log showed >>> >> > clock >>> >> > drift on the monitor that could very easily cause this sort of hang. >>> >> > Can >>> >> > you confirm that that isn't the case with this recent instance of the >>> >> > problem? And/or attach a log? >>> >> > >>> >> > Thanks- >>> >> > sage >>> >> > >>> >> > >>> >> >> >>> >> >> >>> >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >>> >> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >>> >> >> >> Sorry I guess this e-mail got missed. I believe those patches came >>> >> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >>> >> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 >>> >> >> >> and >>> >> >> >> this backport for now until we can figure out what's wrong with >>> >> >> >> 3.6. >>> >> >> >> >>> >> >> >> I typically ignore the wip branches just due to the naming when I'm >>> >> >> >> looking for updates. Where should I typically look for updates >>> >> >> >> that >>> >> >> >> aren't in released kernels? Also, is there anything else in the >>> >> >> >> wip* >>> >> >> >> branches that you think we may find particularly useful? >>> >> >> > >>> >> >> > You were looking in the right place. The problem was we weren't >>> >> >> > super >>> >> >> > organized with our stable patches, and changed our minds about what >>> >> >> > to >>> >> >> > send upstream. These are 'wip' in the sense that they were in >>> >> >> > preparation >>> >> >> > for going upstream. The goal is to push them to the mainline stable >>> >> >> > kernels and ideally not keep them in our tree at all. >>> >> >> > >>> >> >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, >>> >> >> > but >>> >> >> > we're keeping it so that ubuntu can pick it up for quantal. >>> >> >> > >>> >> >> > I'll make sure these are more clearly marked as stable. >>> >> >> > >>> >> >> > sage >>> >> >> > >>> >> >> > >>> >> >> >> >>> >> >> >> >>> >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil >>> >> >> >> wrote: >>> >> >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >>> >> >> >> >> After rem
Re: rbd map command hangs for 15 minutes during system start up
Making 'mon clock drift allowed' very small (0.1) does not reliably reproduce the hang. I started looking at the code for 0.48.2 and it looks like this is only used in Paxos::warn_on_future_time, which only handles the warning, nothing else. On Fri, Nov 16, 2012 at 2:21 PM, Sage Weil wrote: > On Fri, 16 Nov 2012, Nick Bartos wrote: >> Should I be lowering the clock drift allowed, or the lease interval to >> help reproduce it? > > clock drift allowed. > > > >> >> On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: >> > You can safely set the clock drift allowed as high as 500ms. The real >> > limitation is that it needs to be well under the lease interval, which is >> > currently 5 seconds by default. >> > >> > You might be able to reproduce more easily by lowering the threshold... >> > >> > sage >> > >> > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: >> > >> >> How far off do the clocks need to be before there is a problem? It >> >> would seem to be hard to ensure a very large cluster has all of it's >> >> nodes synchronized within 50ms (which seems to be the default for "mon >> >> clock drift allowed"). Does the mon clock drift allowed parameter >> >> change anything other than the log messages? Are there any other >> >> tuning options that may help, assuming that this is the issue and it's >> >> not feasible to get the clocks more than 500ms in sync between all >> >> nodes? >> >> >> >> I'm trying to get a good way of reproducing this and get a trace on >> >> the ceph processes to see what they're waiting on. I'll let you know >> >> when I have more info. >> >> >> >> >> >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: >> >> > I just realized I was mixing up this thread with the other deadlock >> >> > thread. >> >> > >> >> > On Fri, 16 Nov 2012, Nick Bartos wrote: >> >> >> Turns out we're having the 'rbd map' hang on startup again, after we >> >> >> started using the wip-3.5 patch set. How critical is the >> >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >> >> >> removed before which seemed to get rid of the problem (although I'm >> >> >> not completely sure if it completely got rid of it, at least seemed to >> >> >> happen much less often). >> >> >> >> >> >> It seems like we only started having this issue after we started >> >> >> patching the 3.5 ceph client (we started patching to try and get rid >> >> >> of a kernel oops, which the patches seem to have fixed). >> >> > >> >> > Right. That patch fixes a real bug. It also seems pretty unlikely that >> >> > this patch is related to the startup hang. The original log showed >> >> > clock >> >> > drift on the monitor that could very easily cause this sort of hang. >> >> > Can >> >> > you confirm that that isn't the case with this recent instance of the >> >> > problem? And/or attach a log? >> >> > >> >> > Thanks- >> >> > sage >> >> > >> >> > >> >> >> >> >> >> >> >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >> >> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >> >> >> >> Sorry I guess this e-mail got missed. I believe those patches came >> >> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >> >> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and >> >> >> >> this backport for now until we can figure out what's wrong with 3.6. >> >> >> >> >> >> >> >> I typically ignore the wip branches just due to the naming when I'm >> >> >> >> looking for updates. Where should I typically look for updates that >> >> >> >> aren't in released kernels? Also, is there anything else in the >> >> >> >> wip* >> >> >> >> branches that you think we may find particularly useful? >> >> >> > >> >> >> > You were looking in the right place. The problem was we weren't >> >> >> > super >> >> >> > organized with our stable patches, and changed our minds about what >> >> >> > to >> >> >> > send upstream. These are 'wip' in the sense that they were in >> >> >> > preparation >> >> >> > for going upstream. The goal is to push them to the mainline stable >> >> >> > kernels and ideally not keep them in our tree at all. >> >> >> > >> >> >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but >> >> >> > we're keeping it so that ubuntu can pick it up for quantal. >> >> >> > >> >> >> > I'll make sure these are more clearly marked as stable. >> >> >> > >> >> >> > sage >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: >> >> >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >> >> >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, >> >> >> >> >> it >> >> >> >> >> seems we no longer have this hang. >> >> >> >> > >> >> >> >> > Hmm, that's a bit disconcerting. Did this series come from our >> >> >> >> > old 3.5 >> >> >> >> > stable series? I recently prepared a new one that backports >> >> >> >> > *all* of the >> >> >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git.
Re: rbd map command hangs for 15 minutes during system start up
To be clear, the monitor cluster needs to be within this clock drift — the rest of the Ceph cluster can be off by as much as you care to. (Well, there's also a limit imposed by cephx authorization which can keep nodes out of the cluster, but that drift allowance is measured in units of hours.) -Greg On Fri, Nov 16, 2012 at 2:16 PM, Nick Bartos wrote: > Should I be lowering the clock drift allowed, or the lease interval to > help reproduce it? > > On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: >> You can safely set the clock drift allowed as high as 500ms. The real >> limitation is that it needs to be well under the lease interval, which is >> currently 5 seconds by default. >> >> You might be able to reproduce more easily by lowering the threshold... >> >> sage >> >> >> On Fri, 16 Nov 2012, Nick Bartos wrote: >> >>> How far off do the clocks need to be before there is a problem? It >>> would seem to be hard to ensure a very large cluster has all of it's >>> nodes synchronized within 50ms (which seems to be the default for "mon >>> clock drift allowed"). Does the mon clock drift allowed parameter >>> change anything other than the log messages? Are there any other >>> tuning options that may help, assuming that this is the issue and it's >>> not feasible to get the clocks more than 500ms in sync between all >>> nodes? >>> >>> I'm trying to get a good way of reproducing this and get a trace on >>> the ceph processes to see what they're waiting on. I'll let you know >>> when I have more info. >>> >>> >>> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: >>> > I just realized I was mixing up this thread with the other deadlock >>> > thread. >>> > >>> > On Fri, 16 Nov 2012, Nick Bartos wrote: >>> >> Turns out we're having the 'rbd map' hang on startup again, after we >>> >> started using the wip-3.5 patch set. How critical is the >>> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >>> >> removed before which seemed to get rid of the problem (although I'm >>> >> not completely sure if it completely got rid of it, at least seemed to >>> >> happen much less often). >>> >> >>> >> It seems like we only started having this issue after we started >>> >> patching the 3.5 ceph client (we started patching to try and get rid >>> >> of a kernel oops, which the patches seem to have fixed). >>> > >>> > Right. That patch fixes a real bug. It also seems pretty unlikely that >>> > this patch is related to the startup hang. The original log showed clock >>> > drift on the monitor that could very easily cause this sort of hang. Can >>> > you confirm that that isn't the case with this recent instance of the >>> > problem? And/or attach a log? >>> > >>> > Thanks- >>> > sage >>> > >>> > >>> >> >>> >> >>> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >>> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >>> >> >> Sorry I guess this e-mail got missed. I believe those patches came >>> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >>> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and >>> >> >> this backport for now until we can figure out what's wrong with 3.6. >>> >> >> >>> >> >> I typically ignore the wip branches just due to the naming when I'm >>> >> >> looking for updates. Where should I typically look for updates that >>> >> >> aren't in released kernels? Also, is there anything else in the wip* >>> >> >> branches that you think we may find particularly useful? >>> >> > >>> >> > You were looking in the right place. The problem was we weren't super >>> >> > organized with our stable patches, and changed our minds about what to >>> >> > send upstream. These are 'wip' in the sense that they were in >>> >> > preparation >>> >> > for going upstream. The goal is to push them to the mainline stable >>> >> > kernels and ideally not keep them in our tree at all. >>> >> > >>> >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but >>> >> > we're keeping it so that ubuntu can pick it up for quantal. >>> >> > >>> >> > I'll make sure these are more clearly marked as stable. >>> >> > >>> >> > sage >>> >> > >>> >> > >>> >> >> >>> >> >> >>> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: >>> >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >>> >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it >>> >> >> >> seems we no longer have this hang. >>> >> >> > >>> >> >> > Hmm, that's a bit disconcerting. Did this series come from our old >>> >> >> > 3.5 >>> >> >> > stable series? I recently prepared a new one that backports *all* >>> >> >> > of the >>> >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I >>> >> >> > would >>> >> >> > be curious if you see problems with that. >>> >> >> > >>> >> >> > So far, with these fixes in place, we have not seen any unexplained >>> >> >> > kernel >>> >> >> > crashes in this code. >>> >> >> > >>> >> >> > I take it you're going back to a
Re: rbd map command hangs for 15 minutes during system start up
On Fri, 16 Nov 2012, Nick Bartos wrote: > Should I be lowering the clock drift allowed, or the lease interval to > help reproduce it? clock drift allowed. > > On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: > > You can safely set the clock drift allowed as high as 500ms. The real > > limitation is that it needs to be well under the lease interval, which is > > currently 5 seconds by default. > > > > You might be able to reproduce more easily by lowering the threshold... > > > > sage > > > > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > > > >> How far off do the clocks need to be before there is a problem? It > >> would seem to be hard to ensure a very large cluster has all of it's > >> nodes synchronized within 50ms (which seems to be the default for "mon > >> clock drift allowed"). Does the mon clock drift allowed parameter > >> change anything other than the log messages? Are there any other > >> tuning options that may help, assuming that this is the issue and it's > >> not feasible to get the clocks more than 500ms in sync between all > >> nodes? > >> > >> I'm trying to get a good way of reproducing this and get a trace on > >> the ceph processes to see what they're waiting on. I'll let you know > >> when I have more info. > >> > >> > >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: > >> > I just realized I was mixing up this thread with the other deadlock > >> > thread. > >> > > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: > >> >> Turns out we're having the 'rbd map' hang on startup again, after we > >> >> started using the wip-3.5 patch set. How critical is the > >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I > >> >> removed before which seemed to get rid of the problem (although I'm > >> >> not completely sure if it completely got rid of it, at least seemed to > >> >> happen much less often). > >> >> > >> >> It seems like we only started having this issue after we started > >> >> patching the 3.5 ceph client (we started patching to try and get rid > >> >> of a kernel oops, which the patches seem to have fixed). > >> > > >> > Right. That patch fixes a real bug. It also seems pretty unlikely that > >> > this patch is related to the startup hang. The original log showed clock > >> > drift on the monitor that could very easily cause this sort of hang. Can > >> > you confirm that that isn't the case with this recent instance of the > >> > problem? And/or attach a log? > >> > > >> > Thanks- > >> > sage > >> > > >> > > >> >> > >> >> > >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: > >> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: > >> >> >> Sorry I guess this e-mail got missed. I believe those patches came > >> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 > >> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and > >> >> >> this backport for now until we can figure out what's wrong with 3.6. > >> >> >> > >> >> >> I typically ignore the wip branches just due to the naming when I'm > >> >> >> looking for updates. Where should I typically look for updates that > >> >> >> aren't in released kernels? Also, is there anything else in the wip* > >> >> >> branches that you think we may find particularly useful? > >> >> > > >> >> > You were looking in the right place. The problem was we weren't super > >> >> > organized with our stable patches, and changed our minds about what to > >> >> > send upstream. These are 'wip' in the sense that they were in > >> >> > preparation > >> >> > for going upstream. The goal is to push them to the mainline stable > >> >> > kernels and ideally not keep them in our tree at all. > >> >> > > >> >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but > >> >> > we're keeping it so that ubuntu can pick it up for quantal. > >> >> > > >> >> > I'll make sure these are more clearly marked as stable. > >> >> > > >> >> > sage > >> >> > > >> >> > > >> >> >> > >> >> >> > >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: > >> >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: > >> >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, > >> >> >> >> it > >> >> >> >> seems we no longer have this hang. > >> >> >> > > >> >> >> > Hmm, that's a bit disconcerting. Did this series come from our > >> >> >> > old 3.5 > >> >> >> > stable series? I recently prepared a new one that backports *all* > >> >> >> > of the > >> >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. > >> >> >> > I would > >> >> >> > be curious if you see problems with that. > >> >> >> > > >> >> >> > So far, with these fixes in place, we have not seen any > >> >> >> > unexplained kernel > >> >> >> > crashes in this code. > >> >> >> > > >> >> >> > I take it you're going back to a 3.5 kernel because you weren't > >> >> >> > able to > >> >> >> > get rid of the sync problem with 3.6? > >> >> >> > > >> >> >> > sage > >> >> >> > > >> >> >> > > >> >> >> >
Re: rbd map command hangs for 15 minutes during system start up
Should I be lowering the clock drift allowed, or the lease interval to help reproduce it? On Fri, Nov 16, 2012 at 2:13 PM, Sage Weil wrote: > You can safely set the clock drift allowed as high as 500ms. The real > limitation is that it needs to be well under the lease interval, which is > currently 5 seconds by default. > > You might be able to reproduce more easily by lowering the threshold... > > sage > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > >> How far off do the clocks need to be before there is a problem? It >> would seem to be hard to ensure a very large cluster has all of it's >> nodes synchronized within 50ms (which seems to be the default for "mon >> clock drift allowed"). Does the mon clock drift allowed parameter >> change anything other than the log messages? Are there any other >> tuning options that may help, assuming that this is the issue and it's >> not feasible to get the clocks more than 500ms in sync between all >> nodes? >> >> I'm trying to get a good way of reproducing this and get a trace on >> the ceph processes to see what they're waiting on. I'll let you know >> when I have more info. >> >> >> On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: >> > I just realized I was mixing up this thread with the other deadlock >> > thread. >> > >> > On Fri, 16 Nov 2012, Nick Bartos wrote: >> >> Turns out we're having the 'rbd map' hang on startup again, after we >> >> started using the wip-3.5 patch set. How critical is the >> >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >> >> removed before which seemed to get rid of the problem (although I'm >> >> not completely sure if it completely got rid of it, at least seemed to >> >> happen much less often). >> >> >> >> It seems like we only started having this issue after we started >> >> patching the 3.5 ceph client (we started patching to try and get rid >> >> of a kernel oops, which the patches seem to have fixed). >> > >> > Right. That patch fixes a real bug. It also seems pretty unlikely that >> > this patch is related to the startup hang. The original log showed clock >> > drift on the monitor that could very easily cause this sort of hang. Can >> > you confirm that that isn't the case with this recent instance of the >> > problem? And/or attach a log? >> > >> > Thanks- >> > sage >> > >> > >> >> >> >> >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >> >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >> >> >> Sorry I guess this e-mail got missed. I believe those patches came >> >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >> >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and >> >> >> this backport for now until we can figure out what's wrong with 3.6. >> >> >> >> >> >> I typically ignore the wip branches just due to the naming when I'm >> >> >> looking for updates. Where should I typically look for updates that >> >> >> aren't in released kernels? Also, is there anything else in the wip* >> >> >> branches that you think we may find particularly useful? >> >> > >> >> > You were looking in the right place. The problem was we weren't super >> >> > organized with our stable patches, and changed our minds about what to >> >> > send upstream. These are 'wip' in the sense that they were in >> >> > preparation >> >> > for going upstream. The goal is to push them to the mainline stable >> >> > kernels and ideally not keep them in our tree at all. >> >> > >> >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but >> >> > we're keeping it so that ubuntu can pick it up for quantal. >> >> > >> >> > I'll make sure these are more clearly marked as stable. >> >> > >> >> > sage >> >> > >> >> > >> >> >> >> >> >> >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: >> >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >> >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it >> >> >> >> seems we no longer have this hang. >> >> >> > >> >> >> > Hmm, that's a bit disconcerting. Did this series come from our old >> >> >> > 3.5 >> >> >> > stable series? I recently prepared a new one that backports *all* >> >> >> > of the >> >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I >> >> >> > would >> >> >> > be curious if you see problems with that. >> >> >> > >> >> >> > So far, with these fixes in place, we have not seen any unexplained >> >> >> > kernel >> >> >> > crashes in this code. >> >> >> > >> >> >> > I take it you're going back to a 3.5 kernel because you weren't able >> >> >> > to >> >> >> > get rid of the sync problem with 3.6? >> >> >> > >> >> >> > sage >> >> >> > >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin >> >> >> >> wrote: >> >> >> >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> >> >> >> >> >> >> >> >> We are seeing a somewhat random, but frequent hang on our systems >> >> >> >> >> during startup. The hang happens at th
Re: rbd map command hangs for 15 minutes during system start up
You can safely set the clock drift allowed as high as 500ms. The real limitation is that it needs to be well under the lease interval, which is currently 5 seconds by default. You might be able to reproduce more easily by lowering the threshold... sage On Fri, 16 Nov 2012, Nick Bartos wrote: > How far off do the clocks need to be before there is a problem? It > would seem to be hard to ensure a very large cluster has all of it's > nodes synchronized within 50ms (which seems to be the default for "mon > clock drift allowed"). Does the mon clock drift allowed parameter > change anything other than the log messages? Are there any other > tuning options that may help, assuming that this is the issue and it's > not feasible to get the clocks more than 500ms in sync between all > nodes? > > I'm trying to get a good way of reproducing this and get a trace on > the ceph processes to see what they're waiting on. I'll let you know > when I have more info. > > > On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: > > I just realized I was mixing up this thread with the other deadlock > > thread. > > > > On Fri, 16 Nov 2012, Nick Bartos wrote: > >> Turns out we're having the 'rbd map' hang on startup again, after we > >> started using the wip-3.5 patch set. How critical is the > >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I > >> removed before which seemed to get rid of the problem (although I'm > >> not completely sure if it completely got rid of it, at least seemed to > >> happen much less often). > >> > >> It seems like we only started having this issue after we started > >> patching the 3.5 ceph client (we started patching to try and get rid > >> of a kernel oops, which the patches seem to have fixed). > > > > Right. That patch fixes a real bug. It also seems pretty unlikely that > > this patch is related to the startup hang. The original log showed clock > > drift on the monitor that could very easily cause this sort of hang. Can > > you confirm that that isn't the case with this recent instance of the > > problem? And/or attach a log? > > > > Thanks- > > sage > > > > > >> > >> > >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: > >> > On Thu, 15 Nov 2012, Nick Bartos wrote: > >> >> Sorry I guess this e-mail got missed. I believe those patches came > >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 > >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and > >> >> this backport for now until we can figure out what's wrong with 3.6. > >> >> > >> >> I typically ignore the wip branches just due to the naming when I'm > >> >> looking for updates. Where should I typically look for updates that > >> >> aren't in released kernels? Also, is there anything else in the wip* > >> >> branches that you think we may find particularly useful? > >> > > >> > You were looking in the right place. The problem was we weren't super > >> > organized with our stable patches, and changed our minds about what to > >> > send upstream. These are 'wip' in the sense that they were in > >> > preparation > >> > for going upstream. The goal is to push them to the mainline stable > >> > kernels and ideally not keep them in our tree at all. > >> > > >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but > >> > we're keeping it so that ubuntu can pick it up for quantal. > >> > > >> > I'll make sure these are more clearly marked as stable. > >> > > >> > sage > >> > > >> > > >> >> > >> >> > >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: > >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: > >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it > >> >> >> seems we no longer have this hang. > >> >> > > >> >> > Hmm, that's a bit disconcerting. Did this series come from our old > >> >> > 3.5 > >> >> > stable series? I recently prepared a new one that backports *all* of > >> >> > the > >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I > >> >> > would > >> >> > be curious if you see problems with that. > >> >> > > >> >> > So far, with these fixes in place, we have not seen any unexplained > >> >> > kernel > >> >> > crashes in this code. > >> >> > > >> >> > I take it you're going back to a 3.5 kernel because you weren't able > >> >> > to > >> >> > get rid of the sync problem with 3.6? > >> >> > > >> >> > sage > >> >> > > >> >> > > >> >> > > >> >> >> > >> >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin > >> >> >> wrote: > >> >> >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: > >> >> >> >> > >> >> >> >> We are seeing a somewhat random, but frequent hang on our systems > >> >> >> >> during startup. The hang happens at the point where an "rbd map > >> >> >> >> " command is run. > >> >> >> >> > >> >> >> >> I've attached the ceph logs from the cluster. The map command > >> >> >> >> happens > >> >> >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung > >>
Re: rbd map command hangs for 15 minutes during system start up
How far off do the clocks need to be before there is a problem? It would seem to be hard to ensure a very large cluster has all of it's nodes synchronized within 50ms (which seems to be the default for "mon clock drift allowed"). Does the mon clock drift allowed parameter change anything other than the log messages? Are there any other tuning options that may help, assuming that this is the issue and it's not feasible to get the clocks more than 500ms in sync between all nodes? I'm trying to get a good way of reproducing this and get a trace on the ceph processes to see what they're waiting on. I'll let you know when I have more info. On Fri, Nov 16, 2012 at 11:16 AM, Sage Weil wrote: > I just realized I was mixing up this thread with the other deadlock > thread. > > On Fri, 16 Nov 2012, Nick Bartos wrote: >> Turns out we're having the 'rbd map' hang on startup again, after we >> started using the wip-3.5 patch set. How critical is the >> libceph_protect_ceph_con_open_with_mutex commit? That's the one I >> removed before which seemed to get rid of the problem (although I'm >> not completely sure if it completely got rid of it, at least seemed to >> happen much less often). >> >> It seems like we only started having this issue after we started >> patching the 3.5 ceph client (we started patching to try and get rid >> of a kernel oops, which the patches seem to have fixed). > > Right. That patch fixes a real bug. It also seems pretty unlikely that > this patch is related to the startup hang. The original log showed clock > drift on the monitor that could very easily cause this sort of hang. Can > you confirm that that isn't the case with this recent instance of the > problem? And/or attach a log? > > Thanks- > sage > > >> >> >> On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: >> > On Thu, 15 Nov 2012, Nick Bartos wrote: >> >> Sorry I guess this e-mail got missed. I believe those patches came >> >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >> >> branch patches, which seem to all be fine. We'll stick with 3.5 and >> >> this backport for now until we can figure out what's wrong with 3.6. >> >> >> >> I typically ignore the wip branches just due to the naming when I'm >> >> looking for updates. Where should I typically look for updates that >> >> aren't in released kernels? Also, is there anything else in the wip* >> >> branches that you think we may find particularly useful? >> > >> > You were looking in the right place. The problem was we weren't super >> > organized with our stable patches, and changed our minds about what to >> > send upstream. These are 'wip' in the sense that they were in preparation >> > for going upstream. The goal is to push them to the mainline stable >> > kernels and ideally not keep them in our tree at all. >> > >> > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but >> > we're keeping it so that ubuntu can pick it up for quantal. >> > >> > I'll make sure these are more clearly marked as stable. >> > >> > sage >> > >> > >> >> >> >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: >> >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >> >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it >> >> >> seems we no longer have this hang. >> >> > >> >> > Hmm, that's a bit disconcerting. Did this series come from our old 3.5 >> >> > stable series? I recently prepared a new one that backports *all* of >> >> > the >> >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I >> >> > would >> >> > be curious if you see problems with that. >> >> > >> >> > So far, with these fixes in place, we have not seen any unexplained >> >> > kernel >> >> > crashes in this code. >> >> > >> >> > I take it you're going back to a 3.5 kernel because you weren't able to >> >> > get rid of the sync problem with 3.6? >> >> > >> >> > sage >> >> > >> >> > >> >> > >> >> >> >> >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin >> >> >> wrote: >> >> >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> >> >> >> >> >> >> We are seeing a somewhat random, but frequent hang on our systems >> >> >> >> during startup. The hang happens at the point where an "rbd map >> >> >> >> " command is run. >> >> >> >> >> >> >> >> I've attached the ceph logs from the cluster. The map command >> >> >> >> happens >> >> >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung >> >> >> >> can >> >> >> >> be seen in the log as 172.18.0.15:0/1143980479. >> >> >> >> >> >> >> >> It appears as if the TCP socket is opened to the OSD, but then times >> >> >> >> out 15 minutes later, the process gets data when the socket is >> >> >> >> closed >> >> >> >> on the client server and it retries. >> >> >> >> >> >> >> >> Please help. >> >> >> >> >> >> >> >> We are using ceph version 0.48.2argonaut >> >> >> >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). >> >> >> >> >> >> >> >> We are using a 3.5.7 kernel with the follow
Re: rbd map command hangs for 15 minutes during system start up
I just realized I was mixing up this thread with the other deadlock thread. On Fri, 16 Nov 2012, Nick Bartos wrote: > Turns out we're having the 'rbd map' hang on startup again, after we > started using the wip-3.5 patch set. How critical is the > libceph_protect_ceph_con_open_with_mutex commit? That's the one I > removed before which seemed to get rid of the problem (although I'm > not completely sure if it completely got rid of it, at least seemed to > happen much less often). > > It seems like we only started having this issue after we started > patching the 3.5 ceph client (we started patching to try and get rid > of a kernel oops, which the patches seem to have fixed). Right. That patch fixes a real bug. It also seems pretty unlikely that this patch is related to the startup hang. The original log showed clock drift on the monitor that could very easily cause this sort of hang. Can you confirm that that isn't the case with this recent instance of the problem? And/or attach a log? Thanks- sage > > > On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: > > On Thu, 15 Nov 2012, Nick Bartos wrote: > >> Sorry I guess this e-mail got missed. I believe those patches came > >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 > >> branch patches, which seem to all be fine. We'll stick with 3.5 and > >> this backport for now until we can figure out what's wrong with 3.6. > >> > >> I typically ignore the wip branches just due to the naming when I'm > >> looking for updates. Where should I typically look for updates that > >> aren't in released kernels? Also, is there anything else in the wip* > >> branches that you think we may find particularly useful? > > > > You were looking in the right place. The problem was we weren't super > > organized with our stable patches, and changed our minds about what to > > send upstream. These are 'wip' in the sense that they were in preparation > > for going upstream. The goal is to push them to the mainline stable > > kernels and ideally not keep them in our tree at all. > > > > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but > > we're keeping it so that ubuntu can pick it up for quantal. > > > > I'll make sure these are more clearly marked as stable. > > > > sage > > > > > >> > >> > >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: > >> > On Mon, 12 Nov 2012, Nick Bartos wrote: > >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it > >> >> seems we no longer have this hang. > >> > > >> > Hmm, that's a bit disconcerting. Did this series come from our old 3.5 > >> > stable series? I recently prepared a new one that backports *all* of the > >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would > >> > be curious if you see problems with that. > >> > > >> > So far, with these fixes in place, we have not seen any unexplained > >> > kernel > >> > crashes in this code. > >> > > >> > I take it you're going back to a 3.5 kernel because you weren't able to > >> > get rid of the sync problem with 3.6? > >> > > >> > sage > >> > > >> > > >> > > >> >> > >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin > >> >> wrote: > >> >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: > >> >> >> > >> >> >> We are seeing a somewhat random, but frequent hang on our systems > >> >> >> during startup. The hang happens at the point where an "rbd map > >> >> >> " command is run. > >> >> >> > >> >> >> I've attached the ceph logs from the cluster. The map command > >> >> >> happens > >> >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can > >> >> >> be seen in the log as 172.18.0.15:0/1143980479. > >> >> >> > >> >> >> It appears as if the TCP socket is opened to the OSD, but then times > >> >> >> out 15 minutes later, the process gets data when the socket is closed > >> >> >> on the client server and it retries. > >> >> >> > >> >> >> Please help. > >> >> >> > >> >> >> We are using ceph version 0.48.2argonaut > >> >> >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). > >> >> >> > >> >> >> We are using a 3.5.7 kernel with the following list of patches > >> >> >> applied: > >> >> >> > >> >> >> 1-libceph-encapsulate-out-message-data-setup.patch > >> >> >> 2-libceph-dont-mark-footer-complete-before-it-is.patch > >> >> >> 3-libceph-move-init-of-bio_iter.patch > >> >> >> 4-libceph-dont-use-bio_iter-as-a-flag.patch > >> >> >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch > >> >> >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch > >> >> >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch > >> >> >> 8-libceph-protect-ceph_con_open-with-mutex.patch > >> >> >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch > >> >> >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch > >> >> >> 11-rbd-set-image-size-when-header-is-updated.patch > >> >> >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch > >> >> >
Re: rbd map command hangs for 15 minutes during system start up
Turns out we're having the 'rbd map' hang on startup again, after we started using the wip-3.5 patch set. How critical is the libceph_protect_ceph_con_open_with_mutex commit? That's the one I removed before which seemed to get rid of the problem (although I'm not completely sure if it completely got rid of it, at least seemed to happen much less often). It seems like we only started having this issue after we started patching the 3.5 ceph client (we started patching to try and get rid of a kernel oops, which the patches seem to have fixed). On Thu, Nov 15, 2012 at 4:25 PM, Sage Weil wrote: > On Thu, 15 Nov 2012, Nick Bartos wrote: >> Sorry I guess this e-mail got missed. I believe those patches came >> from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 >> branch patches, which seem to all be fine. We'll stick with 3.5 and >> this backport for now until we can figure out what's wrong with 3.6. >> >> I typically ignore the wip branches just due to the naming when I'm >> looking for updates. Where should I typically look for updates that >> aren't in released kernels? Also, is there anything else in the wip* >> branches that you think we may find particularly useful? > > You were looking in the right place. The problem was we weren't super > organized with our stable patches, and changed our minds about what to > send upstream. These are 'wip' in the sense that they were in preparation > for going upstream. The goal is to push them to the mainline stable > kernels and ideally not keep them in our tree at all. > > wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but > we're keeping it so that ubuntu can pick it up for quantal. > > I'll make sure these are more clearly marked as stable. > > sage > > >> >> >> On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: >> > On Mon, 12 Nov 2012, Nick Bartos wrote: >> >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it >> >> seems we no longer have this hang. >> > >> > Hmm, that's a bit disconcerting. Did this series come from our old 3.5 >> > stable series? I recently prepared a new one that backports *all* of the >> > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would >> > be curious if you see problems with that. >> > >> > So far, with these fixes in place, we have not seen any unexplained kernel >> > crashes in this code. >> > >> > I take it you're going back to a 3.5 kernel because you weren't able to >> > get rid of the sync problem with 3.6? >> > >> > sage >> > >> > >> > >> >> >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin >> >> wrote: >> >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> >> >> >> >> We are seeing a somewhat random, but frequent hang on our systems >> >> >> during startup. The hang happens at the point where an "rbd map >> >> >> " command is run. >> >> >> >> >> >> I've attached the ceph logs from the cluster. The map command happens >> >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can >> >> >> be seen in the log as 172.18.0.15:0/1143980479. >> >> >> >> >> >> It appears as if the TCP socket is opened to the OSD, but then times >> >> >> out 15 minutes later, the process gets data when the socket is closed >> >> >> on the client server and it retries. >> >> >> >> >> >> Please help. >> >> >> >> >> >> We are using ceph version 0.48.2argonaut >> >> >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). >> >> >> >> >> >> We are using a 3.5.7 kernel with the following list of patches applied: >> >> >> >> >> >> 1-libceph-encapsulate-out-message-data-setup.patch >> >> >> 2-libceph-dont-mark-footer-complete-before-it-is.patch >> >> >> 3-libceph-move-init-of-bio_iter.patch >> >> >> 4-libceph-dont-use-bio_iter-as-a-flag.patch >> >> >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch >> >> >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch >> >> >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch >> >> >> 8-libceph-protect-ceph_con_open-with-mutex.patch >> >> >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch >> >> >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch >> >> >> 11-rbd-set-image-size-when-header-is-updated.patch >> >> >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch >> >> >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch >> >> >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch >> >> >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch >> >> >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch >> >> >> 17-libceph-check-for-invalid-mapping.patch >> >> >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch >> >> >> 19-rbd-BUG-on-invalid-layout.patch >> >> >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch >> >> >> 21-ceph-avoid-32-bit-page-index-overflow.patch >> >> >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch >> >> >> >> >> >> Any suggestions? >> >> > >> >> > >> >> > The lo
Re: rbd map command hangs for 15 minutes during system start up
On Thu, 15 Nov 2012, Nick Bartos wrote: > Sorry I guess this e-mail got missed. I believe those patches came > from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 > branch patches, which seem to all be fine. We'll stick with 3.5 and > this backport for now until we can figure out what's wrong with 3.6. > > I typically ignore the wip branches just due to the naming when I'm > looking for updates. Where should I typically look for updates that > aren't in released kernels? Also, is there anything else in the wip* > branches that you think we may find particularly useful? You were looking in the right place. The problem was we weren't super organized with our stable patches, and changed our minds about what to send upstream. These are 'wip' in the sense that they were in preparation for going upstream. The goal is to push them to the mainline stable kernels and ideally not keep them in our tree at all. wip-3.5 is an oddity because the mainline stable kernel is EOL'd, but we're keeping it so that ubuntu can pick it up for quantal. I'll make sure these are more clearly marked as stable. sage > > > On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: > > On Mon, 12 Nov 2012, Nick Bartos wrote: > >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it > >> seems we no longer have this hang. > > > > Hmm, that's a bit disconcerting. Did this series come from our old 3.5 > > stable series? I recently prepared a new one that backports *all* of the > > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would > > be curious if you see problems with that. > > > > So far, with these fixes in place, we have not seen any unexplained kernel > > crashes in this code. > > > > I take it you're going back to a 3.5 kernel because you weren't able to > > get rid of the sync problem with 3.6? > > > > sage > > > > > > > >> > >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin > >> wrote: > >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: > >> >> > >> >> We are seeing a somewhat random, but frequent hang on our systems > >> >> during startup. The hang happens at the point where an "rbd map > >> >> " command is run. > >> >> > >> >> I've attached the ceph logs from the cluster. The map command happens > >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can > >> >> be seen in the log as 172.18.0.15:0/1143980479. > >> >> > >> >> It appears as if the TCP socket is opened to the OSD, but then times > >> >> out 15 minutes later, the process gets data when the socket is closed > >> >> on the client server and it retries. > >> >> > >> >> Please help. > >> >> > >> >> We are using ceph version 0.48.2argonaut > >> >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). > >> >> > >> >> We are using a 3.5.7 kernel with the following list of patches applied: > >> >> > >> >> 1-libceph-encapsulate-out-message-data-setup.patch > >> >> 2-libceph-dont-mark-footer-complete-before-it-is.patch > >> >> 3-libceph-move-init-of-bio_iter.patch > >> >> 4-libceph-dont-use-bio_iter-as-a-flag.patch > >> >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch > >> >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch > >> >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch > >> >> 8-libceph-protect-ceph_con_open-with-mutex.patch > >> >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch > >> >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch > >> >> 11-rbd-set-image-size-when-header-is-updated.patch > >> >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch > >> >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch > >> >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch > >> >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch > >> >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch > >> >> 17-libceph-check-for-invalid-mapping.patch > >> >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch > >> >> 19-rbd-BUG-on-invalid-layout.patch > >> >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch > >> >> 21-ceph-avoid-32-bit-page-index-overflow.patch > >> >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch > >> >> > >> >> Any suggestions? > >> > > >> > > >> > The log shows your monitors don't have time sychronized enough among > >> > them to make much progress (including authenticating new connections). > >> > That's probably the real issue. 0.2s is pretty large clock drift. > >> > > >> > > >> >> One thought is that the following patch (which we could not apply) is > >> >> what is required: > >> >> > >> >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch > >> > > >> > > >> > This is certainly useful too, but I don't think it's the cause of > >> > the delay in this case. > >> > > >> > Josh > >> > -- > >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> > the body of a message to majord...@vger.kernel.org > >> > More majordomo
Re: rbd map command hangs for 15 minutes during system start up
Sorry I guess this e-mail got missed. I believe those patches came from the ceph/linux-3.5.5-ceph branch. I'm now using the wip-3.5 branch patches, which seem to all be fine. We'll stick with 3.5 and this backport for now until we can figure out what's wrong with 3.6. I typically ignore the wip branches just due to the naming when I'm looking for updates. Where should I typically look for updates that aren't in released kernels? Also, is there anything else in the wip* branches that you think we may find particularly useful? On Mon, Nov 12, 2012 at 3:16 PM, Sage Weil wrote: > On Mon, 12 Nov 2012, Nick Bartos wrote: >> After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it >> seems we no longer have this hang. > > Hmm, that's a bit disconcerting. Did this series come from our old 3.5 > stable series? I recently prepared a new one that backports *all* of the > fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would > be curious if you see problems with that. > > So far, with these fixes in place, we have not seen any unexplained kernel > crashes in this code. > > I take it you're going back to a 3.5 kernel because you weren't able to > get rid of the sync problem with 3.6? > > sage > > > >> >> On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin wrote: >> > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> >> >> We are seeing a somewhat random, but frequent hang on our systems >> >> during startup. The hang happens at the point where an "rbd map >> >> " command is run. >> >> >> >> I've attached the ceph logs from the cluster. The map command happens >> >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can >> >> be seen in the log as 172.18.0.15:0/1143980479. >> >> >> >> It appears as if the TCP socket is opened to the OSD, but then times >> >> out 15 minutes later, the process gets data when the socket is closed >> >> on the client server and it retries. >> >> >> >> Please help. >> >> >> >> We are using ceph version 0.48.2argonaut >> >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). >> >> >> >> We are using a 3.5.7 kernel with the following list of patches applied: >> >> >> >> 1-libceph-encapsulate-out-message-data-setup.patch >> >> 2-libceph-dont-mark-footer-complete-before-it-is.patch >> >> 3-libceph-move-init-of-bio_iter.patch >> >> 4-libceph-dont-use-bio_iter-as-a-flag.patch >> >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch >> >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch >> >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch >> >> 8-libceph-protect-ceph_con_open-with-mutex.patch >> >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch >> >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch >> >> 11-rbd-set-image-size-when-header-is-updated.patch >> >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch >> >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch >> >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch >> >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch >> >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch >> >> 17-libceph-check-for-invalid-mapping.patch >> >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch >> >> 19-rbd-BUG-on-invalid-layout.patch >> >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch >> >> 21-ceph-avoid-32-bit-page-index-overflow.patch >> >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch >> >> >> >> Any suggestions? >> > >> > >> > The log shows your monitors don't have time sychronized enough among >> > them to make much progress (including authenticating new connections). >> > That's probably the real issue. 0.2s is pretty large clock drift. >> > >> > >> >> One thought is that the following patch (which we could not apply) is >> >> what is required: >> >> >> >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch >> > >> > >> > This is certainly useful too, but I don't think it's the cause of >> > the delay in this case. >> > >> > Josh >> > -- >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> > the body of a message to majord...@vger.kernel.org >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On Mon, 12 Nov 2012, Nick Bartos wrote: > After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it > seems we no longer have this hang. Hmm, that's a bit disconcerting. Did this series come from our old 3.5 stable series? I recently prepared a new one that backports *all* of the fixes from 3.6 to 3.5 (and 3.4); see wip-3.5 in ceph-client.git. I would be curious if you see problems with that. So far, with these fixes in place, we have not seen any unexplained kernel crashes in this code. I take it you're going back to a 3.5 kernel because you weren't able to get rid of the sync problem with 3.6? sage > > On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin wrote: > > On 11/08/2012 02:10 PM, Mandell Degerness wrote: > >> > >> We are seeing a somewhat random, but frequent hang on our systems > >> during startup. The hang happens at the point where an "rbd map > >> " command is run. > >> > >> I've attached the ceph logs from the cluster. The map command happens > >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can > >> be seen in the log as 172.18.0.15:0/1143980479. > >> > >> It appears as if the TCP socket is opened to the OSD, but then times > >> out 15 minutes later, the process gets data when the socket is closed > >> on the client server and it retries. > >> > >> Please help. > >> > >> We are using ceph version 0.48.2argonaut > >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). > >> > >> We are using a 3.5.7 kernel with the following list of patches applied: > >> > >> 1-libceph-encapsulate-out-message-data-setup.patch > >> 2-libceph-dont-mark-footer-complete-before-it-is.patch > >> 3-libceph-move-init-of-bio_iter.patch > >> 4-libceph-dont-use-bio_iter-as-a-flag.patch > >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch > >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch > >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch > >> 8-libceph-protect-ceph_con_open-with-mutex.patch > >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch > >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch > >> 11-rbd-set-image-size-when-header-is-updated.patch > >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch > >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch > >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch > >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch > >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch > >> 17-libceph-check-for-invalid-mapping.patch > >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch > >> 19-rbd-BUG-on-invalid-layout.patch > >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch > >> 21-ceph-avoid-32-bit-page-index-overflow.patch > >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch > >> > >> Any suggestions? > > > > > > The log shows your monitors don't have time sychronized enough among > > them to make much progress (including authenticating new connections). > > That's probably the real issue. 0.2s is pretty large clock drift. > > > > > >> One thought is that the following patch (which we could not apply) is > >> what is required: > >> > >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch > > > > > > This is certainly useful too, but I don't think it's the cause of > > the delay in this case. > > > > Josh > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majord...@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
After removing 8-libceph-protect-ceph_con_open-with-mutex.patch, it seems we no longer have this hang. On Thu, Nov 8, 2012 at 5:43 PM, Josh Durgin wrote: > On 11/08/2012 02:10 PM, Mandell Degerness wrote: >> >> We are seeing a somewhat random, but frequent hang on our systems >> during startup. The hang happens at the point where an "rbd map >> " command is run. >> >> I've attached the ceph logs from the cluster. The map command happens >> at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can >> be seen in the log as 172.18.0.15:0/1143980479. >> >> It appears as if the TCP socket is opened to the OSD, but then times >> out 15 minutes later, the process gets data when the socket is closed >> on the client server and it retries. >> >> Please help. >> >> We are using ceph version 0.48.2argonaut >> (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). >> >> We are using a 3.5.7 kernel with the following list of patches applied: >> >> 1-libceph-encapsulate-out-message-data-setup.patch >> 2-libceph-dont-mark-footer-complete-before-it-is.patch >> 3-libceph-move-init-of-bio_iter.patch >> 4-libceph-dont-use-bio_iter-as-a-flag.patch >> 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch >> 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch >> 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch >> 8-libceph-protect-ceph_con_open-with-mutex.patch >> 9-libceph-reset-connection-retry-on-successfully-negotiation.patch >> 10-rbd-only-reset-capacity-when-pointing-to-head.patch >> 11-rbd-set-image-size-when-header-is-updated.patch >> 12-libceph-fix-crypto-key-null-deref-memory-leak.patch >> 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch >> 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch >> 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch >> 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch >> 17-libceph-check-for-invalid-mapping.patch >> 18-ceph-propagate-layout-error-on-osd-request-creation.patch >> 19-rbd-BUG-on-invalid-layout.patch >> 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch >> 21-ceph-avoid-32-bit-page-index-overflow.patch >> 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch >> >> Any suggestions? > > > The log shows your monitors don't have time sychronized enough among > them to make much progress (including authenticating new connections). > That's probably the real issue. 0.2s is pretty large clock drift. > > >> One thought is that the following patch (which we could not apply) is >> what is required: >> >> 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch > > > This is certainly useful too, but I don't think it's the cause of > the delay in this case. > > Josh > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rbd map command hangs for 15 minutes during system start up
On 11/08/2012 02:10 PM, Mandell Degerness wrote: We are seeing a somewhat random, but frequent hang on our systems during startup. The hang happens at the point where an "rbd map " command is run. I've attached the ceph logs from the cluster. The map command happens at Nov 8 18:41:09 on server 172.18.0.15. The process which hung can be seen in the log as 172.18.0.15:0/1143980479. It appears as if the TCP socket is opened to the OSD, but then times out 15 minutes later, the process gets data when the socket is closed on the client server and it retries. Please help. We are using ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe). We are using a 3.5.7 kernel with the following list of patches applied: 1-libceph-encapsulate-out-message-data-setup.patch 2-libceph-dont-mark-footer-complete-before-it-is.patch 3-libceph-move-init-of-bio_iter.patch 4-libceph-dont-use-bio_iter-as-a-flag.patch 5-libceph-resubmit-linger-ops-when-pg-mapping-changes.patch 6-libceph-re-initialize-bio_iter-on-start-of-message-receive.patch 7-ceph-close-old-con-before-reopening-on-mds-reconnect.patch 8-libceph-protect-ceph_con_open-with-mutex.patch 9-libceph-reset-connection-retry-on-successfully-negotiation.patch 10-rbd-only-reset-capacity-when-pointing-to-head.patch 11-rbd-set-image-size-when-header-is-updated.patch 12-libceph-fix-crypto-key-null-deref-memory-leak.patch 13-ceph-tolerate-and-warn-on-extraneous-dentry-from-mds.patch 14-ceph-avoid-divide-by-zero-in-__validate_layout.patch 15-rbd-drop-dev-reference-on-error-in-rbd_open.patch 16-ceph-Fix-oops-when-handling-mdsmap-that-decreases-max_mds.patch 17-libceph-check-for-invalid-mapping.patch 18-ceph-propagate-layout-error-on-osd-request-creation.patch 19-rbd-BUG-on-invalid-layout.patch 20-ceph-return-EIO-on-invalid-layout-on-GET_DATALOC-ioctl.patch 21-ceph-avoid-32-bit-page-index-overflow.patch 23-ceph-fix-dentry-reference-leak-in-encode_fh.patch Any suggestions? The log shows your monitors don't have time sychronized enough among them to make much progress (including authenticating new connections). That's probably the real issue. 0.2s is pretty large clock drift. One thought is that the following patch (which we could not apply) is what is required: 22-rbd-reset-BACKOFF-if-unable-to-re-queue.patch This is certainly useful too, but I don't think it's the cause of the delay in this case. Josh -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html