Re: app seems to trigger runaway apache threads (was Troubleshooting help)
That does not sound like what I have observed. On 2011-08-24, at 11:42 AM, Tim Worman wrote: > On Aug 22, 2011, at 12:53 PM, Chuck Hill wrote: > >> On 2011-08-22, at 12:48 PM, Tim Worman wrote: >>> On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: >>> On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: > On 2011-08-21, at 12:43 PM, Tim Worman wrote: >> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >>> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >>> Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting >> httpd processes on the server that instantly hit 100% CPU usage of >> one core. This can happen multiple times during times when the app >> is under heavier load. After some time I can have many httpd >> processes where TOP reports each using 100% of a core. When I try to >> log into the app and poke around to try and reproduce the issue, I >> am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? >>> >>> I've seen something like this. It appeared that the woadaptor (i.e. >>> mod_webobjects) did not believe that it had received all of the >>> response from the application. The app had nothing more to send and so >>> the woadaptor just hung there waiting for data that would never come. >>> I did not track down why this happened, but it did seem to be load >>> related. My suspicion was that there is a concurrency bug in the >>> woadaptor. >> >> I'm really at a loss about what to do about it. It's only gotten worse >> as I've included more ajax actions in my app - and, of course, I don't >> experience this behavior in development. I just deployed a major update >> to my app - pretty much unaware that a small problem was going to become >> a big problem with the new version. >> >> In one example, a have a calendar where clicking on a day simply calls >> an AjaxUpdate marking that date as selected to the calendar. The result >> also has to update the entire page though because other things on the >> page need to change in those circumstances. This alone can cause the >> issue - but not always. And it happens even when I'm the only logged in >> user - so the load isn't high. >> >> As one solution, I've considered rolling a custom apache instead of >> using Apple's. But since the server also runs shibboleth, the setup >> isn't exactly simple. But I'm really not sure how to ascertain if the >> problem is the woadaptor or how I can settle it. > > > You could try the CGI adaptor and see if that makes any difference. If > it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? >>> >>> One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot >>> of projects being experiencing this same issue? Obviously, there are wo >>> projects out there taking on a lot more load than mine. I have to think >>> there is something about this specific deployment OR something about my >>> specific requests that's causing my
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 12:53 PM, Chuck Hill wrote: > On 2011-08-22, at 12:48 PM, Tim Worman wrote: >> On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: >> On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: > On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >> >>> Back in January I started this discussion on this same topic: >>> >>> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >>> > I have an app that, during the course of normal usage, is starting > httpd processes on the server that instantly hit 100% CPU usage of > one core. This can happen multiple times during times when the app is > under heavier load. After some time I can have many httpd processes > where TOP reports each using 100% of a core. When I try to log into > the app and poke around to try and reproduce the issue, I am unable. >>> >>> This is an update to my original post hoping to see if there are >>> anymore thoughts on origin. More recently, I have been able to >>> reproduce the issue in my own usage of the app - something I wasn't >>> able to do before. It seems to be easier to generate the issue now that >>> there are more ajax requests. The methods executed by these requests >>> are not intensive or long responses and should return a result in >>> seconds. Some symptoms: >>> >>> - When the actions are executed, busy indicators properly spin while >>> the browser awaits a response from the server. When the issue occurs, >>> the response never comes. >>> - while continuing to await a response there is concurrently an httpd >>> process that pegs he processor at 100% >>> - if I kill the process on the server, the browser immediately updates >>> properly as if the request had run properly >>> >>> It's almost as if apache is somehow receiving an ill-formed request and >>> chokes on it. The problem is, there are no errors in the console or >>> anything strange in any apache logs. Has anyone ever seen behavior like >>> this or have any ideas as to how I could analyze it further? >> >> I've seen something like this. It appeared that the woadaptor (i.e. >> mod_webobjects) did not believe that it had received all of the response >> from the application. The app had nothing more to send and so the >> woadaptor just hung there waiting for data that would never come. I did >> not track down why this happened, but it did seem to be load related. >> My suspicion was that there is a concurrency bug in the woadaptor. > > I'm really at a loss about what to do about it. It's only gotten worse as > I've included more ajax actions in my app - and, of course, I don't > experience this behavior in development. I just deployed a major update > to my app - pretty much unaware that a small problem was going to become > a big problem with the new version. > > In one example, a have a calendar where clicking on a day simply calls an > AjaxUpdate marking that date as selected to the calendar. The result also > has to update the entire page though because other things on the page > need to change in those circumstances. This alone can cause the issue - > but not always. And it happens even when I'm the only logged in user - so > the load isn't high. > > As one solution, I've considered rolling a custom apache instead of using > Apple's. But since the server also runs shibboleth, the setup isn't > exactly simple. But I'm really not sure how to ascertain if the problem > is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. >>> >>> Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other >>> things first - like a possible hardware issue, or other software. I'm going >>> to try a deployment on another server and see if that has any effect - also >>> without shibboleth. Is there any chance that an issue with the id's of page >>> elements could cause an issue like this - say if they're dynamic and the >>> response doesn't find a matching id? >> >> One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot >> of projects being experiencing this same issue? Obviously, there are wo >> projects out there taking on a lot more load than mine. I have to think >> there is something about this specific deployment OR something about my >> specific requests that's causing my little hell here. Obviously, in wo, we >> don't construct our own requests in a fine-grained way so that is why I'm >> wondering about some of the elements on my pages. > > > If it is a bug in the
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 1:01 PM, Chuck Hill wrote: > On 2011-08-22, at 11:51 AM, Tim Worman wrote: >> On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: >> On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: > On 2011-08-20, at 4:02 PM, Tim Worman wrote: > >> Back in January I started this discussion on this same topic: >> >> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. >> >> This is an update to my original post hoping to see if there are anymore >> thoughts on origin. More recently, I have been able to reproduce the >> issue in my own usage of the app - something I wasn't able to do before. >> It seems to be easier to generate the issue now that there are more ajax >> requests. The methods executed by these requests are not intensive or >> long responses and should return a result in seconds. Some symptoms: >> >> - When the actions are executed, busy indicators properly spin while the >> browser awaits a response from the server. When the issue occurs, the >> response never comes. >> - while continuing to await a response there is concurrently an httpd >> process that pegs he processor at 100% >> - if I kill the process on the server, the browser immediately updates >> properly as if the request had run properly >> >> It's almost as if apache is somehow receiving an ill-formed request and >> chokes on it. The problem is, there are no errors in the console or >> anything strange in any apache logs. Has anyone ever seen behavior like >> this or have any ideas as to how I could analyze it further? > > I've seen something like this. It appeared that the woadaptor (i.e. > mod_webobjects) did not believe that it had received all of the response > from the application. The app had nothing more to send and so the > woadaptor just hung there waiting for data that would never come. I did > not track down why this happened, but it did seem to be load related. My > suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. >>> >>> >>> You could try the CGI adaptor and see if that makes any difference. If it >>> IS a bug in mod_webobjects... that could be hard to find and fix. >> >> Yeah, I'd say so. I don't even know C. :-) > > Then extra, extra hard. I do know C, or did, but that code is little much for > what I remember. I can sort of follow how it works. Finding bugs in it... > not so much. > > >> I'm gonna pursue some other things first - like a possible hardware issue, >> or other software. I'm going to try a deployment on another server and see >> if that has any effect - also without shibboleth. > > Is there a newer version of Apache or mod_webobjects you can try? Are you > using the builds from mDimension's site? Yes, I got the one linked from the wiki which is hosted on mDimension. http://webobjects.mdimension.com/wonder/mod_WebObjects/Apache2.2/macosx/10.6/mod_WebObjects.so I updated it again yesterday just hoping that maybe I had something old that had a fix I needed. >> Is there any chance that an issue with the id's of page elements could cause >> an issue like this - say if they're dynamic and the response doesn't find a >> matching id? > > I don't think so. > > Chuck > > > -- > Chuck Hill Senior Consultant / VP Development >
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 12:53 PM, Chuck Hill wrote: > On 2011-08-22, at 12:48 PM, Tim Worman wrote: >> On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: >> On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: > On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >> >>> Back in January I started this discussion on this same topic: >>> >>> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >>> > I have an app that, during the course of normal usage, is starting > httpd processes on the server that instantly hit 100% CPU usage of > one core. This can happen multiple times during times when the app is > under heavier load. After some time I can have many httpd processes > where TOP reports each using 100% of a core. When I try to log into > the app and poke around to try and reproduce the issue, I am unable. >>> >>> This is an update to my original post hoping to see if there are >>> anymore thoughts on origin. More recently, I have been able to >>> reproduce the issue in my own usage of the app - something I wasn't >>> able to do before. It seems to be easier to generate the issue now that >>> there are more ajax requests. The methods executed by these requests >>> are not intensive or long responses and should return a result in >>> seconds. Some symptoms: >>> >>> - When the actions are executed, busy indicators properly spin while >>> the browser awaits a response from the server. When the issue occurs, >>> the response never comes. >>> - while continuing to await a response there is concurrently an httpd >>> process that pegs he processor at 100% >>> - if I kill the process on the server, the browser immediately updates >>> properly as if the request had run properly >>> >>> It's almost as if apache is somehow receiving an ill-formed request and >>> chokes on it. The problem is, there are no errors in the console or >>> anything strange in any apache logs. Has anyone ever seen behavior like >>> this or have any ideas as to how I could analyze it further? >> >> I've seen something like this. It appeared that the woadaptor (i.e. >> mod_webobjects) did not believe that it had received all of the response >> from the application. The app had nothing more to send and so the >> woadaptor just hung there waiting for data that would never come. I did >> not track down why this happened, but it did seem to be load related. >> My suspicion was that there is a concurrency bug in the woadaptor. > > I'm really at a loss about what to do about it. It's only gotten worse as > I've included more ajax actions in my app - and, of course, I don't > experience this behavior in development. I just deployed a major update > to my app - pretty much unaware that a small problem was going to become > a big problem with the new version. > > In one example, a have a calendar where clicking on a day simply calls an > AjaxUpdate marking that date as selected to the calendar. The result also > has to update the entire page though because other things on the page > need to change in those circumstances. This alone can cause the issue - > but not always. And it happens even when I'm the only logged in user - so > the load isn't high. > > As one solution, I've considered rolling a custom apache instead of using > Apple's. But since the server also runs shibboleth, the setup isn't > exactly simple. But I'm really not sure how to ascertain if the problem > is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. >>> >>> Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other >>> things first - like a possible hardware issue, or other software. I'm going >>> to try a deployment on another server and see if that has any effect - also >>> without shibboleth. Is there any chance that an issue with the id's of page >>> elements could cause an issue like this - say if they're dynamic and the >>> response doesn't find a matching id? >> >> One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot >> of projects being experiencing this same issue? Obviously, there are wo >> projects out there taking on a lot more load than mine. I have to think >> there is something about this specific deployment OR something about my >> specific requests that's causing my little hell here. Obviously, in wo, we >> don't construct our own requests in a fine-grained way so that is why I'm >> wondering about some of the elements on my pages. > > > If it is a bug in the
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-22, at 11:51 AM, Tim Worman wrote: > On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: > On 2011-08-21, at 12:43 PM, Tim Worman wrote: >>> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: > Back in January I started this discussion on this same topic: > > http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html > >>> I have an app that, during the course of normal usage, is starting >>> httpd processes on the server that instantly hit 100% CPU usage of one >>> core. This can happen multiple times during times when the app is under >>> heavier load. After some time I can have many httpd processes where TOP >>> reports each using 100% of a core. When I try to log into the app and >>> poke around to try and reproduce the issue, I am unable. > > This is an update to my original post hoping to see if there are anymore > thoughts on origin. More recently, I have been able to reproduce the > issue in my own usage of the app - something I wasn't able to do before. > It seems to be easier to generate the issue now that there are more ajax > requests. The methods executed by these requests are not intensive or > long responses and should return a result in seconds. Some symptoms: > > - When the actions are executed, busy indicators properly spin while the > browser awaits a response from the server. When the issue occurs, the > response never comes. > - while continuing to await a response there is concurrently an httpd > process that pegs he processor at 100% > - if I kill the process on the server, the browser immediately updates > properly as if the request had run properly > > It's almost as if apache is somehow receiving an ill-formed request and > chokes on it. The problem is, there are no errors in the console or > anything strange in any apache logs. Has anyone ever seen behavior like > this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. >>> >>> I'm really at a loss about what to do about it. It's only gotten worse as >>> I've included more ajax actions in my app - and, of course, I don't >>> experience this behavior in development. I just deployed a major update to >>> my app - pretty much unaware that a small problem was going to become a big >>> problem with the new version. >>> >>> In one example, a have a calendar where clicking on a day simply calls an >>> AjaxUpdate marking that date as selected to the calendar. The result also >>> has to update the entire page though because other things on the page need >>> to change in those circumstances. This alone can cause the issue - but not >>> always. And it happens even when I'm the only logged in user - so the load >>> isn't high. >>> >>> As one solution, I've considered rolling a custom apache instead of using >>> Apple's. But since the server also runs shibboleth, the setup isn't exactly >>> simple. But I'm really not sure how to ascertain if the problem is the >>> woadaptor or how I can settle it. >> >> >> You could try the CGI adaptor and see if that makes any difference. If it >> IS a bug in mod_webobjects... that could be hard to find and fix. > > Yeah, I'd say so. I don't even know C. :-) Then extra, extra hard. I do know C, or did, but that code is little much for what I remember. I can sort of follow how it works. Finding bugs in it... not so much. > I'm gonna pursue some other things first - like a possible hardware issue, or > other software. I'm going to try a deployment on another server and see if > that has any effect - also without shibboleth. Is there a newer version of Apache or mod_webobjects you can try? Are you using the builds from mDimension's site? > Is there any chance that an issue with the id's of page elements could cause > an issue like this - say if they're dynamic and the response doesn't find a > matching id? I don't think so. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-22, at 12:48 PM, Tim Worman wrote: > On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: > On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: >>> On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: > On 2011-08-20, at 4:02 PM, Tim Worman wrote: > >> Back in January I started this discussion on this same topic: >> >> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. >> >> This is an update to my original post hoping to see if there are anymore >> thoughts on origin. More recently, I have been able to reproduce the >> issue in my own usage of the app - something I wasn't able to do before. >> It seems to be easier to generate the issue now that there are more ajax >> requests. The methods executed by these requests are not intensive or >> long responses and should return a result in seconds. Some symptoms: >> >> - When the actions are executed, busy indicators properly spin while the >> browser awaits a response from the server. When the issue occurs, the >> response never comes. >> - while continuing to await a response there is concurrently an httpd >> process that pegs he processor at 100% >> - if I kill the process on the server, the browser immediately updates >> properly as if the request had run properly >> >> It's almost as if apache is somehow receiving an ill-formed request and >> chokes on it. The problem is, there are no errors in the console or >> anything strange in any apache logs. Has anyone ever seen behavior like >> this or have any ideas as to how I could analyze it further? > > I've seen something like this. It appeared that the woadaptor (i.e. > mod_webobjects) did not believe that it had received all of the response > from the application. The app had nothing more to send and so the > woadaptor just hung there waiting for data that would never come. I did > not track down why this happened, but it did seem to be load related. My > suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. >>> >>> >>> You could try the CGI adaptor and see if that makes any difference. If it >>> IS a bug in mod_webobjects... that could be hard to find and fix. >> >> Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other >> things first - like a possible hardware issue, or other software. I'm going >> to try a deployment on another server and see if that has any effect - also >> without shibboleth. Is there any chance that an issue with the id's of page >> elements could cause an issue like this - say if they're dynamic and the >> response doesn't find a matching id? > > One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot > of projects being experiencing this same issue? Obviously, there are wo > projects out there taking on a lot more load than mine. I have to think there > is something about this specific deployment OR something about my specific > requests that's causing my little hell here. Obviously, in wo, we don't > construct our own requests in a fine-grained way so that is why I'm wondering > about some of the elements on my pages. If it is a bug in the adaptor it has to be something very specific, maybe a size or the value of some bytes at the start or end or the phase of the moon or... What p
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: > On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: > >> >> On 2011-08-21, at 12:43 PM, Tim Worman wrote: >> >>> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >>> On 2011-08-20, at 4:02 PM, Tim Worman wrote: > Back in January I started this discussion on this same topic: > > http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html > >>> I have an app that, during the course of normal usage, is starting >>> httpd processes on the server that instantly hit 100% CPU usage of one >>> core. This can happen multiple times during times when the app is under >>> heavier load. After some time I can have many httpd processes where TOP >>> reports each using 100% of a core. When I try to log into the app and >>> poke around to try and reproduce the issue, I am unable. > > This is an update to my original post hoping to see if there are anymore > thoughts on origin. More recently, I have been able to reproduce the > issue in my own usage of the app - something I wasn't able to do before. > It seems to be easier to generate the issue now that there are more ajax > requests. The methods executed by these requests are not intensive or > long responses and should return a result in seconds. Some symptoms: > > - When the actions are executed, busy indicators properly spin while the > browser awaits a response from the server. When the issue occurs, the > response never comes. > - while continuing to await a response there is concurrently an httpd > process that pegs he processor at 100% > - if I kill the process on the server, the browser immediately updates > properly as if the request had run properly > > It's almost as if apache is somehow receiving an ill-formed request and > chokes on it. The problem is, there are no errors in the console or > anything strange in any apache logs. Has anyone ever seen behavior like > this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. >>> >>> I'm really at a loss about what to do about it. It's only gotten worse as >>> I've included more ajax actions in my app - and, of course, I don't >>> experience this behavior in development. I just deployed a major update to >>> my app - pretty much unaware that a small problem was going to become a big >>> problem with the new version. >>> >>> In one example, a have a calendar where clicking on a day simply calls an >>> AjaxUpdate marking that date as selected to the calendar. The result also >>> has to update the entire page though because other things on the page need >>> to change in those circumstances. This alone can cause the issue - but not >>> always. And it happens even when I'm the only logged in user - so the load >>> isn't high. >>> >>> As one solution, I've considered rolling a custom apache instead of using >>> Apple's. But since the server also runs shibboleth, the setup isn't exactly >>> simple. But I'm really not sure how to ascertain if the problem is the >>> woadaptor or how I can settle it. >> >> >> You could try the CGI adaptor and see if that makes any difference. If it >> IS a bug in mod_webobjects... that could be hard to find and fix. > > Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things > first - like a possible hardware issue, or other software. I'm going to try a > deployment on another server and see if that has any effect - also without > shibboleth. Is there any chance that an issue with the id's of page elements > could cause an issue like this - say if they're dynamic and the response > doesn't find a matching id? One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot of projects being experiencing this same issue? Obviously, there are wo projects out there taking on a lot more load than mine. I have to think there is something about this specific deployment OR something about my specific requests that's causing my little hell here. Obviously, in wo, we don't construct our own requests in a fine-grained way so that is why I'm wondering about some of the elements on my pages. Tim ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/a
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: > > On 2011-08-21, at 12:43 PM, Tim Worman wrote: > >> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >> >>> >>> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >>> Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd >> processes on the server that instantly hit 100% CPU usage of one core. >> This can happen multiple times during times when the app is under >> heavier load. After some time I can have many httpd processes where TOP >> reports each using 100% of a core. When I try to log into the app and >> poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? >>> >>> I've seen something like this. It appeared that the woadaptor (i.e. >>> mod_webobjects) did not believe that it had received all of the response >>> from the application. The app had nothing more to send and so the >>> woadaptor just hung there waiting for data that would never come. I did >>> not track down why this happened, but it did seem to be load related. My >>> suspicion was that there is a concurrency bug in the woadaptor. >> >> I'm really at a loss about what to do about it. It's only gotten worse as >> I've included more ajax actions in my app - and, of course, I don't >> experience this behavior in development. I just deployed a major update to >> my app - pretty much unaware that a small problem was going to become a big >> problem with the new version. >> >> In one example, a have a calendar where clicking on a day simply calls an >> AjaxUpdate marking that date as selected to the calendar. The result also >> has to update the entire page though because other things on the page need >> to change in those circumstances. This alone can cause the issue - but not >> always. And it happens even when I'm the only logged in user - so the load >> isn't high. >> >> As one solution, I've considered rolling a custom apache instead of using >> Apple's. But since the server also runs shibboleth, the setup isn't exactly >> simple. But I'm really not sure how to ascertain if the problem is the >> woadaptor or how I can settle it. > > > You could try the CGI adaptor and see if that makes any difference. If it IS > a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? Tim Worman UCLA GSE&IS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
Thanks Ramsey. It was owned by root:appservergrp. So, the adaptor is logging now. I'm not sure what I'm seeing but hopefully something will present itself. Tim Worman UCLA GSE&IS On Aug 22, 2011, at 10:05 AM, Ramsey Gurley wrote: > Check the permissions on that file and make sure the appserver user can write > to it. > > Ramsey > > On Aug 22, 2011, at 10:00 AM, Tim Worman wrote: > >> I've touched /tmp/logWebObjects and I've uncommented the log level lines in >> /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf. I've set it at >> the debug level. So, nothing is going into the logs. Is there something I >> need to sacrifice now to actually get this to log like the instructions say? >> :-) >> >> Tim Worman >> UCLA GSE&IS >> >> >> On Aug 21, 2011, at 12:43 PM, Tim Worman wrote: >> >>> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >>> On 2011-08-20, at 4:02 PM, Tim Worman wrote: > Back in January I started this discussion on this same topic: > > http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html > >>> I have an app that, during the course of normal usage, is starting >>> httpd processes on the server that instantly hit 100% CPU usage of one >>> core. This can happen multiple times during times when the app is under >>> heavier load. After some time I can have many httpd processes where TOP >>> reports each using 100% of a core. When I try to log into the app and >>> poke around to try and reproduce the issue, I am unable. > > This is an update to my original post hoping to see if there are anymore > thoughts on origin. More recently, I have been able to reproduce the > issue in my own usage of the app - something I wasn't able to do before. > It seems to be easier to generate the issue now that there are more ajax > requests. The methods executed by these requests are not intensive or > long responses and should return a result in seconds. Some symptoms: > > - When the actions are executed, busy indicators properly spin while the > browser awaits a response from the server. When the issue occurs, the > response never comes. > - while continuing to await a response there is concurrently an httpd > process that pegs he processor at 100% > - if I kill the process on the server, the browser immediately updates > properly as if the request had run properly > > It's almost as if apache is somehow receiving an ill-formed request and > chokes on it. The problem is, there are no errors in the console or > anything strange in any apache logs. Has anyone ever seen behavior like > this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. >>> >>> I'm really at a loss about what to do about it. It's only gotten worse as >>> I've included more ajax actions in my app - and, of course, I don't >>> experience this behavior in development. I just deployed a major update to >>> my app - pretty much unaware that a small problem was going to become a big >>> problem with the new version. >>> >>> In one example, a have a calendar where clicking on a day simply calls an >>> AjaxUpdate marking that date as selected to the calendar. The result also >>> has to update the entire page though because other things on the page need >>> to change in those circumstances. This alone can cause the issue - but not >>> always. And it happens even when I'm the only logged in user - so the load >>> isn't high. >>> >>> As one solution, I've considered rolling a custom apache instead of using >>> Apple's. But since the server also runs shibboleth, the setup isn't exactly >>> simple. But I'm really not sure how to ascertain if the problem is the >>> woadaptor or how I can settle it. >>> >>> Tim >>> Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects >>> >>> ___ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com >>
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-21, at 12:43 PM, Tim Worman wrote: > On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: > >> >> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >> >>> Back in January I started this discussion on this same topic: >>> >>> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >>> > I have an app that, during the course of normal usage, is starting httpd > processes on the server that instantly hit 100% CPU usage of one core. > This can happen multiple times during times when the app is under heavier > load. After some time I can have many httpd processes where TOP reports > each using 100% of a core. When I try to log into the app and poke around > to try and reproduce the issue, I am unable. >>> >>> This is an update to my original post hoping to see if there are anymore >>> thoughts on origin. More recently, I have been able to reproduce the issue >>> in my own usage of the app - something I wasn't able to do before. It seems >>> to be easier to generate the issue now that there are more ajax requests. >>> The methods executed by these requests are not intensive or long responses >>> and should return a result in seconds. Some symptoms: >>> >>> - When the actions are executed, busy indicators properly spin while the >>> browser awaits a response from the server. When the issue occurs, the >>> response never comes. >>> - while continuing to await a response there is concurrently an httpd >>> process that pegs he processor at 100% >>> - if I kill the process on the server, the browser immediately updates >>> properly as if the request had run properly >>> >>> It's almost as if apache is somehow receiving an ill-formed request and >>> chokes on it. The problem is, there are no errors in the console or >>> anything strange in any apache logs. Has anyone ever seen behavior like >>> this or have any ideas as to how I could analyze it further? >> >> I've seen something like this. It appeared that the woadaptor (i.e. >> mod_webobjects) did not believe that it had received all of the response >> from the application. The app had nothing more to send and so the woadaptor >> just hung there waiting for data that would never come. I did not track >> down why this happened, but it did seem to be load related. My suspicion >> was that there is a concurrency bug in the woadaptor. > > I'm really at a loss about what to do about it. It's only gotten worse as > I've included more ajax actions in my app - and, of course, I don't > experience this behavior in development. I just deployed a major update to my > app - pretty much unaware that a small problem was going to become a big > problem with the new version. > > In one example, a have a calendar where clicking on a day simply calls an > AjaxUpdate marking that date as selected to the calendar. The result also has > to update the entire page though because other things on the page need to > change in those circumstances. This alone can cause the issue - but not > always. And it happens even when I'm the only logged in user - so the load > isn't high. > > As one solution, I've considered rolling a custom apache instead of using > Apple's. But since the server also runs shibboleth, the setup isn't exactly > simple. But I'm really not sure how to ascertain if the problem is the > woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
Check the permissions on that file and make sure the appserver user can write to it. Ramsey On Aug 22, 2011, at 10:00 AM, Tim Worman wrote: > I've touched /tmp/logWebObjects and I've uncommented the log level lines in > /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf. I've set it at the > debug level. So, nothing is going into the logs. Is there something I need to > sacrifice now to actually get this to log like the instructions say? :-) > > Tim Worman > UCLA GSE&IS > > > On Aug 21, 2011, at 12:43 PM, Tim Worman wrote: > >> On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: >> >>> >>> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >>> Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd >> processes on the server that instantly hit 100% CPU usage of one core. >> This can happen multiple times during times when the app is under >> heavier load. After some time I can have many httpd processes where TOP >> reports each using 100% of a core. When I try to log into the app and >> poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? >>> >>> I've seen something like this. It appeared that the woadaptor (i.e. >>> mod_webobjects) did not believe that it had received all of the response >>> from the application. The app had nothing more to send and so the >>> woadaptor just hung there waiting for data that would never come. I did >>> not track down why this happened, but it did seem to be load related. My >>> suspicion was that there is a concurrency bug in the woadaptor. >> >> I'm really at a loss about what to do about it. It's only gotten worse as >> I've included more ajax actions in my app - and, of course, I don't >> experience this behavior in development. I just deployed a major update to >> my app - pretty much unaware that a small problem was going to become a big >> problem with the new version. >> >> In one example, a have a calendar where clicking on a day simply calls an >> AjaxUpdate marking that date as selected to the calendar. The result also >> has to update the entire page though because other things on the page need >> to change in those circumstances. This alone can cause the issue - but not >> always. And it happens even when I'm the only logged in user - so the load >> isn't high. >> >> As one solution, I've considered rolling a custom apache instead of using >> Apple's. But since the server also runs shibboleth, the setup isn't exactly >> simple. But I'm really not sure how to ascertain if the problem is the >> woadaptor or how I can settle it. >> >> Tim >> >>> >>> >>> Chuck >>> >>> -- >>> Chuck Hill Senior Consultant / VP Development >>> >>> Practical WebObjects - for developers who want to increase their overall >>> knowledge of WebObjects or who are trying to solve specific problems. >>> http://www.global-village.net/products/practical_webobjects >>> >>> >>> >>> >>> >>> >>> >> >> ___ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com >> >> This email sent to li...@thetimmy.com > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/webobjects-dev/rgurley%40smarthealth
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
I've touched /tmp/logWebObjects and I've uncommented the log level lines in /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf. I've set it at the debug level. So, nothing is going into the logs. Is there something I need to sacrifice now to actually get this to log like the instructions say? :-) Tim Worman UCLA GSE&IS On Aug 21, 2011, at 12:43 PM, Tim Worman wrote: > On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: > >> >> On 2011-08-20, at 4:02 PM, Tim Worman wrote: >> >>> Back in January I started this discussion on this same topic: >>> >>> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >>> > I have an app that, during the course of normal usage, is starting httpd > processes on the server that instantly hit 100% CPU usage of one core. > This can happen multiple times during times when the app is under heavier > load. After some time I can have many httpd processes where TOP reports > each using 100% of a core. When I try to log into the app and poke around > to try and reproduce the issue, I am unable. >>> >>> This is an update to my original post hoping to see if there are anymore >>> thoughts on origin. More recently, I have been able to reproduce the issue >>> in my own usage of the app - something I wasn't able to do before. It seems >>> to be easier to generate the issue now that there are more ajax requests. >>> The methods executed by these requests are not intensive or long responses >>> and should return a result in seconds. Some symptoms: >>> >>> - When the actions are executed, busy indicators properly spin while the >>> browser awaits a response from the server. When the issue occurs, the >>> response never comes. >>> - while continuing to await a response there is concurrently an httpd >>> process that pegs he processor at 100% >>> - if I kill the process on the server, the browser immediately updates >>> properly as if the request had run properly >>> >>> It's almost as if apache is somehow receiving an ill-formed request and >>> chokes on it. The problem is, there are no errors in the console or >>> anything strange in any apache logs. Has anyone ever seen behavior like >>> this or have any ideas as to how I could analyze it further? >> >> I've seen something like this. It appeared that the woadaptor (i.e. >> mod_webobjects) did not believe that it had received all of the response >> from the application. The app had nothing more to send and so the woadaptor >> just hung there waiting for data that would never come. I did not track >> down why this happened, but it did seem to be load related. My suspicion >> was that there is a concurrency bug in the woadaptor. > > I'm really at a loss about what to do about it. It's only gotten worse as > I've included more ajax actions in my app - and, of course, I don't > experience this behavior in development. I just deployed a major update to my > app - pretty much unaware that a small problem was going to become a big > problem with the new version. > > In one example, a have a calendar where clicking on a day simply calls an > AjaxUpdate marking that date as selected to the calendar. The result also has > to update the entire page though because other things on the page need to > change in those circumstances. This alone can cause the issue - but not > always. And it happens even when I'm the only logged in user - so the load > isn't high. > > As one solution, I've considered rolling a custom apache instead of using > Apple's. But since the server also runs shibboleth, the setup isn't exactly > simple. But I'm really not sure how to ascertain if the problem is the > woadaptor or how I can settle it. > > Tim > >> >> >> Chuck >> >> -- >> Chuck Hill Senior Consultant / VP Development >> >> Practical WebObjects - for developers who want to increase their overall >> knowledge of WebObjects or who are trying to solve specific problems. >> http://www.global-village.net/products/practical_webobjects >> >> >> >> >> >> >> > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com > > This email sent to li...@thetimmy.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: > > On 2011-08-20, at 4:02 PM, Tim Worman wrote: > >> Back in January I started this discussion on this same topic: >> >> http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. >> >> This is an update to my original post hoping to see if there are anymore >> thoughts on origin. More recently, I have been able to reproduce the issue >> in my own usage of the app - something I wasn't able to do before. It seems >> to be easier to generate the issue now that there are more ajax requests. >> The methods executed by these requests are not intensive or long responses >> and should return a result in seconds. Some symptoms: >> >> - When the actions are executed, busy indicators properly spin while the >> browser awaits a response from the server. When the issue occurs, the >> response never comes. >> - while continuing to await a response there is concurrently an httpd >> process that pegs he processor at 100% >> - if I kill the process on the server, the browser immediately updates >> properly as if the request had run properly >> >> It's almost as if apache is somehow receiving an ill-formed request and >> chokes on it. The problem is, there are no errors in the console or anything >> strange in any apache logs. Has anyone ever seen behavior like this or have >> any ideas as to how I could analyze it further? > > I've seen something like this. It appeared that the woadaptor (i.e. > mod_webobjects) did not believe that it had received all of the response from > the application. The app had nothing more to send and so the woadaptor just > hung there waiting for data that would never come. I did not track down why > this happened, but it did seem to be load related. My suspicion was that > there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. Tim > > > Chuck > > -- > Chuck Hill Senior Consultant / VP Development > > Practical WebObjects - for developers who want to increase their overall > knowledge of WebObjects or who are trying to solve specific problems. > http://www.global-village.net/products/practical_webobjects > > > > > > > ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-20, at 4:02 PM, Tim Worman wrote: > Back in January I started this discussion on this same topic: > > http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html > >>> I have an app that, during the course of normal usage, is starting httpd >>> processes on the server that instantly hit 100% CPU usage of one core. This >>> can happen multiple times during times when the app is under heavier load. >>> After some time I can have many httpd processes where TOP reports each >>> using 100% of a core. When I try to log into the app and poke around to try >>> and reproduce the issue, I am unable. > > This is an update to my original post hoping to see if there are anymore > thoughts on origin. More recently, I have been able to reproduce the issue in > my own usage of the app - something I wasn't able to do before. It seems to > be easier to generate the issue now that there are more ajax requests. The > methods executed by these requests are not intensive or long responses and > should return a result in seconds. Some symptoms: > > - When the actions are executed, busy indicators properly spin while the > browser awaits a response from the server. When the issue occurs, the > response never comes. > - while continuing to await a response there is concurrently an httpd process > that pegs he processor at 100% > - if I kill the process on the server, the browser immediately updates > properly as if the request had run properly > > It's almost as if apache is somehow receiving an ill-formed request and > chokes on it. The problem is, there are no errors in the console or anything > strange in any apache logs. Has anyone ever seen behavior like this or have > any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
app seems to trigger runaway apache threads (was Troubleshooting help)
Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html >> I have an app that, during the course of normal usage, is starting httpd >> processes on the server that instantly hit 100% CPU usage of one core. This >> can happen multiple times during times when the app is under heavier load. >> After some time I can have many httpd processes where TOP reports each using >> 100% of a core. When I try to log into the app and poke around to try and >> reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? Tim Worman UCLA GSE&IS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
I've seen this too. I'll walk by the server, see all the lights on, scratch my head, run top and there's httpd working very hard at something. Whatever it is, it's very intermittent and seems to coincide when I am truly handling lots of resource requests, but there is obviously something else going on. Whatever processes they are, they must be nice because I can't discern a slowdown in general responsiveness. Then as soon as it started, it stops; maybe an hour or two though. About once a week I stand in front of the server and compliment it on its disposition, work ethic, etc. and it seems to be satisfied. :-/ John On Jan 19, 2011, at 12:50 PM, Tim Worman wrote: > I have an app that, during the course of normal usage, is starting httpd > processes on the server that instantly hit 100% CPU usage of one core. This > can happen multiple times during times when the app is under heavier load. > After some time I can have many httpd processes where TOP reports each using > 100% of a core. When I try to log into the app and poke around to try and > reproduce the issue, I am unable. > > Does anyone have any pointers as to how I can try to narrow down specific > requests or conditions that may be causing these processes to be spawned? > > This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS > 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it > could be caused by some other change I made since but I thought the > architecture could be an important factor if someone else has seen similar > symptoms. > > Tim Worman > UCLA GSE&IS > > > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/webobjects-dev/the_larsons%40mac.com > > This email sent to the_lars...@mac.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
On Jan 20, 2011, at 1:28 PM, Klaus Berkling wrote: > > On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: > >> Thanks Chuck. It's really disconcerting since I have had times where I >> checked on the server and it has 12 hung httpd threads consuming 100% >> processor. >> >> I am running the built-in version on Snow Leopard. It reports that it is >> 2.2.14. If it is WO specific, the same app did not exhibit this behavior on >> another server. >> >> I'm considering whether I could log every request the app generates and see >> if I can match up the request (or even the method that was run) with the >> time that the hung process was spawned? Or maybe httpd has some advanced >> logging that could tell me something. >> >> Love to hear if anyone else has seen anything similar and how it got >> addressed. > > Take a look at the min. spare servers, max spare servers, and num. servers to > start. I keep these numbers low (10/10/10) to avoid old httpd processes. > Theory is that a new connection gets a new, or not as old of an httpd server. > My servers are not normal so YMMV. There is a lot of voodoo in deployments > settings... Thanks Klaus. I think that is good advice and I will likely put some limits on those variables. I did run a backtrace on httpd and it does mention WebObjects. I am not familiar with reading gdb output though. Maybe someone here has some clues as to what might be happening. (gdb) where #0 0x7fff85d10b96 in sendmsg () #1 0x00010039136a in sendBuffers () #2 0x000100390b8e in req_sendRequest () #3 0x00010038d87f in tr_handleRequest () #4 0x0001003997dc in WebObjects_handler () #5 0x000112af in ap_run_handler () #6 0x00011b8e in ap_invoke_handler () #7 0x00010002e784 in ap_process_request () #8 0x00010002aff7 in ap_process_http_connection () #9 0x00010001280b in ap_run_process_connection () #10 0x000100012ca5 in ap_process_connection () #11 0x0001000353f2 in child_main () #12 0x0001000355a8 in make_child () #13 0x00010003582e in perform_idle_server_maintenance () #14 0x000100035d58 in ap_mpm_run () #15 0x000198f5 in main () (gdb) > > I can second Chuck's experience with Apache 1.3. > > kib > > "The era of procrastination, of half-measures, of soothing and baffling > expedients, of delays, is coming to its close. In its place we are entering a > period of consequences." > Winston Churchill > > Klaus Berkling > Web Application Dev. & Systems Administrator > DynEd International, Inc. > www.dyned.com | blog > > > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com > > This email sent to li...@thetimmy.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: > Thanks Chuck. It's really disconcerting since I have had times where I > checked on the server and it has 12 hung httpd threads consuming 100% > processor. > > I am running the built-in version on Snow Leopard. It reports that it is > 2.2.14. If it is WO specific, the same app did not exhibit this behavior on > another server. > > I'm considering whether I could log every request the app generates and see > if I can match up the request (or even the method that was run) with the time > that the hung process was spawned? Or maybe httpd has some advanced logging > that could tell me something. > > Love to hear if anyone else has seen anything similar and how it got > addressed. Take a look at the min. spare servers, max spare servers, and num. servers to start. I keep these numbers low (10/10/10) to avoid old httpd processes. Theory is that a new connection gets a new, or not as old of an httpd server. My servers are not normal so YMMV. There is a lot of voodoo in deployments settings... I can second Chuck's experience with Apache 1.3. kib "The era of procrastination, of half-measures, of soothing and baffling expedients, of delays, is coming to its close. In its place we are entering a period of consequences." Winston Churchill Klaus Berkling Web Application Dev. & Systems Administrator DynEd International, Inc. www.dyned.com | blog ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
Have you checked the Apache error logs and the console logs? Run the hardware tests? Run drive diagnostics? I'd start looking at that. This is definitely not normal for WO 5.4.3 on Snow Leopard. Chuck On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: > Thanks Chuck. It's really disconcerting since I have had times where I > checked on the server and it has 12 hung httpd threads consuming 100% > processor. > > I am running the built-in version on Snow Leopard. It reports that it is > 2.2.14. If it is WO specific, the same app did not exhibit this behavior on > another server. > > I'm considering whether I could log every request the app generates and see > if I can match up the request (or even the method that was run) with the time > that the hung process was spawned? Or maybe httpd has some advanced logging > that could tell me something. > > Love to hear if anyone else has seen anything similar and how it got > addressed. > > Tim Worman > UCLA GSE&IS > > On Jan 19, 2011, at 10:59 AM, Chuck Hill wrote: > >> We have seen this with Apache 1.3. I don't think it was a WO specific issue >> though. >> >> Chuck >> >> >> On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: >> >>> I have an app that, during the course of normal usage, is starting httpd >>> processes on the server that instantly hit 100% CPU usage of one core. This >>> can happen multiple times during times when the app is under heavier load. >>> After some time I can have many httpd processes where TOP reports each >>> using 100% of a core. When I try to log into the app and poke around to try >>> and reproduce the issue, I am unable. >>> >>> Does anyone have any pointers as to how I can try to narrow down specific >>> requests or conditions that may be causing these processes to be spawned? >>> >>> This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS >>> 10.5/PowerPC and I didn't see the same problems there. Truthfully though, >>> it could be caused by some other change I made since but I thought the >>> architecture could be an important factor if someone else has seen similar >>> symptoms. >>> >>> Tim Worman >>> UCLA GSE&IS >>> >>> >>> >>> ___ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >>> >>> This email sent to ch...@global-village.net >> >> -- >> Chuck Hill Senior Consultant / VP Development >> >> Practical WebObjects - for developers who want to increase their overall >> knowledge of WebObjects or who are trying to solve specific problems. >> http://www.global-village.net/products/practical_webobjects >> >> >> >> >> >> >> > -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects smime.p7s Description: S/MIME cryptographic signature ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
Thanks Chuck. It's really disconcerting since I have had times where I checked on the server and it has 12 hung httpd threads consuming 100% processor. I am running the built-in version on Snow Leopard. It reports that it is 2.2.14. If it is WO specific, the same app did not exhibit this behavior on another server. I'm considering whether I could log every request the app generates and see if I can match up the request (or even the method that was run) with the time that the hung process was spawned? Or maybe httpd has some advanced logging that could tell me something. Love to hear if anyone else has seen anything similar and how it got addressed. Tim Worman UCLA GSE&IS On Jan 19, 2011, at 10:59 AM, Chuck Hill wrote: > We have seen this with Apache 1.3. I don't think it was a WO specific issue > though. > > Chuck > > > On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: > >> I have an app that, during the course of normal usage, is starting httpd >> processes on the server that instantly hit 100% CPU usage of one core. This >> can happen multiple times during times when the app is under heavier load. >> After some time I can have many httpd processes where TOP reports each using >> 100% of a core. When I try to log into the app and poke around to try and >> reproduce the issue, I am unable. >> >> Does anyone have any pointers as to how I can try to narrow down specific >> requests or conditions that may be causing these processes to be spawned? >> >> This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS >> 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it >> could be caused by some other change I made since but I thought the >> architecture could be an important factor if someone else has seen similar >> symptoms. >> >> Tim Worman >> UCLA GSE&IS >> >> >> >> ___ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >> >> This email sent to ch...@global-village.net > > -- > Chuck Hill Senior Consultant / VP Development > > Practical WebObjects - for developers who want to increase their overall > knowledge of WebObjects or who are trying to solve specific problems. > http://www.global-village.net/products/practical_webobjects > > > > > > > ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
We have seen this with Apache 1.3. I don't think it was a WO specific issue though. Chuck On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: > I have an app that, during the course of normal usage, is starting httpd > processes on the server that instantly hit 100% CPU usage of one core. This > can happen multiple times during times when the app is under heavier load. > After some time I can have many httpd processes where TOP reports each using > 100% of a core. When I try to log into the app and poke around to try and > reproduce the issue, I am unable. > > Does anyone have any pointers as to how I can try to narrow down specific > requests or conditions that may be causing these processes to be spawned? > > This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS > 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it > could be caused by some other change I made since but I thought the > architecture could be an important factor if someone else has seen similar > symptoms. > > Tim Worman > UCLA GSE&IS > > > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net > > This email sent to ch...@global-village.net -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects smime.p7s Description: S/MIME cryptographic signature ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
troubleshooting help
I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSE&IS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com