Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 12:53 PM, Chuck Hill wrote: On 2011-08-22, at 12:48 PM, Tim Worman wrote: On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot of projects being experiencing this same issue? Obviously, there are wo projects out there taking on a lot more load than mine. I have to think there is something about this specific deployment OR something about my specific requests that's causing my little hell here. Obviously, in wo, we don't construct our own requests in a fine-grained way so that is why I'm wondering about some of the elements on my pages. If it is a bug in the adaptor it has to be something very specific, maybe a size or the value of some bytes at the start or end or the phase of the moon or... What platform do you deploy on? What's the possibility that this could be related? http://www.h-online.com/open/news/item/Tool-causes-Apache-web-server-to-freeze-1330105.html Chuck ___ Do not post admin requests to the list. They will be ignored.
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
That does not sound like what I have observed. On 2011-08-24, at 11:42 AM, Tim Worman wrote: On Aug 22, 2011, at 12:53 PM, Chuck Hill wrote: On 2011-08-22, at 12:48 PM, Tim Worman wrote: On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot of projects being experiencing this same issue? Obviously, there are wo projects out there taking on a lot more load than mine. I have to think there is something about this specific deployment OR something about my specific requests that's causing my little hell here. Obviously, in wo, we don't construct our own requests in a fine-grained way so that is why I'm wondering about some of the elements on my pages. If it is a bug in the adaptor it has to be something very specific, maybe a size or the value of some bytes at the start or end or the phase of the moon or... What platform do you deploy on? What's the possibility that this could be related? http://www.h-online.com/open/news/item/Tool-causes-Apache-web-server-to-freeze-1330105.html Chuck --
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
Thanks Ramsey. It was owned by root:appservergrp. So, the adaptor is logging now. I'm not sure what I'm seeing but hopefully something will present itself. Tim Worman UCLA GSEIS On Aug 22, 2011, at 10:05 AM, Ramsey Gurley wrote: Check the permissions on that file and make sure the appserver user can write to it. Ramsey On Aug 22, 2011, at 10:00 AM, Tim Worman wrote: I've touched /tmp/logWebObjects and I've uncommented the log level lines in /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf. I've set it at the debug level. So, nothing is going into the logs. Is there something I need to sacrifice now to actually get this to log like the instructions say? :-) Tim Worman UCLA GSEIS On Aug 21, 2011, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. Tim Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com This email sent to li...@thetimmy.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/rgurley%40smarthealth.com This email sent to
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot of projects being experiencing this same issue? Obviously, there are wo projects out there taking on a lot more load than mine. I have to think there is something about this specific deployment OR something about my specific requests that's causing my little hell here. Obviously, in wo, we don't construct our own requests in a fine-grained way so that is why I'm wondering about some of the elements on my pages. Tim ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-22, at 12:48 PM, Tim Worman wrote: On Aug 22, 2011, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? One other thing - if this was a bug in mod_webobjects, wouldn't a whole lot of projects being experiencing this same issue? Obviously, there are wo projects out there taking on a lot more load than mine. I have to think there is something about this specific deployment OR something about my specific requests that's causing my little hell here. Obviously, in wo, we don't construct our own requests in a fine-grained way so that is why I'm wondering about some of the elements on my pages. If it is a bug in the adaptor it has to be something very specific, maybe a size or the value of some bytes at the start or end or the phase of the moon or... What platform do you deploy on? Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-22, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) Then extra, extra hard. I do know C, or did, but that code is little much for what I remember. I can sort of follow how it works. Finding bugs in it... not so much. I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there a newer version of Apache or mod_webobjects you can try? Are you using the builds from mDimension's site? Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? I don't think so. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 22, 2011, at 1:01 PM, Chuck Hill wrote: On 2011-08-22, at 11:51 AM, Tim Worman wrote: On Aug 22, 2011, at 11:38 AM, Chuck Hill wrote: On 2011-08-21, at 12:43 PM, Tim Worman wrote: On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. You could try the CGI adaptor and see if that makes any difference. If it IS a bug in mod_webobjects... that could be hard to find and fix. Yeah, I'd say so. I don't even know C. :-) Then extra, extra hard. I do know C, or did, but that code is little much for what I remember. I can sort of follow how it works. Finding bugs in it... not so much. I'm gonna pursue some other things first - like a possible hardware issue, or other software. I'm going to try a deployment on another server and see if that has any effect - also without shibboleth. Is there a newer version of Apache or mod_webobjects you can try? Are you using the builds from mDimension's site? Yes, I got the one linked from the wiki which is hosted on mDimension. http://webobjects.mdimension.com/wonder/mod_WebObjects/Apache2.2/macosx/10.6/mod_WebObjects.so I updated it again yesterday just hoping that maybe I had something old that had a fix I needed. Is there any chance that an issue with the id's of page elements could cause an issue like this - say if they're dynamic and the response doesn't find a matching id? I don't think so. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: app seems to trigger runaway apache threads (was Troubleshooting help)
On Aug 21, 2011, at 11:52 AM, Chuck Hill wrote: On 2011-08-20, at 4:02 PM, Tim Worman wrote: Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? I've seen something like this. It appeared that the woadaptor (i.e. mod_webobjects) did not believe that it had received all of the response from the application. The app had nothing more to send and so the woadaptor just hung there waiting for data that would never come. I did not track down why this happened, but it did seem to be load related. My suspicion was that there is a concurrency bug in the woadaptor. I'm really at a loss about what to do about it. It's only gotten worse as I've included more ajax actions in my app - and, of course, I don't experience this behavior in development. I just deployed a major update to my app - pretty much unaware that a small problem was going to become a big problem with the new version. In one example, a have a calendar where clicking on a day simply calls an AjaxUpdate marking that date as selected to the calendar. The result also has to update the entire page though because other things on the page need to change in those circumstances. This alone can cause the issue - but not always. And it happens even when I'm the only logged in user - so the load isn't high. As one solution, I've considered rolling a custom apache instead of using Apple's. But since the server also runs shibboleth, the setup isn't exactly simple. But I'm really not sure how to ascertain if the problem is the woadaptor or how I can settle it. Tim Chuck -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
app seems to trigger runaway apache threads (was Troubleshooting help)
Back in January I started this discussion on this same topic: http://lists.apple.com/archives/webobjects-dev/2011/Jan/msg00224.html I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. This is an update to my original post hoping to see if there are anymore thoughts on origin. More recently, I have been able to reproduce the issue in my own usage of the app - something I wasn't able to do before. It seems to be easier to generate the issue now that there are more ajax requests. The methods executed by these requests are not intensive or long responses and should return a result in seconds. Some symptoms: - When the actions are executed, busy indicators properly spin while the browser awaits a response from the server. When the issue occurs, the response never comes. - while continuing to await a response there is concurrently an httpd process that pegs he processor at 100% - if I kill the process on the server, the browser immediately updates properly as if the request had run properly It's almost as if apache is somehow receiving an ill-formed request and chokes on it. The problem is, there are no errors in the console or anything strange in any apache logs. Has anyone ever seen behavior like this or have any ideas as to how I could analyze it further? Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
I've seen this too. I'll walk by the server, see all the lights on, scratch my head, run top and there's httpd working very hard at something. Whatever it is, it's very intermittent and seems to coincide when I am truly handling lots of resource requests, but there is obviously something else going on. Whatever processes they are, they must be nice because I can't discern a slowdown in general responsiveness. Then as soon as it started, it stops; maybe an hour or two though. About once a week I stand in front of the server and compliment it on its disposition, work ethic, etc. and it seems to be satisfied. :-/ John On Jan 19, 2011, at 12:50 PM, Tim Worman li...@thetimmy.com wrote: I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/the_larsons%40mac.com This email sent to the_lars...@mac.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
On Jan 20, 2011, at 1:28 PM, Klaus Berkling wrote: On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: Thanks Chuck. It's really disconcerting since I have had times where I checked on the server and it has 12 hung httpd threads consuming 100% processor. I am running the built-in version on Snow Leopard. It reports that it is 2.2.14. If it is WO specific, the same app did not exhibit this behavior on another server. I'm considering whether I could log every request the app generates and see if I can match up the request (or even the method that was run) with the time that the hung process was spawned? Or maybe httpd has some advanced logging that could tell me something. Love to hear if anyone else has seen anything similar and how it got addressed. Take a look at the min. spare servers, max spare servers, and num. servers to start. I keep these numbers low (10/10/10) to avoid old httpd processes. Theory is that a new connection gets a new, or not as old of an httpd server. My servers are not normal so YMMV. There is a lot of voodoo in deployments settings... Thanks Klaus. I think that is good advice and I will likely put some limits on those variables. I did run a backtrace on httpd and it does mention WebObjects. I am not familiar with reading gdb output though. Maybe someone here has some clues as to what might be happening. (gdb) where #0 0x7fff85d10b96 in sendmsg () #1 0x00010039136a in sendBuffers () #2 0x000100390b8e in req_sendRequest () #3 0x00010038d87f in tr_handleRequest () #4 0x0001003997dc in WebObjects_handler () #5 0x000112af in ap_run_handler () #6 0x00011b8e in ap_invoke_handler () #7 0x00010002e784 in ap_process_request () #8 0x00010002aff7 in ap_process_http_connection () #9 0x00010001280b in ap_run_process_connection () #10 0x000100012ca5 in ap_process_connection () #11 0x0001000353f2 in child_main () #12 0x0001000355a8 in make_child () #13 0x00010003582e in perform_idle_server_maintenance () #14 0x000100035d58 in ap_mpm_run () #15 0x000198f5 in main () (gdb) I can second Chuck's experience with Apache 1.3. kib The era of procrastination, of half-measures, of soothing and baffling expedients, of delays, is coming to its close. In its place we are entering a period of consequences. Winston Churchill Klaus Berkling Web Application Dev. Systems Administrator DynEd International, Inc. www.dyned.com | blog ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/lists%40thetimmy.com This email sent to li...@thetimmy.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: Thanks Chuck. It's really disconcerting since I have had times where I checked on the server and it has 12 hung httpd threads consuming 100% processor. I am running the built-in version on Snow Leopard. It reports that it is 2.2.14. If it is WO specific, the same app did not exhibit this behavior on another server. I'm considering whether I could log every request the app generates and see if I can match up the request (or even the method that was run) with the time that the hung process was spawned? Or maybe httpd has some advanced logging that could tell me something. Love to hear if anyone else has seen anything similar and how it got addressed. Take a look at the min. spare servers, max spare servers, and num. servers to start. I keep these numbers low (10/10/10) to avoid old httpd processes. Theory is that a new connection gets a new, or not as old of an httpd server. My servers are not normal so YMMV. There is a lot of voodoo in deployments settings... I can second Chuck's experience with Apache 1.3. kib The era of procrastination, of half-measures, of soothing and baffling expedients, of delays, is coming to its close. In its place we are entering a period of consequences. Winston Churchill Klaus Berkling Web Application Dev. Systems Administrator DynEd International, Inc. www.dyned.com | blog ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
troubleshooting help
I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
We have seen this with Apache 1.3. I don't think it was a WO specific issue though. Chuck On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net This email sent to ch...@global-village.net -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects smime.p7s Description: S/MIME cryptographic signature ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
Thanks Chuck. It's really disconcerting since I have had times where I checked on the server and it has 12 hung httpd threads consuming 100% processor. I am running the built-in version on Snow Leopard. It reports that it is 2.2.14. If it is WO specific, the same app did not exhibit this behavior on another server. I'm considering whether I could log every request the app generates and see if I can match up the request (or even the method that was run) with the time that the hung process was spawned? Or maybe httpd has some advanced logging that could tell me something. Love to hear if anyone else has seen anything similar and how it got addressed. Tim Worman UCLA GSEIS On Jan 19, 2011, at 10:59 AM, Chuck Hill wrote: We have seen this with Apache 1.3. I don't think it was a WO specific issue though. Chuck On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net This email sent to ch...@global-village.net -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: troubleshooting help
Have you checked the Apache error logs and the console logs? Run the hardware tests? Run drive diagnostics? I'd start looking at that. This is definitely not normal for WO 5.4.3 on Snow Leopard. Chuck On Jan 19, 2011, at 1:46 PM, Tim Worman wrote: Thanks Chuck. It's really disconcerting since I have had times where I checked on the server and it has 12 hung httpd threads consuming 100% processor. I am running the built-in version on Snow Leopard. It reports that it is 2.2.14. If it is WO specific, the same app did not exhibit this behavior on another server. I'm considering whether I could log every request the app generates and see if I can match up the request (or even the method that was run) with the time that the hung process was spawned? Or maybe httpd has some advanced logging that could tell me something. Love to hear if anyone else has seen anything similar and how it got addressed. Tim Worman UCLA GSEIS On Jan 19, 2011, at 10:59 AM, Chuck Hill wrote: We have seen this with Apache 1.3. I don't think it was a WO specific issue though. Chuck On Jan 19, 2011, at 10:50 AM, Tim Worman wrote: I have an app that, during the course of normal usage, is starting httpd processes on the server that instantly hit 100% CPU usage of one core. This can happen multiple times during times when the app is under heavier load. After some time I can have many httpd processes where TOP reports each using 100% of a core. When I try to log into the app and poke around to try and reproduce the issue, I am unable. Does anyone have any pointers as to how I can try to narrow down specific requests or conditions that may be causing these processes to be spawned? This app was just recently moved to MOSXS/Intel. Previously it ran on MOSXS 10.5/PowerPC and I didn't see the same problems there. Truthfully though, it could be caused by some other change I made since but I thought the architecture could be an important factor if someone else has seen similar symptoms. Tim Worman UCLA GSEIS ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net This email sent to ch...@global-village.net -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/products/practical_webobjects smime.p7s Description: S/MIME cryptographic signature ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
