Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Tue, Jul 17, 2001 at 10:12:35AM -0700, Aaron Bannert wrote: - Optionally creating a child-pool for each thread. This provides us two things in my mind: 1) Allowing the application to choose an SMS for this thread-pool (which could very well be a special SMS created just for this purpose). for example, you may wish to create a special 'security' thread. it uses an apr_sms_mlock_t instance. this special thread handles all of your security, keeping private keys and plain-text user passwords in-memory. luke
RE: [PATCH] Problems with MPM threaded
As an end-user of apache, I'd like to chime in: please, please, please leave the N processes with M threads model available -- the hypothetical buggy 3rd party modules should be fixed rather than hacked around is fine in theory, but in practice I'd rather have an Apache that can soldier on regardless of the crappy modules it's forced to run. Tuxedo has used the N x M model for a long time with good results for similar reasons. My $0.02 Thanks, -b -Original Message- From: dean gaudet [mailto:[EMAIL PROTECTED]] Sent: Tuesday, July 17, 2001 4:13 AM To: [EMAIL PROTECTED] Subject: Re: [PATCH] Problems with MPM threaded On Sat, 14 Jul 2001 [EMAIL PROTECTED] wrote: Having multiple processes each with multiple threads provides for FAR more robustness than just a single process with multiple threads. ya know, i'm not really convinced of the desirability of this explanation anymore. maybe the hypothetical buggy 3rd party modules should be fixed rather than hacked around. and they won't be fixed as long as apache continues to hide the problem. however i can provide two more justifications which are pretty weak: - userland thread libraries need multiple processes in order to get disk i/o parallelism (and multi-cpu parallelism) - suppose you configure apache with exactly the number of children as you have processors. and suppose that somehow magically the CPU requirements of each process' threads is about even. then a good (kernel) scheduler will likely migrate each process to its own CPU and time slice just the threads within that process. this will tend to decrease the overall memory system traffic (because L2 caches won't be competing for the same process' memory), and increase the TLB hit rate. it may even be possible to avoid interprocessor interrupts every time the memory map is changed (to make sure that remote TLBs are up to date). that's a lot of conditionals to get right. -dean
Re: [PATCH] Problems with MPM threaded
On Sat, 14 Jul 2001 [EMAIL PROTECTED] wrote: Having multiple processes each with multiple threads provides for FAR more robustness than just a single process with multiple threads. ya know, i'm not really convinced of the desirability of this explanation anymore. maybe the hypothetical buggy 3rd party modules should be fixed rather than hacked around. and they won't be fixed as long as apache continues to hide the problem. however i can provide two more justifications which are pretty weak: - userland thread libraries need multiple processes in order to get disk i/o parallelism (and multi-cpu parallelism) - suppose you configure apache with exactly the number of children as you have processors. and suppose that somehow magically the CPU requirements of each process' threads is about even. then a good (kernel) scheduler will likely migrate each process to its own CPU and time slice just the threads within that process. this will tend to decrease the overall memory system traffic (because L2 caches won't be competing for the same process' memory), and increase the TLB hit rate. it may even be possible to avoid interprocessor interrupts every time the memory map is changed (to make sure that remote TLBs are up to date). that's a lot of conditionals to get right. -dean
Re: [PATCH] Problems with MPM threaded
On Sat, 14 Jul 2001, Roy T. Fielding wrote: The correct fix, as I see it, is to kill off the interprocess accept lock by removing the possibility of having other processes in a *threaded* MPM. -- justin That architecture was explored in detail by Netscape. It isn't reliable and slows your web server to a crawl whenever dynamic content is produced. It should only be used for static file servers and caching gateways, and people implementing those might as well use an in-kernel server like TUX. i'm confused... what architecture has that problem with dynamic content? i can believe it if you're referring to a userland threading library, single process server. i can't believe it if the threads are scheduled by the kernel (either as 1:1 or 1:many). NSPR threading was probably to blame, no? -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Normally this would be done (in POSIX) with pthread_cancel(), passing it the pthread_t from the other thread. Unfortunately, this is not a part of APR because many of the current OS implementations of this mechanism will leak resources (aparently in the kernel), and that is bad. -aaron On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote: On Sat, 14 Jul 2001, Sander Striker wrote: The way I see it, each process has a single pool instance as the parent for all the threads. Resetting or destroying that pool should effectively kill all threads. What am I missing? how does a thread kill another thread? -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
On Sun, 15 Jul 2001, Sander Striker wrote:
Why are we so desperate in opting out the child-pool creation?
I don't really have problems with a child pool for each thread. Actually,
it will make the dynamic locking a lot easier to implement if it stays.
all threads MUST have their own private pool root.
otherwise you're just throwing scalability away in locks. (which is
proved by the claim i saw that ian got better performance by defining
ALLOC_USE_MALLOC on an 8-way.)
I totally agree, but only as a solution in httpd.
I also believe that we should provide this [application-specific requirement]
outside of the basic thread support in APR.
Please allow me to use pseudocode:
void * worker_function(void * opaque_application_data) {
apr_pool_t *thread_pool;
create_child_pool(thread_pool, global_root_pool);
do_thread_stuff();
cleanup_child_pool(thread_pool);
if (apr_thread_exit_makes_sense_to_call_from_inside_worker_function) {
int rv = APR_SUCCESS;
apr_thread_exit(rv);
}
return NULL;
}
What I mean by this gibberish is that the pool-handling code can exist
completely outside of our thread_creation-type functions in APR. Is there
any reason for APR threads to do more than this (that is, just simple thread
creation)?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Uh...you knew that already, didn't you... duh... jeez now i'm the smartass ;) -aaron On Tue, Jul 17, 2001 at 08:43:18AM -0700, Aaron Bannert wrote: Normally this would be done (in POSIX) with pthread_cancel(), passing it the pthread_t from the other thread. Unfortunately, this is not a part of APR because many of the current OS implementations of this mechanism will leak resources (aparently in the kernel), and that is bad. -aaron On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote: On Sat, 14 Jul 2001, Sander Striker wrote: The way I see it, each process has a single pool instance as the parent for all the threads. Resetting or destroying that pool should effectively kill all threads. What am I missing? how does a thread kill another thread? -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
From: [EMAIL PROTECTED] Sent: Tuesday, July 17, 2001 11:13 AM I believe that the problem is that the threaded code is creating the pool, and not advertising it to the thread itself. This is an easy thing to fix. I do not agree that every APR app that uses threads should have to create their own thread pools. That is wasted effort by every APR app. I would like to see a conclusion to this thread quickly. So, could people please post their desires quickly, and what they want to see from this. thread-local pools would be a nice improvement, perhaps as an argument to apr_thread_create. They can be flagged such that they avoid locking their own own allocations (of course their 'parent' allocator, malloc, or the parent free-list management, may still need to lock based on platform and so forth). But _unless_ they remain rooted to the top level pool, in apr_process_create they become orphaned, and their cleanups are never executed. That isn't workable. It sounds like (forgive me if I'm stating the obvious) that a pool needs two kinds of parent sms pools. One is the 'heirarchy' for purposes of setup and teardown. The other is the 'allocator', where the blocks will be allocated from. Want to use the conventional 1.3 model? The heirarchial parent is the parent pool, but the allocatation parent is the single top level pool. Want to flip to a malloc, that trusts the clib to 'do the right thing'(sm) with cpu/thread specific free lists? Then we build a very simple malloc/free sms that can be passed as the allocation parent when the pool is created. If I've abused the sms concept here, then correct me, but I don't see any issues remaining beyond semantics and syntax. Bill
Re: Terminating threads in a process,WAS: RE: [PATCH] Problems with MPM threaded
William A. Rowe, Jr. wrote: From: [EMAIL PROTECTED] Sent: Tuesday, July 17, 2001 11:13 AM I believe that the problem is that the threaded code is creating the pool, and not advertising it to the thread itself. This is an easy thing to fix. I do not agree that every APR app that uses threads should have to create their own thread pools. That is wasted effort by every APR app. I would like to see a conclusion to this thread quickly. So, could people please post their desires quickly, and what they want to see from this. thread-local pools would be a nice improvement, perhaps as an argument to apr_thread_create. They can be flagged such that they avoid locking their own own allocations (of course their 'parent' allocator, malloc, or the parent free-list management, may still need to lock based on platform and so forth). But _unless_ they remain rooted to the top level pool, in apr_process_create they become orphaned, and their cleanups are never executed. That isn't workable. I'm not sure that the alternative is workable, either. At the time of the fork, when the child process gets a snapshot of the parent's memory, it's possible that some thread other than the one invoking fork could be halfway through registering a new resource (e.g., file descriptor) in its pool. There's no guarantee that it's safe to attempt a cleanup of that other thread's pool in the child process; if the fork has caught the data structures in an intermediate state, attempting to destroy that pool might yield a segv. --Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
I'm not sure that the alternative is workable, either. At the time of the fork, when the child process gets a snapshot of the parent's memory, it's possible that some thread other than the one invoking fork could be halfway through registering a new resource (e.g., file descriptor) in its pool. There's no guarantee that it's safe to attempt a cleanup of that other thread's pool in the child process; if the fork has caught the data structures in an intermediate state, attempting to destroy that pool might yield a segv. Correct me if I'm wrong, but that would be a property of a buggy MPM. If the MPM chooses to mix non-synchronized fork()s and thread invocation, than that's what it gets. -aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
We have hit an impass in my mind. Dean and I are saying that having each thread have it's own pool is a requirement. Not just for httpd, but for anything using pools. Dean, if I am mis-interpreting you, then I am sorry, and please correct me. Aaron, you disagree. you want each app to make that decision on their own. I see no way to implement that in a thread-safe manner. (Just as a side note, I don't remember seeing anything refuting the technical aspects of such an implementation. I would be interested in one, of course, if you think it is infeasible.) So, I would suggest that we approach this problem in another way. My ONLY goal for APR is to make a library that is useful for cross-platform development and eases the pain of cross-platform development for developers. That was the goal when I created APR, and my original test case was the Apache web server. I believe that the problem is that the threaded code is creating the pool, and not advertising it to the thread itself. This is an easy thing to fix. I do not agree that every APR app that uses threads should have to create their own thread pools. That is wasted effort by every APR app. I would like to see a conclusion to this thread quickly. So, could people please post their desires quickly, and what they want to see from this. In my mind, there are really only two major points that need addressing: - Optionally creating a child-pool for each thread. This provides us two things in my mind: 1) Allowing the application to choose an SMS for this thread-pool (which could very well be a special SMS created just for this purpose). 2) Allowing an application to refrain from creating a child-pool. - Like you [Ryan] said, it is a problem that the thread creation routines right now are not advertising the pool to the thread itself. This to me can be corrected with a better API. The two things I've brought up here are: 1) void * worker_fn(void *opaque_data) should have an additional pool param. 2) apr_thread_exit() is unacceptable. It currently requires a apr_thread_t structure. I would be happy if either the worker_fn was extended to pass in the apr_thread_t OR if we could architect a way for apr_thread_exit() to not need the apr_thread_t struct. I have a feeling the second point will be more palatable, and as such I'll volunteer to cook up a patch illustrating these changes. As for the first point, which is the reason this thread has been going on so long, I'm willing to compromise on the idea of forcefully creating a childpool always, if it necessary, merely because I see it only as overhead for some applications, but not a functional defect. -aaron
Re: Terminating threads in a process,WAS: RE: [PATCH] Problems with MPM threaded
Aaron Bannert wrote: I'm not sure that the alternative is workable, either. At the time of the fork, when the child process gets a snapshot of the parent's memory, it's possible that some thread other than the one invoking fork could be halfway through registering a new resource (e.g., file descriptor) in its pool. There's no guarantee that it's safe to attempt a cleanup of that other thread's pool in the child process; if the fork has caught the data structures in an intermediate state, attempting to destroy that pool might yield a segv. Correct me if I'm wrong, but that would be a property of a buggy MPM. If the MPM chooses to mix non-synchronized fork()s and thread invocation, than that's what it gets. Synchronizing the forks with pool resource registration isn't a scalable design; it requires locking a process-wide lock every time you want to register a resource in a per-thread pool. It would be a huge mistake to slow down the routine operation of the httpd in order to optimize for fork. --Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote: Aaron Bannert wrote: I'm not sure that the alternative is workable, either. At the time of the fork, when the child process gets a snapshot of the parent's memory, it's possible that some thread other than the one invoking fork could be halfway through registering a new resource (e.g., file descriptor) in its pool. There's no guarantee that it's safe to attempt a cleanup of that other thread's pool in the child process; if the fork has caught the data structures in an intermediate state, attempting to destroy that pool might yield a segv. Correct me if I'm wrong, but that would be a property of a buggy MPM. If the MPM chooses to mix non-synchronized fork()s and thread invocation, than that's what it gets. Synchronizing the forks with pool resource registration isn't a scalable design; it requires locking a process-wide lock every time you want to register a resource in a per-thread pool. It would be a huge mistake to slow down the routine operation of the httpd in order to optimize for fork. I wasn't suggesting that, I was suggesting that it's a bad idea to start fork()ing off processes from an MPM that's also doing threads and you want that fork()ed of process to later do cleanups. My point is it's solely an MPM architecture decision. Are you saying that this is a typical scenario? -aaron
Re: Terminating threads in a process,WAS: RE: [PATCH] Problems with MPM threaded
Aaron Bannert wrote: On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote: Aaron Bannert wrote: I'm not sure that the alternative is workable, either. At the time of the fork, when the child process gets a snapshot of the parent's memory, it's possible that some thread other than the one invoking fork could be halfway through registering a new resource (e.g., file descriptor) in its pool. There's no guarantee that it's safe to attempt a cleanup of that other thread's pool in the child process; if the fork has caught the data structures in an intermediate state, attempting to destroy that pool might yield a segv. Correct me if I'm wrong, but that would be a property of a buggy MPM. If the MPM chooses to mix non-synchronized fork()s and thread invocation, than that's what it gets. Synchronizing the forks with pool resource registration isn't a scalable design; it requires locking a process-wide lock every time you want to register a resource in a per-thread pool. It would be a huge mistake to slow down the routine operation of the httpd in order to optimize for fork. I wasn't suggesting that, I was suggesting that it's a bad idea to start fork()ing off processes from an MPM that's also doing threads and you want that fork()ed of process to later do cleanups. I agree. My point is it's solely an MPM architecture decision. Are you saying that this is a typical scenario? I think it's an atypical scenario, and one that can't be implemented very well anyway. (The synchronized approach has a performance problem, and the unsynchronized approach can crash due to a race condition.) Thus my argument is that this scenario (forking from a threaded MPM and then trying to clean up the other threads' resources in the child process) isn't a valid use case for the design of the pools. --Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
From: dean gaudet [EMAIL PROTECTED] Sent: Tuesday, July 17, 2001 6:15 PM if you assume that you want some form of notification, but you want to leave it unspecified because you're not sure what each apr thread will be used for, then you can make a somewhat generic kill off other threads cleanup. so for example, when an httpd thread is created it would register http_thread_cleanup which would use whatever magic global int die_now_please = 1, and release some condition variable, or throw something into a queue / and so on. that would be registered in the parent thread's pool -- and would only be invoked by the parent thread. pools let you do this, you don't need the mutexes for it, you just have to be explicit about parallelism. (combine that with a root pool per thread and then we can remove alloc_mutex and free lists and push the real gnarly problems into the libc malloc where it's probably best solved.) Yes, yes, yes. Can we please split the concept of a heirarchial parent (the 'creator' thread's or process pool, in this case) from the allocation parent (the actual give me memory for my pool from ... here!) Then we have an OS Knows Best malloc/free mpm for threading, just as you suggest. This solves your thread-specific requirements and our scoping issues, along with fixing the 'walk the chain of pools for a block' problem, both at once.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
[snip] that would be registered in the parent thread's pool -- and would only be invoked by the parent thread. pools let you do this, you don't need the mutexes for it, you just have to be explicit about parallelism. (combine that with a root pool per thread and then we can remove alloc_mutex and free lists and push the real gnarly problems into the libc malloc where it's probably best solved.) Yes, yes, yes. Can we please split the concept of a heirarchial parent (the 'creator' thread's or process pool, in this case) from the allocation parent (the actual give me memory for my pool from ... here!) Then we have an OS Knows Best malloc/free mpm for threading, just as you suggest. This solves your thread-specific requirements and our scoping issues, along with fixing the 'walk the chain of pools for a block' problem, both at once. It's probably just me, but I'm having trouble parsing this (I think I'm getting a cold :( ). Are you saying you want the thread function to have access to both a scope pool as well as an allocator pool, in case they are different? -aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
From: Aaron Bannert [EMAIL PROTECTED] Sent: Tuesday, July 17, 2001 6:41 PM [snip] that would be registered in the parent thread's pool -- and would only be invoked by the parent thread. pools let you do this, you don't need the mutexes for it, you just have to be explicit about parallelism. (combine that with a root pool per thread and then we can remove alloc_mutex and free lists and push the real gnarly problems into the libc malloc where it's probably best solved.) Yes, yes, yes. Can we please split the concept of a heirarchial parent (the 'creator' thread's or process pool, in this case) from the allocation parent (the actual give me memory for my pool from ... here!) Then we have an OS Knows Best malloc/free mpm for threading, just as you suggest. This solves your thread-specific requirements and our scoping issues, along with fixing the 'walk the chain of pools for a block' problem, both at once. It's probably just me, but I'm having trouble parsing this (I think I'm getting a cold :( ). Are you saying you want the thread function to have access to both a scope pool as well as an allocator pool, in case they are different? I've officially graduated to the rbb (insert a decent name here) club :-) Thank you, yes, scope pool defines teardowns such as cleanups, while the allocator pool addresses our performance concerns :) An obvious test is that 'allocator' is unique (e.g. thread private), a parent (at any depth) of the 'scope' pool, or the 'scope' pool itself.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
I totally agree, but only as a solution in httpd.
no, everywhere.
I also believe that we should provide this [application-specific requirement]
outside of the basic thread support in APR.
Please allow me to use pseudocode:
void * worker_function(void * opaque_application_data) {
apr_pool_t *thread_pool;
create_child_pool(thread_pool, global_root_pool);
now you've got mutexes in global_root_pool. see my performance comment
above.
I'm not advocating this scenario as a real solution, but merely trying
to illustrate how the child-pool creation routines don't really have to
be in apr_thread_create(). The application using APR threads can do this
before calling apr_thread_create() or it can do it in it's worker_function.
Either way, it is better to leave it totally up to the application, no?
-aaron
Re: [PATCH] Problems with MPM threaded
That architecture was explored in detail by Netscape. It isn't reliable and slows your web server to a crawl whenever dynamic content is produced. It should only be used for static file servers and caching gateways, and people implementing those might as well use an in-kernel server like TUX. i'm confused... what architecture has that problem with dynamic content? i can believe it if you're referring to a userland threading library, single process server. i can't believe it if the threads are scheduled by the kernel (either as 1:1 or 1:many). NSPR threading was probably to blame, no? Does NSPR do userland threading on Solaris? I thought they were using native threads. But you are right, the dynamic content problem was for libraries that are not thread-safe and for platforms that had userland threads. They ended up with some funky stuff in NSAPI that made the generic API slower than mollasses (internal modules used another API). Roy
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
I have some use cases that just want a thread. Since no child-pool is required in this case, it becomes unnecessary overhead (that is currently broken, as the child-pool is only cleaned in apr_thread_exit() but that whole thing is screwey). Again, that is incorrect. The THREAD-pool is cleaned whenever the parent pool is cleaned. Yes, there is some extra overhead, but remember that the problem with pools (as the originally existed, and are still implemented in the server), is that you can't have two threads working with the same pool. You are correct, I failed to mention that it is not *only* cleaned in apr_thread_exit(), but also when the parent pool is cleaned. It is, however, inappropriate to assume that the parent pool will ever get cleaned. Right now, if my application were to have a global/root pool that repeatedly created short-lived threads (as it does, BTW), it would eventually run out of memory. So, like I said, if apr_thread_exit() is not called (and in all of apr, apr-util, and httpd is is never called, not even in the test* programs), then that application is wholly dependent on the ability of the thread's parent pool to clean up the memory. IF SMS's replace all pools, then that might change. As things stand today though, that hasn't happened, and may not happen quickly. So if apr threads didn't know anything about child-pools, then they would just simply create a thread that called my worker_function(). I could pass in my opaque data to that worker_function(), and do the child-pool creations and destruction right there. I just don't see how child-pool creation MUST be in apr_thread_create() Yes, you could do that, but then EVERYBODY who wanted to create a thread would have to do that, and why are we forcing people to duplicate that code everytime a thread is created? I agree that this is a useful function *at some level*, and so I propose that this particular utility be built on top of a lower-level set of APR thread routines. I will even write these routines myself, just because I believe they are useful. -aaron
RE: [PATCH] Add intraprocess mutex to threaded MPM (WAS Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded)
Since a lot has happened in the mailing list, since the last time I checked (on Friday), I am responding only to the questions that still hold significance, from the original mail I had sent.. Do find my answers embedded below. -Original Message- From: Justin Erenkrantz [mailto:[EMAIL PROTECTED]] Sent: Friday, July 13, 2001 11:49 PM To: [EMAIL PROTECTED] Subject: Re: [PATCH] Problems with MPM threaded On Sat, Jul 14, 2001 at 12:42:29AM -0400, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote: snip Have you taken a look at the patch I posted that merges the POD code in threaded with the version in mpm_common.c? Threaded shouldn't be doing POD checks in threaded.c. It's redundant and it's done incorrectly anyway in the threaded MPM. Admittedly, this doesn't fix the issue of having a child who received a POD kill its sibling threads. More on that in a sec. Yes, I did try the patch - as you mention, it did not resolve the problem. Probably I was missing the right mpm_common.c file, since I had to tinker around with the code a little to make it work with your patch. Where can I find the right one? snip A pthread_cond_wait() and fcntl() are viable options according to the manpage. But, I'd much prefer us to use pthread_mutex_t if at all possible for cross-process locking. pthread_cond_wait() is what I was thinking of - I am still not convinced that it is a bad alternative (it does use pthread_mutex_t). I would try to make it as clean as possible. Lets see how it pans out. The current patch that you have posted is good, in the sense that it will work. It has one problem though: At any given time, there are only 5 worker threads (one from each child process), waiting to grab accept_mutex - others are waiting for their per-process worker_accept_mutex. Also, each thread would have to grab two locks, instead of one. I am sure there would be a noticeable performance hit at peak loads. Any comments? M
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote: APR threads, when created, would now take an additional parameter that is the mechanism (an sms implementation) by which it should create child pools. As it is now, the pool that is passed in to apr_thread_create() serves as the parent pool for the new thread-specific sms. If this parameter were null, the apr_thread_create() function would not create a sub-pool or register cleanup routines (which satisfies my requirements). if at all possible, the behaviour should match as closely as possible the existing situation, when this parameter is NULL, even if the current situation has bugs / is not very nice. this is mainly so that people do not object to having the behaviour of existing code disrupted. luke
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sun, Jul 15, 2001 at 06:49:51PM +0200, Luke Kenneth Casson Leighton wrote: On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote: APR threads, when created, would now take an additional parameter that is the mechanism (an sms implementation) by which it should create child pools. As it is now, the pool that is passed in to apr_thread_create() serves as the parent pool for the new thread-specific sms. If this parameter were null, the apr_thread_create() function would not create a sub-pool or register cleanup routines (which satisfies my requirements). if at all possible, the behaviour should match as closely as possible the existing situation, when this parameter is NULL, even if the current situation has bugs / is not very nice. this is mainly so that people do not object to having the behaviour of existing code disrupted. Fair enough. It's just that in order to opt-out of the child-pool creating process in apr_thread_create, we're going to have to add a parameter to the call and that means changing programs that currently using it, like httpd. Would it be nicer to just add new parallel apr_smsthread_create [name is not important] functions that do this and allow for the opting-out also? I can provide a patch for this. -aaron
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Fair enough. It's just that in order to opt-out of the child-pool creating process in apr_thread_create, we're going to have to add a parameter Why are we so desperate in opting out the child-pool creation? I don't really have problems with a child pool for each thread. Actually, it will make the dynamic locking a lot easier to implement if it stays. to the call and that means changing programs that currently using it, like httpd. Would it be nicer to just add new parallel apr_smsthread_create [name is not important] functions that do this and allow for the opting-out also? I can provide a patch for this. Hmmm, the thought of a double interface doesn't bring out happy feelings with me I'm afraid. Sander
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote: Fair enough. It's just that in order to opt-out of the child-pool creating process in apr_thread_create, we're going to have to add a parameter Why are we so desperate in opting out the child-pool creation? I don't really have problems with a child pool for each thread. Actually, it will make the dynamic locking a lot easier to implement if it stays. I have some use cases that just want a thread. Since no child-pool is required in this case, it becomes unnecessary overhead (that is currently broken, as the child-pool is only cleaned in apr_thread_exit() but that whole thing is screwey). So if apr threads didn't know anything about child-pools, then they would just simply create a thread that called my worker_function(). I could pass in my opaque data to that worker_function(), and do the child-pool creations and destruction right there. I just don't see how child-pool creation MUST be in apr_thread_create() -aaron
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Why are we so desperate in opting out the child-pool creation?
I don't really have problems with a child pool for each thread.
Actually,
it will make the dynamic locking a lot easier to implement if it stays.
I have some use cases that just want a thread. Since no child-pool is
required in this case, it becomes unnecessary overhead (that is currently
broken, as the child-pool is only cleaned in apr_thread_exit() but that
whole thing is screwey).
So if apr threads didn't know anything about child-pools, then they
would just simply create a thread that called my worker_function().
I could pass in my opaque data to that worker_function(), and do
the child-pool creations and destruction right there. I just don't see
how child-pool creation MUST be in apr_thread_create()
Ah, ok, I'm getting the picture now. Well, for my purposes, the child
pool creation doesn't have to be in apr_thread_create. There doesn't
have to be child pool creation at all. But, I do want the worker fn
to be able to create a child pool from the pool passed into
apr_thread_create. So, we need a way to pass in the data to the
worker fn.
The only thing I need is something like this for the sms locking scheme:
struct thread_data
{
...
apr_sms_t *sms;
void (*worker_fn)(void *data);
void *data;
};
void thread_stub(void *data)
{
apr_sms_t *pms;
apr_os_thread_t thread;
struct thread_data *td = (struct thread_data *)data;
thread = apr_os_thread_current();
pms = td-sms;
apr_sms_thread_register(pms, thread);
td-worker_fn(td-data);
apr_sms_thread_unregister(pms, thread);
}
In apr_thread_create an instance of thread_data is created
and initialized, then instead of running the worker function
in the thread, we launch the stub which in turn calls the
worker.
Possibly we can use the stub for more than only this?
Sander
Re: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:42:29AM -0400, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote:
I propose the following patch [PATCH A]: It will partially fix the unwanted
child deaths problem (symptoms mentioned in the mails included below). It
fixes the problem by making sure that perform_idle_server_maintenance() does
not count the threads of the process that recd the POD, in the calculation
of idle_thread_count. To do that I have used
ap_scoreboard_image-parent[process_slot]-process_status field. I am
temporarily using an already defined value, SB_IDLE_DIE. If the general
idea is acceptable, I can work on solidifying the details. PATCH A is
attached below.
Have you taken a look at the patch I posted that merges the POD code in
threaded with the version in mpm_common.c? Threaded shouldn't be doing
POD checks in threaded.c. It's redundant and it's done incorrectly
anyway in the threaded MPM.
Admittedly, this doesn't fix the issue of having a child who received a
POD kill its sibling threads. More on that in a sec.
However this patch exposes another problem in the code - by this new fix,
although the untargetted childs do not get a POD, the targetted child
process does not die immediately either. Here is why that happens: In the
worker_thread() routine in threaded.c, at the instant when worker 1.0
(represented in the process_slot.thread_slot format, i.e, 1 is the
process_slot and 0 is the thread_slot) gets the POD, the remaining threads
of process 1 are all waiting at the apr_lock_acquire(accept_mutex). If the
web-server is really idle, the chances are slim that all the remaining
worker threads for process 1 would acquire the lock in a very short time.
As an effect, the remaining worker threads of process 1 do not die
immediately. To resolve this:
SOLUTION 1: I plan to temporarily implement a new
apr_lock_acquire_timeout() function, which would cause the threads waiting
on the mutex to give-up after sometime. The piece of code would look
something like this:
while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
!= APR_SUCCESS) {
if (check_if_timer_popped)
if (workers_may_exit)
break;
else { /* apr_lock_acquire failed */
ap_log_error();
workers_may_exit = 1;
}
}
I know that this would cause some performance impact, but my guess is that
it would not be a lot, especially if we keep the timevalue reasonably high.
However, in order to get the functionality of
perform_idle_server_maintenance() working right, we _will_ have to implement
the above solution (or maybe something similar)!
-1 (my veto only matters in APR-land). This is ugly. How are you going
to implement this? The problem is that we need to wake the system up from
holding the mutex. But, that is almost impossible to implement portably
AFAIK. Please enlighten me if you know of a way to do this. You can't
do a trylock (assuming you can do so with all variants) and then sleep
for the timeout value. That doesn't work either. (All of the threads are
sleeping after trying a held lock when someone comes along? Oops.)
The only alternative that I can think of is pthread_cancel(), BUT (at
least Solaris) says (man cancellation):
A mutex is explicitly not a cancellation point and should
be held for only the minimal essential time.
A pthread_cond_wait() and fcntl() are viable options according to the
manpage. But, I'd much prefer us to use pthread_mutex_t if at all
possible for cross-process locking. (Yeah, yeah, we require Solaris 8+
to do pthread_mutex_t for a cross-process lock now for robust locking.
That's what you get for having child processes - see below...)
SOLUTION 2: One could use a slightly more _involved_ approach, where the
dying thread could send a signal to its sibling threads, each of which will
then handle that signal with a graceful exit.
How? A thread doesn't have a process id associated with it. And, I'm
curious to whether the signal would somehow kick us out of the acquire
function. Doubtful. I may be wrong (probably am). Please correct me.
We need to be able to tell the threads they need to exit, but you can't
force them to just exit (which is why pthread_cancel wouldn't work).
They need to exit on their own - otherwise, they won't cleanup.
SOLUTION 3: We could turn our heads the other way by not worrying about this
situation at all, since these threads would eventually die (when more
requests are handled by the webserver). However, by ignoring the problem,
the purpose of perform_idle_server_maintenance() would be lost!!
And, based on my interpretation and analysis of the threaded MPM and its
intentions, we shouldn't even need to worry about this case. Let me
explain:
The problem you are stating is that the POD is received by one thread in
one child process, it marks workers_may_exit and then
Re: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:42:29AM -0400, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote:
[snip]
SOLUTION 1: I plan to temporarily implement a new
apr_lock_acquire_timeout() function, which would cause the threads waiting
on the mutex to give-up after sometime. The piece of code would look
something like this:
while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
!= APR_SUCCESS) {
if (check_if_timer_popped)
if (workers_may_exit)
break;
else { /* apr_lock_acquire failed */
ap_log_error();
workers_may_exit = 1;
}
}
I know that this would cause some performance impact, but my guess is that
it would not be a lot, especially if we keep the timevalue reasonably high.
However, in order to get the functionality of
perform_idle_server_maintenance() working right, we _will_ have to implement
the above solution (or maybe something similar)!
This looks like a job for condition variables. Using condition
variables you can control exactly when threads get woken up to check
on the condition. For example, when one thread returns from accept(),
it can signal another thread to move to the accept() state (and as an
optimization that I think Ryan suggested, now you don't need to have
an intraprocess accept lock, since only one thread can be listening in
accept() at a time). The other threads are blocked waiting for some
condition, either allowing them to pass to the accept() state or the
workers_may_exit state.
If this sounds like something you could use here, I can probably elaborate
on this some more, maybe even give some [pseudo]code.
-aaron
Re: [PATCH] Problems with MPM threaded
SOLUTION 2: One could use a slightly more _involved_ approach, where the dying thread could send a signal to its sibling threads, each of which will then handle that signal with a graceful exit. How? A thread doesn't have a process id associated with it. And, I'm curious to whether the signal would somehow kick us out of the acquire function. Doubtful. I may be wrong (probably am). Please correct me. We need to be able to tell the threads they need to exit, but you can't force them to just exit (which is why pthread_cancel wouldn't work). They need to exit on their own - otherwise, they won't cleanup. SOLUTION 3: We could turn our heads the other way by not worrying about this situation at all, since these threads would eventually die (when more requests are handled by the webserver). However, by ignoring the problem, the purpose of perform_idle_server_maintenance() would be lost!! And, based on my interpretation and analysis of the threaded MPM and its intentions, we shouldn't even need to worry about this case. Let me explain: The problem you are stating is that the POD is received by one thread in one child process, it marks workers_may_exit and then quits. The accept mutex is then shifted over to another child process which doesn't have workers_may_exit set and goes on its merry way. The only time we check for workers_may_exit is after the mutex has been acquired. Therefore, it can take a while to have all of the threads in that POD-receiving process to see workers_may_exit. Threaded MPM is designed (rather should be - the implementation doesn't suggest this) to only have ONE child process that does the No! The threaded MPM does NOT and was not intended to implement a thread-only server. It was designed to implement a hybrid thread/process server. Because we expect both threads and processes, the rest of this message is based on a misconception. Having multiple processes each with multiple threads provides for FAR more robustness than just a single process with multiple threads. If you want that, then you want the perchild MPM, so that the number of threads can grow and shrink correctly. Ryan _ Ryan Bloom [EMAIL PROTECTED] Covalent Technologies [EMAIL PROTECTED] -
Re: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 09:19:17AM -0700, [EMAIL PROTECTED] wrote: No! The threaded MPM does NOT and was not intended to implement a thread-only server. It was designed to implement a hybrid thread/process server. Because we expect both threads and processes, the rest of this message is based on a misconception. Having multiple processes each with multiple threads provides for FAR more robustness than just a single process with multiple threads. If you want that, then you want the perchild MPM, so that the number of threads can grow and shrink correctly. What are we gaining by having multiple children processes? I certainly don't think the robustness gained by having multiple children processes is all that great. And, it serves to make the threaded MPM code almost unmaintainble. Perchild looks a little closer, but I still think it is tainted by having multiple children processes. Oh, and it needs the POD fix as well. By having the possibility of having other children processes, you now need a mechanism to kill all threads in the same process efficiently. You'd need to kick them out of the accept mutex, but I'm not seeing how that's going to happen quickly. -- justin
Re: [PATCH] Problems with MPM threaded
On Sat, 14 Jul 2001, Justin Erenkrantz wrote: On Sat, Jul 14, 2001 at 09:19:17AM -0700, [EMAIL PROTECTED] wrote: No! The threaded MPM does NOT and was not intended to implement a thread-only server. It was designed to implement a hybrid thread/process server. Because we expect both threads and processes, the rest of this message is based on a misconception. Having multiple processes each with multiple threads provides for FAR more robustness than just a single process with multiple threads. If you want that, then you want the perchild MPM, so that the number of threads can grow and shrink correctly. What are we gaining by having multiple children processes? I certainly don't think the robustness gained by having multiple children processes is all that great. And, it serves to make the threaded MPM code almost Apache allows 3rd-party modules, which makes it much more likely that the server will seg fault. If you only have one process, then your server won't be serving any requests during the time it takes to start a second server. Add to that the time it takes for the parent process to notice that the child has died (up to 1 second), and you have missed serving a lot of requests. Now, add scalability concerns. The threaded MPM creates a set number of threads per child process. This means that at ALL times, you have to have enough threads to handle the maximum number of concurrent connections. That is bogus. I realize that threads are less expensive than processes, but come on. Why should have my enough threads in my server to handle my peak load at all times? The Perchild MPM was designed to allow you to grow the number of threads in a given process. So, you can bring the number of threads down to 10 or 20, and grow as needed. unmaintainble. Perchild looks a little closer, but I still think it is tainted by having multiple children processes. Oh, and it needs the POD fix as well. Perchild can have as few children as you want. Set it up to run 1 child, and it will just have one child. By having the possibility of having other children processes, you now need a mechanism to kill all threads in the same process efficiently. You'd need to kick them out of the accept mutex, but I'm not seeing how that's going to happen quickly. -- justin You need to be able to kill all threads in a process efficiently regardless. We will always need to be able to re-spawn processes that die, and we will always have the concept of MaxRequestsPerChild. If a process is going to die and be replaced, then we need to kill all the threads in that process as quickly as possible. Only allowing one process at a time doesn't remove that requirement. Feel free to create an MPM that only allows one process at a time, that is why the MPM mechanism exists, but Threaded and Perchild were both designed to allow multiple child processes for a reason. Ryan _ Ryan Bloom [EMAIL PROTECTED] Covalent Technologies [EMAIL PROTECTED] -
Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
By having the possibility of having other children processes, you now need a mechanism to kill all threads in the same process efficiently. You'd need to kick them out of the accept mutex, but I'm not seeing how that's going to happen quickly. -- justin You need to be able to kill all threads in a process efficiently regardless. We will always need to be able to re-spawn processes that die, and we will always have the concept of MaxRequestsPerChild. If a process is going to die and be replaced, then we need to kill all the threads in that process as quickly as possible. Only allowing one process at a time doesn't remove that requirement. The way I see it, each process has a single pool instance as the parent for all the threads. Resetting or destroying that pool should effectively kill all threads. What am I missing? Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 09:13:08PM +0200, Sander Striker wrote: The way I see it, each process has a single pool instance as the parent for all the threads. Resetting or destroying that pool should effectively kill all threads. What am I missing? As I see it, the problem is: [ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't apply here. ] All threads are stuck in the accept mutex except for one. It reads the POD. And, it is time to die. It sets workers_may_exit to 1. It then releases the accept mutex and exits. Another *child process* acquires the accept mutex. Depending upon how many child processes you have, it may take a while for all of the threads in the doomed child process to acquire the mutex, wake up, check workers_may_exit, release the mutex, and then exit. Only when all of the threads in the doomed child process are exited does any cleanup occur. These sibling threads are blocked until they receive the mutex. They aren't going to be going anywhere until the mutex is turned over to them. They can't check the value of workers_may_exit *until* they acquire the mutex. Condition variables help *somewhat*, but the problem is now of scope - you have an interprocess condition and a intraprocess condition to check. Does this work? I don't know. And, you can't kick the thread out of the mutex acquire (pthread_cancel or similar strategies don't cancel a mutex operation), so you are screwed. And, destroying its parent pool does *NOT* destroy the thread. Look at the code again. The only person that can call pthread_exit() is the actual thread itself. You can't call pthread_exit on behalf of another thread (i.e. from the thread that knows it is doomed or a cleanup thread). It just doesn't work like that. I just don't agree with Ryan's assessment here. But, it's possible I'm missing some major piece of code. -- justin
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
The way I see it, each process has a single pool instance as the parent for all the threads. Resetting or destroying that pool should effectively kill all threads. What am I missing? As I see it, the problem is: [ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't apply here. ] All threads are stuck in the accept mutex except for one. It reads the POD. And, it is time to die. It sets workers_may_exit to 1. It then releases the accept mutex and exits. Another *child process* acquires the accept mutex. Depending upon how many child processes you have, it may take a while for all of the threads in the doomed child process to acquire the mutex, wake up, check workers_may_exit, release the mutex, and then exit. Only when all of the threads in the doomed child process are exited does any cleanup occur. These sibling threads are blocked until they receive the mutex. They aren't going to be going anywhere until the mutex is turned over to them. They can't check the value of workers_may_exit *until* they acquire the mutex. Condition variables help *somewhat*, but the problem is now of scope - you have an interprocess condition and a intraprocess condition to check. Does this work? I don't know. And, you can't kick the thread out of the mutex acquire (pthread_cancel or similar strategies don't cancel a mutex operation), so you are screwed. Err, doesn't destruction of the mutex wake everyone up? Oh, wait, does every process share the same mutex? And, destroying its parent pool does *NOT* destroy the thread. Look at the code again. Well, it should, although it may not do so now. The only person that can call pthread_exit() is the actual thread itself. You can't call pthread_exit on behalf of another thread (i.e. from the thread that knows it is doomed or a cleanup thread). It just doesn't work like that. I just don't agree with Ryan's assessment here. But, it's possible I'm missing some major piece of code. -- justin Could be I'm way off too. Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:27:05PM -0700, Justin Erenkrantz wrote: And, you can't kick the thread out of the mutex acquire (pthread_cancel or similar strategies don't cancel a mutex operation), so you are screwed. And, destroying its parent pool does *NOT* destroy the thread. Look at the code again. The only person that can call pthread_exit() is the actual thread itself. You can't call pthread_exit on behalf of another thread (i.e. from the thread that knows it is doomed or a cleanup thread). It just doesn't work like that. Here's what my box's pthread_cancel man page says: Cancellation points are those points in the program execu tion where a test for pending cancellation requests is performed and cancellation is executed if positive. The following POSIX threads functions are cancellation points: pthread_join(3) pthread_cond_wait(3) pthread_cond_timedwait(3) pthread_testcancel(3) sem_wait(3) sigwait(3) Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll let it be cancel()ed. Anyway, yes the pool cleanup routines that are supposed to be calling apr_thread_exit() are broken because nothing in httpd is calling apr_thread_exit() (AFA-my-grep-says). So let's either let the thread creator (aka httpd) do the cleanup registration, or put it in apr (as an optional feature). -aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 09:43:53PM +0200, Sander Striker wrote: Err, doesn't destruction of the mutex wake everyone up? Oh, wait, does every process share the same mutex? Yup. accept_mutex is CROSS_PROCESS. See? That's the rub. You can't destroy it. Otherwise, you'd screw the other child processes which shouldn't be killed off. Which is why I'm saying it is invalid to have multiple child process in a threaded MPM. And, destroying its parent pool does *NOT* destroy the thread. Look at the code again. Well, it should, although it may not do so now. That's impossible to do. Another thread can not indicate that a thread should exit with the threading libraries (safely). It must have some OOB way of doing so (which we have). The only problem is that our thread is stuck in an *uninterruptable* operation. If that operation were interruptable OR that thread received the mutex, then it'd check the workers_may_exit flag. Then, it'd exit. -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:49:51PM -0700, Aaron Bannert wrote: Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll let it be cancel()ed. Nope. As I posted earlier, man cancellation on Solaris 8 says: A mutex is explicitly not a cancellation point and should be held for only the minimal essential time. pthread_cancel wouldn't work. And as Ryan pointed out, it is bad to try and forcibly kill a thread. -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Async cancellation of threads is VERY bad ju-ju. We don't do it because it causes most, if not all, thread libraries to leak. The cleanups are called because the child pool is destroyed when all the threads die. Right, but we're about to kill the processes anyway, so who cares if it leaks on some platforms. Anyway, we're writing to a specification, not an implementation. If Apache isn't going to encourage OS vendors to properly implement their own APIs, who is? (Just think how easy it would be for us if all these platforms fully supported a single API, be it posix or whatever. Heaven!) Anyway, to me this is what pools were designed for. -aaron
Standards compliance in Apache code WAS Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 01:19:43PM -0700, Aaron Bannert wrote: Async cancellation of threads is VERY bad ju-ju. We don't do it because it causes most, if not all, thread libraries to leak. The cleanups are called because the child pool is destroyed when all the threads die. Right, but we're about to kill the processes anyway, so who cares if it leaks on some platforms. Anyway, we're writing to a specification, not an implementation. If Apache isn't going to encourage OS vendors to properly implement their own APIs, who is? Eh, it's probably the other way around. Otherwise, we'd have no need for #ifdefs. =) Standards are only as good as their implementations. Most users of Apache couldn't care less if their favorite OS didn't implement a particular feature correctly. It's *our* problem to work around it. Ideally, yes, we could just write to a spec, but that's in fantasy land. I live in the real world with lots of sub-par implementations flying around. (Just think how easy it would be for us if all these platforms fully supported a single API, be it posix or whatever. Heaven!) Anyway, to me this is what pools were designed for. I'd still like to know how you are going to interrupt the cross-process mutex. It doesn't matter if it implemented Posix or whatever. That's not the problem here. We're in the middle of an uninterruptable call and that mutex can not be destroyed because it is *cross-process*. -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 01:18:04PM -0700, Justin Erenkrantz wrote: On Sat, Jul 14, 2001 at 12:49:51PM -0700, Aaron Bannert wrote: Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll let it be cancel()ed. Nope. As I posted earlier, man cancellation on Solaris 8 says: A mutex is explicitly not a cancellation point and should be held for only the minimal essential time. Hmmm..that sounds like we're using mutexes incorrectly. -aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 01:29:35PM -0700, [EMAIL PROTECTED] wrote: What you are missing is the history. All of the problems that we are trying to solve were solved already in the original code. These problem are solved most easily by using two mutexes, one for keeping only one process in the inner loop, and the inner one for the threads in the process. Then, when you are killing a process, you can easily wake up all the threads in the process that are sitting on a lock. Ah. Yes, that makes more sense. But, that's not what's there now. Shall I submit a patch to threaded MPM to do this? -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 01:29:35PM -0700, [EMAIL PROTECTED] wrote: This is why the original code, in apache-apr, written by Manoj back at IBM used two mutexes. The first was cross-process, the second was cross-threads. It allowed us to handle this much cleaner. I didn't realize this had been removed. [snip] What you are missing is the history. All of the problems that we are trying to solve were solved already in the original code. These problem are solved most easily by using two mutexes, one for keeping only one process in the inner loop, and the inner one for the threads in the process. Then, when you are killing a process, you can easily wake up all the threads in the process that are sitting on a lock. [bird's eye view observation follows:] I'm beginning to understand why I keep running across parts of APR that seem inconsistent, overly complicated, disjoint, not generally applicable, and sometimes seemingly suboptimal. They are holdovers from when they provided a solution in httpd. I can see how it would be difficult to form APR for general use when it started out just being code ripped out of apache. I do think that many of these problems are because APR is in some places still striving to support the needs of httpd directly, instead of just providing the basic building blocks that both httpd and other programs need. So where to start? -aaron
[PATCH] Add intraprocess mutex to threaded MPM WAS Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Ah. Yes, that makes more sense. But, that's not what's there now.
Shall I submit a patch to threaded MPM to do this? -- justin
Please.
This also includes my POD patch. I can separate it out if you don't
want the POD code merged yet. -- justin
Index: threaded.c
===
RCS file: /home/cvs/httpd-2.0/server/mpm/threaded/threaded.c,v
retrieving revision 1.44
diff -u -r1.44 threaded.c
--- threaded.c 2001/07/03 13:58:10 1.44
+++ threaded.c 2001/07/14 20:48:04
@@ -143,10 +143,7 @@
char ap_coredump_dir[MAX_STRING_LEN];
-static apr_file_t *pipe_of_death_in = NULL;
-static apr_file_t *pipe_of_death_out = NULL;
-static apr_lock_t *pipe_of_death_mutex; /* insures that a child process only
- consumes one character */
+static ap_pod_t *pipe_of_death;
/* *Non*-shared http_main globals... */
@@ -178,7 +175,13 @@
static int worker_thread_count;
static apr_lock_t *worker_thread_count_mutex;
-/* Locks for accept serialization */
+/* Locks for accept serialization
+ * worker_accept_mutex ensures that only one thread in a child process
+ * may be in accept. It also ensures that when we have
+ * workers_may_exit=1 that we exit.
+ * accept_mutex is the cross-process mutex which all children have.
+ */
+static apr_lock_t *worker_accept_mutex;
static apr_lock_t *accept_mutex;
static apr_lockmech_e_np accept_lock_mech = APR_LOCK_DEFAULT;
static const char *lock_fname;
@@ -494,29 +497,6 @@
}
}
-/* Sets workers_may_exit if we received a character on the pipe_of_death */
-static void check_pipe_of_death(void)
-{
-apr_lock_acquire(pipe_of_death_mutex);
-if (!workers_may_exit) {
-apr_status_t ret;
-char pipe_read_char;
- apr_size_t n = 1;
-
-ret = apr_recv(listensocks[0], pipe_read_char, n);
-if (APR_STATUS_IS_EAGAIN(ret)) {
-/* It lost the lottery. It must continue to suffer
- * through a life of servitude. */
-}
-else {
-/* It won the lottery (or something else is very
- * wrong). Embrace death with open arms. */
-workers_may_exit = 1;
-}
-}
-apr_lock_release(pipe_of_death_mutex);
-}
-
static void * worker_thread(void * dummy)
{
proc_info * ti = dummy;
@@ -539,8 +519,8 @@
worker_thread_count++;
apr_lock_release(worker_thread_count_mutex);
-apr_poll_setup(pollset, num_listensocks+1, tpool);
-for(n=0 ; n = num_listensocks ; ++n)
+apr_poll_setup(pollset, num_listensocks, tpool);
+for(n = 0; n num_listensocks; ++n)
apr_poll_socket_add(pollset, listensocks[n], APR_POLLIN);
/* TODO: Switch to a system where threads reuse the results from earlier
@@ -553,7 +533,10 @@
(void) ap_update_child_status(process_slot, thread_slot, SERVER_READY,
(request_rec *) NULL);
-if ((rv = SAFE_ACCEPT(apr_lock_acquire(accept_mutex)))
+
+apr_lock_acquire(worker_accept_mutex);
+if (!workers_may_exit
+(rv = SAFE_ACCEPT(apr_lock_acquire(accept_mutex)))
!= APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
apr_lock_acquire failed. Attempting to shutdown
@@ -580,12 +563,8 @@
if (workers_may_exit) break;
- apr_poll_revents_get(event, listensocks[0], pollset);
-if (event APR_POLLIN) {
-/* A process got a signal on the shutdown pipe. Check if we're
- * the lucky process to die. */
-check_pipe_of_death();
-continue;
+if (!ap_mpm_pod_check(pipe_of_death)) {
+workers_may_exit = 1;
}
if (num_listensocks == 1) {
@@ -624,6 +603,12 @@
process gracefully.);
workers_may_exit = 1;
}
+if ((rv = apr_lock_release(worker_accept_mutex)) != APR_SUCCESS) {
+ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
+ apr_lock_release failed. Attempting to shutdown
+ process gracefully.);
+workers_may_exit = 1;
+}
if (csd != NULL) {
process_socket(ptrans, csd, process_slot, thread_slot);
requests_this_child--;
@@ -637,6 +622,12 @@
process gracefully.);
workers_may_exit = 1;
}
+if ((rv = apr_lock_release(worker_accept_mutex)) != APR_SUCCESS) {
+ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
+ apr_lock_release failed. Attempting to shutdown
+ process gracefully.);
+workers_may_exit = 1;
+}
break;
}
Re: [PATCH] Problems with MPM threaded
The correct fix, as I see it, is to kill off the interprocess accept lock by removing the possibility of having other processes in a *threaded* MPM. -- justin That architecture was explored in detail by Netscape. It isn't reliable and slows your web server to a crawl whenever dynamic content is produced. It should only be used for static file servers and caching gateways, and people implementing those might as well use an in-kernel server like TUX. Roy
