Re: how to debug worker mpm httpd with gdb?
My 2 cents: Keep using Linux when a fast build time is imperative or you're debugging non-threaded apps. Use AIX or Solaris with native tools when you want to debug threaded apps. It sucks :( -- Jeff Trawick | [EMAIL PROTECTED] Born in Roswell... married an alien...
Re: how to debug worker mpm httpd with gdb?
I realize that the problem is in the debugger, not the MPM's cooperation with it. If it was able to automatically switch to the thread when a breakpoint is reached and stay there that's all I need and then I don't care how many threads are running. Moreover I definitely agree with Ryan, that in order to debug threads you need more than one worker thread, otherwise you can do most of the debugging with prefork mpm. Not that the potential problems will always manifest them during the debugging, but it raises the chance that some problems will be exposed early. I guess my request was more of asking tips and tricks that I'm not aware of to improve the threads debugging experience. I'm new to threads (especially debugging threads), and I believe many others too. So having a doc explaining debugging threads will save a lot of questions when other module authors will start porting their modules to 2.0. I volunteer to create such a doc based on what I learn, so if you can help me that would be great! Are there any books covering threads debugging in details? I've read "Programming with POSIX(R) Threads" so far but it doesn't cover debugging. > but I'd make sure you are running > the latest gdb version (5.1.1?). I've found that the threading > support was much improved from 4.x->5.x. -- justin yes, I've the latest one. Are there any other debuggers that are better than gdb concerning threads, including commercial ones and running on Linux? Thanks _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 04:26:19PM -0800, Justin Erenkrantz wrote: > > Just change ThreadsPerChild to 1 if you want this. If you really want it, > > I see no reason we can't create a -DONE_WORKER or similiar MPM-specific > > flag to do the same. > > If you read Stas's message, he set TPC to 1, but it didn't work > right. (The threads weren't peristent - they were always killed > off.) -- justin This sounds like a problem with gdb on linux w/ threads. I've set this in solaris w/ gdb and not seen problems. I'd be surprised if threads were being "killed off" after a request. (Perhaps Max and MinSpareThreads shouldn't both be 1?) I would expect there to be at least 3 threads when setting ThreadsPerChild to 1: worker thread listener thread POD thread/signal thread (signal thread is for -X) Also, most thread libraries must create management threads for dealing with LWP mapping, thread multiplexing, etc... which are also probably showing up in the gdb output. -aaron
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 04:23:47PM -0800, Aaron Bannert wrote: > On Tue, Mar 12, 2002 at 03:37:20PM -0800, Justin Erenkrantz wrote: > > -X should only create two threads: the manager thread and a single > > worker thread. However, IIRC, it'll create ThreadsPerChild for that > > one process. It should limit itself to just one worker thread. > > > > By limiting us to two httpd-created threads, it should be somewhat > > easier for the debugger/user to determine which thread they should > > attach to. IMHO, the current solution of creating one process with > > a normal amount of workers isn't very nice. -- justin > > Just change ThreadsPerChild to 1 if you want this. If you really want it, > I see no reason we can't create a -DONE_WORKER or similiar MPM-specific > flag to do the same. If you read Stas's message, he set TPC to 1, but it didn't work right. (The threads weren't peristent - they were always killed off.) -- justin
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 03:37:20PM -0800, Justin Erenkrantz wrote: > -X should only create two threads: the manager thread and a single > worker thread. However, IIRC, it'll create ThreadsPerChild for that > one process. It should limit itself to just one worker thread. > > By limiting us to two httpd-created threads, it should be somewhat > easier for the debugger/user to determine which thread they should > attach to. IMHO, the current solution of creating one process with > a normal amount of workers isn't very nice. -- justin Just change ThreadsPerChild to 1 if you want this. If you really want it, I see no reason we can't create a -DONE_WORKER or similiar MPM-specific flag to do the same. -aaron
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 03:14:52PM -0800, Ryan Bloom wrote: > Okay, but creating two threads won't solve the problem, and it may > actually hinder people from using -X to debug thread related problems. > The hardest part of debugging threaded code, is that the threads > interact. By artificially limiting the number of threads, you are > effectively removing the ability to debug thread interaction problems. Correct which is the argument made why the MPM is doing the right thing now. My take is that there are two potential "debuggers" types - our module writers and our MPM/thread experts. I'd prefer we default to a mode that is beneficial to the module writers (one worker thread only), and add another mode (say -DTHREAD_DEBUG - name sucks) which allows the worker MPM to create its normal amount of threads even though it is only using ONE_PROCESS. I think there will be far more people attempting to debug their modules than our MPM. > Also, on a platform like Linux, reducing to two threads really doesn't > help anything, because the debuggers suck at dealing with threads. Recent gdbs are much better at threads in Linux. > What you really want is a good debugger, which will allow you to set a > breakpoint, and the debugger will switch to the right thread when the > breakpoint is hit. Well, yeah, I'd like a nice debugger too. I'm a poor college student, so I don't see that happening anytime soon unless gdb gets an infusion of support. =) -- justin
RE: how to debug worker mpm httpd with gdb?
> On Tue, Mar 12, 2002 at 03:06:13PM -0800, Ryan Bloom wrote: > > How would you fix this? By having the worker MPM go into a single > > threaded, single process mode with -X? That won't actually solve the > > problem, because the OS will still create the manager thread. This is > > just part of the pain of doing threaded debugging on Unix. > > > > -X should only create two threads: the manager thread and a single > worker thread. However, IIRC, it'll create ThreadsPerChild for that > one process. It should limit itself to just one worker thread. > > By limiting us to two httpd-created threads, it should be somewhat > easier for the debugger/user to determine which thread they should > attach to. IMHO, the current solution of creating one process with > a normal amount of workers isn't very nice. -- justin Okay, but creating two threads won't solve the problem, and it may actually hinder people from using -X to debug thread related problems. The hardest part of debugging threaded code, is that the threads interact. By artificially limiting the number of threads, you are effectively removing the ability to debug thread interaction problems. Also, on a platform like Linux, reducing to two threads really doesn't help anything, because the debuggers suck at dealing with threads. What you really want is a good debugger, which will allow you to set a breakpoint, and the debugger will switch to the right thread when the breakpoint is hit. Ryan
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 03:06:13PM -0800, Ryan Bloom wrote: > How would you fix this? By having the worker MPM go into a single > threaded, single process mode with -X? That won't actually solve the > problem, because the OS will still create the manager thread. This is > just part of the pain of doing threaded debugging on Unix. > -X should only create two threads: the manager thread and a single worker thread. However, IIRC, it'll create ThreadsPerChild for that one process. It should limit itself to just one worker thread. By limiting us to two httpd-created threads, it should be somewhat easier for the debugger/user to determine which thread they should attach to. IMHO, the current solution of creating one process with a normal amount of workers isn't very nice. -- justin
RE: how to debug worker mpm httpd with gdb?
> On Tue, Mar 12, 2002 at 06:21:27PM +0800, Stas Bekman wrote: > > Any guidelines for debugging threaded (worker) mpm? I've read the > > relevant gdb section, but it isn't enough. I couldn't find any relevant > > info in httpd-docs-2.0/manual/developer. > > > > I'm interested in tips making the debug process easier. Does anybody > > have a cheatlist with useful tips/notes and willing to share it? > > > During the first request yet another thread starts. And one of the old > > ones goes away. > > > > It seems that the newest thread is the one I need to attach to. But when > > I attach to it once, in a while I get switched to the previous thread, > > etc. This is very confusing and non-productive :( > > Yeah, well, I've maintained that worker doesn't have the right > support for debugging. Nobody else seems to agree with me. =( > > AIUI (without placing words in anyone's mouth), it is that if you > are using -X on worker, you want to debug the MPM *not* Apache. > So, if you want to reliably test your code (i.e. a module), you > should use the prefork MPM. If you are using worker, you need to > attach to the "right" worker thread and hope you get the right > thread. I've complained to the appropriate parties and they > maintain their position is correct. Whatever. Feel free to > join the complaint line. Worker is useless in -X mode. How would you fix this? By having the worker MPM go into a single threaded, single process mode with -X? That won't actually solve the problem, because the OS will still create the manager thread. This is just part of the pain of doing threaded debugging on Unix. Ryan
Re: how to debug worker mpm httpd with gdb?
On Tue, Mar 12, 2002 at 06:21:27PM +0800, Stas Bekman wrote: > Any guidelines for debugging threaded (worker) mpm? I've read the > relevant gdb section, but it isn't enough. I couldn't find any relevant > info in httpd-docs-2.0/manual/developer. > > I'm interested in tips making the debug process easier. Does anybody > have a cheatlist with useful tips/notes and willing to share it? > During the first request yet another thread starts. And one of the old > ones goes away. > > It seems that the newest thread is the one I need to attach to. But when > I attach to it once, in a while I get switched to the previous thread, > etc. This is very confusing and non-productive :( Yeah, well, I've maintained that worker doesn't have the right support for debugging. Nobody else seems to agree with me. =( AIUI (without placing words in anyone's mouth), it is that if you are using -X on worker, you want to debug the MPM *not* Apache. So, if you want to reliably test your code (i.e. a module), you should use the prefork MPM. If you are using worker, you need to attach to the "right" worker thread and hope you get the right thread. I've complained to the appropriate parties and they maintain their position is correct. Whatever. Feel free to join the complaint line. Worker is useless in -X mode. > Any ideas how to tell gdb not to resolve so's automatically and still > have threads working? Don't have an idea about this, but I'd make sure you are running the latest gdb version (5.1.1?). I've found that the threading support was much improved from 4.x->5.x. -- justin
how to debug worker mpm httpd with gdb?
Any guidelines for debugging threaded (worker) mpm? I've read the relevant gdb section, but it isn't enough. I couldn't find any relevant info in httpd-docs-2.0/manual/developer. I'm interested in tips making the debug process easier. Does anybody have a cheatlist with useful tips/notes and willing to share it? In particular how do I know which thread is going to process the next request? If I start with -DONE_PROCESS and have: StartServers 1 MaxClients 1 MinSpareThreads 1 MaxSpareThreads 1 ThreadsPerChild 1 MaxRequestsPerChild 0 httpd still keeps 4 threads, that's what gdb says: (gdb) info thread * 4 Thread 2051 (LWP 31642) apr_poll (aprset=0x81861f8, nsds=0xbf5ffaec, timeout=-1) at poll.c:143 3 Thread 1026 (LWP 31641) 0x402267a2 in sigsuspend () from /lib/libc.so.6 2 Thread 2049 (LWP 31640) 0x402dfd94 in poll () from /lib/libc.so.6 1 Thread 1024 (LWP 31592) 0x402267a2 in sigsuspend () from /lib/libc.so.6 During the first request yet another thread starts. And one of the old ones goes away. It seems that the newest thread is the one I need to attach to. But when I attach to it once, in a while I get switched to the previous thread, etc. This is very confusing and non-productive :( Another question is regarding: set auto-solib-add 0 without this setting gdb tries to load/resolve symbols of many shared libs and the startup may take 1 minute and more under mod_perl, which sucks. With this setting I only need to load sharedlib which I need for debugging, and the startup takes a few seconds. The problem is that with worker-mpm and this setting in effect gdb won't work properly. It doesn't see threads. 'info threads' returns nothing. If I don't set auto-solib-add to 0, it works but takes ages to start. Any ideas how to tell gdb not to resolve so's automatically and still have threads working? Thanks! _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/