My question is that if mod_wsgi should wipe out all meomry inherited from parent once it forks? I am not clear if a module inherits an C++ object from parent, does it trigger a destructor call?
Alex On Monday, May 12, 2014 5:42:27 PM UTC-7, Graham Dumpleton wrote: > > Okay. So this isn't an atexit() callback but global C++ object destructors > kicking in from the automatic execution of finaliser sections on the object > files. > > Same issue applies though in part. It looks like the page speed module > could be making some assumption that certain data will always be > initialised by the time the process is terminated, but possibly because > Apache module child init handlers are not called for the page speed module > in the mod_wsgi daemon processes, then that data isn't initialised and as a > result it crashes. > > When this happens though it is usual to see a NULL pointer dereference or > low memory access due to relative reference to NULL pointer. I can't see an > obvious case of that, but is hard to tell what the module is doing. > > Another problem with this thought is that since the page speed module > doesn't get to do anything at all in the mod_wsgi daemon mode process, then > can't see how this issue wouldn't also arise in the Apache parent process > unless the fact that the module might be unloaded from memory by Apache > first before shutdown (can't remember) might mean that global C++ > destructors aren't called in that case. > > Now one could argue that if this is happening that the page speed module > is being sloppy, but at the same time, under normal circumstances an Apache > module would never need to contend with possibility that something like the > Apache child init handler might not be called in a child process. That is > an oddity caused by mod_wsgi daemon mode. > > Anyway, all can do right now is confirm whether it is the page speed > module by disabling that module temporarily. > > Will then need to work out what to do and perhaps raise issue with page > speed module authors if that is where it is arising and see if they want to > say not their problem since mod_wsgi does weird stuff. :-) > > Graham > > On 13/05/2014, at 9:51 AM, Alex Wu <[email protected] <javascript:>> wrote: > > Here is one example: > > warning: Can't read pathname for load map: Input/output error. > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `(wsgi:dataplane) -D Dataplane -D > pagespeed -D fwd_proxy -D DAT'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fb46da18e4a in ?? () from /usr/lib/libpython2.7.so.1.0 > (gdb) info threads > Id Target Id Frame > 5 Thread 0x7fb458fe9700 (LWP 25847) 0x00007fb4777566e0 in sigprocmask > () from /lib/x86_64-linux-gnu/libc.so.6 > 4 Thread 0x7fb4785bb740 (LWP 22886) 0x00007fb46cadd678 in (anonymous > namespace)::scribble (ptr=0x7fb478f13a38, size=34008, > scribble_word=-559038737) > at pagespeed/kernel/base/mem_debug.cc:81 > 3 Thread 0x7fb469870700 (LWP 22895) 0x00007fb477814a93 in epoll_wait > () from /lib/x86_64-linux-gnu/libc.so.6 > 2 Thread 0x7fb46aa76700 (LWP 22893) 0x00007fb47780d763 in select () > from /lib/x86_64-linux-gnu/libc.so.6 > * 1 Thread 0x7fb46a071700 (LWP 22894) 0x00007fb46da18e4a in ?? () from > /usr/lib/libpython2.7.so.1.0 > (gdb) thread 4 > [Switching to thread 4 (Thread 0x7fb4785bb740 (LWP 22886))] > #0 0x00007fb46cadd678 in (anonymous namespace)::scribble > (ptr=0x7fb478f13a38, size=34008, scribble_word=-559038737) at > pagespeed/kernel/base/mem_debug.cc:81 > 81 pagespeed/kernel/base/mem_debug.cc: No such file or directory. > (gdb) bt > #0 0x00007fb46cadd678 in (anonymous namespace)::scribble > (ptr=0x7fb478f13a38, size=34008, scribble_word=-559038737) at > pagespeed/kernel/base/mem_debug.cc:81 > #1 0x00007fb46cadd827 in (anonymous namespace)::debug_free > (ptr=0x7fb478f13a38) at pagespeed/kernel/base/mem_debug.cc:100 > #2 0x00007fb46cadd9f9 in operator delete[] (ptr=0x7fb478f13a38) at > pagespeed/kernel/base/mem_debug.cc:142 > #3 0x00007fb46ce2256e in re2::Prog::~Prog (this=0x7fb478c260e8, > __in_chrg=<optimized out>) at third_party/re2/src/re2/prog.cc:123 > #4 0x00007fb46cdf5402 in re2::RE2::~RE2 (this=0x7fb478ff3dd8, > __in_chrg=<optimized out>) at third_party/re2/src/re2/re2.cc:272 > #5 0x00007fb46d1033af in > pagespeed::js::JsTokenizerPatterns::~JsTokenizerPatterns > (this=0x7fb478ff3dd8, __in_chrg=<optimized out>) > at pagespeed/kernel/js/js_tokenizer.cc:1096 > #6 0x00007fb46cf9f00c in > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns>::operator() > (this=0x7fb46d6a6fe8, ptr=0x7fb478ff3dd8) > at third_party/chromium/src/base/memory/scoped_ptr.h:137 > #7 0x00007fb46cf9efc2 in > base::internal::scoped_ptr_impl<pagespeed::js::JsTokenizerPatterns, > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns> >::~scoped_ptr_impl > (this=0x7fb46d6a6fe8, __in_chrg=<optimized out>) at > third_party/chromium/src/base/memory/scoped_ptr.h:220 > #8 0x00007fb46cf9ef6c in scoped_ptr<pagespeed::js::JsTokenizerPatterns, > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns> >::~scoped_ptr > (this=0x7fb46d6a6fe8, > __in_chrg=<optimized out>) at > third_party/chromium/src/base/memory/scoped_ptr.h:310 > #9 0x00007fb46cf9ef33 in net_instaweb::ProcessContext::~ProcessContext > (this=0x7fb46d6a6fe8, __in_chrg=<optimized out>) at > net/instaweb/rewriter/process_context.cc:54 > #10 0x00007fb46cad3969 in net_instaweb::(anonymous > namespace)::ApacheProcessContext::~ApacheProcessContext > (this=0x7fb46d6a6fe0, __in_chrg=<optimized out>) > at net/instaweb/apache/mod_instaweb.cc:313 > #11 0x00007fb47775b901 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #12 0x00007fb47775b985 in exit () from /lib/x86_64-linux-gnu/libc.so.6 > #13 0x00007fb46dddfd96 in wsgi_start_process (p=<optimized out>, > daemon=<optimized out>) at mod_wsgi.c:11969 > #14 0x00007fb46dde1344 in wsgi_start_daemons (p=0x7fb478bac138) at > mod_wsgi.c:12166 > #15 wsgi_hook_init (pconf=0x7fb478bac138, ptemp=<optimized out>, > plog=<optimized out>, s=<optimized out>) at mod_wsgi.c:13737 > #16 0x00007fb478633113 in ap_run_post_config (pconf=0x7fb478bac138, > plog=0x7fb478bd9378, ptemp=0x7fb478bd7348, s=0x7fb478bd5538) at config.c:106 > #17 0x00007fb478608993 in main (argc=15, argv=0x7fff2ee8cfd8) at main.c:765 > On Monday, May 12, 2014 4:07:35 PM UTC-7, Graham Dumpleton wrote: >> >> Can you point out to me where in the Apache 2.4 code base it calls >> atexit() to register anything on process shutdown? >> >> Neither Apache nor the underlying APR/APU libraries that it uses rely on >> atexit() to have anything triggered on process shutdown that I know of and >> I cannot find anything in the code I have handy for those which uses >> atexit() in such a generic way. >> >> Normally Apache relies on cleanup actions attached to deletion of memory >> pools and not atexit(). Thus it requires orderly Apache process shutdown >> and for memory pools to be destroyed for actions to be performed on process >> shutdown. The destruction of memory pools is not triggered via atexit(). >> >> Do you also have a more extensive stack trace that that one line so I can >> see in what actual code the crash occurs? That may give me more clues. >> >> Graham >> >> On 13/05/2014, at 8:58 AM, Alex Wu <[email protected]> wrote: >> >> we do not specifically add hook to atexit. It is called/triggered by >> apache frame work when a module is written within the apache 2.4 frame >> work. Also, mod_pagespeed used scoped point on their server context, it >> triggers auto clean once exit is called and library is unloaded. >> >> Alex >> >> >> >> On Monday, May 12, 2014 3:40:26 PM UTC-7, Graham Dumpleton wrote: >>> >>> If your own Apache modules are using atexit() to perform cleanup on >>> process exit, rather than Apache's own mechanisms for performing cleanup >>> actions when the pool the module uses is cleaned up, then the atexit() >>> callback will have to take into consideration that under mod_wsgi when >>> using daemon mode, that the Apache module child init handler will not be >>> called in the daemon process for your Apache module. Thus the callback >>> should check whether global data pointers are in fact non NULL before >>> trying to do things with them. >>> >>> Can you confirm you are using atexit() callbacks in C code with your >>> Apache modules and explain at what point you are registering the callback >>> with atexit()? >>> >>> Is there a specific reason you are using atexit() callbacks rather than >>> doing the normal thing of in the Apache module child init handler >>> registering a cleanup callback on the memory pool given to the Apache >>> module on child init and relying on that being triggered by Apache when >>> shutting things down? >>> >>> Graham >>> >>> On 13/05/2014, at 8:23 AM, Alex Wu <[email protected]> wrote: >>> >>> some are our own, one is mod_pagespeed. We use python 2.7.3 with apache >>> 2.4.7 in MPM mode. The segmentation fault is cleanup routine of each >>> modules other than mod_wsgi after exit call. >>> >>> Alex >>> >>> >>> On Monday, May 12, 2014 1:50:35 PM UTC-7, Graham Dumpleton wrote: >>>> >>>> On 13/05/2014, at 4:40 AM, Alex Wu <[email protected]> wrote: >>>> >>>> > We have observed various segmentation fault caused by exit call from >>>> mod_wsgi 3.5: >>>> > >>>> > #20 0x00007f9490a94d96 in wsgi_start_process (p=<optimized out>, >>>> daemon=<optimized out>) at mod_wsgi.c:11969 >>>> > >>>> > The exit call triggers cleanup from other modules, that cleanup >>>> caused segmentation fault, >>>> >>>> What version of Apache and Python are you using? >>>> >>>> What other non standard Apache modules are you using? For example, is >>>> PHP being used in the same Apache instance? >>>> >>>> Graham >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
