Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
On 12/02/2009 09:09 PM, Rainer Jung wrote: > Platform: Solaris 8 (sic!) > MPM: worker dynamically loaded > APR etc: Bundled > PCRE: 7.8 > > During testing of 2.3.4 I noticed crashes after restart. > > I did a build with lots of modules, especially including mod_logio. The > scoreboard uses in ap_increment_counts() the optional function > ap_logio_get_last_bytes() from mod_logio if available. > > In my case after restart the memory location of mod_logio and the > address of the optional function changed, but the scoreboard still tries > to call to the original address retrieved after the initial start. > > I don't know about the full implementation of the optional functions, > but it seems either > > APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes); > > in register_hooks in mod_logio needs to run after restarts too, or there > is a problem resulting in an unwanted change of load order of the > modules during restart. I did not edit the config files between start > and restart. > > The problem happens with normal restarts and graceful restarts. > > Wild guess: it might have to do with dynamic MPM loading. I am not so sure. I guess the problem is that we call APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes); only in ap_calc_scoreboard_size which is IMHO not called at restarts. Does the following totally untested patch fix your issue? Index: scoreboard.c === --- scoreboard.c(Revision 886294) +++ scoreboard.c(Arbeitskopie) @@ -284,6 +284,8 @@ apr_status_t rv; #endif +pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes); + if (ap_scoreboard_image) { running_gen = ap_scoreboard_image->global->running_generation; ap_scoreboard_image->global->restart_time = apr_time_now(); Regards Rüdiger
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
On Wed, Dec 2, 2009 at 3:33 PM, Ruediger Pluem wrote: > > > On 12/02/2009 09:09 PM, Rainer Jung wrote: >> Platform: Solaris 8 (sic!) >> MPM: worker dynamically loaded >> APR etc: Bundled >> PCRE: 7.8 >> >> During testing of 2.3.4 I noticed crashes after restart. >> >> I did a build with lots of modules, especially including mod_logio. The >> scoreboard uses in ap_increment_counts() the optional function >> ap_logio_get_last_bytes() from mod_logio if available. >> >> In my case after restart the memory location of mod_logio and the >> address of the optional function changed, but the scoreboard still tries >> to call to the original address retrieved after the initial start. >> >> I don't know about the full implementation of the optional functions, >> but it seems either >> >> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes); >> >> in register_hooks in mod_logio needs to run after restarts too, or there >> is a problem resulting in an unwanted change of load order of the >> modules during restart. I did not edit the config files between start >> and restart. >> >> The problem happens with normal restarts and graceful restarts. >> >> Wild guess: it might have to do with dynamic MPM loading. > > I am not so sure. I guess the problem is that we call > > APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes); > > only in ap_calc_scoreboard_size which is IMHO not called at restarts. That was my guess. Which reminds me of this old challenge for a module I wrote: * XXX * our daemon must be created in the pre-mpm hook so that the scoreboard * is available in the daemon process; the scoreboard isn't created until * the core module's pre-mpm hook * BUT: the pre-mpm hook is not called for graceful restart, so we can't * start a new instance of the daemon then * BUT: request processing from the daemon MUST use the new configuration, * so we'd be busted for graceful restart
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
Hi Rüdiger, On 02.12.2009 21:33, Ruediger Pluem wrote: On 12/02/2009 09:09 PM, Rainer Jung wrote: Platform: Solaris 8 (sic!) MPM: worker dynamically loaded APR etc: Bundled PCRE: 7.8 During testing of 2.3.4 I noticed crashes after restart. I did a build with lots of modules, especially including mod_logio. The scoreboard uses in ap_increment_counts() the optional function ap_logio_get_last_bytes() from mod_logio if available. In my case after restart the memory location of mod_logio and the address of the optional function changed, but the scoreboard still tries to call to the original address retrieved after the initial start. I don't know about the full implementation of the optional functions, but it seems either APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes); in register_hooks in mod_logio needs to run after restarts too, or there is a problem resulting in an unwanted change of load order of the modules during restart. I did not edit the config files between start and restart. The problem happens with normal restarts and graceful restarts. Wild guess: it might have to do with dynamic MPM loading. I am not so sure. I guess the problem is that we call APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes); only in ap_calc_scoreboard_size which is IMHO not called at restarts. Does the following totally untested patch fix your issue? Index: scoreboard.c === --- scoreboard.c(Revision 886294) +++ scoreboard.c(Arbeitskopie) @@ -284,6 +284,8 @@ apr_status_t rv; #endif +pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes); + if (ap_scoreboard_image) { running_gen = ap_scoreboard_image->global->running_generation; ap_scoreboard_image->global->restart_time = apr_time_now(); Yes, that fixes it, makes a lot of sense. What's interesting is, that the function address changes during the first restart, but not any more when doing further restarts. So the initial start ends up with a different load order than all of the following restarts. Something to keep in mind at least in case it's new in 2.3 - and about 60 other places to check which use optional functions, whether the function pointers are correctly reinitialized. Thanks! Will you commit? Rainer
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
On 12/02/2009 09:53 PM, Rainer Jung wrote: > > Yes, that fixes it, makes a lot of sense. > > What's interesting is, that the function address changes during the > first restart, but not any more when doing further restarts. So the > initial start ends up with a different load order than all of the > following restarts. Something to keep in mind at least in case it's new > in 2.3 - and about 60 other places to check which use optional > functions, whether the function pointers are correctly reinitialized. > > Thanks! > > Will you commit? Done as r886324. Regards Rüdiger