Re: Problem with DNS lookup caching in reverse proxy
- Original Message - > Hello everybody, > > I have tried to solve my issue by contacting the users@ mailing list > but meanwhile I think this is the more appropriate list to address > it. > I have issues with the mod_proxy resolving a full qualified > servername via DNS and caching it in its connection pool so a change > in the resolver doesn't get noticed by mod_proxy. After trying all > the different possibilities that came to mind I'm back at where it > all started. > I still cannot find a solution to the problem that mod_proxy caches > the IP address resolved by DNS and thus doesn't acknowledge all > changes to it. With the help of Igor Galić on the users@ mailing > list and by means of Bug Report 50551 I am able to get about 95% of > the cases solved. But there still are cases where the resolved > address won't be changed. > But first let me describe the configuration more deeply: > > What we have is a configuration where the Apache Reverse Proxy (RPX) > is used to allow transparent switch of the backend JEE server. We > solve this by configuring an alias to the service on the backend. > The alias is then entered in whatever DNS resolver we use to point > to the one system which is currently meant to receive the request. > This allows us to make application changes on one system while the > other is still working, then transparently switching to the updates > server to do maintenance on the other. > > Now the problem with this is that Apache, or more exactly the > mod_proxy, has a connection pool mechanism which apparently binds to > a once resolved IP address "forever". See the type struct > proxy_conn_pool in file mod_proxy.h or just my description in Bug > Report 50551. > > It appears to me, and I believe I was able to show this by extensive > testing, that the member variable apr_sockaddr_t *addr /* Preparsed > remote address info */ > in this struct is responsible for this behavior and that this > preparsed remote address is invalidated only if the last pooled > connection in this connection pool is timed out. > > In my opinion the best solution to this problem would be to add an > additional perameter to mod_proxy to force a renewal of the DNS > resolve procedure for this address after a set timeout like it is > possible in mod_weblogic. Unfortunately this is way beyond my Instead of re-resolving after a set timeout, it would make sense to re-resolve after a timeout occured, marking the worker in error. The only question is: Is that the struct shared across the pool? > capabilities at the moment to implement it myself. But I wonder why > there seems to be no need for such an option so nobody implemented > it yet. Perhaps there is another I'm still not aware of? > > Regards > Slawo > > > > -Ursprüngliche Nachricht- > Von: Slawomir R. Janotta [mailto:s.jano...@aviope.de] > Gesendet: Freitag, 3. Dezember 2010 16:54 > An: us...@httpd.apache.org > Betreff: Re: [users@httpd] Problem with DNS lookup caching in reverse > proxy > > > > > Would it be a viable option for you to not use the DNS method, and > > instead do something like: > > > > > > BalanceMember https://backend1.mydomain.com:8080/ > > BalanceMember https://backend1.mydomain.com:8080/ status=+H > > > > > > ProxyPass / balancer://notsohotcluster > > ProxyPassReverse / balancer://notsohotcluster > > > > This will (probably) simply mitigate the DNS problem. > > > > > > i > > > > -- > > Igor Galić > > > > Tel: +43 (0) 664 886 22 883 > > Mail: i.ga...@brainsware.org > > URL: http://brainsware.org/ > > > > > Thank you very much, Igor. > Indeed, your suggestion solves most of the cases where I encounter > the > switch-over of the backend hosts. > However, this does *not* help in all cases, as this works only if the > backend is no more reachable and thus an error condition occurs. > Sometimes > I just want to switch from one backend to the other without shutting > any > of them. A working TTL in the connection pool or in the DNS cache > would be > ideal for this. > Perhaps I have to open a new Bug report in Bugzilla. > Regards > Slawo. > > > Sicherheitshinweis: > Dieses E-Mail von PostFinance ist signiert. Weitere Informationen > finden Sie unter: > https://www.postfinance.ch/e-signature. > Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt. -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
Re: svn commit: r1057048 - in /httpd/httpd/trunk: CHANGES modules/generators/mod_status.c
On 26.04.2011 21:57, Jeff Trawick wrote: On Sun, Jan 9, 2011 at 6:00 PM, wrote: Author: sf Date: Sun Jan 9 23:00:33 2011 New Revision: 1057048 URL: http://svn.apache.org/viewvc?rev=1057048&view=rev Log: mod_status: Don't show slots which are disabled by MaxClients as open. They now get displayed as ' ' instead of '.'. No complaints here about not displaying unused slots as open, but the result can look a bit odd. -cut here--- 1 requests currently being processed, 49 idle workers _ __W__ Scoreboard Key: ---cut here-- On Windows, with huge default ThreadLimit relative to ThreadsPerChild, you're left with lots of unexpected whitespace. What about just surpressing the ' ' when printing out the worker map? After a graceful restart which shrinks MaxClients or ThreadsPerChild you might have the map gradually get smaller for a short time as workers in those "beyond" scoreboard entries exit, but generally it would look better. Comments? (Do folks want the space used to reflect ThreadLimit * ServerLimit?) Modified: httpd/httpd/trunk/modules/generators/mod_status.c URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/generators/mod_status.c?rev=1057048&r1=1057047&r2=1057048&view=diff == --- httpd/httpd/trunk/modules/generators/mod_status.c (original) +++ httpd/httpd/trunk/modules/generators/mod_status.c Sun Jan 9 23:00:33 2011 @@ -483,7 +492,8 @@ static int status_handler(request_rec *r ap_rputs("\"L\" Logging, \n", r); ap_rputs("\"G\" Gracefully finishing, \n", r); ap_rputs("\"I\" Idle cleanup of worker, \n", r); -ap_rputs("\".\" Open slot with no current process\n", r); +ap_rputs("\".\" Open slot with no current process,\n", r); +ap_rputs("\" \" Slot disabled by MaxClients setting\n", r); (and/or by ThreadsPerChild) I'd say the additional slots are more of an implementation detail (pre-allocated to allow increasing the actual available processes and threads by graceful restart) and do not provide relevant information for mod_status. So I am +1 for dropping, but can live with white space too. Regards, Rainer
Re: svn commit: r1057048 - in /httpd/httpd/trunk: CHANGES modules/generators/mod_status.c
On Sun, Jan 9, 2011 at 6:00 PM, wrote: > Author: sf > Date: Sun Jan 9 23:00:33 2011 > New Revision: 1057048 > > URL: http://svn.apache.org/viewvc?rev=1057048&view=rev > Log: > mod_status: Don't show slots which are disabled by MaxClients as open. They now get displayed as ' ' instead of '.'. No complaints here about not displaying unused slots as open, but the result can look a bit odd. -cut here--- 1 requests currently being processed, 49 idle workers _ __W__ Scoreboard Key: ---cut here-- On Windows, with huge default ThreadLimit relative to ThreadsPerChild, you're left with lots of unexpected whitespace. What about just surpressing the ' ' when printing out the worker map? After a graceful restart which shrinks MaxClients or ThreadsPerChild you might have the map gradually get smaller for a short time as workers in those "beyond" scoreboard entries exit, but generally it would look better. Comments? (Do folks want the space used to reflect ThreadLimit * ServerLimit?) > Modified: httpd/httpd/trunk/modules/generators/mod_status.c > URL: > http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/generators/mod_status.c?rev=1057048&r1=1057047&r2=1057048&view=diff > == > --- httpd/httpd/trunk/modules/generators/mod_status.c (original) > +++ httpd/httpd/trunk/modules/generators/mod_status.c Sun Jan 9 23:00:33 2011 > @@ -483,7 +492,8 @@ static int status_handler(request_rec *r > ap_rputs("\"L\" Logging, \n", r); > ap_rputs("\"G\" Gracefully finishing, \n", > r); > ap_rputs("\"I\" Idle cleanup of worker, \n", r); > - ap_rputs("\".\" Open slot with no current > process\n", r); > + ap_rputs("\".\" Open slot with no current > process,\n", r); > + ap_rputs("\" \" Slot disabled by MaxClients > setting\n", r); (and/or by ThreadsPerChild)
Re: mod_fcgid can kill all the services on the server via kill -15 -1
Ryan, I like this approach. Do you want us to prepare a patch against latest version? Also, we never committed to apache project. Would it be enough if we post a patch here. Or what would be the process? On Sun, Apr 17, 2011 at 9:58 PM, pqf wrote: > Hi, all > Another question, does proc_wait_process() should update procnode->proc_id > to 0 too? or else mod_fcgid may send a signal to another irrelevant process > while apache is shutting down? I don't follow up mod_fcgid for a while, I > just took a glance, maybe it's updated somewhere else? > By the way, procnode->proc_id is set to 0 while apache startup, so why not > update procnode->proc_id to 0 while fcgid_create_privileged_process() is > fail? And then check magic number 0 rather than both -1 and 0, in both > proc_kill_gracefully() and proc_kill_force(). > > Cheers. > Ryan > > > > Hello, > > There is a very interesting, and quite a rare bug in mod_fcgid. It is easy > to reproduce if you can cause fork to fail (which can be done with > CloudLinux -- if anyone wants to replicate it). > > *Here is how it works: * > mod_fcgid tries to spawn a new process (proc_spawn_process in > fcgid_proc_unix.c), but fork returns -1. > More exactly fcgid_create_privileged_process function call returns error, > and fills in tmpproc.pid with -1 & tmpproc is assiged to procnode->proc_id). > > Now, if at the same time service httpd restart is executed, function > kill_all_subprocess in fcgid_pm_main.c will execute, and it will try to go > through all procnodes, sending SIGTERM via proc_kill_gracefully, (and then > SIGKILL via proc_kill_force) to procnode->proc_id.pid > Yet, one procnode will be pointing to procnode->proc_id.pid, causing kill > -15 -1 (kill all). > The end results all services on the server failing, including SSH, apache, > syslogd, etc.. > > I guess the problem is really rare for most people. Also it is quite hard > to diagnose, as it is completely not clear where the signal came from, and > it took us some time to correlate them with apache restarts.. Yet due to our > OS being used by shared hosts (where httpd restart is common thing), and our > ability to limit memory available to processes on per virtual host bases > (which causes fork to fail once that virtual host reaches memory limit), we > see the issue quite often. > > The solution is quite simple (not sure if it is the best / right solution), > in file: fcgid_proc_unix.c, in methods proc_kill_gracefully, line: > > rv = apr_proc_kill(&(procnode->proc_id), SIGTERM); > > should be changed to: >if (procnode->proc_id.pid != -1) { > rv = apr_proc_kill(&(procnode->proc_id), SIGTERM); >} else { > rv = APR_SUCCESS; >} > > Similarly in proc_kill_force > rv = apr_proc_kill(&(procnode->proc_id), SIGKILL); > should be changed to: >if (procnode->proc_id.pid != -1) { > rv = apr_proc_kill(&(procnode->proc_id), SIGKILL); >} else { > rv = APR_SUCCESS; >} > > Regards, > Igor Seletskiy > CEO @ Cloud Linux Inc > > >
Re: svn commit: r1096577 - in /httpd/httpd/trunk: CHANGES docs/manual/mod/mod_ldap.xml modules/ldap/util_ldap.c modules/ldap/util_ldap_cache.c modules/ldap/util_ldap_cache_mgr.c
On Tue, 26 Apr 2011, Ruediger Pluem wrote: On 04/25/2011 10:00 PM, s...@apache.org wrote: Author: sf Date: Mon Apr 25 20:00:43 2011 New Revision: 1096577 URL: http://svn.apache.org/viewvc?rev=1096577&view=rev Log: mod_ldap: Make LDAPSharedCacheSize 0 create a non-shared-memory cache per process as opposed to disabling caching completely. This allows to use the non-shared-memory cache as a workaround for the shared memory cache not being available during graceful restarts PR: 48958 Modified: httpd/httpd/trunk/CHANGES httpd/httpd/trunk/docs/manual/mod/mod_ldap.xml httpd/httpd/trunk/modules/ldap/util_ldap.c httpd/httpd/trunk/modules/ldap/util_ldap_cache.c httpd/httpd/trunk/modules/ldap/util_ldap_cache_mgr.c Modified: httpd/httpd/trunk/modules/ldap/util_ldap_cache_mgr.c URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ldap/util_ldap_cache_mgr.c?rev=1096577&r1=1096576&r2=1096577&view=diff == --- httpd/httpd/trunk/modules/ldap/util_ldap_cache_mgr.c (original) +++ httpd/httpd/trunk/modules/ldap/util_ldap_cache_mgr.c Mon Apr 25 20:00:43 2011 @@ -363,6 +366,14 @@ util_ald_cache_t *util_ald_create_cache( cache->nodes = (util_cache_node_t **)util_ald_alloc(cache, cache->size * sizeof(util_cache_node_t *)); if (!cache->nodes) { util_ald_free(cache, cache); +#if APR_HAS_SHARED_MEMORY +if (!st->cache_rmm) +free(cache); +else +apr_rmm_free(st->cache_rmm, block); +#else +free(cache); Do we ever alloc cache #if !APR_HAS_SHARED_MEMORY? Yes, it's simply allocated with calloc. But my commit was wrong anyway, because the util_ald_free(cache, cache) in the line above already frees cache. Thanks for the review. Cheers, Stefan
Re: [PATCH] check_forensic re tweak
- Original Message - > With mod_unique_id installed there's the possibility > of having a - in the forensic id, so here's a minor > patch to ensure check_forensic does the right thing. > > Index: support/check_forensic > === > --- support/check_forensic(revision 1094142) > +++ support/check_forensic(working copy) > @@ -43,8 +43,8 @@ > trap "rm -f -- \"$all\" \"$in\" \"$out\";" 0 1 2 3 13 15 > > cut -f 1 -d '|' $F > $all > -grep + < $all | cut -c2- | sort > $in > -grep -- - < $all | cut -c2- | sort > $out > +grep ^+ < $all | cut -c2- | sort > $in > +grep -- ^- < $all | cut -c2- | sort > $out > > # use -i instead of -I for GNU xargs > join -v 1 $in $out | xargs -I xx egrep "^\\+xx" $F Thanks joe - applied in r1096775 i -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: i.ga...@brainsware.org URL: http://brainsware.org/