Hi everybody, I'm happy to announce that I was now able to find a solution to this problem which seems to work for me. I would like to discuss my solution with you experts for possible corrections and/or improvements.
1. As I didn't find any place where I could put an arbitrarily executed timer,
I decided to go with a Boolean option to enable/disable the DNS address issue
resolving.
2. I have put the handling code into the function ap_proxy_release_connection()
of module proxy_util.c
3. The code makes a call to apr_sockaddr_info_get() to check if the destination
address has changed. If it has changed, the change is propagated to all places
I thought to be relevant and then the connection in question is marked as being
due to be closed. This is the code I came up with:
if (dnsto) {
apr_status_t return_value;
const char *hostname = conn->worker->hostname;
const apr_port_t port = conn->worker->port;
apr_sockaddr_t *old_sock_addr = conn->addr;
apr_sockaddr_t *new_sock_addr;
apr_pool_t *pool;
return_value = apr_pool_create(&pool, conn->pool);
/*
* Obtain the currently valid address from the OS
*/
apr_sockaddr_info_get(&new_sock_addr, hostname, APR_UNSPEC, port, 0,
pool);
get_addr(old_address, old_sock_addr);
get_addr(new_address, new_sock_addr);
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
"proxy: ip of connection is %pI, the current resolved
ip is %pI",
old_sock_addr, new_sock_addr);
if (strcmp(old_address, new_address)) {
ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,
"proxy: Invalidating the connection due to change in
dns resolved address.");
/*
* If the address has changed, store the changed
* address in all relevant places and mark the
* connection as due to be closed.
*/
conn->worker->cp->addr = new_sock_addr;
conn->addr = new_sock_addr;
conn->connection->remote_addr = new_sock_addr;
conn->close = 1;
}
}
This code is located right before the call to connection_cleanup()
The get_addr() function isn't really necessary and is used here just for
debugging purposes. The comparison (at the moment a strcmp()) could easily be
replaced by a check based on the equality of the two apr_sockaddr_t-s.
The code as presented here has been tested on Linux. I hope, although I didn't
test it yet, that it also will work on Solaris, as this is my real target
platform. With this hack, as soon as the DNS resolver switches the ip address
of the target, all the connections in the proxy being released are checked and
marked to be closed. They are rebuilt at the time of the next access with the
new address. This results in a correct switch-over. Backed up with the already
possible smax=0 and ttl=30 settings I have a double-strategy where all active
connections are closed more or less immediately (as soon as they are released),
while all less active connections are "garbage collected" by the normal
connection inactivity timeout. This is exactly what I want to achieve.
There are still some questions, though:
A. Is this the right place to put such a check?
B. Is it possible to have a timer to do this check instead?
C. Is the pool handling correct or does it lead to a memory leak?
D. Is the method to clean up and close the connections with obsolete ip
addresses (the last four lines of the code above) the right approach?
Thanks for any help.
Regards
Slawo.
-----Ursprüngliche Nachricht-----
Von: Janotta Slawomir Richard, PF54 extern
Gesendet: Montag, 2. Mai 2011 11:26
An: i.ga...@brainsware.org; dev@httpd.apache.org
Betreff: AW: Problem with DNS lookup caching in reverse proxy
Hi Igor and everybody :-)
Unfortunately, this is exactly what won't work for me.
The point is: In our case the remote address is still valid. The "old" node is
not to be shut down so the connection to it is technically still valid.
Instead, the full qualified address if looked up via the resolver now points to
a "new" node. If I'm correct, in this case there won't be any error. And if
there is a lot of traffic on the connection, there also won't be any timeout
and the connection will be happy accessing the "old" node because it simply
uses the stored ip-address and doesn't see any reason to discard it. If I set
ttl to say 1 second (way too small for me in fact) and the connection is
accessed every 100ms, there is no way to timeout the connection. The only
solution I can think of is to force the expiry after a set timeout.
Having said that I would like to ask some additional help with implementing
such a solution. Unfortunately, I'm pretty new to Apache and have to work it
out "the hard way". So far I have worked out how to enable the additional
parameter for the mod_proxy. I have done it by adding the setting dnsto (for
DNS timeout) to the structure proxy_worker in mod_proxy.h. The question is:
where can I register the timeout to be used and how do I force the worker to
re-resolve the address.
Regards
Slawo.
-----Ursprüngliche Nachricht-----
Von: Igor Galić [mailto:i.ga...@brainsware.org]
Gesendet: Dienstag, 26. April 2011 23:45
An: dev@httpd.apache.org
Betreff: Re: Problem with DNS lookup caching in reverse proxy
----- Original Message -----
> Hello everybody,
>
> I have tried to solve my issue by contacting the users@ mailing list
> but meanwhile I think this is the more appropriate list to address
> it.
> I have issues with the mod_proxy resolving a full qualified
> servername via DNS and caching it in its connection pool so a change
> in the resolver doesn't get noticed by mod_proxy. After trying all
> the different possibilities that came to mind I'm back at where it
> all started.
> I still cannot find a solution to the problem that mod_proxy caches
> the IP address resolved by DNS and thus doesn't acknowledge all
> changes to it. With the help of Igor Galić on the users@ mailing
> list and by means of Bug Report 50551 I am able to get about 95% of
> the cases solved. But there still are cases where the resolved
> address won't be changed.
> But first let me describe the configuration more deeply:
>
> What we have is a configuration where the Apache Reverse Proxy (RPX)
> is used to allow transparent switch of the backend JEE server. We
> solve this by configuring an alias to the service on the backend.
> The alias is then entered in whatever DNS resolver we use to point
> to the one system which is currently meant to receive the request.
> This allows us to make application changes on one system while the
> other is still working, then transparently switching to the updates
> server to do maintenance on the other.
>
> Now the problem with this is that Apache, or more exactly the
> mod_proxy, has a connection pool mechanism which apparently binds to
> a once resolved IP address "forever". See the type struct
> proxy_conn_pool in file mod_proxy.h or just my description in Bug
> Report 50551.
>
> It appears to me, and I believe I was able to show this by extensive
> testing, that the member variable apr_sockaddr_t *addr /* Preparsed
> remote address info */
> in this struct is responsible for this behavior and that this
> preparsed remote address is invalidated only if the last pooled
> connection in this connection pool is timed out.
>
> In my opinion the best solution to this problem would be to add an
> additional perameter to mod_proxy to force a renewal of the DNS
> resolve procedure for this address after a set timeout like it is
> possible in mod_weblogic. Unfortunately this is way beyond my
Instead of re-resolving after a set timeout, it would make sense
to re-resolve after a timeout occured, marking the worker in error.
The only question is: Is that the struct shared across the pool?
> capabilities at the moment to implement it myself. But I wonder why
> there seems to be no need for such an option so nobody implemented
> it yet. Perhaps there is another I'm still not aware of?
>
> Regards
> Slawo
Sicherheitshinweis:
Dieses E-Mail von PostFinance ist signiert. Weitere Informationen finden Sie
unter:
https://www.postfinance.ch/e-signature.
Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt.
smime.p7s
Description: S/MIME Cryptographic Signature
