Problem with DNS lookup caching in reverse proxy
Tnks,
I found it in my Spambox
Met vriendelijke groet,
Gilbert van Houten
Wijn.org
Woerden
i...@wijn.org
T 0348-483838
0622-488964
http://shop.wijn.org
<>
On 13-mei-2011, at 11:19, slawomir.jano...@postfinance.ch wrote:
Hi everybody,
I'm happy to announce that I was now able to find a solution to this
problem which seems to work for me.
I would like to discuss my solution with you experts for possible
corrections and/or improvements.
1. As I didn't find any place where I could put an arbitrarily
executed timer, I decided to go with a Boolean option to enable/
disable the DNS address issue resolving.
2. I have put the handling code into the function
ap_proxy_release_connection() of module proxy_util.c
3. The code makes a call to apr_sockaddr_info_get() to check if the
destination address has changed. If it has changed, the change is
propagated to all places I thought to be relevant and then the
connection in question is marked as being due to be closed. This is
the code I came up with:
if (dnsto) {
apr_status_t return_value;
const char *hostname = conn->worker->hostname;
const apr_port_t port = conn->worker->port;
apr_sockaddr_t *old_sock_addr = conn->addr;
apr_sockaddr_t *new_sock_addr;
apr_pool_t *pool;
return_value = apr_pool_create(&pool, conn->pool);
/*
* Obtain the currently valid address from the OS
*/
apr_sockaddr_info_get(&new_sock_addr, hostname, APR_UNSPEC, port, 0,
pool);
get_addr(old_address, old_sock_addr);
get_addr(new_address, new_sock_addr);
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
"proxy: ip of connection is %pI, the current resolved ip is
%pI",
old_sock_addr, new_sock_addr);
if (strcmp(old_address, new_address)) {
ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,
"proxy: Invalidating the connection due to change in dns resolved
address.");
/*
* If the address has changed, store the changed
* address in all relevant places and mark the
* connection as due to be closed.
*/
conn->worker->cp->addr = new_sock_addr;
conn->addr = new_sock_addr;
conn->connection->remote_addr = new_sock_addr;
conn->close = 1;
}
}
This code is located right before the call to connection_cleanup()
The get_addr() function isn't really necessary and is used here just
for debugging purposes. The comparison (at the moment a strcmp())
could easily be replaced by a check based on the equality of the two
apr_sockaddr_t-s.
The code as presented here has been tested on Linux. I hope, although
I didn't test it yet, that it also will work on Solaris, as this is
my real target platform. With this hack, as soon as the DNS resolver
switches the ip address of the target, all the connections in the
proxy being released are checked and marked to be closed. They are
rebuilt at the time of the next access with the new address. This
results in a correct switch-over. Backed up with the already possible
smax=0 and ttl=30 settings I have a double-strategy where all active
connections are closed more or less immediately (as soon as they are
released), while all less active connections are "garbage collected"
by the normal connection inactivity timeout. This is exactly what I
want to achieve.
There are still some questions, though:
A. Is this the right place to put such a check?
B. Is it possible to have a timer to do this check instead?
C. Is the pool handling correct or does it lead to a memory leak?
D. Is the method to clean up and close the connections with obsolete
ip addresses (the last four lines of the code above) the right approach?
Thanks for any help.
Regards
Slawo.
-Ursprüngliche Nachricht-
Von: Janotta Slawomir Richard, PF54 extern
Gesendet: Montag, 2. Mai 2011 11:26
An: i.ga...@brainsware.org; dev@httpd.apache.org
Betreff: AW: Problem with DNS lookup caching in reverse proxy
Hi Igor and everybody :-)
Unfortunately, this is exactly what won't work for me.
The point is: In our case the remote address is still valid. The
"old" node is not to be shut down so the connection to it is
technically still valid. Instead, the full qualified address if
looked up via the resolver now points to a "new" node. If I'm
correct, in this case there won't be any error. And if there is a lot
of traffic on the connection, there also won't be any timeout and the
connection will be happy accessing the "old" node because it simply
uses the stored ip-address and doesn't see any reason to discard it.
If I set ttl to say 1 second (way too small for me in fact
RE: Problem with DNS lookup caching in reverse proxy
Hi everybody,
I'm happy to announce that I was now able to find a solution to this problem
which seems to work for me.
I would like to discuss my solution with you experts for possible corrections
and/or improvements.
1. As I didn't find any place where I could put an arbitrarily executed timer,
I decided to go with a Boolean option to enable/disable the DNS address issue
resolving.
2. I have put the handling code into the function ap_proxy_release_connection()
of module proxy_util.c
3. The code makes a call to apr_sockaddr_info_get() to check if the destination
address has changed. If it has changed, the change is propagated to all places
I thought to be relevant and then the connection in question is marked as being
due to be closed. This is the code I came up with:
if (dnsto) {
apr_status_t return_value;
const char *hostname = conn->worker->hostname;
const apr_port_t port = conn->worker->port;
apr_sockaddr_t *old_sock_addr = conn->addr;
apr_sockaddr_t *new_sock_addr;
apr_pool_t *pool;
return_value = apr_pool_create(&pool, conn->pool);
/*
* Obtain the currently valid address from the OS
*/
apr_sockaddr_info_get(&new_sock_addr, hostname, APR_UNSPEC, port, 0,
pool);
get_addr(old_address, old_sock_addr);
get_addr(new_address, new_sock_addr);
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
"proxy: ip of connection is %pI, the current resolved
ip is %pI",
old_sock_addr, new_sock_addr);
if (strcmp(old_address, new_address)) {
ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,
"proxy: Invalidating the connection due to change in
dns resolved address.");
/*
* If the address has changed, store the changed
* address in all relevant places and mark the
* connection as due to be closed.
*/
conn->worker->cp->addr = new_sock_addr;
conn->addr = new_sock_addr;
conn->connection->remote_addr = new_sock_addr;
conn->close = 1;
}
}
This code is located right before the call to connection_cleanup()
The get_addr() function isn't really necessary and is used here just for
debugging purposes. The comparison (at the moment a strcmp()) could easily be
replaced by a check based on the equality of the two apr_sockaddr_t-s.
The code as presented here has been tested on Linux. I hope, although I didn't
test it yet, that it also will work on Solaris, as this is my real target
platform. With this hack, as soon as the DNS resolver switches the ip address
of the target, all the connections in the proxy being released are checked and
marked to be closed. They are rebuilt at the time of the next access with the
new address. This results in a correct switch-over. Backed up with the already
possible smax=0 and ttl=30 settings I have a double-strategy where all active
connections are closed more or less immediately (as soon as they are released),
while all less active connections are "garbage collected" by the normal
connection inactivity timeout. This is exactly what I want to achieve.
There are still some questions, though:
A. Is this the right place to put such a check?
B. Is it possible to have a timer to do this check instead?
C. Is the pool handling correct or does it lead to a memory leak?
D. Is the method to clean up and close the connections with obsolete ip
addresses (the last four lines of the code above) the right approach?
Thanks for any help.
Regards
Slawo.
-Ursprüngliche Nachricht-
Von: Janotta Slawomir Richard, PF54 extern
Gesendet: Montag, 2. Mai 2011 11:26
An: i.ga...@brainsware.org; dev@httpd.apache.org
Betreff: AW: Problem with DNS lookup caching in reverse proxy
Hi Igor and everybody :-)
Unfortunately, this is exactly what won't work for me.
The point is: In our case the remote address is still valid. The "old" node is
not to be shut down so the connection to it is technically still valid.
Instead, the full qualified address if looked up via the resolver now points to
a "new" node. If I'm correct, in this case there won't be any error. And if
there is a lot of traffic on the connection, there also won't be any timeout
and the connection will be happy accessing the "old" node because it simply
uses the stored ip-address and doesn't see any reason to discard it. If I set
ttl to say 1 second (way too small for me in fact) and the connection is
accessed every 100ms, there is no way to timeout the connection. The only
solution I can think of is to force the expiry after a set timeout.
Having said that I would like to ask some additional help with implementing
such a solution. U
AW: Problem with DNS lookup caching in reverse proxy
Hi Igor and everybody :-) Unfortunately, this is exactly what won't work for me. The point is: In our case the remote address is still valid. The "old" node is not to be shut down so the connection to it is technically still valid. Instead, the full qualified address if looked up via the resolver now points to a "new" node. If I'm correct, in this case there won't be any error. And if there is a lot of traffic on the connection, there also won't be any timeout and the connection will be happy accessing the "old" node because it simply uses the stored ip-address and doesn't see any reason to discard it. If I set ttl to say 1 second (way too small for me in fact) and the connection is accessed every 100ms, there is no way to timeout the connection. The only solution I can think of is to force the expiry after a set timeout. Having said that I would like to ask some additional help with implementing such a solution. Unfortunately, I'm pretty new to Apache and have to work it out "the hard way". So far I have worked out how to enable the additional parameter for the mod_proxy. I have done it by adding the setting dnsto (for DNS timeout) to the structure proxy_worker in mod_proxy.h. The question is: where can I register the timeout to be used and how do I force the worker to re-resolve the address. Regards Slawo. -Ursprüngliche Nachricht- Von: Igor Galić [mailto:i.ga...@brainsware.org] Gesendet: Dienstag, 26. April 2011 23:45 An: dev@httpd.apache.org Betreff: Re: Problem with DNS lookup caching in reverse proxy - Original Message - > Hello everybody, > > I have tried to solve my issue by contacting the users@ mailing list > but meanwhile I think this is the more appropriate list to address > it. > I have issues with the mod_proxy resolving a full qualified > servername via DNS and caching it in its connection pool so a change > in the resolver doesn't get noticed by mod_proxy. After trying all > the different possibilities that came to mind I'm back at where it > all started. > I still cannot find a solution to the problem that mod_proxy caches > the IP address resolved by DNS and thus doesn't acknowledge all > changes to it. With the help of Igor Galić on the users@ mailing > list and by means of Bug Report 50551 I am able to get about 95% of > the cases solved. But there still are cases where the resolved > address won't be changed. > But first let me describe the configuration more deeply: > > What we have is a configuration where the Apache Reverse Proxy (RPX) > is used to allow transparent switch of the backend JEE server. We > solve this by configuring an alias to the service on the backend. > The alias is then entered in whatever DNS resolver we use to point > to the one system which is currently meant to receive the request. > This allows us to make application changes on one system while the > other is still working, then transparently switching to the updates > server to do maintenance on the other. > > Now the problem with this is that Apache, or more exactly the > mod_proxy, has a connection pool mechanism which apparently binds to > a once resolved IP address "forever". See the type struct > proxy_conn_pool in file mod_proxy.h or just my description in Bug > Report 50551. > > It appears to me, and I believe I was able to show this by extensive > testing, that the member variable apr_sockaddr_t *addr /* Preparsed > remote address info */ > in this struct is responsible for this behavior and that this > preparsed remote address is invalidated only if the last pooled > connection in this connection pool is timed out. > > In my opinion the best solution to this problem would be to add an > additional perameter to mod_proxy to force a renewal of the DNS > resolve procedure for this address after a set timeout like it is > possible in mod_weblogic. Unfortunately this is way beyond my Instead of re-resolving after a set timeout, it would make sense to re-resolve after a timeout occured, marking the worker in error. The only question is: Is that the struct shared across the pool? > capabilities at the moment to implement it myself. But I wonder why > there seems to be no need for such an option so nobody implemented > it yet. Perhaps there is another I'm still not aware of? > > Regards > Slawo Sicherheitshinweis: Dieses E-Mail von PostFinance ist signiert. Weitere Informationen finden Sie unter: https://www.postfinance.ch/e-signature. Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt. smime.p7s Description: S/MIME Cryptographic Signature
Re: Problem with DNS lookup caching in reverse proxy
- Original Message - > Hello everybody, > > I have tried to solve my issue by contacting the users@ mailing list > but meanwhile I think this is the more appropriate list to address > it. > I have issues with the mod_proxy resolving a full qualified > servername via DNS and caching it in its connection pool so a change > in the resolver doesn't get noticed by mod_proxy. After trying all > the different possibilities that came to mind I'm back at where it > all started. > I still cannot find a solution to the problem that mod_proxy caches > the IP address resolved by DNS and thus doesn't acknowledge all > changes to it. With the help of Igor Galić on the users@ mailing > list and by means of Bug Report 50551 I am able to get about 95% of > the cases solved. But there still are cases where the resolved > address won't be changed. > But first let me describe the configuration more deeply: > > What we have is a configuration where the Apache Reverse Proxy (RPX) > is used to allow transparent switch of the backend JEE server. We > solve this by configuring an alias to the service on the backend. > The alias is then entered in whatever DNS resolver we use to point > to the one system which is currently meant to receive the request. > This allows us to make application changes on one system while the > other is still working, then transparently switching to the updates > server to do maintenance on the other. > > Now the problem with this is that Apache, or more exactly the > mod_proxy, has a connection pool mechanism which apparently binds to > a once resolved IP address "forever". See the type struct > proxy_conn_pool in file mod_proxy.h or just my description in Bug > Report 50551. > > It appears to me, and I believe I was able to show this by extensive > testing, that the member variable apr_sockaddr_t *addr /* Preparsed > remote address info */ > in this struct is responsible for this behavior and that this > preparsed remote address is invalidated only if the last pooled > connection in this connection pool is timed out. > > In my opinion the best solution to this problem would be to add an > additional perameter to mod_proxy to force a renewal of the DNS > resolve procedure for this address after a set timeout like it is > possible in mod_weblogic. Unfortunately this is way beyond my Instead of re-resolving after a set timeout, it would make sense to re-resolve after a timeout occured, marking the worker in error. The only question is: Is that the struct shared across the pool? > capabilities at the moment to implement it myself. But I wonder why > there seems to be no need for such an option so nobody implemented > it yet. Perhaps there is another I'm still not aware of? > > Regards > Slawo > > > > -----Ursprüngliche Nachricht----- > Von: Slawomir R. Janotta [mailto:s.jano...@aviope.de] > Gesendet: Freitag, 3. Dezember 2010 16:54 > An: us...@httpd.apache.org > Betreff: Re: [users@httpd] Problem with DNS lookup caching in reverse > proxy > > > > > Would it be a viable option for you to not use the DNS method, and > > instead do something like: > > > > > > BalanceMember https://backend1.mydomain.com:8080/ > > BalanceMember https://backend1.mydomain.com:8080/ status=+H > > > > > > ProxyPass / balancer://notsohotcluster > > ProxyPassReverse / balancer://notsohotcluster > > > > This will (probably) simply mitigate the DNS problem. > > > > > > i > > > > -- > > Igor Galić > > > > Tel: +43 (0) 664 886 22 883 > > Mail: i.ga...@brainsware.org > > URL: http://brainsware.org/ > > > > > Thank you very much, Igor. > Indeed, your suggestion solves most of the cases where I encounter > the > switch-over of the backend hosts. > However, this does *not* help in all cases, as this works only if the > backend is no more reachable and thus an error condition occurs. > Sometimes > I just want to switch from one backend to the other without shutting > any > of them. A working TTL in the connection pool or in the DNS cache > would be > ideal for this. > Perhaps I have to open a new Bug report in Bugzilla. > Regards > Slawo. > > > Sicherheitshinweis: > Dieses E-Mail von PostFinance ist signiert. Weitere Informationen > finden Sie unter: > https://www.postfinance.ch/e-signature. > Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt. -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
Problem with DNS lookup caching in reverse proxy
Hello everybody, I have tried to solve my issue by contacting the users@ mailing list but meanwhile I think this is the more appropriate list to address it. I have issues with the mod_proxy resolving a full qualified servername via DNS and caching it in its connection pool so a change in the resolver doesn't get noticed by mod_proxy. After trying all the different possibilities that came to mind I'm back at where it all started. I still cannot find a solution to the problem that mod_proxy caches the IP address resolved by DNS and thus doesn't acknowledge all changes to it. With the help of Igor Galić on the users@ mailing list and by means of Bug Report 50551 I am able to get about 95% of the cases solved. But there still are cases where the resolved address won't be changed. But first let me describe the configuration more deeply: What we have is a configuration where the Apache Reverse Proxy (RPX) is used to allow transparent switch of the backend JEE server. We solve this by configuring an alias to the service on the backend. The alias is then entered in whatever DNS resolver we use to point to the one system which is currently meant to receive the request. This allows us to make application changes on one system while the other is still working, then transparently switching to the updates server to do maintenance on the other. Now the problem with this is that Apache, or more exactly the mod_proxy, has a connection pool mechanism which apparently binds to a once resolved IP address "forever". See the type struct proxy_conn_pool in file mod_proxy.h or just my description in Bug Report 50551. It appears to me, and I believe I was able to show this by extensive testing, that the member variable apr_sockaddr_t *addr /* Preparsed remote address info */ in this struct is responsible for this behavior and that this preparsed remote address is invalidated only if the last pooled connection in this connection pool is timed out. In my opinion the best solution to this problem would be to add an additional perameter to mod_proxy to force a renewal of the DNS resolve procedure for this address after a set timeout like it is possible in mod_weblogic. Unfortunately this is way beyond my capabilities at the moment to implement it myself. But I wonder why there seems to be no need for such an option so nobody implemented it yet. Perhaps there is another I'm still not aware of? Regards Slawo -Ursprüngliche Nachricht- Von: Slawomir R. Janotta [mailto:s.jano...@aviope.de] Gesendet: Freitag, 3. Dezember 2010 16:54 An: us...@httpd.apache.org Betreff: Re: [users@httpd] Problem with DNS lookup caching in reverse proxy > > Would it be a viable option for you to not use the DNS method, and > instead do something like: > > > BalanceMember https://backend1.mydomain.com:8080/ > BalanceMember https://backend1.mydomain.com:8080/ status=+H > > > ProxyPass / balancer://notsohotcluster > ProxyPassReverse / balancer://notsohotcluster > > This will (probably) simply mitigate the DNS problem. > > > i > > -- > Igor Galić > > Tel: +43 (0) 664 886 22 883 > Mail: i.ga...@brainsware.org > URL: http://brainsware.org/ > > Thank you very much, Igor. Indeed, your suggestion solves most of the cases where I encounter the switch-over of the backend hosts. However, this does *not* help in all cases, as this works only if the backend is no more reachable and thus an error condition occurs. Sometimes I just want to switch from one backend to the other without shutting any of them. A working TTL in the connection pool or in the DNS cache would be ideal for this. Perhaps I have to open a new Bug report in Bugzilla. Regards Slawo. Sicherheitshinweis: Dieses E-Mail von PostFinance ist signiert. Weitere Informationen finden Sie unter: https://www.postfinance.ch/e-signature. Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt. smime.p7s Description: S/MIME Cryptographic Signature
