Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:54, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, No, the Content-Length doesn't include the size of the headers. Yes of course, but you could add the length of all r-headers_in. It's not very reliable. In headers_out you have things like key and value, while the resulting header is key: value\r\n. Then, I am not sure if the ap_http_header_filter that transforms the apr_table_t *headers_out to strings changes/inserts headers. Then there are two other filters, the byte-range and the content-length filter. The content-length filter sets the value of the Content-Length header while I don't know what the byte-range filter does (maybe it is applicable in the case of chunked transport). You would need to count the size of the headers_out _after_ such filters alter the headers. So... too hairy. I just wondered if the content-type reflects the actual size, for example if the request is compressed. If the request is compressed, Content-Length indicates the length of the _compressed_ body. Content-Type should stay unmodified regardless if the body is compressed or not. However, a compressed body is flagged by the Content-Encoding: gzip header (or deflate). S
Re: Logging authentication requests
To clarify: Is it currently possible to log authentication requests (ideally both success and failure, individually)? If not, is it possible? Meant: is it currently possible to do this with Apache 2.2 (perhaps using an existing module)? If not, is it even theoretically possible? Dave
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:44, Andrej van der Zee [EMAIL PROTECTED] wrote: Thanks for your reply! Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. No, the Content-Length doesn't include the size of the headers. While I was typing the message, Dave Ingram's message arrived. I think the solution he proposes is much better. Sorin Sorin
RE: Logging authentication requests
Given what I learned writing my module, that would certainly work. I think you'd be hooking check_user_id with the very first call that happens in that phase. That said, I don't know if there might be a better way to handle this... Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message- From: Dave Ingram [mailto:[EMAIL PROTECTED] Sent: Thursday, October 09, 2008 10:11 AM To: modules-dev@httpd.apache.org Subject: Logging authentication requests Hi all, Just thinking about logging... Is it currently possible to log authentication requests (ideally both success and failure, individually)? If not, is it possible? And where should the module fit into the authentication chain? Is this related to Rick Houser's earlier posts about wrapping an existing hook? Dave
APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN
Hi, I was wondering what the purpose is of the macros APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN. For example, in mod_authn_dbd APR_RETRIEVE_OPTIONAL_FN is used to get a hold on the functions ap_dbd_acquire and ap_dbd_prepare declared in in mod_dbd with APR_DECLARE_OPTIONAL_FN. I could also directly invoke these functions as they are declared in the header file mod_dbd.h and let the loading of the module fail with an unresolved symbol if mod_dbd is not loaded. What is the benefit for using those macros? Cheers, Andrej
Re: APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN
Hi, Without them, it became impossible to build plugable frameworks without linking one module to another. Yes of course, that would be necessary if you compile your module outside the httpd source tree with apxs I guess. Thanks a lot, Andrej
Equivalents from Apache-1.3 to Apache2.2
Hello *, I'm trying to build my old module written under Apache-1.3.X under Apache-2.2.X. Problem is that during the build more errors occurs because of missing function ap_ctx_get and ap_global_ctx. 1) Do you know if the function are replaced somewho in Apache-2.2.X? 2) In the header file of ap_ctx.h (mod_ssl for apache-1.3.X) is written that the function serves for getting generic Context Interface for Apache. Could you someone help me? regards / S pozdravem Ing. Petr Hráček IT Services and Enterprise Communications Developer Bidláky 20, budova Mediahall 639 00 Brno Tel.: +420 533 337 267 Fax.: +420 533 337 252 mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] www.isec-it.com http://www.isec-it.com/ Společnost: iSEC - IT Services and Enterprise Communications s.r.o. Sídlo: Brno, Bidláky 20, 639 00 Obchodní firma je zapsána v obchodním rejstříku u Krajského soudu v Brně v oddílu C, ve složce 54602. IČ: 27721710 POZOR! Tento e-mail může obsahovat obchodní tajemství nebo jiné utajované a důvěrné informace. Pokud jste tento e-mail obdrželi omylem, oznamujeme Vám tímto, že je přísně zakázáno jej jakýmkoli způsobem upravovat, kopírovat nebo šířit. Neprodleně nás prosím informujte a tento e-mail zničte. Děkujeme za spolupráci. Important Note! This e-mail may contain trade secrets or privileged, undisclosed or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.
Re: proxy_ajp connect timeout fix.
On 10/08/2008 10:55 PM, Matt Stevenson wrote: Hi, I've used mod_jk (1/2) for years. I've always had an issue when a backend server goes down, not tomcat/jboss stopped but dead. Recently some people I work with have been using mod_proxy and mod_proxy_ajp. This seems to have the same issue. The code (proxy_util.c) assumes that apr_socket_timeout_set works for all connects. I don't believe it does, not unless it is in non blocking mode. I wrote the code below forap_proxy_connect_backend before I looked deeper into the apr libs (sorry its not in diff format, and for the hard 2 sec timeout). The code seems to work fine for linux (and probably other unix). I've basically redone the apr code in apr_wait_for_io_or_timeout (should have dug deeper first). Anyway the current release code doesn't seem to work for me for down boxes (to test point an ajp proxy at a non existant IP on the network and a live server). I think if you put the socket in non-blocking mode first and with a timeout apr will try to handle a connect timeout (I haven't had a chance to try), switch back to non blocking after connect. Exactly. This might be the only thing missing in ap_proxy_connect_backend: Setting the socket to non blocking before calling apr_socket_connect and setting it to blocking again afterwards. Regards Rüdiger
Re: SNI in 2.2.x (Re: Time for 2.2.10?)
On Thu, Oct 9, 2008 at 5:59 AM, Ian G [EMAIL PROTECTED] wrote: As we all know, this will not be in 2.2.10... Please recall that things must be in -trunk before being viable for backport to 2.2.x. It's impossible to even express how disappointing this is ;( There are only two changes in TLS on the server side that have been identified to have any effect on phishing [1]. TLS/SNI is the easy one. What's the effect beyond making mass-vhosting easier? A httpd fix will almost work by itself; the browsers already did their part [2]. Only the config changes implemented by all here are needed on the web server to turn the LAMPs on in a million small but secured sites. There's still the issue of certificates and CPU time. What are the blockages? Mozo have offered money but don't know what to do or who to talk to? Review has been public. Nobody's opposed to SNI in the webserver, but AIUI the patch that implements it seems to have a troubled history with respect to integrating with all the per-directory quriks of SSL renegotiation in mod_ssl. IMO the merits of SNI isn't the operative argument. -- Eric Covener [EMAIL PROTECTED]
Re: load balancer workers
On 10/08/2008 11:21 PM, Vinicius Petrucci wrote: Hi, ok. but if I cannot change the assignment of workers to balancer during runtime, how do I duplicate the workers at startup? where should I write the code for duplication? I've tried using the httpd.conf, e.g., Proxy balancer://webapp1 BalancerMember http://ampere:81 route=ampere loadfactor=20 BalancerMember http://coulomb:81 route=coulomb loadfactor=10 BalancerMember http://hertz:81 route=hertz loadfactor=10 ProxySet lbmethod=byrequests /Proxy Proxy balancer://webapp2 BalancerMember http://ampere:81 route=ampere loadfactor=20 BalancerMember http://coulomb:81 route=coulomb loadfactor=10 BalancerMember http://hertz:81 route=hertz loadfactor=10 ProxySet lbmethod=byrequests /Proxy but when Apache starts it outputs the following warning: [Wed Oct 08 18:14:48 2008] [warn] worker http://ampere:81 already used by another worker [Wed Oct 08 18:14:48 2008] [warn] worker http://coulomb:81 already used by another worker [Wed Oct 08 18:14:48 2008] [warn] worker http://hertz:81 already used by another worker Is there any problem using this approach? The workers in your second balancer are not copies, but are the *same* workers as in your first balancer. So what can you do to create copies? If you web applications have context pathes do something like Proxy balancer://webapp1/webapp1 BalancerMember http://ampere:81/webapp1 route=ampere loadfactor=20 BalancerMember http://coulomb:81/webapp1 route=coulomb loadfactor=10 BalancerMember http://hertz:81/webapp1 route=hertz loadfactor=10 ProxySet lbmethod=byrequests /Proxy Proxy balancer://webapp2/webapp2 BalancerMember http://ampere:81/webapp2 route=ampere loadfactor=20 BalancerMember http://coulomb:81/webapp2 route=coulomb loadfactor=10 BalancerMember http://hertz:81/webapp2 route=hertz loadfactor=10 ProxySet lbmethod=byrequests /Proxy If not use different hostnames for your backend that point to the same IP addresses, e.g. ampere-webapp1 and ampere-webapp2 BTW: I'm planing to support multiple services (or applications) associated to different (dynamic) group of servers. My cluster configuration needs to be dynamic because the workload for the different running applications varies according to the time. The point is to change the overall cluster configuration (increase/decrease servers on balancer X) depending the their loads. As Paul said, the best approach here is to setup a configuration like you did (with the small modifications I proposed) and let your module do some kind of virtual operator of the mod_proxy_balancer web interface by dynamically enabling and disabling the workers you like. Regards Rüdiger
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 14:40, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, I was wondering what would be the right way to get the number of bytes / packets that are sent / received in an Apache module for httpd-2.1 and higher (including the HTTP headers). Check server/core.c and server/core_filters.c in the apache sources. There are two filters there, ap_core_input_filter and ap_core_output_filter. They are the first input/last output filters. They are responsible of packaging the byte stream from the socket into bucket brigades and of writing the contents of bucket brigades to the socket. So, the output of ap_core_input_filter is really was comes from the socket, and the input of ap_core_output_filter is really was goes to the socket. So you'll need to write your own filters that count the bytes and insert them immediately after/before ap_core_input/output_filter. This, if you want to count the headers. If not, for output it would be easier to parse the access logs. For input, I don't know. Wouldn't it be easier if you don't do this in apache but with some sort of network monitoring tools? iptables, snmp, or something like that? I'm not an expert, but I suppose some smart tools must be out there, allowing you to filter by source IP, port, etc. Sorin
Re: SNI in 2.2.x (Re: Time for 2.2.10?)
As we all know, this will not be in 2.2.10... Please recall that things must be in -trunk before being viable for backport to 2.2.x. It's impossible to even express how disappointing this is ;( There are only two changes in TLS on the server side that have been identified to have any effect on phishing [1]. TLS/SNI is the easy one. A httpd fix will almost work by itself; the browsers already did their part [2]. Only the config changes implemented by all here are needed on the web server to turn the LAMPs on in a million small but secured sites. Which makes this the #1 easy fix for security in existing code bases, today, and since around 2004 [3]. This massive injection of activity will flow through in dozens of ways, e.g., by pulling more and more Linux guys into thinking about securing systems. What are the blockages? Mozo have offered money but don't know what to do or who to talk to? iang [1] The other is the PSK one which requires re-coding on the client side to be useful. Nice idea, but may take years to roll out. [2] in a concerted and serious effort from 2006 or so, the browser manufacturers implemented TLS/SNI. They also deprecated SSL v2, and actively chased sites to switch. TLS/SNI was one of the two reasons for actively going out there and badgering the sites, I forget the other reason. [3] The rest is really hard, like KCM and security UI. It requires end to end changes, and a lot of security re-thinking. smime.p7s Description: S/MIME Cryptographic Signature
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:11, Sorin Manolache [EMAIL PROTECTED] wrote: On Thu, Oct 9, 2008 at 14:40, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, I was wondering what would be the right way to get the number of bytes / packets that are sent / received in an Apache module for httpd-2.1 and higher (including the HTTP headers). Check server/core.c and server/core_filters.c in the apache sources. There are two filters there, ap_core_input_filter and ap_core_output_filter. They are the first input/last output filters. They are responsible of packaging the byte stream from the socket into bucket brigades and of writing the contents of bucket brigades to the socket. So, the output of ap_core_input_filter is really was comes from the socket, and the input of ap_core_output_filter is really was goes to the socket. So you'll need to write your own filters that count the bytes and insert them immediately after/before ap_core_input/output_filter. This, if you want to count the headers. If not, for output it would be easier to parse the access logs. For input, I don't know. Wouldn't it be easier if you don't do this in apache but with some sort of network monitoring tools? iptables, snmp, or something like that? I'm not an expert, but I suppose some smart tools must be out there, allowing you to filter by source IP, port, etc. Sorin Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. Sorin
Re: load balancer workers
On Oct 8, 2008, at 5:26 PM, Vinicius Petrucci wrote: I don't think the approach you are suggesting will work. I would suggest duplicating the workers at startup, enable/disable in each balancer to 'move' them. Correct. So you cannot change the assignment of workers to a balancer during runtime. BTW: What is the goal that rises the need for you to change the assignment of workers to balancers during runtime? Hi, ok. but if I cannot change the assignment of workers to balancer during runtime, how do I duplicate the workers at startup? where should I write the code for duplication? Off the top of my head: create a shared memory segment that recreates both the shared and non-shared LB parameters. This will allow you change the assignments at runtime since those assignments will be now me shared among all processes. Then, have your module run before mod_proxy, so that your assignments are used and mod_proxy is pretty much bypassed. In other words, just have your module be a fork of mod_proxy et.al. with all aspects shared.
Re: number of bytes/packets sent/received
Thanks for your reply! Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. Cheers, Andrej
Re: Apache httpd 2.2.10 test tarballs available...
On Oct 8, 2008, at 4:44 PM, Paul Querna wrote: William A. Rowe, Jr. wrote: Oden Eriksson wrote: Den Wednesday 08 October 2008 19:50:06 skrev William A. Rowe, Jr.: Akins, Brian wrote: On 10/7/08 8:49 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Sure sounds like this is a re-initialization of mysql, with apr and php fighting for the honors. I thought the official support of php was fastcgi only in httpd 2.2 Given the headaches he is encountering, I'd think that php- fastcgi would be the ideal solution. vhosting does not work very well with apache, without band-aids like fastcgi etc. that's a shame. If you mean mass vhosting of untrusted content, and are still letting authors write perl and php without knowing where to hunt them down when their scripts or they personally mess around in their in-process, non-sandbox environment, you are being foolish. In-process modperl/modphp is for hosting trusted content. You'll have to give them a very restricted language, such as sed or awk, if you want to keep their fingers away from the dangerous buttons. Or host them under [fast]cgi which is what that environment is created for. I agree completely. Maybe we should finish our mod_proxy_fcgi module or try to import mod_fcgid :-) But, I don't have time to work on mod_proxy_fcgi. H I'll take that on... Not that I have a lot of time, but I'd also like to see it become viable. So Can we ask the mod_fcgid project if we could import it? If they are willing to change the license :)
Re: number of bytes/packets sent/received
I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. Would mod_logio be at all helpful? http://httpd.apache.org/docs/2.2/mod/mod_logio.html I haven't looked into it, but the source might give you some ideas. You may even have a direct solution if you use a module that logs directly to a database. I must admit that I don't know offhand if such a module exists. Dave
Re: number of bytes/packets sent/received
Hi Dave, Would mod_logio be at all helpful? http://httpd.apache.org/docs/2.2/mod/mod_logio.html I haven't looked into it, but the source might give you some ideas. You may even have a direct solution if you use a module that logs directly to a database. I must admit that I don't know offhand if such a module exists. You really made it easy for me huh? Thanks alot. Cheers, Andrej
Re: number of bytes/packets sent/received
Hi, No, the Content-Length doesn't include the size of the headers. Yes of course, but you could add the length of all r-headers_in. I just wondered if the content-type reflects the actual size, for example if the request is compressed. While I was typing the message, Dave Ingram's message arrived. I think the solution he proposes is much better. Yes, thanks alot for your time anyway. Cheers, Andrej
Re: number of bytes/packets sent/received
Would mod_logio be at all helpful? You really made it easy for me huh? Thanks alot. No use in doing more work than you need to ;-) To be honest, I only found out about it today while researching some obscure logging options. A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Dave
Re: number of bytes/packets sent/received
Hi, A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Since Apache 2.1 there is mod_dbd that makes it pretty easy to log to a MySQL, sqlite2/3 and Postgres database. It also takes care of connection pooling. Cheers, Andrej
Re: number of bytes/packets sent/received
A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Since Apache 2.1 there is mod_dbd that makes it pretty easy to log to a MySQL, sqlite2/3 and Postgres database. It also takes care of connection pooling. Ah. I'm planning to use mod_dbd myself for a project, but I wasn't aware it provided any logging ability. Dave
Logging authentication requests
Hi all, Just thinking about logging... Is it currently possible to log authentication requests (ideally both success and failure, individually)? If not, is it possible? And where should the module fit into the authentication chain? Is this related to Rick Houser's earlier posts about wrapping an existing hook? Dave
Re: PR45605 and event MPM
On Tue, Oct 7, 2008 at 7:03 PM, Ruediger Pluem [EMAIL PROTECTED] wrote: One comment in the event version of ap_queue_info_wait_for_idler confuses me: * A negative value in queue_info-idlers tells how many * threads are waiting on an idle worker. IMHO this would require ap_queue_info_wait_for_idler to be called by different threads, but I can only see calls via get_worker from the listener thread and AFAICT there is only one listener thread in event like in worker. Do I miss something here? No, you didn't miss anything. With the original event patch to the worker MPM there were separate listener and event threads. When Paul moved the patch into the repository as the Event MPM he merged the two threads. There didn't seem to be a good reason to revert wait_for_idler back to the worker behavior at the time. I will take a look at the bug report and see if I understand the problem. Greg
Re: proxy_ajp connect timeout fix.
Had a bit more time, here is a patch that should work for Unix which have
apr_wait_for_io_or_timeout available. I can't test windows/others so that's
the reason for the ifdef.
ProxyPass / balance://hotcluster/
Proxy balance://hotcluster
# defaultish tomcat
BalancerMember ajp://10.136.130.111:7009 loadfactor=1 connectiontimeout=2
# below IP is not reachable, acts like a down box
BalancerMember ajp://192.168.0.7:7010 loadfactor=1 connectiontimeout=2
/Proxy
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 703219)
+++ modules/proxy/proxy_util.c (working copy)
@@ -2358,9 +2358,18 @@
proxy: %s: fam %d socket created to connect to %s,
proxy_function, backend_addr-family, worker-hostname);
+/* use non blocking for connect timeouts to work. The ifdef
+ limits to unix systems which have apr_wait_for_io_or_timeout.
+ TODO: remove the ifdef and see what works/breaks */
+#ifdef USE_WAIT_FOR_IO
+apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 1);
+#endif
/* make the connection out of the socket */
rv = apr_socket_connect(newsock, backend_addr);
+#ifdef USE_WAIT_FOR_IO
+apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 0);
+#endif
/* if an error occurred, loop round and try again */
if (rv != APR_SUCCESS) {
apr_socket_close(newsock);
Regards
matt
- Original Message
From: Matt Stevenson [EMAIL PROTECTED]
To: dev@httpd.apache.org
Sent: Wednesday, October 8, 2008 9:55:43 PM
Subject: proxy_ajp connect timeout fix.
Hi,
I've used mod_jk (1/2) for years. I've always had an issue when a backend
server goes down, not tomcat/jboss stopped but dead. Recently some people I
work with have been using mod_proxy and mod_proxy_ajp. This seems to have the
same issue.
The code (proxy_util.c) assumes that apr_socket_timeout_set works for all
connects. I don't believe it does, not unless it is in non blocking mode. I
wrote the code below forap_proxy_connect_backend before I looked deeper into
the apr libs (sorry its not in diff format, and for the hard 2 sec timeout).
The code seems to work fine for linux (and probably other unix). I've basically
redone the apr code in apr_wait_for_io_or_timeout (should have dug deeper
first).
Anyway the current release code doesn't seem to work for me for down boxes (to
test point an ajp proxy at a non existant IP on the network and a live server).
I think if you put the socket in non-blocking mode first and with a timeout apr
will try to handle a connect timeout (I haven't had a chance to try), switch
back to non blocking after connect.
Regards
Matt
if (worker-keepalive) {
if ((rv = apr_socket_opt_set(newsock,
APR_SO_KEEPALIVE, 1)) != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s,
apr_socket_opt_set(SO_KEEPALIVE): Failed to set
Keepalive);
}
}
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
proxy: %s: fam %d socket created to connect to %s,
proxy_function, backend_addr-family, worker-hostname);
apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 1);
apr_socket_timeout_set(newsock, 0);
rv = apr_socket_connect(newsock, backend_addr);
if( rv != APR_SUCCESS APR_STATUS_IS_EINPROGRESS(rv)){
apr_pollfd_t pfds[1];
apr_status_t status;
apr_int32_t nfds;
pfds[0].reqevents = APR_POLLOUT;
pfds[0].desc_type = APR_POLL_SOCKET;
pfds[0].desc.s = newsock;
rv = apr_poll(pfds[0], 1, nfds, apr_time_from_sec(2));
}
/* if an error occurred, loop round and try again */
if (rv != APR_SUCCESS) {
apr_socket_close(newsock);
loglevel = backend_addr-next ? APLOG_DEBUG : APLOG_ERR;
ap_log_error(APLOG_MARK, loglevel, rv, s,
proxy: %s: attempt to connect to %pI (%s) failed,
proxy_function,
backend_addr,
worker-hostname);
backend_addr = backend_addr-next;
continue;
}
apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 0);
/* Set a timeout on the socket */
Re: proxy_ajp connect timeout fix.
On 10/09/2008 10:11 PM, Matt Stevenson wrote:
Had a bit more time, here is a patch that should work for Unix which have
apr_wait_for_io_or_timeout available. I can't test windows/others so that's
the reason for the ifdef.
ProxyPass / balance://hotcluster/
Proxy balance://hotcluster
# defaultish tomcat
BalancerMember ajp://10.136.130.111:7009 loadfactor=1 connectiontimeout=2
# below IP is not reachable, acts like a down box
BalancerMember ajp://192.168.0.7:7010 loadfactor=1 connectiontimeout=2
/Proxy
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 703219)
+++ modules/proxy/proxy_util.c (working copy)
@@ -2358,9 +2358,18 @@
proxy: %s: fam %d socket created to connect to %s,
proxy_function, backend_addr-family, worker-hostname);
+/* use non blocking for connect timeouts to work. The ifdef
+ limits to unix systems which have apr_wait_for_io_or_timeout.
+ TODO: remove the ifdef and see what works/breaks */
+#ifdef USE_WAIT_FOR_IO
+apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 1);
+#endif
/* make the connection out of the socket */
rv = apr_socket_connect(newsock, backend_addr);
+#ifdef USE_WAIT_FOR_IO
+apr_socket_opt_set(newsock, APR_SO_NONBLOCK, 0);
+#endif
/* if an error occurred, loop round and try again */
if (rv != APR_SUCCESS) {
apr_socket_close(newsock);
According to the APR source files only older versions of BeOS do not provide
apr_wait_for_io_or_timeout.
So I would say that it is pretty much save to remove the ifdef's
Regards
Rüdiger
Re: PR45605 and event MPM
On Tue, Oct 7, 2008 at 7:03 PM, Ruediger Pluem [EMAIL PROTECTED] wrote: I am currently looking at PR45605 ( https://issues.apache.org/bugzilla/show_bug.cgi?id=45605) and the analysis and the resulting patch in Comment 4 look good to me (https://issues.apache.org/bugzilla/show_bug.cgi?id=45605#c4). As worker and event MPM are very similar I had a look if and how this patch is applicable to the event MPM. I noticed that ap_queue_info_wait_for_idler is quite different in worker and event MPM. I don't think the problem reported by PR45605 exists in the Event MPM. I had to stare at ap_queue_info_wait_for_idler for a long time back when it could be called by two separate threads. The unserialized access to queue_info-idlers followed by the unconditional decrement outside of the mutex in that section of code made my head twitch when I thought about races on SMP systems. The two major improvements in Event's fdqueue are: * doing the atomic decrement first before making a decision about waiting. The decrement is visible to other threads immediately. * using the negative value of idlers to distinguish between when threads (just the listener today) are waiting for idle workers vs. when all workers are busy but nobody else (the listener) is blocked. That eliminates a situation where the condition variable is signalled unnecessarily, which happens in the PR scenario. Greg
Re: PR45605 and event MPM
On 10/09/2008 10:53 PM, Greg Ames wrote: On Tue, Oct 7, 2008 at 7:03 PM, Ruediger Pluem [EMAIL PROTECTED] wrote: I am currently looking at PR45605 ( https://issues.apache.org/bugzilla/show_bug.cgi?id=45605) and the analysis and the resulting patch in Comment 4 look good to me (https://issues.apache.org/bugzilla/show_bug.cgi?id=45605#c4). As worker and event MPM are very similar I had a look if and how this patch is applicable to the event MPM. I noticed that ap_queue_info_wait_for_idler is quite different in worker and event MPM. I don't think the problem reported by PR45605 exists in the Event MPM. I had to stare at ap_queue_info_wait_for_idler for a long time back when it could be called by two separate threads. The unserialized access to queue_info-idlers followed by the unconditional decrement outside of the mutex in that section of code made my head twitch when I thought about races on SMP systems. The two major improvements in Event's fdqueue are: * doing the atomic decrement first before making a decision about waiting. The decrement is visible to other threads immediately. * using the negative value of idlers to distinguish between when threads (just the listener today) are waiting for idle workers vs. when all workers are busy but nobody else (the listener) is blocked. That eliminates a situation where the condition variable is signalled unnecessarily, which happens in the PR scenario. Thanks for checking the code again Greg. I also went through it again and agree that the situation described in the PR does not result in any harmful behaviour but that the correct thing is done. As event and worker are still very similar I would tend to copy ap_queue_info_wait_for_idler and ap_queue_info_set_idle from event over to worker to use the same code for both which should make maintaining easier. Does anybody see problems with this or are we still too worried about the correct handling of signed vs. unsigned variables by apr_atomic_ to use this in a non experimental MPM? Regards Rüdiger
Speeding up mod_proxy_balancer on Windows
I had previously discovered that mod_proxy_balancer takes over 1 second on Windows to determine that nothing is listening on the target port. This becomes problematic if you are balancing over a sparsely populated set of proxy ports. A Windows guru here found the Windows GetTcpTable which would appear to offer a quicker way to determine a port's status -- whereas doing the obvious thing and attempting to connect takes over a second to fail. I'd like to experiment with using this API to address this issue upon attempted formation of the first connection for a given worker one is balancing over. Can anyone suggest where I should look to do add such a call? Eventually this should presumably be an APR-level thing, but in the short term I'm just looking for where I can experiment with inserting it in an #ifdef in the proxy code -- and getting a little lost here, unfortunately. -- Jess Holle
Re: Speeding up mod_proxy_balancer on Windows
P.S. Yes, I know this approach only has any hope of working when Apache and the proxy backends are on the same host. Jess Holle wrote: I had previously discovered that mod_proxy_balancer takes over 1 second on Windows to determine that nothing is listening on the target port. This becomes problematic if you are balancing over a sparsely populated set of proxy ports. A Windows guru here found the Windows GetTcpTable which would appear to offer a quicker way to determine a port's status -- whereas doing the obvious thing and attempting to connect takes over a second to fail. I'd like to experiment with using this API to address this issue upon attempted formation of the first connection for a given worker one is balancing over. Can anyone suggest where I should look to do add such a call? Eventually this should presumably be an APR-level thing, but in the short term I'm just looking for where I can experiment with inserting it in an #ifdef in the proxy code -- and getting a little lost here, unfortunately. -- Jess Holle
Re: Apache httpd 2.2.10 test tarballs available...
On Tue, Oct 7, 2008 at 2:37 PM, Jim Jagielski [EMAIL PROTECTED] wrote: ... at the usual location: http://httpd.apache.org/dev/dist/ The availability of these test tarballs does not constitute an official release, however please download and test as a VOTE will be called for in the next few days regarding their release. +1 AIX 5.3 z/OS 1.7 Greg
Re: Speeding up mod_proxy_balancer on Windows
On 10/09/2008 11:50 PM, Jess Holle wrote: P.S. Yes, I know this approach only has any hope of working when Apache and the proxy backends are on the same host. Jess Holle wrote: I had previously discovered that mod_proxy_balancer takes over 1 second on Windows to determine that nothing is listening on the target port. This becomes problematic if you are balancing over a sparsely populated set of proxy ports. A Windows guru here found the Windows GetTcpTable which would appear to offer a quicker way to determine a port's status -- whereas doing the obvious thing and attempting to connect takes over a second to fail. I'd like to experiment with using this API to address this issue upon attempted formation of the first connection for a given worker one is balancing over. Can anyone suggest where I should look to do add such a call? Eventually this should presumably be an APR-level thing, but in the short term I'm just looking for where I can experiment with inserting it in an #ifdef in the proxy code -- and getting a little lost here, unfortunately. Did you check whether the currently running thread proxy_ajp connect timeout fix. (http://mail-archives.apache.org/mod_mbox/httpd-dev/200810.mbox/[EMAIL PROTECTED] and http://mail-archives.apache.org/mod_mbox/httpd-dev/200810.mbox/[EMAIL PROTECTED]) does fix your issue on Windows? If httpd and the backends are running on the same machine this shouldn't take a second. The connect call should return immediately with an error code indicating that the connection was refused (if the port is down). If not is it possible that there is a local firewall that causes this trouble? Regards Rüdiger
Re: Apache httpd 2.2.10 test tarballs available...
Jim Jagielski wrote: On Oct 8, 2008, at 4:44 PM, Paul Querna wrote: So Can we ask the mod_fcgid project if we could import it? If they are willing to change the license :) You totally miss the point. We aren't entirely clear if this author even has the IP they claim to have (talk about IP import processes). He claims GPL. He adds no restrictions, and; 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. however, if the work was at all based on the code authored by the crew for mod_fastcgi, as opposed to the coding logic of mod_fastcgi, we have; This FastCGI application library source and object code (the Software) and its documentation (the Documentation) are copyrighted by Open Market, Inc (Open Market). The following terms apply to all files associated with the Software and Documentation unless explicitly disclaimed in individual files. Open Market permits you to use, copy, modify, distribute, and license this Software and the Documentation solely for the purpose of implementing the FastCGI specification defined by Open Market or derivative specifications publicly endorsed by Open Market and promulgated by an open standards organization and for no other purpose, provided that existing copyright notices are retained in all copies and that this notice is included verbatim in any distributions. No written agreement, license, or royalty fee is required for any of the authorized uses. Modifications to this Software and Documentation may be copyrighted by their authors and need not follow the licensing terms described here, but the modified Software and Documentation must be used for the sole purpose of implementing the FastCGI specification defined by Open Market or derivative specifications publicly endorsed by Open Market and promulgated by an open standards organization and for no other purpose. If modifications to this Software and Documentation have new licensing terms, the new terms must protect Open Market's proprietary rights in the Software and Documentation to the same extent as these licensing terms and must be clearly indicated on the first page of each file where they apply. Open Market shall retain all right, title and interest in and to the Software and Documentation, including without limitation all patent, copyright, trade secret and other proprietary rights. OPEN MARKET MAKES NO EXPRESS OR IMPLIED WARRANTY WITH RESPECT TO THE SOFTWARE OR THE DOCUMENTATION, INCLUDING WITHOUT LIMITATION ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL OPEN MARKET BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY DAMAGES ARISING FROM OR RELATING TO THIS SOFTWARE OR THE DOCUMENTATION, INCLUDING, WITHOUT LIMITATION, ANY INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES OR SIMILAR DAMAGES, INCLUDING LOST PROFITS OR LOST DATA, EVEN IF OPEN MARKET HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE SOFTWARE AND DOCUMENTATION ARE PROVIDED AS IS. OPEN MARKET HAS NO LIABILITY IN CONTRACT, TORT, NEGLIGENCE OR OTHERWISE ARISING OUT OF THIS SOFTWARE OR THE DOCUMENTATION. -- which is altogether out of sorts with either the GPL or the AL. So before we ask the authors if they would relicense, we first need to ask them if they've offered a legitimate license in the first place.
Re: Speeding up mod_proxy_balancer on Windows
Ruediger Pluem wrote: Did you check whether the currently running thread proxy_ajp connect timeout fix. (http://mail-archives.apache.org/mod_mbox/httpd-dev/200810.mbox/[EMAIL PROTECTED] and http://mail-archives.apache.org/mod_mbox/httpd-dev/200810.mbox/[EMAIL PROTECTED]) does fix your issue on Windows? I was watching this. I'll have to give this a try as it just became clear from the latest of these messages that the fix applied to Windows. If httpd and the backends are running on the same machine this shouldn't take a second. The connect call should return immediately with an error code indicating that the connection was refused (if the port is down). Yes, that's what I'd assumed. It has become clear from testing on multiple machines that this is not the case. Another engineer did testing with Windows APIs directly in .NET app and came up with the same result -- over 1 second per port refusal. If not is it possible that there is a local firewall that causes this trouble? I tried disabling that and other such steps. I certainly concur -- the refusal should be immediate and certainly /far/ faster than 1 second per port. I've been left wondering if this isn't an odd-ball hack by Microsoft to slow down remote port scans. I'll give the timeout fix a try, but I'm not hopeful given the data so far. -- Jess Holle
