[STATUS] (flood) Wed Jul 28 23:45:39 EDT 2004
flood STATUS: -*-text-*- Last modified at [$Date: 2003/07/01 20:55:12 $] Release: 1.0: Released July 23, 2002 milestone-03: Tagged January 16, 2002 ASF-transfer: Released July 17, 2001 milestone-02: Tagged August 13, 2001 milestone-01: Tagged July 11, 2001 (tag lost during transfer) RELEASE SHOWSTOPPERS: * Everything needs to work perfectly Other bugs that need fixing: * I get a SIGBUS on Darwin with our examples/round-robin-ssl.xml config, on the second URL. I'm using OpenSSL 0.9.6c 21 dec 2001. * iPlanet sends Content-length - there is a hack in there now to recognize it. However, all HTTP headers need to be normalized before checking their values. This isn't easy to do. Grr. * OpenSSL 0.9.6 Segfaults under high load. Upgrade to OpenSSL 0.9.6b. Aaron says: I just found a big bug that might have been causing this all along (we weren't closing ssl sockets). How can I reproduce the problem you were seeing to verify if this was the fix? * SEGVs when /tmp/.rnd doesn't exist are bad. Make it configurable and at least bomb with a good error message. (See Doug's patch.) Status: This is fixed, no? * If APR has disabled threads, flood should as well. We might want to have an enable/disable parameter that does this also, providing an error if threads are desired but not available. * flood needs to clear pools more often. With a long running test it can chew up memory very quickly. We should just bite the bullet and create/destroy/clear pools for each level of our model: farm, farmer, profile, url/request-cycle, etc. * APR needs to have a unified interface for ephemeral port exhaustion, but aparently Solaris and Linux return different errors at the moment. Fix this in APR then take advantage of it in flood. * The examples/analyze-relative scripts fail when there are less than 5 unique URLs. Other features that need writing: * More analysis and graphing scripts are needed * Write robust tool (using tethereal perhaps) to take network dumps and convert them to flood's XML format. Status: Justin volunteers. Aaron had a script somewhere that is a start. Jacek is working on a Mozilla application, codename Flood URL bag (much like Live HTTP Headers) and small HTTP proxy. * Get chunked encoding support working. Status: Justin volunteers. He got sidetracked by the httpd implementation of input filtering and never finished this. This is a stopgap until apr-serf is completed. * Maybe we should make randfile and capath runtime directives that come out of the XML, instead of autoconf parameters. * We are using apr_os_thread_current() and getpid() in some places when what we really want is a GUID. The GUID will be used to correlate raw output data with each farmer. We may wish to print a unique ID for each of farm, farmer, profile, and url to help in postprocessing. * We are using strtol() in some places and strtoll() in others. Pick one (Aaron says strtol(), but he's not sure). * Validation of responses (known C-L, specific strings in response) Status: Justin volunteers * HTTP error codes (ie. teach it about 302s) Justin says: Yeah, this won't be with round_robin as implemented. Need a linked list-based profile where we can insert new URLs into the sequence. * Farmer (Single-thread, multiple profiles) Status: Aaron says: If you have threads, then any Farmer can be run as part of any Farm. If you don't have threads, you can currently only run one Farmer named Joe right now (this will be changed so that if you don't have threads, flood will attempt to run all Farmers in serial under one process). * Collective (Single-host, multiple farms) This is a number of Farms that have been fork()ed into child processes. * Megaconglomerate (Multiple hosts each running a collective) This is a number of Collectives running on a number of hosts, invoked via RSH/SSH or maybe even some proprietary mechanism. * Other types of urllists a) Random / Random-weighted b) Sequenced (useful with cookie propogation) c) Round-robin d) Chaining of the above strategies Status: Round-robin is complete. * Other types of reports Status: Aaron says: simple reports are functional. Justin added a new type that simply prints the approx. timestamp when the test was run, and the result as OK/FAIL; it is called easy reports (see flood_easy_reports.h). Furthermore,
[STATUS] (perl-framework) Wed Jul 28 23:45:40 EDT 2004
httpd-test/perl-framework STATUS: -*-text-*- Last modified at [$Date: 2002/03/09 05:22:48 $] Stuff to do: * finish the t/TEST exit code issue (ORed with 0x2C if framework failed) * change existing tests that frob the DocumentRoot (e.g., t/modules/access.t) to *not* do that; instead, have Makefile.PL prepare appropriate subdirectory configs for them. Why? So t/TEST can be used to test a remote server. * problems with -d perl mode, doesn't work as documented Message-ID: [EMAIL PROTECTED] Date: Sat, 20 Oct 2001 12:58:33 +0800 Subject: Re: perldb Tests to be written: * t/apache - simulations of network failures (incomplete POST bodies, chunked and unchunked; missing POST bodies; slooow client connexions, such as taking 1 minute to send 1KiB; ...) * t/modules/autoindex - something seems possibly broken with inheritance on 2.0 * t/ssl - SSLPassPhraseDialog exec: - SSLRandomSeed exec:
Re: cookie_jar ignored with requests_redirectable = 1
Stas Bekman wrote: David? Boris Zentner wrote: We are all at the OSCon at the moment, so we hardly get a chance to sleep. I suppose David will follow up soonish :) -- __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: cookie_jar ignored with requests_redirectable = 1
On Jul 28, 2004, at 1:29 AM, Stas Bekman wrote:
Boris Zentner wrote:
Hi,
recently I try to use Apache::Test with HTTP::Cookies. But it did not
work as expected. I suppose the way to add a cookie_jar to A::T was
with 'Apache::TestRequest::user_agent'. I read the docs from
Apache::TestRequest::user_agent. Here is the relevant part of it.
And finally, the semantics of the requests_redirectable
parameter is different than for LWP::UserAgent: It
either follows redirects for a request, or it doesn't.
Thus requests_redirectable is a boolean value instead of
the array reference that LWP::UserAgent expects. To
This implies to me that I have the choice to enable or disable
redirects for this useragent with a bool value.
Apache::TestRequest::user_agent(
reset = 1, cookie_jar = $cookie_jar, requests_redirectable =
1 );
But this way the cookies are ignored. I expected that
HTTP::Cookies-extract_cookies is called after every request.
Therefor I create the cookie_jar from
Oops.
package My::Cookies;
use base 'HTTP::Cookies';
sub extract_cookies {
warn extract_cookies!!!;
shift-SUPER::extract_cookies(@_);
}
To get it work, I need to parse the cookie headers myself or
Apache::TestRequest::user_agent(
reset = 1, cookie_jar = $cookie_jar, requests_redirectable =
0 );
But here I need to redirect myself or do it with the undocumented
Apache::TestRequest::user_agent(
reset = 1, cookie_jar = $cookie_jar, requests_redirectable =
[qw~x y~]
);
that does anything I want but is undocumented!
Here is a part from Apache::TestRequest::user_agent that looks wrong
to me.
my $redir = $args-{requests_redirectable};
if (ref $redir and (@$redir 1 or $redir-[0] ne 'POST')) {
$RedirectOK = 1;
} else {
$RedirectOK = 0;
}
Does this address the issue?
--- TestRequest.pm.~1.96.~ Thu May 6 12:11:33 2004
+++ TestRequest.pm Thu Jul 29 14:03:58 2004
@@ -115,7 +115,7 @@
if (exists $args-{requests_redirectable}) {
my $redir = $args-{requests_redirectable};
-if (ref $redir and (@$redir 1 or $redir-[0] ne 'POST')) {
+if ((ref $redir and (@$redir 1 or $redir-[0] ne 'POST')) or
$redir) {
$RedirectOK = 1;
} else {
$RedirectOK = 0;
I think that I might have changed this code to work this way, and
overlooked that it could be passed as a simple boolean, even though
that's the _only_ way it worked before I got my hands on it.
So does passing the array reference actually affect the way the LWP
user agent object operates? If so, that's cool, but it's undocumented.
I'm not sure whether it should be documented, though, since if LWP
isn't installed it certainly won't work with the simple request
interface that Apache::TestRequest uses in its place.
Regards,
David
smime.p7s
Description: S/MIME cryptographic signature
Re: cvs commit: httpd-2.0 STATUS
At 05:55 PM 7/28/2004, [EMAIL PROTECTED] wrote:
minfrin 2004/07/28 15:55:15
Modified:.Tag: APACHE_2_0_BRANCH STATUS
Log:
Propose a backport
+*) Add load balancer support to the scoreboard in preparation for
+ load balancing support in mod_proxy.
+ include/scoreboard.h: 1.52
+ server/scoreboard.c: 1.75
+ +1: minfrin
Rather than vote... afraid I have to veto (for 2.0)...
--- scoreboard.h 9 Feb 2004 20:38:21 - 1.51
+++ scoreboard.h 28 Jul 2004 22:50:54 - 1.52
@@ -32,6 +32,7 @@
#include apr_thread_proc.h
#include apr_portable.h
#include apr_shm.h
+#include apr_optional.h
/* Scoreboard file, if there is one */
#ifndef DEFAULT_SCOREBOARD
@@ -118,6 +119,7 @@
typedef struct {
int server_limit;
int thread_limit;
+int lb_limit;
ap_scoreboard_e sb_type;
ap_generation_t running_generation; /* the generation of children which
* should still be serving requests. */
This definitely breaks binary compatibility.
Bill
RE: cvs commit: httpd-2.0 STATUS
William A. Rowe, Jr. wrote:
/* Scoreboard file, if there is one */
#ifndef DEFAULT_SCOREBOARD
@@ -118,6 +119,7 @@
typedef struct {
int server_limit;
int thread_limit;
+int lb_limit;
ap_scoreboard_e sb_type;
ap_generation_t running_generation; /* the
generation of children which
* should still
be serving
requests. */
This definitely breaks binary compatibility.
Moving the lb_limit to the end of the struct will not break the binary
compatibility. Correct?
MT.
smime.p7s
Description: S/MIME cryptographic signature
Re: cvs commit: httpd-2.0 STATUS
* Mladen Turk [EMAIL PROTECTED] wrote:
William A. Rowe, Jr. wrote:
/* Scoreboard file, if there is one */
#ifndef DEFAULT_SCOREBOARD
@@ -118,6 +119,7 @@
typedef struct {
int server_limit;
int thread_limit;
+int lb_limit;
ap_scoreboard_e sb_type;
ap_generation_t running_generation; /* the
generation of children which
* should still
be serving
requests. */
This definitely breaks binary compatibility.
Moving the lb_limit to the end of the struct will not break the binary
compatibility. Correct?
Not Correct. It *may* be the case. Depending on who allocates the stuff.
nd
--
Das Verhalten von Gates hatte mir bewiesen, dass ich auf ihn und seine
beiden Gefährten nicht zu zählen brauchte -- Karl May, Winnetou III
Im Westen was neues: http://pub.perlig.de/books.html#apache2
Hi
Important document! Norton AntiVirus Deleted1.txt Description: plain/text
[PATCH] Bug #9037 reminder
Hi, I've received yet another mail from someone asking me about the patch for bug #9037 at http://nagoya.apache.org/bugzilla/show_bug.cgi?id=9037 The patch is attacched to the bug report. Also many distributions (e.g. Debian) ship apache with a patch for this issue. So far it seems to me that there is enough interest to have this problem fixed. I'm interested to bring the patch to an appliable state if it isn't ok as it is. Regards. Note: probably the severity (set to minor) of this problem ins't appropriate because when you hit it in a productive environment it causes a major pain. -- Federico Mennite
RE: cvs commit: httpd-2.0 STATUS
André Malo wrote: William A. Rowe, Jr. wrote: This definitely breaks binary compatibility. Moving the lb_limit to the end of the struct will not break the binary compatibility. Correct? Not Correct. It *may* be the case. Depending on who allocates the stuff. Can you explain that a bit. Seems to me that there is a single ap_scoreboard_image-global, and it is allocated when the scoreboard gets created. IMO if the lb_limit is moved to the end of the struct it will not affect the existing binaries. Cannot speak on all builds but FreeBSD 4.1.10 and WIN32 are binary compatible (mod_status is the only that directly uses global). The only way where it might create problems is ap_scoreboard_image::servers not global, for some 3rd party module that directly uses this pointer array instead using ap_get_scoreboard_worker. Scoreboard file format is not binary compatible for sure, and this is the only part that might rise concerns, but only if that compatibility is required after all. MT. smime.p7s Description: S/MIME cryptographic signature
RE: cvs commit: httpd-2.0 STATUS
At 02:24 AM 7/29/2004, Mladen Turk wrote:
William A. Rowe, Jr. wrote:
/* Scoreboard file, if there is one */
#ifndef DEFAULT_SCOREBOARD
@@ -118,6 +119,7 @@
typedef struct {
int server_limit;
int thread_limit;
+int lb_limit;
ap_scoreboard_e sb_type;
ap_generation_t running_generation; /* the
generation of children which
* should still
be serving
requests. */
This definitely breaks binary compatibility.
Moving the lb_limit to the end of the struct will not break the binary
compatibility. Correct?
Yes, in the case of global_score, that would be safer. It seems that
the additional lb_score 's element point was better placed.
If you changed worker_score, it would still have broken (or if you change
balancers in the future.) Agents reviewing the scoreboard are presuming
scoreboard_entry *psb can be accessed as psb[0..n] and that -will- be
broken with any size/alignment change to the struct.
E.g.
Bill
RE: cvs commit: httpd-2.0 STATUS
William A. Rowe, Jr. wrote: Moving the lb_limit to the end of the struct will not break the binary compatibility. Correct? Yes, in the case of global_score, that would be safer. It seems that the additional lb_score 's element point was better placed. If you changed worker_score, it would still have broken (or if you change balancers in the future.) Agents reviewing the scoreboard are presuming scoreboard_entry *psb can be accessed as psb[0..n] and that -will- be broken with any size/alignment change to the struct. I was afraid you've gonna said that. Those 3rd party... :). How about adding extra data to the end of the entire scoreboard withouth touching any existing structs or data? It will require an extra copy on child_init, but there will be no compatibility issues, thought. Will that rise any objections? MT. smime.p7s Description: S/MIME cryptographic signature
RE: cvs commit: httpd-2.0 STATUS
At 10:26 AM 7/29/2004, William A. Rowe, Jr. wrote: If you changed worker_score, it would still have broken (or if you change balancers in the future.) Agents reviewing the scoreboard are presuming scoreboard_entry *psb can be accessed as psb[0..n] and that -will- be broken with any size/alignment change to the struct. E.g. [...?] we have individual element accessors, but older style code presumes it can loop through the array of individual worker threads/procs. Adding another element to worker_score would break the starting offset of the members psb[1..n]. Although we create the pointer-pointer logic in the children to avoid this, it's still possible that the code would break some modules. I can't envision a case where any of the scoreboard entries are allocated outside of our scoreboard.c code. At 11:01 AM 7/29/2004, Mladen Turk wrote: I was afraid you've gonna said that. Those 3rd party... :). How about adding extra data to the end of the entire scoreboard withouth touching any existing structs or data? It will require an extra copy on child_init, but there will be no compatibility issues, thought. You aren't changing worker_score so this isn't an issue. The patch already creates (new) a fourth lb_score array. Bill
RE: cvs commit: httpd-2.0 STATUS
William A. Rowe, Jr. wrote: Although we create the pointer-pointer logic in the children to avoid this, it's still possible that the code would break some modules. I can't envision a case where any of the scoreboard entries are allocated outside of our scoreboard.c code. OK, then. I'll revert the patch and for 2.1 too, and we'll use our own shm for loadbalancer. Can we then create two extra hooks aka on_scoreboard_create and on_scoreboard_attach with ap_scoreboard_e, thread and server limit params. It will also help other modules like mod_ssl (currently using pool tag to supress double creation on post_config). MT. smime.p7s Description: S/MIME cryptographic signature
[PATCH] scoreboard WAS RE: cvs commit: httpd-2.0 STATUS
Hi all,
Since there was some concerns regarding binary compatibility, here is the
patch that uses different approach.
1. Revert the patch with changes to scoreboard structures
2. Add an extra hook that is run during ap_reopen_scoreboard with detached
param.
We will use our own shm, opening by hooking pre_mpm, and attaching in child
on reopen_scoreboard.
This will resolve any binary compatibility concerns.
Regards,
MT.
Index: scoreboard.h
===
RCS file: /home/cvspublic/httpd-2.0/include/scoreboard.h,v
retrieving revision 1.53
diff -u -r1.53 scoreboard.h
--- scoreboard.h29 Jul 2004 15:18:40 - 1.53
+++ scoreboard.h29 Jul 2004 18:28:21 -
@@ -124,7 +124,6 @@
* should still be serving requests.
*/
apr_time_t restart_time;
-int lb_limit;
} global_score;
/* stuff which the parent generally writes and the children rarely read */
@@ -138,13 +137,6 @@
*/
};
-/* stuff which is lb specific */
-typedef struct lb_score lb_score;
-struct lb_score{
-/* TODO: make a real stuct from this */
-unsigned char data[1024];
-};
-
/* Scoreboard is now in 'local' memory, since it isn't updated once created,
* even in forked architectures. Child created-processes (non-fork) will
* set up these indicies into the (possibly relocated) shmem records.
@@ -153,7 +145,6 @@
global_score *global;
process_score *parent;
worker_score **servers;
-lb_score **balancers;
} scoreboard;
typedef struct ap_sb_handle_t ap_sb_handle_t;
@@ -179,7 +170,6 @@
AP_DECLARE(worker_score *) ap_get_scoreboard_worker(int x, int y);
AP_DECLARE(process_score *) ap_get_scoreboard_process(int x);
AP_DECLARE(global_score *) ap_get_scoreboard_global(void);
-AP_DECLARE(lb_score *) ap_get_scoreboard_lb(int child_num, int lb_num);
AP_DECLARE_DATA extern scoreboard *ap_scoreboard_image;
AP_DECLARE_DATA extern const char *ap_scoreboard_fname;
@@ -198,11 +188,13 @@
AP_DECLARE_HOOK(int, pre_mpm, (apr_pool_t *p, ap_scoreboard_e sb_type))
/**
- * proxy load balancer
- * @return the number of load balancer workers.
+ * Hook for reopening scoreboard, reopen scoreboard.
+ * @param detached Non-zero if this is a seperate child process.
+ * @ingroup hooks
+ * @return OK or DECLINE on success; anything else is a error
*/
-APR_DECLARE_OPTIONAL_FN(int, ap_proxy_lb_workers,
-(void));
+AP_DECLARE_HOOK(int, reopen_scoreboard, (int detached))
+
/* for time_process_request() in http_main.c */
#define START_PREQUEST 1
Index: scoreboard.c
===
RCS file: /home/cvspublic/httpd-2.0/server/scoreboard.c,v
retrieving revision 1.75
diff -u -r1.75 scoreboard.c
--- scoreboard.c28 Jul 2004 22:50:54 - 1.75
+++ scoreboard.c29 Jul 2004 18:33:01 -
@@ -53,21 +53,23 @@
APR_HOOK_STRUCT(
APR_HOOK_LINK(pre_mpm)
+APR_HOOK_LINK(reopen_scoreboard)
)
AP_IMPLEMENT_HOOK_RUN_ALL(int,pre_mpm,
(apr_pool_t *p, ap_scoreboard_e sb_type),
(p, sb_type),OK,DECLINED)
-static APR_OPTIONAL_FN_TYPE(ap_proxy_lb_workers)
-*proxy_lb_workers;
+AP_IMPLEMENT_HOOK_RUN_ALL(int,reopen_scoreboard,
+ (int detached),
+ (detached),OK,DECLINED)
struct ap_sb_handle_t {
int child_num;
int thread_num;
};
-static int server_limit, thread_limit, lb_limit;
+static int server_limit, thread_limit;
static apr_size_t scoreboard_size;
/*
@@ -93,18 +95,9 @@
ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, thread_limit);
ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, server_limit);
-if (!proxy_lb_workers)
-proxy_lb_workers = APR_RETRIEVE_OPTIONAL_FN(ap_proxy_lb_workers);
-if (proxy_lb_workers)
-lb_limit = proxy_lb_workers();
-else
-lb_limit = 0;
-
scoreboard_size = sizeof(global_score);
scoreboard_size += sizeof(process_score) * server_limit;
scoreboard_size += sizeof(worker_score) * server_limit * thread_limit;
-if (lb_limit)
-scoreboard_size += sizeof(lb_score) * server_limit * lb_limit;
return scoreboard_size;
}
@@ -116,8 +109,7 @@
ap_calc_scoreboard_size();
ap_scoreboard_image =
-calloc(1, sizeof(scoreboard) + server_limit * sizeof(worker_score *) +
- server_limit * lb_limit * sizeof(lb_score *));
+calloc(1, sizeof(scoreboard) + server_limit * sizeof(worker_score *));
more_storage = shared_score;
ap_scoreboard_image-global = (global_score *)more_storage;
more_storage += sizeof(global_score);
@@ -129,19 +121,9 @@
ap_scoreboard_image-servers[i] = (worker_score *)more_storage;
more_storage += thread_limit * sizeof(worker_score);
}
-if
Ideas for Smart Filtering
The filter architecture periodically gets discussed here, and I've been meaning to write up my thoughts for some time. I'm using a module that implements a slightly different filter API, primarily for filtering in a proxy context. I've now written a brief discussion document on the subject. It's mostly an abstraction of what I'm currently using, although it does propose some additional improvements, primarily with regard to protocol handling (reflecting the recent byteranges thread here). It generated an interesting discussion, including some interesting alternative ideas, last night on IRC. Perhaps it can lead to a general-purpose module for 2.0 and an architecture update for 2.2? http://www.apachetutor.org/dev/smart-filter -- Nick Kew
Re: cvs commit: httpd-2.0 STATUS
On Thursday, July 29, 2004, at 05:58 AM, André Malo wrote:
* Mladen Turk [EMAIL PROTECTED] wrote:
William A. Rowe, Jr. wrote:
/* Scoreboard file, if there is one */
#ifndef DEFAULT_SCOREBOARD
@@ -118,6 +119,7 @@
typedef struct {
int server_limit;
int thread_limit;
+int lb_limit;
ap_scoreboard_e sb_type;
ap_generation_t running_generation; /* the
generation of children which
* should still
be serving
requests. */
This definitely breaks binary compatibility.
Moving the lb_limit to the end of the struct will not break the binary
compatibility. Correct?
Not Correct. It *may* be the case. Depending on who allocates the
stuff.
Then the question to ask is whether any independent modules
(those that are not installed when the server is installed)
are likely to use that structure, and how they are expected
to use it.
I'd be surprised if it were even possible for an independent
module to allocate a scoreboard struct, but it has been a while
since I looked at that code.
Roy
