[STATUS] (perl-framework) Wed May 29 23:45:28 EDT 2002
httpd-test/perl-framework STATUS: -*-text-*- Last modified at [$Date: 2002/03/09 05:22:48 $] Stuff to do: * finish the t/TEST exit code issue (ORed with 0x2C if framework failed) * change existing tests that frob the DocumentRoot (e.g., t/modules/access.t) to *not* do that; instead, have Makefile.PL prepare appropriate subdirectory configs for them. Why? So t/TEST can be used to test a remote server. * problems with -d perl mode, doesn't work as documented Message-ID: [EMAIL PROTECTED] Date: Sat, 20 Oct 2001 12:58:33 +0800 Subject: Re: perldb Tests to be written: * t/apache - simulations of network failures (incomplete POST bodies, chunked and unchunked; missing POST bodies; slooow client connexions, such as taking 1 minute to send 1KiB; ...) * t/modules/autoindex - something seems possibly broken with inheritance on 2.0 * t/ssl - SSLPassPhraseDialog exec: - SSLRandomSeed exec:
[STATUS] (flood) Wed May 29 23:45:27 EDT 2002
flood STATUS: -*-text-*- Last modified at [$Date: 2002/01/17 01:09:45 $] Release: milestone-04: In development milestone-03: Tagged January 16, 2002 ASF-transfer: Released July 17, 2001 milestone-02: Tagged August 13, 2001 milestone-01: Tagged July 11, 2001 (tag lost during transfer) RELEASE SHOWSTOPPERS: * Everything needs to work perfectly Other bugs that need fixing: * DNS lookup failures in any urllist cause segfault. Justin says: Wow. Why? Aaron says: Is this still happening? It's been awhile. * Misformed URLs cause a segfault. When we fix this we should make sure to print out a helpful message when this occurs. (This is not something that can be detected with a validating XML parser, unfortunately.) * iPlanet sends Content-length - there is a hack in there now to recognize it. However, all HTTP headers need to be normalized before checking their values. This isn't easy to do. Grr. * OpenSSL 0.9.6 Segfaults under high load. Upgrade to OpenSSL 0.9.6b. Aaron says: I just found a big bug that might have been causing this all along (we weren't closing ssl sockets). How can I reproduce the problem you were seeing to verify if this was the fix? * SEGVs when /tmp/.rnd doesn't exist are bad. Make it configurable and at least bomb with a good error message. (See Doug's patch.) Status: This is fixed, no? * If APR has disabled threads, flood should as well. We might want to have an enable/disable parameter that does this also, providing an error if threads are desired but not available. * flood needs to clear pools more often. With a long running test it can chew up memory very quickly. We should just bite the bullet and create/destroy/clear pools for each level of our model: farm, farmer, profile, url/request-cycle, etc. Other features that need writing: * Write robust tool (using tethereal perhaps) to take network dumps and convert them to flood's XML format. Status: Justin volunteers. Aaron had a script somewhere that is a start. * Get chunked encoding support working. Status: Justin volunteers. He got sidetracked by the httpd implementation of input filtering and never finished this. This is a stopgap until apr-serf is completed. * Maybe we should make randfile and capath runtime directives that come out of the XML, instead of autoconf parameters. * Migrate all locks to APRs new lock api. * We are using apr_os_thread_current() and getpid() in some places when what we really want is a GUID. The GUID will be used to correlate raw output data with each farmer. We may wish to print a unique ID for each of farm, farmer, profile, and url to help in postprocessing. * We are using strtol() in some places and strtoll() in others. Pick one (Aaron says strtol(), but he's not sure). * Validation of responses (known C-L, specific strings in response) Status: Justin volunteers * HTTP error codes (ie. teach it about 302s) Justin says: Yeah, this won't be with round_robin as implemented. Need a linked list-based profile where we can insert new URLs into the sequence. * Farmer (Single-thread, multiple profiles) Status: Aaron says: If you have threads, then any Farmer can be run as part of any Farm. If you don't have threads, you can currently only run one Farmer named Joe right now (this will be changed so that if you don't have threads, flood will attempt to run all Farmers in serial under one process). * Collective (Single-host, multiple farms) This is a number of Farms that have been fork()ed into child processes. * Megaconglomerate (Multiple hosts each running a collective) This is a number of Collectives running on a number of hosts, invoked via RSH/SSH or maybe even some proprietary mechanism. * Other types of urllists a) Random / Random-weighted b) Sequenced (useful with cookie propogation) c) Round-robin d) Chaining of the above strategies Status: Round-robin is complete. * Other types of reports Status: Aaron says: simple reports are functional. Justin added a new type that simply prints the approx. timestamp when the test was run, and the result as OK/FAIL; it is called easy reports (see flood_easy_reports.h). Furthermore, simple_reports and easy_reports both print out the current requesting URI line. Documentation that needs writing: * Documentation? What documentation? RTFS? Status: Justin volunteers. He'll probably use Anakia
Modules using the input brigade calls directly
On Thu, May 30, 2002 at 07:34:00AM -, [EMAIL PROTECTED] wrote: jerenkrantz02/05/30 00:34:00 Modified:.CHANGES modules/proxy mod_proxy.c proxy_http.c Log: Switch mod_proxy to using the brigade/filter calls directly rather than the *_client_block calls. Okay, with mod_proxy and the cgi variants done, I think I've transformed the majority of our uses of ap_*_client_block to natively use the input filtering API. (And, mod_deflate's new filter follows a similar strategy.) In case you are interested, here's a summary of what/why I've done: - These modules will now accept chunked encoding (which will be dechunked by HTTP_IN). Previously, this was forbidden due to ap_setup_client_block(). This is probably the major reason why I did this. I don't believe we should necessarily limit to only accepting C-L denoted bodies. As I've stated before, if a module wishes to get the raw chunks, they should remove the HTTP_IN filter. - As a side effect, a lot of 8k static buffer allocations have been removed. (I can hear the Netware guys cheering from here.) - This should also greatly reduce the number of copies that the input data should have to go through as it proceeds up to these modules. In the case of mod_proxy, the code is zero-copy from the standpoint of the module - it just gets a brigade from the input and passes it on to the output filter - no copies needed. - A large amount of code in the get_client_block can be bypassed. So, this should reduce our input overhead. And, we don't have to parse the header information twice for each request body (since we don't call ap_setup_client_block anymore). - As discussed previously on here (thanks to Greg and Aaron), HTTP_IN (aka ap_http_filter) will now handle the protocol logic for you. If there should be no body, it will return EOS. Note that due to the RFC interpretation, I had to introduce a special case for proxy's response using HTTP_IN so that the response case with just a Connection: Close would be deemed valid (this case is only valid when dealing with a response body). Please let me know if you encounter any problems or have any suggestions. I have tried to test all cases that I could think of (incl. httpd-test), but it's very possible something slipped through. -- justin
Re: Client socket
On Thu, May 30, 2002 at 11:03:05AM +0530, Vinod Panicker wrote: I need the actual socket that apache uses to communicate with the client in the php module. Is it available somewhere in the request_rec structure? You've already found it in r-connection-client-fd. Tony. -- f.a.n.finch [EMAIL PROTECTED] http://dotat.at/ SOUTH FITZROY: NORTH OR NORTHEAST 4 OR 5, OCCASIONALLY 6 NEAR CAPE FINISTERRE. FAIR. GOOD.
RE: Client socket
From: Tony Finch [mailto:[EMAIL PROTECTED]]On Behalf Of 'Tony Finch' Sent: 30 May 2002 11:01 On Thu, May 30, 2002 at 11:03:05AM +0530, Vinod Panicker wrote: I need the actual socket that apache uses to communicate with the client in the php module. Is it available somewhere in the request_rec structure? You've already found it in r-connection-client-fd. Question: why do you need the socket? Staying away from the socket is in general a better idea. You get all the data through the input filters, bypassing those means you lose out on a lot of functionality. Sander
RE: Client socket
I feel like kicking myself! Forgot all that I had learnt abt sockets! What I actually wanted was the ability to use the socket from another process. I obviously cant do that in this case since the socket that has been opened is local to the process (DUH!) Now I want to pass this socket to another application, which can write to it when required. I'm sitting with stevens right now, hunting for ways in which I can pass open file descriptors to another process :) Any help would be welcome. I don't require the input filters, since I want to write to the socket of a keep-alive connection later. Was just wondering if I can do it in a simple way, or I'll have to use libnet to do it Tx, Vinod. --- Vinod Panicker [EMAIL PROTECTED] Sr. Software Designer Geodesic Information Systems Ltd. -Original Message- From: Sander Striker [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 30, 2002 2:54 PM To: [EMAIL PROTECTED]; Vinod Panicker Subject: RE: Client socket From: Tony Finch [mailto:[EMAIL PROTECTED]]On Behalf Of 'Tony Finch' Sent: 30 May 2002 11:01 On Thu, May 30, 2002 at 11:03:05AM +0530, Vinod Panicker wrote: I need the actual socket that apache uses to communicate with the client in the php module. Is it available somewhere in the request_rec structure? You've already found it in r-connection-client-fd. Question: why do you need the socket? Staying away from the socket is in general a better idea. You get all the data through the input filters, bypassing those means you lose out on a lot of functionality. Sander
Re: Renames pending, WAS: RE: cvs commit: httpd-2.0/server/mpm/worker worker.c
* Sander Striker ([EMAIL PROTECTED]) wrote : From: Cliff Woolley [mailto:[EMAIL PROTECTED]] Sent: 30 May 2002 02:41 [...] Log: Catch up with the apr_allocator_set_owner - apr_allocator_owner_set renames in APR. This requires an MMN bump (which is fine with me, since we've already done one in 2.0.37-dev, and I'm starting to see little point in going back now). If there are other renames of this sort, we should get them in all at once. What about these: apr_pool_set_abort apr_pool_get_abort apr_pool_get_parent And the ones left in renames_pending? Also, Thom May posted a long list of suggestions a while back. Noone seems to have responded to that. Attached. -- Thom May - [EMAIL PROTECTED] Stibbons pants is like fuck. It can be good or bad. Index: renames_pending === RCS file: /home/cvspublic/apr/renames_pending,v retrieving revision 1.9 diff -u -u -r1.9 renames_pending --- renames_pending 22 Apr 2002 10:24:40 - 1.9 +++ renames_pending 30 May 2002 09:46:38 - @@ -4,3 +4,51 @@ apr_user_id_get from apr_get_userid apr_user_name_getfrom apr_get_username +-- +apr_time_exp_gmt_get from apr_implode_gmt +apr_time_interval_t from apr_interval_time_t +apr_time_interval_short_tfrom apr_short_interval_time_t +apr_file_info_t from apr_finfo_t +apr_file_statfrom apr_stat +apr_file_lstat from apr_lstat +apr_file_path_root from apr_filepath_root +apr_file_path_merge from apr_file_path_merge +apr_file_path_setfrom apr_filepath_set +apr_file_path_getfrom apr_filepath_get +apr_file_attrs_t from apr_fileattrs_t +apr_file_seek_where_tfrom apr_seek_where_t + +apr_fnmatch_test from apr_is_fnmatch + +apr_lock_scope_e from apr_lockscope_e +apr_lock_type_e from apr_locktype_e +apr_lock_mech_e from apr_lockmech_e +apr_lock_readerwriter_e from apr_readerwriter_e + +apr_socket_shutdown from apr_shutdown +apr_socket_bind from apr_bind +apr_socket_listenfrom apr_listen +apr_socket_acceptfrom apr_accept +apr_socket_connect from apr_connect +apr_sockaddr_name_info_get from apr_getnameinfo +apr_port_addr_parse from apr_parse_addr_port +apr_hostname_get from apr_gethostname +apr_socket_send from apr_send +apr_socket_sendv from apr_sendv +apr_socket_sendtofrom apr_sendto +apr_socket_recv_from from apr_recvfrom +apr_socket_file_send from apr_sendfile +apr_socket_recv from apr_recv +apr_socket_opt_set from apr_setsocketopt +apr_socket_opt_get from apr_getsocketopt +apr_socket_file_create from apr_socket_from_file +apr_service_byname_get from apr_getservbyname +apr_socket_filter_accept from apr_socket_accept_filter +apr_socket_inherit_set from apr_socket_set_inherit +apr_socket_inherit_unset from apr_socket_unset_inherit + +apr_user_id_current from apr_current_userid +apr_user_compare from apr_compare_users +apr_group_id_get from apr_get_groupid +apr_group_comparefrom apr_compare_groups +
Re: cvs commit: httpd-2.0/include ap_mmn.h
On Thu, May 30, 2002 at 05:57:33AM -, [EMAIL PROTECTED] wrote: jwoolley02/05/29 22:57:33 Modified:include ap_mmn.h Log: Imagine the horror. I go to try compiling PHP4, and it bombs out on r-boundary. BUT WAIT, I say, we have a test in there for that: #if !MODULE_MAGIC_AT_LEAST(20020506,0). The test doesn't seem to be working. That's odd, I think. So I go look at the ap_mmn.h. Egad, the tests are totally backwards. How on earth are we going to handle this? Surely it's wrong in 1.3.x as well. So I go look. Nope, turns out it got fixed TWO YEARS AGO in that repository. Sigh. Anyway, thanks Greg. :) Submitted by: Greg Stein, circa Apache 1.3.13 hehe... :-) I had no idea it never made it to 2.0... woah. Thanks for the port! :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: [1.3] Proxy fixes and FWD: Re: [apache-modules] Setting bytes_sent in Request Record while generating all headers by myself in Apache 1.3]
Martin Kraemer wrote: About the X-Forwarded-* stuff: It's non-standard anyway (you can add any X-whatever header and still be RFC2616 compliant) so I'd rather not see it in 1.3 now (maybe in the next release ;-) I've seen proxies like squid hang because of nonstandard headers (proxy-connection: keep-alive) so I am not very fond of polluting the header namespace with more non-standard cruft. These headers have been present for a long time in the httpd-2.0 mod_proxy, and are also added as standard by Squid, so they do definitely work. As there is not likely to be another release unless new bugs are found, I would rather add it now and see it included, rather than add it later and have it never released. I'll commit it now (as I am in an internet cafe and cannot commit it later), however if there are any -1's please revert the commit before the release. Regards, Graham -- - [EMAIL PROTECTED]There's a moon over Bourbon Street tonight... smime.p7s Description: S/MIME Cryptographic Signature
Re: don't try this at home
Jeff Trawick [EMAIL PROTECTED] writes: okay, do try it, but (unlike somebody last night) don't try it on daedalus GET / HTTP/1.1 Accept: */* Host: test Content-Type: application/x-www-form-urlencoded Transfer-Encoding: chunked AAA Is this test (with cr-lf added where appropriate) working consistently for everybody that tries? Yesterday afternoon: on one build I was getting a 200 followed by a 500 (access_log said 500 was for method A...) on another build I was getting a segfault I'll try to understand this more later today. -- Jeff Trawick | [EMAIL PROTECTED] Born in Roswell... married an alien...
Re: setsockopt bug
I'm running Apache 2.0.36-dev on Win2K, with SSL enabled. Most clients are Win2K systems using either Netscape 6.x, IE5.5 or Mozilla RC1, and they connect and browse without problems. One client is a Mac OS X using IE5.0. It is constantly receiving 'Data Encryption Errors'. On pages where there are images, the images appear as broken links. I've tried all of the 'nokeepalive' tips (I believe) related to using IE, but no apparent difference in behavior. In my log file, there are several lines (actually many) of the error [Thu May 30 07:53:16 2002] [warn] (32538)Socket operation on non-socket: setsockopt(SO_UPDATE_ACCEPT_CONTEXT) failed. I now know that the Mac is not the only client generating these errors on the server, but the other clients appear to be working fine, even though these errors show up in the server log file. I'm in the process of upgrading the Mac to a more recent version of IE. For what it's worth, this Mac, using IE can reference our Netscape Enterprise Server, SSL enabled sites without problems. I was remember seeing the above error in the list recently, but searching the bug database did not find a hit. Since I have a system that can repeatedly generate both the client and the server problems, I was going to do some probing into the code to see if I could figure out what was going on ( before I upgrade and cover up the problem). Thanks Jeff Trawick wrote: Dwayne Miller [EMAIL PROTECTED] writes: Wasn't there recently a bug on WinNT/Win2k with a setsockopt call that failed. I'm getting that error when the client is IE5 on Mac OSx and SSL is enabled. Tks Exactly what error are you getting? Was there something in your error log? Show that entry here.
Re: Modules using the input brigade calls directly
Justin Erenkrantz wrote: On Thu, May 30, 2002 at 07:34:00AM -, [EMAIL PROTECTED] wrote: jerenkrantz02/05/30 00:34:00 Modified:.CHANGES modules/proxy mod_proxy.c proxy_http.c Log: Switch mod_proxy to using the brigade/filter calls directly rather than the *_client_block calls. Okay, with mod_proxy and the cgi variants done, I think I've transformed the majority of our uses of ap_*_client_block to natively use the input filtering API. (And, mod_deflate's new filter follows a similar strategy.) In case you are interested, here's a summary of what/why I've done: ... ... Wow, great! (*) Finally, Apache completed the migration to the modular model of I/O filtering... A historical day that should be remembered! (*) (*) (The first and the last paragraphs are NOT sarcastic...) Thanks, Justin (I need no more patches in the core source of Apache... Now I can do everything cleanly...) -- Eli Marmor [EMAIL PROTECTED] CTO, Founder Netmask (El-Mar) Internet Technologies Ltd. __ Tel.: +972-9-766-1020 8 Yad-Harutzim St. Fax.: +972-9-766-1314 P.O.B. 7004 Mobile: +972-50-23-7338 Kfar-Saba 44641, Israel
Re: Modules using the input brigade calls directly
A small wish from the field: Will Justin's stuff be included with the RC3 of 2.0.37? -- Eli Marmor [EMAIL PROTECTED] CTO, Founder Netmask (El-Mar) Internet Technologies Ltd. __ Tel.: +972-9-766-1020 8 Yad-Harutzim St. Fax.: +972-9-766-1314 P.O.B. 7004 Mobile: +972-50-23-7338 Kfar-Saba 44641, Israel
Re: don't try this at home
Jeff Trawick [EMAIL PROTECTED] writes: Jeff Trawick [EMAIL PROTECTED] writes: okay, do try it, but (unlike somebody last night) don't try it on daedalus GET / HTTP/1.1 Accept: */* Host: test Content-Type: application/x-www-form-urlencoded Transfer-Encoding: chunked AAA Is this test (with cr-lf added where appropriate) working consistently for everybody that tries? Yesterday afternoon: on one build I was getting a 200 followed by a 500 (access_log said 500 was for method A...) on another build I was getting a segfault Running HEAD, we get a segfault when the bogus request is for a resource for which we aren't authorized. Here is the traceback: #0 0x4014bd41 in __kill () from /lib/libc.so.6 #1 0x4014b9b6 in raise (sig=6) at ../sysdeps/posix/raise.c:27 #2 0x4014d0d8 in abort () at ../sysdeps/generic/abort.c:88 #3 0x80be80c in ap_log_assert (szExp=0x80e945b readbytes 0, szFile=0x80e93e4 http_protocol.c, nLine=957) at log.c:689 #4 0x807dc8c in ap_http_filter (f=0x81b46a8, b=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=-1) at http_protocol.c:957 #5 0x80cae79 in ap_get_brigade (next=0x81b46a8, bb=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at util_filter.c:507 #6 0x80d5125 in net_time_filter (f=0x81b3f68, b=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at core.c:3324 #7 0x80cae79 in ap_get_brigade (next=0x81b3f68, bb=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at util_filter.c:507 #8 0x807f46d in ap_get_client_block (r=0x81b3808, buffer=0xbfffb5c0, bufsiz=8192) at http_protocol.c:1769 #9 0x807f765 in ap_discard_request_body (r=0x81b3808) at http_protocol.c:1874 #10 0x8081df9 in ap_die (type=401, r=0x81b3808) at http_request.c:153 #11 0x807eaf8 in ap_http_header_filter (f=0x81b3fb0, b=0x81b4b48) at http_protocol.c:1448 #12 0x80caefd in ap_pass_brigade (next=0x81b3fb0, bb=0x81b4b48) at util_filter.c:534 #13 0x80ce3aa in ap_content_length_filter (f=0x81b3f98, b=0x81b4b48) at protocol.c:1292 #14 0x80caefd in ap_pass_brigade (next=0x81b3f98, bb=0x81b4b48) at util_filter.c:534 #15 0x8081223 in ap_byterange_filter (f=0x81b3f80, bb=0x81b4b48) at http_protocol.c:2745 #16 0x80caefd in ap_pass_brigade (next=0x81b3f80, bb=0x81b4b48) at util_filter.c:534 #17 0x807d94c in ap_http_filter (f=0x81b46a8, b=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at http_protocol.c:888 #18 0x80cae79 in ap_get_brigade (next=0x81b46a8, bb=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at util_filter.c:507 #19 0x80d5125 in net_time_filter (f=0x81b3f68, b=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at core.c:3324 #20 0x80cae79 in ap_get_brigade (next=0x81b3f68, bb=0x81b3f48, mode=AP_MODE_READBYTES, block=APR_BLOCK_READ, readbytes=8192) at util_filter.c:507 #21 0x807f46d in ap_get_client_block (r=0x81b3808, buffer=0xbfffd878, bufsiz=8192) at http_protocol.c:1769 #22 0x807f765 in ap_discard_request_body (r=0x81b3808) at http_protocol.c:1874 #23 0x8081df9 in ap_die (type=401, r=0x81b3808) at http_request.c:153 #24 0x80820c2 in ap_process_request (r=0x81b3808) at http_request.c:277 #25 0x807bbe1 in ap_process_http_connection (c=0x81ad848) at http_core.c:291 #26 0x80c7ce0 in ap_run_process_connection (c=0x81ad848) at connection.c:85 #27 0x80c8145 in ap_process_connection (c=0x81ad848, csd=0x81ad788) at connection.c:207 #28 0x80b80a4 in child_main (child_num_arg=4) at prefork.c:671 #29 0x80b8259 in make_child (s=0x8158c50, slot=4) at prefork.c:765 #30 0x80b82f2 in startup_children (number_to_start=1) at prefork.c:783 #31 0x80b87f1 in ap_mpm_run (_pconf=0x810cfe8, plog=0x814d0e8, s=0x8158c50) at prefork.c:999 #32 0x80bfe7c in main (argc=3, argv=0xba84) at main.c:646 (gdb) cute recursion, huh? I'll try to see what is up with the weird 200-followed-by-500 problem. -- Jeff Trawick | [EMAIL PROTECTED] Born in Roswell... married an alien...
Re: Client socket
On Thu, May 30, 2002 at 03:00:21PM +0530, Vinod Panicker wrote: I feel like kicking myself! Forgot all that I had learnt abt sockets! What I actually wanted was the ability to use the socket from another process. I obviously cant do that in this case since the socket that has been opened is local to the process (DUH!) Now I want to pass this socket to another application, which can write to it when required. I'm sitting with stevens right now, hunting for ways in which I can pass open file descriptors to another process :) Any help would be welcome. Unix domain sockets can be used to pass file descriptors, as can doors and stream pipes (see Advanced Programming in the Unix Environment, by W. Richard Stevens, Chapter 15, p. 475 for the domain sockets/stream pipes). Others and I have been working on adding this functionality to APR so that it will be available across unix platforms as well as Windows and others, FWIW. -aaron
Re: Modules using the input brigade calls directly
On Thu, 2002-05-30 at 01:23, Justin Erenkrantz wrote: - As a side effect, a lot of 8k static buffer allocations have been removed. (I can hear the Netware guys cheering from here.) - This should also greatly reduce the number of copies that the input data should have to go through as it proceeds up to these modules. In the case of mod_proxy, the code is zero-copy from the standpoint of the module - it just gets a brigade from the input and passes it on to the output filter - no copies needed. Thanks, this is a great improvement. --Brian
Re: httpd-2.0 STATUS
At 01:07 AM 5/30/2002, you wrote: On Thu, 30 May 2002, William A. Rowe, Jr. wrote: is modules/ssl/README even valuable anymore? yes. fine to remove the stale stuff, but not the whole damn thing. there was a useful roadmap of the source in there ... Then I'll ask a simple question before I resurrect ... Do we want individual READMEs or LAYOUTs that describe all the files throughout each Apache source tree directory, only one top-level LAYOUT that describes the entire directory tree and it's contents, or neither? List, what is your preference? I'll be happy to create the single template or all the LAYOUT skeletons and the group can populate all of them over time? ... and everything that was in the TODO section is still valid: o SSL renegotiations in combination with POST request o Port all remaining code (code inside #if 0...#endif blocks) o Do we need SSL_set_read_ahead()? o the ssl_expr api is NOT THREAD SAFE. race conditions exist: -in ssl_expr_comp() if SSLRequire is used in .htaccess (ssl_expr_info is global) -is ssl_expr_eval() if there is an error (ssl_expr_error is global) o SSLRequire directive (parsing of) leaks memory o Diffie-Hellman-Parameters for temporary keys are hardcoded in ssl_engine_dh.c, while the comment in ssl_engine_kernel.c says: it is suggested that keys be changed daily or every 500 transactions, and more often if possible. o ssl_var_lookup could be rewritten to be MUCH faster o CRL callback should be pluggable o session cache store should be pluggable o init functions should return status code rather than ssl_die() o ssl_engine_pphrase.c needs to be reworked so it is generic enough to also decrypt proxy keys o the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops Adding to our top-level STATUS, thanks Doug!!! Bill
Re: Modules using the input brigade calls directly
Which module would be a good example for someone who wants to rewrite an existing ap_*_client_block based module? Tks Brian Pane wrote: On Thu, 2002-05-30 at 01:23, Justin Erenkrantz wrote: - As a side effect, a lot of 8k static buffer allocations have been removed. (I can hear the Netware guys cheering from here.) - This should also greatly reduce the number of copies that the input data should have to go through as it proceeds up to these modules. In the case of mod_proxy, the code is zero-copy from the standpoint of the module - it just gets a brigade from the input and passes it on to the output filter - no copies needed. Thanks, this is a great improvement. --Brian
Re: httpd-2.0 STATUS
On Thu, May 30, 2002 at 10:43:54AM -0500, William A. Rowe, Jr. wrote: Then I'll ask a simple question before I resurrect ... Do we want individual READMEs or LAYOUTs that describe all the files throughout each Apache source tree directory, only one top-level LAYOUT that describes the entire directory tree and it's contents, or neither? List, what is your preference? +1 for one file at the top-level. -- justin
RE: httpd-2.0 STATUS
From: Justin Erenkrantz [mailto:[EMAIL PROTECTED]] Sent: 30 May 2002 18:38 On Thu, May 30, 2002 at 10:43:54AM -0500, William A. Rowe, Jr. wrote: Then I'll ask a simple question before I resurrect ... Do we want individual READMEs or LAYOUTs that describe all the files throughout each Apache source tree directory, only one top-level LAYOUT that describes the entire directory tree and it's contents, or neither? List, what is your preference? +1 for one file at the top-level. -- justin I'll see your +1 and raise you 1. Sander
Re: cvs commit: httpd-2.0/modules/generators mod_cgid.c
jerenkrantz02/05/29 22:42:46
Modified:modules/generators mod_cgid.c
Log:
Apply same patch to mod_cgid that was applied to mod_cgi so that it
bypasses the *_client_block calls in favor of using the input filters
and brigades directly.
Revision ChangesPath
1.134 +70 -35httpd-2.0/modules/generators/mod_cgid.c
Index: mod_cgid.c
===
snip
+do {
+apr_bucket *bucket;
+apr_status_t rv;
+
+rv = ap_get_brigade(r-input_filters, bb, AP_MODE_READBYTES,
+APR_BLOCK_READ, HUGE_STRING_LEN);
+
+if (rv != APR_SUCCESS) {
+return rv;
+}
+
+APR_BRIGADE_FOREACH(bucket, bb) {
+const char *data;
+apr_size_t len;
+
+if (APR_BUCKET_IS_EOS(bucket)) {
+seen_eos = 1;
+break;
+}
+
+/* We can't do much with this. */
+if (APR_BUCKET_IS_FLUSH(bucket)) {
+continue;
+}
+
+/* If the child stopped, we still must read to EOS. */
+if (child_stopped_reading) {
+continue;
+}
The above section of code looks like a potential endless loop if the brigade does not
contain an EOS bucket. Should the check for child_stopped_reading occur after the
apr_bucket_read below?
Bill
-if (ap_should_client_block(r)) {
-int dbsize, len_read;
+/* read */
+apr_bucket_read(bucket, data, len, APR_BLOCK_READ);
+
+if (conf-logname dbpos conf-bufbytes) {
+int cursize;
+
+if ((dbpos + len) conf-bufbytes) {
+cursize = conf-bufbytes - dbpos;
+}
+else {
+cursize = len;
+}
+memcpy(dbuf + dbpos, data, cursize);
+dbpos += cursize;
+}
+
+/* Keep writing data to the child until done or too much time
+ * elapses with no progress or an error occurs.
+ */
+rv = apr_file_write_full(tempsock, data, len, NULL);
+
+if (rv != APR_SUCCESS) {
+/* silly script stopped reading, soak up remaining message */
+child_stopped_reading = 1;
+}
+}
+apr_brigade_cleanup(bb);
+}
+while (!seen_eos);
+
+if (conf-logname) {
+dbuf[dbpos] = '\0';
+}
-if (conf-logname) {
-dbuf = apr_pcalloc(r-pool, conf-bufbytes + 1);
-dbpos = 0;
-}
-
-while ((len_read =
-ap_get_client_block(r, argsbuffer, HUGE_STRING_LEN)) 0) {
-if (conf-logname) {
-if ((dbpos + len_read) conf-bufbytes) {
-dbsize = conf-bufbytes - dbpos;
-}
-else {
-dbsize = len_read;
-}
-memcpy(dbuf + dbpos, argsbuffer, dbsize);
-dbpos += dbsize;
-}
-nbytes = len_read;
-apr_file_write(tempsock, argsbuffer, nbytes);
-if (nbytes len_read) {
-/* silly script stopped reading, soak up remaining message */
-while (ap_get_client_block(r, argsbuffer, HUGE_STRING_LEN) 0) {
-/* dump it */
-}
-break;
-}
-}
-}
/* we're done writing, or maybe we didn't write at all;
* force EOF on child's stdin so that the cgi detects end (or
* absence) of data
Re: cvs commit: httpd-2.0/modules/generators mod_cgid.c
On Thu, May 30, 2002 at 12:54:39PM -0400, Bill Stoddard wrote: The above section of code looks like a potential endless loop if the brigade does not contain an EOS bucket. Should the check for child_stopped_reading occur after the apr_bucket_read below? An EOS should be returned at some point - we get a brigade from the input filters, read through the brigade, if we didn't see EOS, then we repeat. So, I'm not concerned about that - we'll have to get EOS. If the client socket is disconnected or errors out, we'll return. However, on the second point, you may well be right. I wasn't 100% sure if the check should be done before or after the apr_bucket_read. Any reason why it should be after? I was trying to save the read call. -- justin
Re: httpd-2.0 STATUS
i see value the old modules/ssl/README. it has been very handy in the past, and i would expect it to be for anybody coming from mod_ssl 1.3 based sources to contribute to 2.0 or even just being brand new to the 2.0 source. now they have lost the source roadmap, summary of major changes, incompatibilities, etc. todo items being in modules/ssl or in the top-level STATUS, i don't really care. but why blow away the rest of the useful info that was in there?
Re: Modules using the input brigade calls directly
On Thu, 30 May 2002, Eli Marmor wrote: A small wish from the field: Will Justin's stuff be included with the RC3 of 2.0.37? I'm still procrastinating on tagging RC2. Most of Justin's stuff is already in, but I'm considering backing it out and waiting until RC3 since some questions have surfaced. I see no reason why it shouldn't go in the final 2.0.37 though, assuming we get enough testing time on it between now and then. --Cliff
Re: httpd-2.0 STATUS
At 12:02 PM 5/30/2002, you wrote: i see value the old modules/ssl/README. it has been very handy in the past, and i would expect it to be for anybody coming from mod_ssl 1.3 based sources to contribute to 2.0 or even just being brand new to the 2.0 source. now they have lost the source roadmap, summary of major changes, incompatibilities, etc. todo items being in modules/ssl or in the top-level STATUS, i don't really care. but why blow away the rest of the useful info that was in there? The TODO list has moved to STATUS so that's good. Based on feedback, I'll create a top-level LAYOUT describing the directory trees within the httpd-2.0 layout, and fill in the SSL items from the old README. This allows any developers to grasp any part of the server [including mod_ssl] without jumping between different files and different conventions for each of the server's components. Perhaps we could resurrect the porting history [although I believe it's horribly incomplete] as modules/ssl/HISTORY? OTOH, those parts that are correct aught to have been committed to CHANGES if they were not in the first place. What I -don't- want is to continue special-casing modules where each has their own conventions for additional developer documentation and files. This will continue to be an issue when modules like proxy jump from their own sub-projects for development into the httpd-2.0/modules repository for general inclusion in the server. Bill
Re: httpd-2.0 STATUS
On Thu, 30 May 2002, William A. Rowe, Jr. wrote: Perhaps we could resurrect the porting history [although I believe it's horribly incomplete] as modules/ssl/HISTORY? OTOH, those parts that are correct aught to have been committed to CHANGES if they were not in the first place. they are in CHANGES, but it is HUGE. a summary is nice rather than having to sift through that file. the httpd CHANGES file is not very useful imho. too much info for users, to little for developers. i like what Perl and PHP do, Changes is generated from the perforce/cvs logs, has filenames, dates, submitter, committer, etc., handy for developers. then perldelta.pod contains a brief summary of the Changes that is useful and easy for users to understand. PHP has something similar, but calls it NEWS i think.
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 08:20:30PM -, [EMAIL PROTECTED] wrote: + 2) edit the path to LIBTOOL in config_vars.mk to reflect the + place where it was actually installed In this case, I think it would be best if it used apr-config's --apr-libtool option. % /pkg/httpd-2.0/bin/apr-config --apr-libtool /bin/sh /pkg/httpd-2.0/build/libtool % srclib/apr/apr-config --apr-libtool /bin/sh /home/jerenkrantz/cvs-apache/build/prefork/srclib/apr/libtool This might be better than relying on config_vars.mk. -- justin
Re: don't try this at home
On Thu, May 30, 2002 at 11:02:09AM -0400, Jeff Trawick wrote: Running HEAD, we get a segfault when the bogus request is for a resource for which we aren't authorized. Here is the traceback: Oh, that's not good. Something broke after I initially fixed it. I'm trying to chase down what happened now. -- justin
Re: don't try this at home
On Thu, May 30, 2002 at 01:33:45PM -0700, Justin Erenkrantz wrote: On Thu, May 30, 2002 at 11:02:09AM -0400, Jeff Trawick wrote: Running HEAD, we get a segfault when the bogus request is for a resource for which we aren't authorized. Here is the traceback: Oh, that's not good. Something broke after I initially fixed it. I'm trying to chase down what happened now. -- justin As an update: My problem was related to mod_bucketeer eating the error buckets. I fixed that and it works for me now. (Perhaps other output filters need to pay attention to the error bucket as well?) I'll try with a URL that generates a 401 now. -- justin
[PATCH] Sanity checking for htpasswd.c
Hi,
just reposting this patch.
Cheers,
-Thom
--
Thom May - [EMAIL PROTECTED]
Gman hum, I wonder if the cargo bar is looking for staff?
thom gman: cargo bar sydney has an insane staff turn over rate :)
jdub yeah, for turning over in the morning and seeing what you've woken up with.
hadess jdub woke up with Gman in his bed :P
Index: htpasswd.c
===
RCS file: /home/cvspublic/httpd-2.0/support/htpasswd.c,v
retrieving revision 1.43
diff -u -u -r1.43 htpasswd.c
--- htpasswd.c 16 May 2002 19:57:11 - 1.43
+++ htpasswd.c 17 May 2002 07:43:49 -
@@ -77,6 +77,7 @@
* 5: Failure; buffer would overflow (username, filename, or computed
* record too long)
* 6: Failure; username contains illegal or reserved characters
+ * 7: Failure: file is not a valid htpasswd file
*/
#include apr.h
@@ -133,6 +134,7 @@
#define ERR_INTERRUPTED 4
#define ERR_OVERFLOW 5
#define ERR_BADUSER 6
+#define ERR_INVALID 7
/*
* This needs to be declared statically so the signal handler can
@@ -582,6 +584,39 @@
perror(fopen);
exit(ERR_FILEPERM);
}
+/*
+ * Now we need to confirm that this is a valid htpasswd file
+ */
+if (! newfile){
+
+fpw = fopen(pwfilename, r);
+while (! (get_line(line, sizeof(line), fpw))) {
+char *testcolon;
+
+if ((line[0] == '#') || (line[0] == '\0')) {
+continue;
+}
+testcolon = strchr(line, ':');
+if (testcolon != NULL){
+/*
+ * We got a valid line. keep going
+ */
+continue;
+}
+else {
+/*
+ * no colon in the line, and it's not a comment
+ * Time to bail out before we do damage.
+ */
+fprintf(stderr, %s: The file %s does not appear
+to be a valid htpasswd file.\n,
+argv[0], pwfilename);
+fclose(fpw);
+exit(ERR_INVALID);
+}
+}
+fclose(fpw);
+}
}
/*
@@ -678,7 +713,7 @@
/*
* The temporary file now contains the information that should be
* in the actual password file. Close the open files, re-open them
- * in the appropriate mode, and copy them file to the real one.
+ * in the appropriate mode, and copy the temp file to the real one.
*/
fclose(ftemp);
fpw = fopen(pwfilename, w+);
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
From: Justin Erenkrantz [mailto:[EMAIL PROTECTED]] On Thu, May 30, 2002 at 02:43:05PM -0700, Ryan Bloom wrote: This doesn't fix the underlying problem. Mod_bucketeer needs to be smart enough to copy buckets if it doesn't recognize their type. If the bucket can't be copied, then the filter should remove the bucket from one brigade and add it to the other. Any filter that silently removes buckets is broken. True, but I thought part of the idea behind buckets is so that the backing type doesn't have to be known. Exactly! The fact here is that not all buckets can be read such as EOS, flush, and error. Perhaps these buckets should return an error if they are read? Then, if a filter sees APR_ENOTIMPL, it can copy them over. Every bucket can be read. That is a mandatory function that all buckets must implement. If the back-end bucket-type isn't know, the filter can ALWAYS do a read/apr_bucket_make_heap. Ryan
Re: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
On Thu, 30 May 2002, Justin Erenkrantz wrote: The fact here is that not all buckets can be read such as EOS, flush, and error. Sure they can. They just contain zero data bytes. Perhaps these buckets should return an error if they are read? Then, if a filter sees APR_ENOTIMPL, it can copy them over. -- justin No way. *All* buckets can be read from, that's an important part of the design. Having to test for APR_ENOTIMPL all over the place would be a major drag. --Cliff
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
On Thu, 30 May 2002, Ryan Bloom wrote: True, but I thought part of the idea behind buckets is so that the backing type doesn't have to be known. Exactly! Exactly. All you really need to test here is length. If the bucket is zero bytes, that's all you need to know. That gets you all metadata buckets handled in one fell swoop. If there are certain kinds of metadata buckets a filter *does* know about and want to handle specially (eg EOS or FLUSH), it can make the choice to special case those. Every bucket can be read. That is a mandatory function that all buckets must implement. If the back-end bucket-type isn't know, the filter can ALWAYS do a read/apr_bucket_make_heap. True. Though we've specified the additional semantics that apr_bucket_read() causes the bucket to auto-morph into one that is in-memory. Either way. --Cliff
Re: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
On Thu, May 30, 2002 at 05:50:45PM -0400, Cliff Woolley wrote: On Thu, 30 May 2002, Justin Erenkrantz wrote: The fact here is that not all buckets can be read such as EOS, flush, and error. Sure they can. They just contain zero data bytes. Right, but the problem is how are these metadata buckets identified if their types aren't checked for explicitly? It'd be nice if we came up with a way to prevent every module for checking for EOS, flush, and error each time. -- justin
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
Every bucket can be read. That is a mandatory function that all buckets must implement. If the back-end bucket-type isn't know, the filter can ALWAYS do a read/apr_bucket_make_heap. True. Though we've specified the additional semantics that apr_bucket_read() causes the bucket to auto-morph into one that is in-memory. Either way. I didn't think it _had_ to auto-morph. My understanding is that the default buckets do, because we assume the performance will be better if they do. That makes sense, because a file_bucket is likely to be read multiple times, so it makes sense to morph the bucket on the first read, so that we don't have to go to the disk twice. Ryan
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
On Thu, 30 May 2002, Ryan Bloom wrote: I didn't think it _had_ to auto-morph. My understanding is that the default buckets do, because we assume the performance will be better if they do. That makes sense, because a file_bucket is likely to be read multiple times, so it makes sense to morph the bucket on the first read, so that we don't have to go to the disk twice. If it didn't, how could you possibly pass back a buffer containing the data? In other words, I guess there's no hard and fast rule that it *has* to actually morph, but since it's doing all the work of copying that data into an in-memory buffer anyway, it doesn't make much sense *not* to morph to a bucket type that handles in-memory buffers. Sure it could allocate a new buffer and redo the copy every single time it was read from, but why would it want to? shrug --Cliff
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
From: Justin Erenkrantz [mailto:[EMAIL PROTECTED]] On Thu, May 30, 2002 at 05:50:45PM -0400, Cliff Woolley wrote: On Thu, 30 May 2002, Justin Erenkrantz wrote: The fact here is that not all buckets can be read such as EOS, flush, and error. Sure they can. They just contain zero data bytes. Right, but the problem is how are these metadata buckets identified if their types aren't checked for explicitly? It'd be nice if we came up with a way to prevent every module for checking for EOS, flush, and error each time. -- Justin All of the metadata buckets have a common interface. They all have a 0 length. So, filters should be written so that any 0 length bucket is just moved from one brigade to the next. If 0 length buckets are removed from the brigade, then the filter is most likely doing something wrong. The only time this isn't true, is when the bucket is a 0 length heap/mmap/file/transient bucket, but those don't really matter, because the lowest level filters will remove those cleanly. Since the lowest level filters know which are the metadata buckets, they also know that any 0 length bucket that isn't a metadata bucket can be removed. Ryan
Re: cvs commit: httpd-2.0/modules/ssl ssl_engine_pphrase.c
On 30 May 2002 [EMAIL PROTECTED] wrote: jwoolley02/05/30 15:39:08 Modified:modules/ssl ssl_engine_pphrase.c Log: This definitely gets the award for least useful error message of the month. Index: ssl_engine_pphrase.c === RCS file: /home/cvs/httpd-2.0/modules/ssl/ssl_engine_pphrase.c,v retrieving revision 1.40 retrieving revision 1.41 PS: I totally eyeballed this... not tested. I'm hoping somebody with Win32 will make sure it compiles for me. :) --Cliff
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
From: Cliff Woolley [mailto:[EMAIL PROTECTED]] On Thu, 30 May 2002, Ryan Bloom wrote: I didn't think it _had_ to auto-morph. My understanding is that the default buckets do, because we assume the performance will be better if they do. That makes sense, because a file_bucket is likely to be read multiple times, so it makes sense to morph the bucket on the first read, so that we don't have to go to the disk twice. If it didn't, how could you possibly pass back a buffer containing the data? In other words, I guess there's no hard and fast rule that it *has* to actually morph, but since it's doing all the work of copying that data into an in-memory buffer anyway, it doesn't make much sense *not* to morph to a bucket type that handles in-memory buffers. Sure it could allocate a new buffer and redo the copy every single time it was read from, but why would it want to? shrug Agreed, but I can think of one case. (a bit of hand-waving, but a possibility, nonetheless) :-) If I write two filters that work in tandem to solve a problem, I may want to use a special bucket type to facilitate them working together. For example, if I have a bucket type that encodes graphics in lists of RGB values, then the read function MUST return a binary buffer of data, but later filters will most likely find it MUCH easier to deal with the RGB values directly. So, in this case, I could have a handler that served graphics files by filling out this special bucket, and a filter later on that added a watermark. Any filter in between the two will still have to be able to read from the bucket, but the watermark filter will be able to operate much better on the RGB values. That is the only type of case that would make sense to keep the data in a different structure. :-) Ryan
RE: cvs commit: httpd-2.0/modules/test mod_bucketeer.c
On Thu, 30 May 2002, Ryan Bloom wrote: Agreed, but I can think of one case. (a bit of hand-waving, but a possibility, nonetheless) :-) If I write two filters that work in tandem to solve a problem, I may want to use a special bucket type to facilitate them working together. Fair enough. I was thinking of a similar situation a few minutes ago. Though I think in practice it's likely that such a bucket type would hang on to the buffer it copied the data into in addition to the native format and would therefore meet my earlier assertion by virtue of having the data in both formats (binary buffer and native format) after the first call to apr_bucket_read(). It wouldn't *have* to, but it probably would anyway. :) --Cliff
Re: cvs commit: httpd-2.0 STATUS
On 30 May 2002 [EMAIL PROTECTED] wrote: + +* 413 (invalid chunk size) followed by another request segfaults. + Message-ID: [EMAIL PROTECTED] + Status: Justin is completely confounded by this. It looks like a + bucket lifetime bug, but somehow an operation on one + brigade is altering another brigade and corrupting it. Ouch. I'll take a peek. Your lifetime guess is probably a good one. I'd think that most likely what's happening is that a bucket (or a bucket buffer) is getting freed but still being used afterward; the bucket freelists would reassign that storage thereby making it easy for the first to corrupt the second. Can you give me an exact sequence of requests to duplicate this? --Cliff
release strategy stuff (was: cvs commit: httpd-2.0 STATUS)
On Sun, May 26, 2002 at 11:24:49AM -0700, Brian Pane wrote: ... I didn't yet cast a vote. So, -0 for including the pool fix in 2.0.37 +1 for including it in 2.0.38. Note: the real procedure here is that Sander gets to just check the stuff in whenever he would like. (which, I believe he did as soon as Brian said the perf was fine) After that, if the RM wants to include the change or not, it is up to him. Personally, I'm finding that it seems we're getting back to the old, slow it takes a lot of pain to make a release process. Rather than a nice, easy, snap and release. Remember: even though we're GA, we can still snap new releases, test them, and *then* decide whether to release them (and whether to call them GA or intemediate beta releases). Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 07:26:02PM -0400, Cliff Woolley wrote: On 30 May 2002 [EMAIL PROTECTED] wrote: + +* 413 (invalid chunk size) followed by another request segfaults. + Message-ID: [EMAIL PROTECTED] + Status: Justin is completely confounded by this. It looks like a + bucket lifetime bug, but somehow an operation on one + brigade is altering another brigade and corrupting it. Ouch. I'll take a peek. Your lifetime guess is probably a good one. I'd think that most likely what's happening is that a bucket (or a bucket buffer) is getting freed but still being used afterward; the bucket freelists would reassign that storage thereby making it easy for the first to corrupt the second. Can you give me an exact sequence of requests to duplicate this? Take Jeff's sequence of: GET / HTTP/1.0 Transfer-Encoding: chunked AAA You'll get a 413 back (correctly). Then the connection is closed and reconnect and do: GET / HTTP/1.0 Segfault. As I hinted at in a message a few minutes ago, the b brigade in ap_http_header_filter is fine before calling basic_http_header. After that function returns, the b brigade is corrupted. You'll probably want to run with -X so that you are sure that there is only one process. Let me know if you can reproduce this. (Aaron says he can't.) -- justin
Re: release strategy stuff (was: cvs commit: httpd-2.0 STATUS)
On Thu, 30 May 2002, Greg Stein wrote: Personally, I'm finding that it seems we're getting back to the old, slow it takes a lot of pain to make a release process. Rather than a nice, easy, snap and release. Remember: even though we're GA, we can still snap new releases, test them, and *then* decide whether to release them (and whether to call them GA or intemediate beta releases). I like to think we're going with an in-between approach, which is that in practice it's hard for any one person to know exactly when the tree is stable on all platforms. So one person that volunteers to RM can tag the tree when he thinks it's in pretty good shape, but then reality sets in and other platforms need a chance to catch up. In order to manage the release process, giving the RM the power to incorporate patches for some period of time and then doing the snap, release rather than trying to pick one moment in time when everybody on every platform speaks up at once saying yes, it's working seems to be helpful. shrug. --Cliff
APR versioning (was: cvs commit: httpd-2.0/server/mpm/worker worker.c)
On Wed, May 29, 2002 at 08:40:55PM -0400, Cliff Woolley wrote: On 30 May 2002 [EMAIL PROTECTED] wrote: striker 02/05/29 17:21:27 Modified:server/mpm/beos beos.c server/mpm/experimental/leader leader.c server/mpm/experimental/threadpool threadpool.c server/mpm/netware mpm_netware.c server/mpm/prefork prefork.c server/mpm/worker worker.c Log: Catch up with the apr_allocator_set_owner - apr_allocator_owner_set renames in APR. This requires an MMN bump (which is fine with me, since we've already done one in 2.0.37-dev, and I'm starting to see little point in going back now). If there are other renames of this sort, we should get them in all at once. Is the APR versioning system in place yet? The header and the code is in there, but we also need to integrate it into the build process so that we generate versioned .so files (e.g. libapr.so.1) The mechanism has been added to apr, but not apr-util (yet). But the versioning doc states that we are not (strictly) bound to retain API compatibility as long as we have not reached version 1.0. Currently, APR is labelled as 0.9.0. So APR(UTIL) is somewhere on the line of free to change the API at will and please don't mess up httpd releases. If we made a formal 1.0 release of APR(UTIL), then we could start applying the versioning rules to it quite strictly. A separate decision is whether to carry the versioning scheme into httpd. At this point, I'd say no. Until we show that it is working well for APR and company, we can/should just let httpd stick to its current versioning model. As a true action item, I might suggest making an official 0.9 release of APR and APRUTIL, to get the release procedures set up and oiled. I'd suggest that we do the release with the caveat of API changes until 1.0 is released (just as a fallback; I doubt we'd really make any, but if we *do*, we don't have to worry about all the rules), then we apply the thumbscrews to its API. Note that I'd also suggest calling it 0.9 so we don't give false impressions of a real 1.0 release. We need more field testing before then (specifically with things like: where do the includes go? do we have vsn numbers on the .so files? etc) Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 11:17:23PM -, [EMAIL PROTECTED] wrote: jerenkrantz02/05/30 16:17:23 Modified:.STATUS Log: showstoppers++; (groan) ... RELEASE SHOWSTOPPERS: + +* 413 (invalid chunk size) followed by another request segfaults. + Message-ID: [EMAIL PROTECTED] + Status: Justin is completely confounded by this. It looks like a + bucket lifetime bug, but somehow an operation on one + brigade is altering another brigade and corrupting it. IMO, this isn't a showstopper. Any current client that happens to *send* chunked data is not going to be sending invalid chunk sizes. So we aren't really fixing a problem here, but a potential DOS attack. But when you stop and think about it: rather than crashing servers, a client could simply attach and wait on the socket. They can tie up *way* more processes that way (until the server times them out, but that is 15 seconds later; a *lot* longer than it would take to restart a crashed child) So... while it should be fixed, I wouldn't call it a showstopper. Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: don't try this at home
On Thu, May 30, 2002 at 11:02:09AM -0400, Jeff Trawick wrote:
Running HEAD, we get a segfault when the bogus request is for a
resource for which we aren't authorized. Here is the traceback:
In particular, this problem is related to the fact that we have
two errors on this request: 401 and 413. The problem is that
since 401 was reported first, the 413 is lost in ap_die()
when we find the recursive error (http_request.c:122) and we
reset r-status to 401.
However, 401 isn't listed as a status code that drops the
connection (line 146). Therefore, ap_die() calls
ap_discard_request_body again - which causes HTTP_IN to get
called again. But, since we are 413, we're not supposed to
re-read the body. Oops.
My suggestion is as follows: if the ap_die() code is one that
forces us to drop the connection, we don't report the recursive
error, but instead just report this one.
So the conditional on http_request.c:121 may work as:
if (r-status != HTTP_OK !ap_status_drops_connection(type)) {
Can you try this? -- justin
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 11:17:23PM -, Justin Erenkrantz wrote: +* 413 (invalid chunk size) followed by another request segfaults. + Message-ID: [EMAIL PROTECTED] + Status: Justin is completely confounded by this. It looks like a + bucket lifetime bug, but somehow an operation on one + brigade is altering another brigade and corrupting it. Actually, there seem to be two different errors. The one Justin is seeing has to do with two sequential requests. The one Jeff (and I) are seeing has to do with a single request for an entity that has access requirements. I'm seeing a SEGV (or the child is puking and dying) here, but I haven't tried hard enough to reproduce Justin's problem. -aaron
Re: cvs commit: httpd-2.0/modules/http http_protocol.c
On Thu, May 30, 2002 at 05:29:26PM -0700, Justin Erenkrantz wrote: Per my earlier message, I believe this is the wrong way to do this. If we get a neg chunk length, we shouldn't *ever* get back into ap_http_filter (HTTP_IN). The whole point of the 413 error is that we refuse to read the request. -- justin Yeah, I agree that we need to fix how we deal with fatal errors like this, but we still shouldn't leave bogus values lying around in things like r-remaining. I'm only trying to fix the assert, and I agree that this doesn't fix the problem you're describing. I'm testing the conditional you posted right now, but if it works for you then commit it. :) -aaron
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 11:17:23PM -, [EMAIL PROTECTED] wrote: jerenkrantz02/05/30 16:17:23 Modified:.STATUS Log: showstoppers++; (groan) ... RELEASE SHOWSTOPPERS: + +* 413 (invalid chunk size) followed by another request segfaults. + Message-ID: [EMAIL PROTECTED] + Status: Justin is completely confounded by this. It looks like a + bucket lifetime bug, but somehow an operation on one + brigade is altering another brigade and corrupting it. IMO, this isn't a showstopper. Any current client that happens to *send* chunked data is not going to be sending invalid chunk sizes. So we aren't really fixing a problem here, but a potential DOS attack. But when you stop and think about it: rather than crashing servers, a client could simply attach and wait on the socket. They can tie up *way* more processes that way (until the server times them out, but that is 15 seconds later; a *lot* longer than it would take to restart a crashed child) And all that child's threads? If we are voting, I vote this is a showstopper. A segfaulting process can leave an awful lot of cruft laying around. Bill
Re: Need a new feature: Listing of CGI-enabled directories.
Hi Ronald, Let me preface this by noting that I agree with your goals, however I believe that you may not have considered the nature of the problem in sufficient depth. On Thu, May 30, 2002 at 07:45:42PM -0700, Ronald F. Guilmette wrote: Most sites and web hosting companies _do_ want to eliminate these exploit- able FormMail scripts from their web servers, but when you are a big web hosting company with thousands of virtually-hosted web sites, the task of just finding the bad scripts in your filesystem isn't terribly easy. In fact in can get downright convoluted. The first step in finding all such scripts however may often be the most difficult one. That first step consists of simply gathering into one big list a list of all of the CGI-enabled directories on the local web server. Once such a list has been compiled, other standard UNIX tools such as `find' and `file' and `grep' and be set to work, plowing through all of the files in those (CGI) directories and finding all of the bad FormMail scripts. How are you defining bad FormMail scripts here? In the simplest case, you could just run 'find' across the filesystem containing the web content looking for formmail.cgi or formmail.pl and checking those found against a list of known good/bad versions. This doesn't require any specific knowledge of the Apache configuration in use, and is an entirely viable approach even on filesystems of several hundred gigabytes. A more thorough check would involve testing all executable ascii files, perhaps excluding .html/.php and variants thereof. But seriously, is there already a way to do what I need to do with the Apache server? Looking at the command line options on the man page for httpd, there doesn't seem to be an option to request httpd to just parse the config file, list the CGI directories, and then exit. But that's exactly what I need. It isn't possible in the generic case. Apache allows many aspects of the configuration to be modified by the inclusion of .htaccess files beneath the DocumentRoot of the web server. Unless Apache has been configured not to allow the ExecCGI option to be overridden, you would need to parse both httpd.conf AND every .htaccess file on the filesystem. Apache itself does not do this at startup - it is done only as required. P.S. Note that FormMail is far from the only dangerous CGI script that a big web hosting company might find that some of its less-clued end lusers may have uploaded and installed into CGI-enabled directories. And the overall comprehensive search process that I'm trying to develop could be (and arguably should be) applied to searching for other dangerous CGI scripts too. You also can't assume that only files in designated CGI directories are going to be problematic. There's a long history of people using all sorts of redirection and other techniques to access/execute things that they shouldn't be able to. This does, admittedly, involve far more effort (and good luck) than the straightforward formmail abuse you're concerned about. It should be apparent at this point that what you're looking at here is a reduction of the abuse of formmail etc, as it is almost impossible to stamp out entirely. In this particular example, the best solution is for the web host to replace the sendmail binary with something which performs more rigorous checking of its input and, ideally, tags outbound messages with identifying information. This is the solution implemented at a former employer - the sendmail replacement checks that the From: address is valid and sane and writes a Received: header which includes information which can be used to identify the VirtualHost responsible for generating the message. This also removes the ability for poorly-written PHP scripts and the like to be abused in a similar manner. Although the script was developed in-house (and, unfortunately, cannot be released) I believe there are open-source alternatives available to accomplish the same ends. I hope these thoughts provide you with a different perspective to consider, and I wish you well in your efforts to reduce the amount of spam on the Internet. Cheers, Zac
Re: Need a new feature: Listing of CGI-enabled directories.
At 09:45 PM 5/30/2002, you wrote: I am directing this message to the developer's list because I strongly suspect that it may require some new development. ... The first step in finding all such scripts however may often be the most difficult one. That first step consists of simply gathering into one big list a list of all of the CGI-enabled directories on the local web server. Once such a list has been compiled, other standard UNIX tools such as `find' and `file' and `grep' and be set to work, plowing through all of the files in those (CGI) directories and finding all of the bad FormMail scripts. I've been thinking along this line for a different reason [not just limited to the cgi enabled bit.] However, I have no time to implement at the moment, but I'll share my thoughts. Many, many users have trouble figuring out 1. why their scripts don't run 2. how the +/- options syntax was really parsed 3. how a given request will be handled What about an htls command [httpd ls file listing utility] that would actually use the core Apache code to run directories and files through the Location, Directory and Files tests? It doesn't have to be blindingly fast [it isn't the daemon itself, it's an administrator's tool] and it can be rather simple. One complexity is the difference introduced by Location and VirtualHost directives. My original though was to list the files, but those listings are only complete in the context of a host. If we take the logic in mod_autoindex one step further, and set up whatever hooks through mod_alias to 'query' the possible matches under a given pattern, we should be able to make this reasonably complete. Perhaps the same sort of hook would work for mod_rewrite, although the logic to work those backwards is probably as complex as the rewrite code itself. Recursion is one complication, depending on how the patterns interact. Anyway, I'm not suggesting that htls would ever be a web app!!! This would be a local tool, similar to mod_info, only providing a list of resources and the flags that affect them. One might invoke the utility with; htls -r www.apache.org/dist/ and using whatever abbrieviations, you might end up with something like IN(D)EXES (I)NCLUDES INC(N)OEXEC (S)YM_LINKS OPT_SYM_(O)WNER (E)XECCGI (M)ULTI 128 D--SO-M www.apache.org/dist/project1/ D--SO-M www.apache.org/dist/project1/file.html DEM www.apache.org/dist/project1/search.pl D--SO-M www.apache.org/dist/project2/ ... You could further extend the concept to include the optional listing of the handler toggled by such a request, etc. BTW - you might also want to consider simply reviewing your log files. It would be really trivial if you could add the handler invoked (or a flag to indicate that a given filter were invoked) to the logs, so you could look specifically for requests that trigger mod_cgi. It might be a straighter line to identifying the risks you are concerned with.
Re: cvs commit: httpd-2.0 STATUS
On Thu, May 30, 2002 at 09:49:26PM -0400, Bill Stoddard wrote: And all that child's threads? If we are voting, I vote this is a showstopper. A segfaulting process can leave an awful lot of cruft laying around. I would tend to agree, but as far as I know the segv has been fixed (and the other was found to only occur when mod_bucketter has been inserted). -aaron
