Re: Sending multiple responses
Thanks Bill for the reply. My 2 cents (I am filter novice :) ) >We need to create a new metadata bucket to pass on the Continue, or >Processing, or other 1xx-style response codes. It needs to be processed >by the HTTP filter so that these responses aren't inserted into 1.0 streams, >or into the middle of a chunked response, etc. I observed one more thing : Filters such as http_header_filter removes itself from filter chain by calling "ap_remove_output_filter". So this filter should take care that if I am sending 1xx kind of responses it should not remove it from chain, else in subsequent responses there are no headers. > >It's a very interesting request, thank you for posting it to the list, Amit! Actually it will become necessary when mod_dav will be pushed more into commercial arena. COPYing large collection etc are normal operations and often clients sets time-out of 120sec or so. Its necessary from UI perspective (could show progress bar etc.). I hope there would be more cleaner way of doing it somewhere in future. :) Thanks a lot Amit FREE ADHD DVD or CD-Rom (your choice) - click here! http://ad.doubleclick.net/clk;6413623;3807821;f?http://mocda2.com/1/c/563632/131726/311392/311392 AOL users go here: http://ad.doubleclick.net/clk;6413623;3807821;f?http://mocda2.com/1/c/563632/131726/311392/311392 This offer applies to U.S. Residents Only
Re: DDOS protection features request
Sergey Ignatchenko wrote: There exists (and I know of several instances when it was successfully used) a trivial kind of DDOS attack that seems to be quite poorly handled by Apache: if multiple clients just create TCP connections to port 80 and do not send anything over those connections, it will take TimeOut (300 sec by default) to drop such a connection. So, it becomes trivial to fill all 256 slots (default MaxClients for prefork MPM) and even thousands of slots for worker MPM, therefore effectively preventing legitimate users from requesting data. Two features IMO would help (sorry, if they were already recently added - pls refer me to appropriate directives then): - split TimeOut directive into several (which is already planned according to Apache docs: http://httpd.apache.org/docs-2.0/mod/core.html#timeout). To deal with such an attack, it is necessary to reduce "The total amount of time it takes to receive a GET request." to several seconds, but unfortunately reducing current TimeOut value to several seconds will have all kinds of ill effects, including effective inability to serve relatively big files (1M+). This has been discussed before on the list. The conclusion of the discussion has always been something like 'there are other very effective ways to DoS the server that cannot be defended against at the HTTP protocol level so implementing DoS protections in the web server is pointless'. Or something like that. However... I an in favor of splitting the timeout directive into two separate directives. This seems relatively easy to do and it will protect against a trivial attack. - implementing restriction on number of connections from single source IP (something similar to mod_conn that existed for Apache 1.3). Note: I have no idea how mod_conn was implemented, but to deal with the attack, IP session limit check must be performed _before_ GET request completed. This otoh, would play havoc with folks accessing websites via a proxy (think employees of a large company accessing internet sites. They all might look like they are coming from the same IP addr.) I would not object to someone starting an httpd subproject for a general purpose module to implement some of the defences that can be implemented at the HTTP level. Bill
DDOS protection features request
There exists (and I know of several instances when it was successfully used) a trivial kind of DDOS attack that seems to be quite poorly handled by Apache: if multiple clients just create TCP connections to port 80 and do not send anything over those connections, it will take TimeOut (300 sec by default) to drop such a connection. So, it becomes trivial to fill all 256 slots (default MaxClients for prefork MPM) and even thousands of slots for worker MPM, therefore effectively preventing legitimate users from requesting data. Two features IMO would help (sorry, if they were already recently added - pls refer me to appropriate directives then): - split TimeOut directive into several (which is already planned according to Apache docs: http://httpd.apache.org/docs-2.0/mod/core.html#timeout). To deal with such an attack, it is necessary to reduce "The total amount of time it takes to receive a GET request." to several seconds, but unfortunately reducing current TimeOut value to several seconds will have all kinds of ill effects, including effective inability to serve relatively big files (1M+). - implementing restriction on number of connections from single source IP (something similar to mod_conn that existed for Apache 1.3). Note: I have no idea how mod_conn was implemented, but to deal with the attack, IP session limit check must be performed _before_ GET request completed. Thx in advance __ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree
Re: Sending multiple responses
12:44 AM 11/6/2003, amit athavale wrote: >Hi, > >Requirement : >--- > >I want to change the mod_dav code so that it sends "102 Processing" responses after >copying say 5 resources. This is necessary when COPYing or DELETEing large >collections.(say depth > 20 and total resources = 1. In such cases webdav clients >times-out but server continues to process . This might be confusing for the user. > >Issue : >- >How can I use apache APIs/filter APIs so that from "handler" it'll continue to send >"102 Processing" responses and after everything is deleted, it will send "204" as >usual and come out of handler. > >Looking at the code I can think of the ugly way of doing it by writing response using >ap_r* and simulate the behavior of finalize_request_protocol and >check_pipeline_flush. (and then reset r->eos_sent to 0) > >I know this is not the clean way of doing it but couldnt come up with it. The clean way, I just realized, doesn't exist today. The logical way at the moment is to take your filter* chain, and walk it until you get past the content. But you also need to walk past the http filter so that it isn't considered part of the 'body'. No further, because if there was a transfer encoding schema going on, you would need this compressed, and you want it ssl crypted if mod_ssl is in the loop. Never mind that additional http protocol data may be required, and that your answer is *wrong* if someone has plugged in an alternate transport in place of HTTP/1.1. (It would even be wrong for HTTP/1.0 responses.) So I think this is only 1/2 an answer. You are sending a response, so it should be sent down the filter chain. But it isn't body, it's metadata. Filter gurus know where I'm going with this ... We need to create a new metadata bucket to pass on the Continue, or Processing, or other 1xx-style response codes. It needs to be processed by the HTTP filter so that these responses aren't inserted into 1.0 streams, or into the middle of a chunked response, etc. An interesting side effect - should this be passed to the head of the filter stack (in which case *every* filter needs to pass it immediately even if that filter hasn't composed it's next content bucket), or should we have some accessor ap_pass_protocol_brigade that skips the content/body filters? It's a very interesting request, thank you for posting it to the list, Amit! Bill
Re: mod_ldap SEGV while caching on FreeBSD 4.8-STABLE
Matthieu Estrade wrote:
Hi Albert,
Could you try this little patch posted on bugzilla:
http://nagoya.apache.org/bugzilla/showattachment.cgi?attach_id=8185
Matthieu, can you take a look at the attached patch?
I replaced a call to unlink() with apr_file_remove(). I also axed a bunch of
tab chars and some other style issues in your patch ;) There were such issues
in the ldap code before and there still are, but we might as well move it in
the right direction on all fronts.
Thanks,
Jeff
Index: include/util_ldap.h
===
RCS file: /home/cvs/httpd-2.0/include/util_ldap.h,v
retrieving revision 1.11
diff -u -r1.11 util_ldap.h
--- include/util_ldap.h 14 Feb 2003 16:04:00 - 1.11
+++ include/util_ldap.h 6 Nov 2003 17:30:18 -
@@ -75,6 +75,10 @@
#include "http_protocol.h"
#include "http_request.h"
+#if APR_HAS_SHARED_MEMORY
+#include "apr_rmm.h"
+#include "apr_shm.h"
+#endif
/* Create a set of LDAP_DECLARE(type), LDLDAP_DECLARE(type) and
* LDAP_DECLARE_DATA with appropriate export and import tags for the platform
@@ -97,7 +101,6 @@
#define LDAP_DECLARE_DATA __declspec(dllimport)
#endif
-
/*
* LDAP Connections
*/
@@ -138,9 +141,11 @@
apr_pool_t *pool; /* pool from which this state is allocated */
#if APR_HAS_THREADS
apr_thread_mutex_t *mutex; /* mutex lock for the connection list */
+apr_thread_rwlock_t *util_ldap_cache_lock;
#endif
apr_size_t cache_bytes; /* Size (in bytes) of shared memory cache */
+char *cache_file; /* filename for shm */
long search_cache_ttl; /* TTL for search cache */
long search_cache_size; /* Size (in entries) of search cache */
long compare_cache_ttl; /* TTL for compare cache */
@@ -150,6 +155,15 @@
char *cert_auth_file;
int cert_file_type;
int ssl_support;
+
+#if APR_HAS_SHARED_MEMORY
+apr_shm_t *cache_shm;
+apr_rmm_t *cache_rmm;
+#endif
+
+/* cache ald */
+void *util_ldap_cache;
+
} util_ldap_state_t;
@@ -286,21 +300,21 @@
* @param reqsize The size of the shared memory segement to request. A size
*of zero requests the max size possible from
*apr_shmem_init()
- * @deffunc void util_ldap_cache_init(apr_pool_t *p)
+ * @deffunc void util_ldap_cache_init(apr_pool_t *p, util_ldap_state_t *st)
* @return The status code returned is the status code of the
* apr_smmem_init() call. Regardless of the status, the cache
* will be set up at least for in-process or in-thread operation.
*/
-apr_status_t util_ldap_cache_init(apr_pool_t *pool, apr_size_t reqsize);
+apr_status_t util_ldap_cache_init(apr_pool_t *pool, util_ldap_state_t *st);
/**
* Display formatted stats for cache
* @param The pool to allocate the returned string from
* @tip This function returns a string allocated from the provided pool that describes
* various stats about the cache.
- * @deffunc char *util_ald_cache_display(apr_pool_t *pool)
+ * @deffunc char *util_ald_cache_display(apr_pool_t *pool, util_ldap_state_t *st)
*/
-char *util_ald_cache_display(apr_pool_t *pool);
+char *util_ald_cache_display(apr_pool_t *pool, util_ldap_state_t *st);
/* from apr_ldap_cache_mgr.c */
@@ -310,9 +324,9 @@
* @param The pool to allocate the returned string from
* @tip This function returns a string allocated from the provided pool that describes
* various stats about the cache.
- * @deffunc char *util_ald_cache_display(apr_pool_t *pool)
+ * @deffunc char *util_ald_cache_display(apr_pool_t *pool, util_ldap_state_t *st)
*/
-char *util_ald_cache_display(apr_pool_t *pool);
+char *util_ald_cache_display(apr_pool_t *pool, util_ldap_state_t *st);
#endif /* APU_HAS_LDAP */
#endif /* UTIL_LDAP_H */
Index: modules/experimental/util_ldap.c
===
RCS file: /home/cvs/httpd-2.0/modules/experimental/util_ldap.c,v
retrieving revision 1.14
diff -u -r1.14 util_ldap.c
--- modules/experimental/util_ldap.c4 Apr 2003 13:47:13 - 1.14
+++ modules/experimental/util_ldap.c6 Nov 2003 17:30:18 -
@@ -141,6 +141,7 @@
*/
int util_ldap_handler(request_rec *r)
{
+util_ldap_state_t *st = (util_ldap_state_t
*)ap_get_module_config(r->server->module_config, &ldap_module);
r->allowed |= (1 << M_GET);
if (r->method_number != M_GET)
@@ -171,7 +172,7 @@
"\n", r
);
-ap_rputs(util_ald_cache_display(r->pool), r);
+ap_rputs(util_ald_cache_display(r->pool, st), r);
ap_rputs("\n\n", r);
@@ -506,9 +507,7 @@
LDAPMessage *res, *entry;
char *searchdn;
-util_ldap_state_t *st =
-(util_ldap_state_t *)ap_get_module_config(r->server->module_config,
-&ldap_module);
+util_ldap_state_t *st = (util_ldap_state_t
*)ap_get_module_config(r->server->module_config, &ldap_module);
/* read
Best place to log error 500 errors
Hello, what would be the "best" way to log all error 500 (all status 50x responses in fact) into a separate logfile ? One way could be a piped log, but depending of the format the user has configured the output can be very different. I think of something like a (transparent) filter in the output chain. Thanks. André
Re: OT: Freebsd 3.4 binaries on 4.8/4.9
On Thu, Nov 06, 2003 at 11:26:55AM -0500, Jim Jagielski wrote: > Because I want to provide some FreeBSD binaries of the > latest release, and I'm lazy, I need to ask: Will binaries > compiled under FreeBSD 3.4 work under 4.8/4.9 ? The binaries will run, but if they use any kernel structures which have changed, it's a no-hoper. If memory serves me correctly, at least the pwd/utmp/wtmp structures changed and some of the process info ones. So that could affect mod_userdir, suexec and so on, and probably more. Then again it's so long ago that I was sure of any of this, it could be complete nonsense :) -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
OT: Freebsd 3.4 binaries on 4.8/4.9
Because I want to provide some FreeBSD binaries of the latest release, and I'm lazy, I need to ask: Will binaries compiled under FreeBSD 3.4 work under 4.8/4.9 ?
Re: Structure of apache source
--- Pedro Picapiedra <[EMAIL PROTECTED]> wrote: - I want to know if there's a document about the structure of Apache source, i must change something, and i don't know how start with it. This is the file I found on it: http://www.math.uwaterloo.ca/~oadragoi/CS746G/a2/caa.html __ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree
Re: piped log files
On Thu, Nov 06, 2003 at 11:02:16AM -0500, Jeff Trawick wrote: > Jeff Trawick wrote: > >Bastiaan van der Put wrote: > > > >>CustomLog "|/usr/local/apache2/bin/logresolve >> > >>/home/accounts/x/x/logs/access_log" combined > > > > > >unless somebody speaks up soon, I'll commit the patch to Apache 2.1-dev > > status: I hit a bit of a hangup that must be debugged :( > > there are two pieces of code handling piped loggers: error log and > mod_log_config... in 1.3, both handled shell operations... in 2.0, as you > discovered, neither handles shell operations > > the patch you are using is fine AFAICT, but trying the same change to the > code to handle piped error log results in quite a bit of unhappiness (no > logging at all)... that needs to be tracked down before trying to fix half > the problem Maybe out in left field, but could ErrorLog be moved from the core and into mod_log_config? Then, if mod_log_config is not present, or no ErrorLog directive is specified, then stderr can be used. Any special reason to keep it in the core? Thanks. Glenn
Re: piped log files
Jeff Trawick wrote: Bastiaan van der Put wrote: CustomLog "|/usr/local/apache2/bin/logresolve >> /home/accounts/x/x/logs/access_log" combined unless somebody speaks up soon, I'll commit the patch to Apache 2.1-dev status: I hit a bit of a hangup that must be debugged :( there are two pieces of code handling piped loggers: error log and mod_log_config... in 1.3, both handled shell operations... in 2.0, as you discovered, neither handles shell operations the patch you are using is fine AFAICT, but trying the same change to the code to handle piped error log results in quite a bit of unhappiness (no logging at all)... that needs to be tracked down before trying to fix half the problem
Structure of apache source
I want to know if there's a document about the structure of Apache source, i must change something, and i don't know how start with it. ThanksUna mejor experiencia en Internet. Prueba gratis dos meses MSN 8.
Re: Problem with ordering of modules in IHS 1x
Sander Striker wrote: From: Swapan Gupta [mailto:[EMAIL PROTECTED] Sent: Thursday, November 06, 2003 2:50 PM Then this is not an Apache issue and this question should be asked the IHS1x support team. Yes, I should have said that yet again instead of asking further questions. --/-- If you have an issue with Apache httpd, then Apache httpd mailing lists and bug dbs are appropriate venues. If you have an issue with IHS, use the IBM support line. If you have an issue with IHS *and* you can recreate the issue with pure Apache httpd, then feel free to use Apache httpd resources in addition to the IBM support line, but don't mention IHS.
module load order
Hi, I was just asked again by a user about the load order of modules. While it seems that it is now with Apache2 less important, and also ClearModuleList and AddModule are removed now with Apache2, I think it is at least still important for auth modules. So I searched the manual for any hints, specially about the fact that the last module loaded is the first module in the auth chain - but I couldnt find anything about that, or I'm too blind to find... also in the 1.3 manual I found nothing, even not where ClearModuleList and AddModule are documented... I think that's an important piece of information the user should be able to read somewhere in the docs... comments? Guenter.
Re: sctp related compile errors (resolved, but see last paragraph)
On Sat, Nov 01, 2003 at 10:29:58PM -0500, James H.Cloos Jr. wrote: > I figured this out shortly after posting ... :( > > What is happening is that something in the latest glibc or kernel > headers (not linked into /usr/include, but glibc was built against > the 2.6 kernel and probably grabbed some headers then) convinces apr > that sctp is supported, even when the lksctp library is not installed. > > So I grabbed the latest release from lksctp.sf.net and installed that; > then apache compiled w/o further complaint. > > So apr's configure needs to do a better job of detecting sctp support > and turn it off if the necessary defines are not available. Can you file a bug report on this, and attach the configure and make output for the failure case? http://nagoya.apache.org/bugzilla/enter_bug.cgi?product=APR&component=APR joe
RE: Problem with ordering of modules in IHS 1x
> From: Swapan Gupta [mailto:[EMAIL PROTECTED] > Sent: Thursday, November 06, 2003 2:50 PM > Yes, my config file does have ClearModule List and AddModule directives. > Moreover, the ordering of LoadModule directives and AddModule directives is in the > same as I want it to be, ie., entries > for my module are in the end in the respective sets. > But still it is the same behaviour. > > Basically my module intends to use some HTTP request headers that are being set by > the other module, but the functions > which are doing this job in both the modules are in the same phase (check access). > > In the case of Apache1x, my function gets the HTTP headers which are set by the > function of the other module, but in the > case of IHS1x it is not able to get that HTTP header. Then this is not an Apache issue and this question should be asked the IHS1x support team. Sander
RE: Problem with ordering of modules in IHS 1x
Yes, my config file does have ClearModule List and AddModule directives. Moreover, the ordering of LoadModule directives and AddModule directives is in the same as I want it to be, ie., entries for my module are in the end in the respective sets. But still it is the same behaviour. Basically my module intends to use some HTTP request headers that are being set by the other module, but the functions which are doing this job in both the modules are in the same phase (check access). In the case of Apache1x, my function gets the HTTP headers which are set by the function of the other module, but in the case of IHS1x it is not able to get that HTTP header. Thanks, Swapan. -Original Message- From: Jeff Trawick [mailto:[EMAIL PROTECTED] Sent: Thursday, November 06, 2003 7:00 PM To: [EMAIL PROTECTED] Subject: Re: Problem with ordering of modules in IHS 1x Swapan Gupta wrote: > I am having a problem with the ordering of modules with IHS, based upon > Apache1x. > > I have two modules which have their functions scheduled to be executed > during the same phase (Check Access). > > The functions get executed in a reverse order than the one I intend to. > The same is true even if I swap the positions of loading the modules in > the httpd.conf file. Do you have just LoadModule for the modules, or LoadModule and ClearModuleList and AddModule? If your config file has ClearModuleList then the order of the LoadModule directives doesn't matter and instead the order of the AddModule matters. > Is there a way by which I could explicitly specify the order of > execution of the functions in the same phase vis-à-vis other modules > having functions defined for the execution in the same phase? Basically > I want to execute my function at the end of a particular phase (check > access), so that even if there are other modules defining function for > check access phase, my function gets executed at last. Is there a way > out to do this? no, in Apache 1.3 the order of LoadModule or AddModule determines the order in which the hooks will be called
Re: Problem with ordering of modules in IHS 1x
Swapan Gupta wrote: I am having a problem with the ordering of modules with IHS, based upon Apache1x. I have two modules which have their functions scheduled to be executed during the same phase (Check Access). The functions get executed in a reverse order than the one I intend to. The same is true even if I swap the positions of loading the modules in the httpd.conf file. Do you have just LoadModule for the modules, or LoadModule and ClearModuleList and AddModule? If your config file has ClearModuleList then the order of the LoadModule directives doesn't matter and instead the order of the AddModule matters. Is there a way by which I could explicitly specify the order of execution of the functions in the same phase vis-à-vis other modules having functions defined for the execution in the same phase? Basically I want to execute my function at the end of a particular phase (check access), so that even if there are other modules defining function for check access phase, my function gets executed at last. Is there a way out to do this? no, in Apache 1.3 the order of LoadModule or AddModule determines the order in which the hooks will be called
Problem with ordering of modules in IHS 1x
Hi, I am having a problem with the ordering of modules with IHS, based upon Apache1x. I have two modules which have their functions scheduled to be executed during the same phase (Check Access). The functions get executed in a reverse order than the one I intend to. The same is true even if I swap the positions of loading the modules in the httpd.conf file. Is there a way by which I could explicitly specify the order of execution of the functions in the same phase vis-à-vis other modules having functions defined for the execution in the same phase? Basically I want to execute my function at the end of a particular phase (check access), so that even if there are other modules defining function for check access phase, my function gets executed at last. Is there a way out to do this? I know there is a way to do that with Apache2x/IHS 2x, but my problem is related to IHS 1x/Apache 1x. Thanks, Swapan.
Re: should input filter return the exact amount of bytes asked for?
Stas Bekman wrote: I'm trying to get rid of ap_get_client_block(), but I don't understand a few things. ap_get_client_block() asks for readbytes from the upstream filter. What happens if the filter returns less bytes (while there is still more data coming?) What happens if the filter returns more bytes than requested (e.g. because it uncompressed some data). After all the incoming filters all propogate a request for N bytes read to the core_in filter, which returns that exact number if it can. Now as the data flows up the filter chain its length may change. Does it mean that if the filter didn't return the exact amount asked for it's broken? Is that the case when it returns less data than requested? Or when it returns more data? I'm trying to deal with the case where a user call wants N bytes and I've to give that exact number in a single call. I'm not sure whether I should buffer things if I've got too much data or on the opposite ask for more bbs if I don't have enough data. Are there any modules I can look at to learn from? The doc for ap_get_brigade doesn't say anything about ap_get_brigade satisfying 'readbytes' argument. /** * Get the current bucket brigade from the next filter on the filter * stack. The filter returns an apr_status_t value. If the bottom-most * filter doesn't read from the network, then ::AP_NOBODY_READ is returned. * The bucket brigade will be empty when there is nothing left to get. * @param filter The next filter in the chain * @param bucket The current bucket brigade. The original brigade passed * to ap_get_brigade() must be empty. * @param mode The way in which the data should be read * @param block How the operations should be performed * ::APR_BLOCK_READ, ::APR_NONBLOCK_READ * @param readbytes How many bytes to read from the next filter. */ AP_DECLARE(apr_status_t) ap_get_brigade(ap_filter_t *filter, apr_bucket_brigade *bucket, ap_input_mode_t mode, apr_read_type_e block, apr_off_t readbytes); What bothers me most is the case where a filter may return more data than it has been asked for in the AP_MODE_READBYTES mode. ap_get_client_block() doesn't deal with buffering such data and drops it on the floor. So it either has to be fixed to do the buffering or the filter spec (ap_get_brigade) needs to clearly state that no more than requested amount of data should be returned in the AP_MODE_READBYTES. And ap_get_client_block needs to assert if it gets more. __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Problem with ordering of modules in IHS 1x
Hi, I am having a problem with the ordering of modules with IHS, based upon Apache1x. I have two modules which have their functions scheduled to be executed during the same phase (Check Access). The functions get executed in a reverse order than the one I intend to. The same is true even if I swap the positions of loading the modules in the httpd.conf file. Is there a way by which I could explicitly specify the order of execution of the functions in the same phase vis-à-vis other modules having functions defined for the execution in the same phase? Basically I want to execute my function at the end of a particular phase (check access), so that even if there are other modules defining function for check access phase, my function gets executed at last. Is there a way out to do this? I know there is a way to do that with Apache2x/IHS 2x, but my problem is related to IHS 1x/Apache 1x. Thanks, Swapan.
