Re: default conf and ScriptAlias
… both +1 and -1. A change in version number or major version can imply significant changes in the base configuration, and I see this suggestion as a fit for a httpd-2.5, -3.0 or the likes. Hence, +1. However changing such widely used setting on the existing 10 year old 2.4 tree will cause operators headaches as the one outlined by Noel - more so as this setting is there for way longer than 2.4 and therefore -1. Alex > On Oct 9, 2021, at 20:30, Noel Butler wrote: > > >> >> On 10/10/2021 03:39, Eric Covener wrote: >> >> Relative to the recent CVEs, should we replace ScriptAlias in the >> default conf with Alias + SetHandler cgi-script in the corresponding >> Directory section? >> >> And .. should ScriptAlias be deprecated/discouraged in some way if the >> expanded version is safer by avoiding the equivalent of setting the >> handler in Location vs. Directory? >> >> I am assuming it is not possible/feasible to make ScriptAlias just >> work as if it was in the 2nd arguments Directory config. > > -1 > > > > You are talking about changing a httpd life long option, thats used in > millions of settings around the world. > > Scriptalias setting is not used in any directory setting in my case, its used > in a global way > > DocumentRoot "/var/www/html" > > > AllowOverride None > Options SymlinksIfOwnerMatch > Require all granted > > > Alias /icons/ "/var/www/icons/" > > > AllowOverride None > Require all granted > > > ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" > > > AllowOverride None > Options None > Require all granted > > > > > and more globally used in every service provider i've been at (not all my > doing but end result is identical) inside virtual hosts confs > > > ServerName xxx > ServerAlias www. > DocumentRoot /var/www/vhost/xxx/www/html > ScriptAlias /cgi-bin/ /var/www/vhost/x/www/cgi-bin/ > > ...snip... > > > > This is how every person expects it. > > So you want to go make that more convoluted? > > > > -- > Regards, > Noel Butler > > This Email, including attachments, may contain legally privileged > information, therefore at all times remains confidential and subject to > copyright protected under international law. You may not disseminate this > message without the authors express written authority to do so. If you are > not the intended recipient, please notify the sender then delete all copies > of this message including attachments immediately. Confidentiality, > copyright, and legal privilege are not waived or lost by reason of the > mistaken delivery of this message. > >
Re: [VOTE] Release httpd-2.4.51-rc1 as httpd-2.4.51
+1 on Slackware64 -current Alex > On Oct 7, 2021, at 09:17, ste...@eissing.org wrote: > > Hi all, > > due to found security weaknesses in our 2.4.50 release, the security team > feels it is necessary to do a new release on very short notice. We will skip > the usual 3 day voting period and close the vote once we feel comfortable > with our testing. > > Please find below the proposed release tarball and signatures: > > https://dist.apache.org/repos/dist/dev/httpd/ > > I would like to call a VOTE over the next few days^h^h^h^hhours to release > this candidate tarball httpd-2.4.51-rc1 as 2.4.51: > [ ] +1: It's not just good, it's hopefully good enough! > [ ] +0: Let's have a talk. > [ ] -1: There's trouble in paradise. Here's what's wrong. > > The computed digests of the tarball up for vote are: > sha1: 516128e5acb7311e6e4d32d600664deb0d12e61f *httpd-2.4.51-rc1.tar.gz > sha256: c2cedb0b47666bea633b44d5b3a2ebf3c466e0506955fbc3012a5a9b078ca8b4 > *httpd-2.4.51-rc1.tar.gz > sha512: > 507fd2bbc420e8a1f0a90737d253f1aa31000a948f7a840fdd4797a78f7a4f1bd39250b33087485213a3bed4d11221e98eabfaf4ff17c7d0380236f8a52ee157 > *httpd-2.4.51-rc1.tar.gz > > The SVN candidate source is found at tags/candidate-2.4.51-rc1. > > Kind Regards, > Stefan
Re: [VOTE] Release httpd-2.4.46
I don’t see why a verbiage similar to “Fixed in Apache httpd-2.4.44 (not released to the public)” couldn’t be used: this is, after all, a true statement. While it should be common understanding that newer code versions carry improvements and fixes from previous ones, maybe this should be clarified on the initial paragraphs of the vulnerabilities page. Last but not least, this also resolves thoughts of “where is 2.4.44, I cannot find it” (although only if one browses to the vulnerabilities page). What I am not sure, however, is how much this affects the existing automation workflow. Alex > On Aug 8, 2020, at 08:27, Daniel Ruggeri wrote: > > Hi, Bill; > I wondered about this myself. I agree that we allow for ambiguity > when we say an issue is fixed in 2.4.44 and 2.4.45 (which weren't > released). Perhaps we should just bump the 'fixed' version up to the > released version... but then we should also add to the 'affected' > versions the version numbers we burned during QA. That's odd, too, > because we didn't release those versions so they aren't really 'affected'. > > I could go either way... the vulnerability reporting is enough "after > work" for a release that makes it a prime candidate for processing it > with announce.sh, so I'm happy to encode whatever we consider the best > way forward into that script. > > -- > Daniel Ruggeri > >> On 8/7/2020 8:56 AM, William A Rowe Jr wrote: >> Following the announcement link, it isn't clear that >> https://httpd.apache.org/security/vulnerabilities_24.html >> fixes issues in 2.4.46. >> Should the fixed-in be promoted to the revision of Apache HTTP Server >> actually published (released) by the project? It almost reads like >> "fixed in >> 2.4.46-dev" (which 0-day disclosures are described as, until a release >> is actually published.) >> On Wed, Aug 5, 2020 at 6:32 AM Daniel Ruggeri > <mailto:dan...@bitnebula.com>> wrote: >> Hi, all; >> With 12 binding PMC +1 votes, two additional +1 votes from the >> community, and no -1 votes, I'm pleased to report that the vote has >> PASSED to release 2.4.46. I will begin the process of pushing to the >> distribution mirrors which should enable us for a Friday >> announcement - >> a great way to wrap up the week! >> Here are the votes I recorded during the thread: >> PMC >> jailletc36, steffenal, elukey, jorton, jfclere, ylavic, covener, >> gbechis, gsmith, druggeri, jblond, rjung >> Community >> Noel Butler, wrowe >> -- >> Daniel Ruggeri >>> On 8/1/2020 9:13 AM, Daniel Ruggeri wrote: >>> Hi, all; >>> Third time is a charm! Please find below the proposed release >> tarball >>> and signatures: >>> https://dist.apache.org/repos/dist/dev/httpd/ >>> I would like to call a VOTE over the next few days to release this >>> candidate tarball as 2.4.46: >>> [ ] +1: It's not just good, it's good enough! >>> [ ] +0: Let's have a talk. >>> [ ] -1: There's trouble in paradise. Here's what's wrong. >>> The computed digests of the tarball up for vote are: >>> sha1: 15adb7eb3dc97e89c8a4237901a9d6887056ab98 *httpd-2.4.46.tar.gz >>> sha256: >> 44b759ce932dc090c0e75c0210b4485ebf6983466fb8ca1b446c8168e1a1aec2 >>> *httpd-2.4.46.tar.gz >>> sha512: >> >> 5801c1dd0365f706a5e2365e58599b5adac674f3c66b0f39249909841e6cdf16bfdfe001fbd668f323bf7b6d14b116b5e7af49867d456336fad5e685ba020b15 >>> *httpd-2.4.46.tar.gz >>> The SVN tag is '2.4.46' at r1880505.
iOS 14 / macOS 11 and HTTP/3 support
From Apple’s developer beta release notes, the newest Apple code is now shipping with HTTP/3 support. Disabled by default, but can be enabled by users. As of today, HTTP/3 Draft 29 isn’t yet supported. Alex
SSLStrictSNIVHostCheck not being enforced by mod_ssl
"SSLStrictSNIVHostCheck On” directive in either _default_ or specific vhost entry has no effect: clients not forced to provide SNI data for web site access. Environment: - Standard HTTPD 2.4.43 and OpenSSL 1.1.1f builds from Pat Volkerdi’ Slackware -current. Enabled "LogLevel debug”, not seeing *anything* SNI related in the logs. Ideas? -- Alex
Support for RFC7627 (Extended Master Secret) on mod_ssl
Does mod_ssl supports this RFC for EMS? I’ve found no references to this on the current documentation. Seems this has been added to NIST recommendations a while back, and from an old OpenSSL thread[1], this is a client and server combination, most modern clients are now using. More folks might be looking at this now that consumer grade systems[2] are reporting if their servers does or does not support this. [1] https://github.com/openssl/openssl/issues/7246#issuecomment-452360968 <https://github.com/openssl/openssl/issues/7246#issuecomment-452360968> [2] https://www.immuniweb.com/ <https://www.immuniweb.com/> — Alex
Re: svn commit: r1875349 - /httpd/site/trunk/tools/roll.sh
From OpenSSL download page (https://www.openssl.org/source/): Note: The latest stable version is the 1.1.1 series. This is also our Long Term Support (LTS) version, supported until 11th September 2023. All other versions (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used. Users of these older versions are encourage to upgrade to 1.1.1 as soon as possible. Extended support for 1.0.2 to gain access to security fixes for that version is available. While I understand they do offer paid support for previous versions, I don’t think it is wise for httpd to openly support a discouraged code. Previous OpenSSL versions were fun, but it is time to move on. Just my $.02. Alex > On Mar 18, 2020, at 09:44, jean-frederic clere wrote: > > On 18/03/2020 11:09, Ruediger Pluem wrote: >>> On 3/18/20 9:36 AM, jfcl...@apache.org wrote: >>> Author: jfclere >>> Date: Wed Mar 18 08:36:46 2020 >>> New Revision: 1875349 >>> >>> URL: http://svn.apache.org/viewvc?rev=1875349&view=rev >>> Log: >>> Add sha512 >>> >>> Modified: >>> httpd/site/trunk/tools/roll.sh >>> >>> Modified: httpd/site/trunk/tools/roll.sh >>> URL: >>> http://svn.apache.org/viewvc/httpd/site/trunk/tools/roll.sh?rev=1875349&r1=1875348&r2=1875349&view=diff >>> == >>> --- httpd/site/trunk/tools/roll.sh (original) >>> +++ httpd/site/trunk/tools/roll.sh Wed Mar 18 08:36:46 2020 >>> @@ -103,9 +103,11 @@ openssl="`which openssl 2> /dev/null | h >>> md5sum="`which md5sum 2> /dev/null | head -1`" >>> sha1sum="`which sha1sum 2> /dev/null | head -1`" >>> sha256sum="`which sha256sum 2> /dev/null | head -1`" >>> +sha512sum="`which sha512sum 2> /dev/null | head -1`" >>> md5="`which md5 2> /dev/null | head -1`" >>> sha1="`which sha1 2> /dev/null | head -1`" >>> sha256="`which sha256 2> /dev/null | head -1`" >>> +sha512sum="`which sha512sum 2> /dev/null | head -1`" >> Should the above be sha512 instead of sha512sum? >> Are we sure that openssl / gpg are capable of sha512 for a reasonable span >> of versions or is it worth checking for a >> minimal version? > > gpg looks good, openssl > 1.0.0 is good too and 10 years old no? > >> Regards >> Rüdiger > > > -- > Cheers > > Jean-Frederic
Re: Opt in(/out?) for TLS protocol per vhost (was: svn commit: r1868645)
IMHO, it *is* intuitive. If you want no default configuration, do not set one by default, otherwise inheritance applies - just as everything else in this daemon. Or are you all planning to opt in/out every other settings as well, to make a standard "intuitive-driven” configuration approach? > On Oct 25, 2019, at 10:18 AM, Yann Ylavic wrote: > > The current status is that, without an opt-in/out, it takes the root > value if configured, or the base server's otherwise. Not very > intuitive...
Re: Cloudflare, Google Chrome, and Firefox Add HTTP/3 Support
Although I should had made a few things clear, seems some good discussion happened. Amongst the same lines: When I asked about Apache, I should had stated HTTPD. There is a QUIC implementation on Apache.org under ATS (Apache Traffic Server), a reverse proxy, load balancer daemon. While definitely an interesting approach that takes a ton of overhead from the web server, it adds much more than “just a mod_h3” to be maintained. Not that a mod_h3 wouldn’t be enough work to be maintained. A motivator for the implementation is the continuity and evolution of the web we all know and rely on, which this very ancient dinosaur daemon helped to build and solidify. Apache may or may not have the largest market share amongst HTTP servers anymore, but it does not means it is stuck in time. As of h2, h3 is another evolution that should be looked at, when the time is right. And while all is well said, it needs done. I too agree it might be a fun project for anyone with enough time, motivation and skills to do so. I fall short on at least one of these, so as much as the enthusiast of me would love to turn it on at my systems, I’m yet to remember how to code anything other than a bash script or minor automation tools in pre-made, 3rd party Python modules. Besides, h3 is not a full formal standard yet, so while it is showing signs it will be some day, it might be as QUIC is/was for a while before it settles up as standard. But it never hurt to be checked out. Last but not least, thanks Stefan for your h2 work. Alex > On Sep 27, 2019, at 17:12, Helmut K. C. Tessarek wrote: > > On 2019-09-27 11:47, Eric Covener wrote: >> I don't think market share is a big motivating factor for active >> contributors. > > Maybe not, but I remember a discussion a while back on this list that had to > do with features vs stability, about market shares and why other web servers > are gaining. > >> HTTP/3 would be a lot of work, a lot of shoehorning into HTTPD, and >> likely be de-stabilizing for quite some time. There is >> simply nobody (seemingly) working on it. > > I get that, I was simply saying that I didn't understand why there wasn't a > plan. That's all. > I also do understand that this might be a highly complex topic, especially > since it will touch many components. > I'm very grateful that Stefan took the initiative to get h2 into httpd. > >> HTTPD is great and familiar to lots of people, but HTTPD'S age brings >> a lot of baggage. Lots of other great servers have much >> less baggage and currently have much more commercial interest and buzz. > > I've been using Apache httpd since the early days and I won't be switching to > another web server. But the "baggage" can't be the reason for stagnation. A > web server's main functionality is to serve web pages. If the protocol evolves > so must the server, otherwise the server will be obsolete at one point in the > future. > > Cheers, > K. C. > > -- > regards Helmut K. C. Tessarek KeyID 0x172380A011EF4944 > Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944 > > /* > Thou shalt not follow the NULL pointer for chaos and madness > await thee at its end. > */ >
Cloudflare, Google Chrome, and Firefox Add HTTP/3 Support
https://news.slashdot.org/story/19/09/26/1710239/cloudflare-google-chrome-and-firefox-add-http3-support With that, the obvious question: what about Apache?
Re: Default log file locations
$ tail -F /var/log/apache/access_log /var/log/apache/error_log There’s your stdout output, no code modifications needed. You are welcome. Jokes aside, you could make use of a web socket to pull these logs out in a way code doesn’t need to be changed. Or you could dump them straight into a database. Or, IDK, there are so many ways to extract the logs from this ancient daemon, that defaulting them to stdout/stderr sounds like a bad idea - for one sole user/reason, is too much of a change. If *maybe* the suggestion was to allow usage of stdout/stderr instead of defaulting them to those, it would be a less dramatic change, but hey, caveats are starting to appear, as Eric just pointed one out. Alex > On Jun 27, 2019, at 17:24, Eric Covener wrote: > > >> >> From my perspective it would be advantageous to have Apache write to >> the terminal by default (i.e. no hardcoded log file locations) and >> allow to override this behavior via the Apache configuration file. > >> Is there any reason why the default behavior is not that way yet? > > I think it's useful as opt-in, but I wouldn't want to see any > "defaults" changed (whether that's how the code behaves in the absence > of logging related directives, or how our default httpd.conf looks) > > One wrinkle I recall here is that the closing of stdout and stderr > happens in APR (apr_proc_detach?) and it requires a new APR release to > provide any alternate options there.
Re: Plan to add sandbox branch
How often are nodes updating themselves? Are they only updating their main proxy server or each other in a “multicast” fashion? How about if one of the nodes crash before sending back an update? Are you locking a session to a specific backend host, keep it tracked on the proxy/front end server and if so, for how long? When I read your message, in particular the STATUS info statement, this plan of yours looked to me like how a regular load balancer product works, and if that is the intent, how different from and/or how integrated to ATS is this going to be? Alex On 2018/11/27 11:23:25, Jim Jagielski wrote: > In the coming week or so, I will be committing my load balance,> > load determination and discovery work to a sandbox trunk. Many> > people have asked for more info, so here we go.> > > Basically, this new feature uses nanomsg (nng) to implement the> > SURVEY protocol between workers (nodes) and the front end server.> > The proxy server will send out MEMBERSHIP and STATUS surveys, that> > nodes can respond to; thus new nodes can automagically add themselves> > as they come up as well as remove themselves when they restart or> > shutdown via MEMBERSHIP. STATUS can be used to provide real-time> > load on each node, which will be sent via a __packed__ struct,> > ala NTP and other common usage patterns, in which 32bit fields are> > converted from endian-ness to network and back. That STATUS info can> > be used for "intelligent" load balancing and will provide some> > typical server metrics such as memory, # of cpus, etc...> > smime.p7s Description: S/MIME cryptographic signature
Server Token: None
Can we have an empty SERVER header instead of the minimalistic but yet “revealing“ issued by the token when set as Prod? Most people are change this header either by patching themselves (and maintaining their patches), or by installing extra modules/plugins, but it would be very, very handy if this was an option from the main source itself. I did a quick and dirty patch for the latest release code, and as someone who doesn’t code anything past a hello world for quite a few years, it was simple enough I’m surprised how nobody cared to do it. Or perhaps this had been discussed before and the general consensus was to leave the bare minimum to Prod: if so, people that want to keep low would find their ways anyway, but giving us choice is not unusual from the spirit of FOSS. Alex httpd-server-header-none.diff.gz Description: GNU Zip compressed data smime.p7s Description: S/MIME cryptographic signature
Wiki Contribution Request
Dear Httpd Wiki Admin, My name is Alex Owen and my wiki name is AlexOwen. I am an Linux and apache httpd administrator of over 10 years professional experience in a UK University setting. Along the way I have picked up a few tips and tricks and thought I could make the world a marginally better place by sharing some of them on the httpd wiki. My main motivation to have wiki access is to document my recipe to make the SSLRequireSSL directive redirect from the http to the https version of a page. I'd be obliged if someone could add me to the appropriate group on the wiki so I could add this new document. If you wish to make further assesment of my suitability as a wiki contributor then please check out a debian wiki page I first authored in 2008 and have continued to take an active part in updating since. https://wiki.debian.org/DebianInstaller/NetbootFirmware https://wiki.debian.org/DebianInstaller/NetbootFirmware?action=info Many Thanks Alex Owen -- Private web search without a filter bubble: https://duckduckgo.com/
Threaded vs non-threaded dev package
I am compiling a module I have written on Ubuntu Precise. The module will always be run on apache-mpm-prefork (i.e. the non-threaded mpm), but the module itself uses threads (apr_thread*). Should I be compiling against apache2-threaded-dev or apache2-prefork-dev? Or doesn't it matter? -- Alex Bligh
range request support using ap_send_fd
Hi,
I'm working on a problem where mod_perl doesn't seem to accept range
requests documented here:
http://www.gossamer-threads.com/lists/modperl/dev/104360
Working with 2.2.22, my issue seems to be the mod_perl sendfile
implementation uess ap_send_fd, and ap_send_fd does not create an EOS
bucket, so in byterange_filter.c:
/*
* Don't attempt to do byte range work if this brigade doesn't
* contain an EOS, or if any of the buckets has an unknown length;
* this avoids the cases where it is expensive to perform
* byteranging (i.e. may require arbitrary amounts of memory).
*/
if (!APR_BUCKET_IS_EOS(e) || clength <= 0) {
ap_remove_output_filter(f);
return ap_pass_brigade(f->next, bb);
}
we don't handle range requests, and so my mod_perl handler does not work
with range requests.
My question is should ap_send_fd be inserting an eos bucket? i.e.
alex@alex ~/httpd-2.2.22 $ diff -u server/protocol.c.orig server/protocol.c
--- server/protocol.c.orig 2012-01-24 12:02:19.0 -0800
+++ server/protocol.c 2012-05-24 18:24:57.914018451 -0700
@@ -1386,6 +1386,8 @@
bb = apr_brigade_create(r->pool, c->bucket_alloc);
b = apr_bucket_file_create(fd, offset, len, r->pool, c->bucket_alloc);
APR_BRIGADE_INSERT_TAIL(bb, b);
+b = apr_bucket_eos_create(c->bucket_alloc);
+APR_BRIGADE_INSERT_TAIL(bb, b);
rv = ap_pass_brigade(r->output_filters, bb);
if (rv != APR_SUCCESS) {
alex@alex ~/httpd-2.2.22 $
which does "fix" the problem for me (i.e. range requests work), but I
have no idea the implications behind this and the warning in the comment
about arbitrary amounts of memory. =)
Thanks for any help/guidance you can provide.
Cheers,
Alex
Re: Question about APR SHM
Hi Igor, Thanks for the suggestion. I have meanwhile informed the APR developers via the appropriate list. Cheers, Alex Op 08-08-10 21:47, Igor Galić schreef: > - "Alex Wulms" wrote: > > >> Op 31-07-10 18:07, Alex Wulms schreef: >> >>> Hi, >>> > Hi Alex, > > despite the fact that there are a lot of APR developers subscribed to > this list, you're probably better off sending this to > > d...@apr.apache.org > > So long, > >
Re: Question about APR SHM
Op 31-07-10 18:07, Alex Wulms schreef: > Hi, > > I have got a question about APR SHM. The comments of the function > apr_shm_baseaddr_get(const apr_shm_t *m) indicate that the resulting > address is only valid in the callers address space, since the API does > not guarantee that other attaching processes will maintain the same > address mapping. > ... > Hi, In order to play it safe, I have made my code function completely on top of the RMM (relocatable memory management) API already present in APR. As a side-product, I have written a re-usable RMM hash table (derived from apr_hash.c). You can find the code in the GIT repository located here: http://repo.or.cz/w/httpd-crcsyncproxy.git It concerns files crccache/rmm_hash.h and crccache/rmm_hash.c in the repository. Please feel free to include them in a future version of APR. Kind regards, Alex
Question about APR SHM
Hi, I have got a question about APR SHM. The comments of the function apr_shm_baseaddr_get(const apr_shm_t *m) indicate that the resulting address is only valid in the callers address space, since the API does not guarantee that other attaching processes will maintain the same address mapping. When I look at the implementation of modules that use SHM (like mod_ssl and mod_ldap), it seems like the address returned by the function is re-used as-is in the worker child processes. In both modules, the above function is invoked from the post_config handler and the resulting memory structure is then used in the worker child processes. So my question is: when is the note about the validity of the resulting address applicable? Is it only applicable when a new process has attached itself explicitly to an already existing SHM segment with the apr_shm_attach(...) function? And can I safely ignore it when it concerns an implicit attachment (inherited) by the child process? And is it like that on all supported platforms and not only on Unix (I know for a fact that it is indeed like this on Unix)? Thanks and kind regards, Alex
Re: Error log format configuration syntax
Op 28-07-10 15:41, Rainer Jung schreef: > On 28.07.2010 13:44, Dan Poirier wrote: >> On 2010-07-28 at 03:51, Alex Wulms wrote: >> >>> Hi, >>> >>> While adding some debug log statements to a module I'm working on, I >>> ran >>> into the problem that ap_log_error (in apache 2.2) does not support %zd >>> or %zu conversion for type_t arguments. This was problematic to make >>> the >>> code compile on both 32 and 64 bit platforms. >> >>> * platform (32-bit or 64-bit). This violates the whole purpose of >>> type_t, which >>> * was introduced in C exactly to provide cross-platform >>> compatibility... >> >> I'm confused. Neither c89 nor c99 define a type "type_t", as far as I >> can see. >> >> But you might find the *_FMT macro definitions from APR helpful, or else >> explain your problem in more detail? > > It seems in C99 a length specifier "z" means: For integer types, > causes printf to expect a size_t sized integer argument. Citing: > > Specifies that a following d, i, o, u, x, or X conversion specifier > applies to a size_t or the corresponding signed integer type argument; > or that a following n conversion specifier applies to a pointer to a > signed integer type corresponding to size_t argument. > > At least cross checking on Solaris 10 shows it is actually implemented > for printf() there. > > So: type_t -> size_t and the OP should be able to use APR_SIZE_T_FMT > or APR_SSIZE_T_FMT (signed) as you suggested. > Thanks for the tip about the APR formatting macro's (I'm still getting up to speed on the features of the APR API). It is indeed what I need. I'll adapt the code accordingly. Cheers, Alex
Re: Error log format configuration syntax
Hi,
While adding some debug log statements to a module I'm working on, I ran
into the problem that ap_log_error (in apache 2.2) does not support %zd
or %zu conversion for type_t arguments. This was problematic to make the
code compile on both 32 and 64 bit platforms. So I made a small wrapper
(see below) to workaround the problem. I suggest to build support for
%zd and %zu conversion into the unified logger.
/**
* ap_log_error does not support %zd or %zu conversion for type_t arguments
* So with ap_log_error one would have to specify either %d or %ld,
depending on the
* platform (32-bit or 64-bit). This violates the whole purpose of
type_t, which
* was introduced in C exactly to provide cross-platform compatibility...
* This wrapper function supports %zd and %zu conversion parameters.
* Note that it truncates the logged message to 1000 bytes, so don't use
it to log messages that might
* be longer
*/
void ap_log_error_wrapper(const char *file, int line, int level,
apr_status_t status, const server_rec *s,
const char *fmt, ...)
{
char msg[1000];
va_list ap;
va_start(ap, fmt);
vsnprintf(msg, sizeof(msg), fmt, ap);
ap_log_error(file, line, level, status, s, "%s", msg);
}
Cheers,
Alex
Op 27-07-10 23:11, Stefan Fritsch schreef:
> On Wednesday 21 July 2010, William A. Rowe Jr. wrote:
>
>> On 7/21/2010 4:35 PM, Stefan Fritsch wrote:
>>
>>> And I agree that a unified logging formatter would be nice, but
>>> this is not something that will be done before 2.3.7 and it
>>> would likely mean an incompatible API change for modules
>>> providing handlers for mod_log_config. IMHO, this can wait until
>>> 3.0.
>>>
>> IMHO, it must not. The simple reason is that the more code
>> duplication we introduce, the more opportunity for flaws and
>> maintenance headaches during the 2.4 lifecycle. I'd accept
>> waiting for this entire custom error log feature for 3.0, but
>> really would rather introduce it sooner and not later.
>>
> I fear a unified logging formatter may either delay 2.4 or not make it
> into 2.4. In 2.4, we really need some way to omit some of the fields
> we currently have in the error log prefix (> 100 chars of prefix is
> insane). But as I had less time in the last few days than I hoped, it
> does not look like my (non-unified) errorlog formatter would be ready
> for 2.3.7 in any case. So we will have time until 2.3.8 to decide what
> to do.
>
>
>> If there is a resulting API change, I think everyone is willing to
>> accept this during the 2.3-alpha/beta cycle. 2.4.0 is our
>> 'lockdown'.
>>
>> I'm willing to help with this code, although I'm just starting to
>> dig out of the hole from my two recent hand surgeries. 6-finger
>> typing isn't all that efficient :)
>>
> I think there are more important things that could use your and my
> attention than avoiding this bit of code duplication.
>
>
> Buf if you want to start a bit of technical discussion: The main
> technical difference between error and access log is that for the
> access log, everything is allocated from the request pool and then
> written to the log file (i.e. there is no length limit). For the error
> log, there is a fixed 8K buffer (allocated on the stack). For the
> error log, it is not possible to use any of the existing pools to
> avoid unbounded memory usage. The unified log formatter would either
> have to use a pre-allocated buffer or a temp pool.
>
> For my work on the error log formatter, I have stayed with the pre-
> allocated buffer. Would this be a reasonable solution for the unified
> logger? It would mean a fixed limit for the access logs lines (though
> the access log could use a larger buffer than the error log, I guess
> 16 or 32K would be enough even for the access log). The pool approach
> would require a per-thread temp logging pool (using
> apr_threadkey_private_create or the like) or creating and destroying a
> sub-pool for every log line.
>
> Which solution looks better to you?
>
>
> Cheers,
> Stefan
>
>
Re: Apache devs attending FOSDEM 2010?
Hi, Martin and myself are meeting sunday at 10:30 at the Mozilla room, where we will meet up with Gerv from Mozilla. Please feel free to join. The purpose is to discuss how to progress the delta-http protocol support; more specifically the integration in Apache webserver through mod-crcsync that I have been working on with Toby based on the rolling-crc library from Rusty Russell and support for the protocol into a future version of Firefox. The driver for the delta protocol is OLPC deployments, to accelerate internet access for schools in rural area's with slow internet uplinks. The protocol may also be useful for mobile internet, as long as 4G is not yet widespread. See http://wiki.laptop.org/go/Apache_Proxy_CRCsync for more details. Cheers, Alex
Re: Apache devs attending FOSDEM 2010?
Any proposals for where/when we could meet exactly? Op 28-01-10 00:47, Jorge Schrauwen schreef: > > On Wed, Jan 27, 2010 at 11:58 PM, Alex Wulms <mailto:alex.wu...@scarlet.be>> wrote: > > Op donderdag 14 januari 2010 14:28:42 schreef Martin Langhoff: > > On Mon, Jan 11, 2010 at 8:49 PM, Nick Kew <mailto:n...@webthing.com>> wrote: > > > I've booked into the renaissance hotel - > > > > http://www.marriott.com/hotels/travel/brubr-renaissance-brussels-hotel/ > > > > Cool -- that's in my neighbourhood, though a tad long to walk > from home :-) > > > > >> If so, it'd be great to get together > > > > > > +1. Will you be at the Friday beer event? Were you thinking > in terms > > > of an apache-specific get-together on Saturday or Sunday evening? > > > Or other ??? > > > > I'll probably miss the Friday beer (I'll be returning from a > trip that > > very night), but Saturday afternoon or Sunday anytime we can stage a > > get together. I'll be the guy running around with laptops with green > > ears ;-) > Hi, > > I prefer Saturday afternoon or evening. > > > Kind regards, > Alex > > > I'm 90% certain I'll be there on Saturday. > > Would be a nice ending to my exam period :) >
Re: Apache devs attending FOSDEM 2010?
Op donderdag 14 januari 2010 14:28:42 schreef Martin Langhoff: > On Mon, Jan 11, 2010 at 8:49 PM, Nick Kew wrote: > > I've booked into the renaissance hotel - > > http://www.marriott.com/hotels/travel/brubr-renaissance-brussels-hotel/ > > Cool -- that's in my neighbourhood, though a tad long to walk from home :-) > > >> If so, it'd be great to get together > > > > +1. Will you be at the Friday beer event? Were you thinking in terms > > of an apache-specific get-together on Saturday or Sunday evening? > > Or other ??? > > I'll probably miss the Friday beer (I'll be returning from a trip that > very night), but Saturday afternoon or Sunday anytime we can stage a > get together. I'll be the guy running around with laptops with green > ears ;-) Hi, I prefer Saturday afternoon or evening. Kind regards, Alex
Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.
2009/8/21 Alex Stapleton : > 2009/8/20 Roy T. Fielding : >> On Aug 20, 2009, at 2:01 AM, Alex Stapleton wrote: >>> >>> 2009/8/19 Roy T. Fielding : >>>> >>>> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote: >>>> >>>>> (This has been cross posted to us...@. I apologise if this mail isn't >>>>> relevant to the dev list.) >>>>> >>>>> First some background. We use Apache HTTPD 2.0 over a high-latency, >>>>> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL. >>>>> We also use Transfer-Encoding: chunked sometimes. This is a machine >>>>> monitoring application. We are using iframe streaming to push real >>>>> time data to operators browsers. >>>>> >>>>> I have noticed after much faffing around with wireshark that httpd >>>>> will transmit 3 Application Data fragments for each chunk in a chunked >>>>> HTTP stream. The fragments are the HTTP chunk size, the chunk data, >>>>> and then a CRLF. All fragments in an SSL session are rounded to the >>>>> length of the ciphers block size. This results in the 4+2 bytes for >>>>> the chunk frame ending up being 64 bytes of data over TCP. A waste of >>>>> 58 bytes per chunk in this case. >>>> >>>> That's odd -- we don't do this with non-SSL writes (last I checked). >>>> Perhaps we are relying on a TCP cork for the non-SSL case? >>>> What is your operating system and platform? >>> >>> I initially discovered this issue on a fairly old Ubuntu 6 machine >>> running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X >>> 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also >>> point out that I am using PHP 5.2.9 to generate the chunked data. If >>> there's a way of doing chunked output using "plain" Apache let me know >>> and I can test that too if needed. >> >> Er, have you tested it with 2.2.x? The likelihood of us making any >> non-security changes to the 2.0.x branch is extremely small. > > I have tested with 2.2.11 from MacPorts on my iMac and it also > exhibits this behaviour. I'll try and do a 2.2.13 build to test with > :) Confirmed on 2.2.13. >> Apache does chunked output by default if no content-length is >> provided and the client says it is HTTP/1.1. >> >> Roy >> >> >
Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.
2009/8/20 Roy T. Fielding : > On Aug 20, 2009, at 2:01 AM, Alex Stapleton wrote: >> >> 2009/8/19 Roy T. Fielding : >>> >>> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote: >>> >>>> (This has been cross posted to us...@. I apologise if this mail isn't >>>> relevant to the dev list.) >>>> >>>> First some background. We use Apache HTTPD 2.0 over a high-latency, >>>> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL. >>>> We also use Transfer-Encoding: chunked sometimes. This is a machine >>>> monitoring application. We are using iframe streaming to push real >>>> time data to operators browsers. >>>> >>>> I have noticed after much faffing around with wireshark that httpd >>>> will transmit 3 Application Data fragments for each chunk in a chunked >>>> HTTP stream. The fragments are the HTTP chunk size, the chunk data, >>>> and then a CRLF. All fragments in an SSL session are rounded to the >>>> length of the ciphers block size. This results in the 4+2 bytes for >>>> the chunk frame ending up being 64 bytes of data over TCP. A waste of >>>> 58 bytes per chunk in this case. >>> >>> That's odd -- we don't do this with non-SSL writes (last I checked). >>> Perhaps we are relying on a TCP cork for the non-SSL case? >>> What is your operating system and platform? >> >> I initially discovered this issue on a fairly old Ubuntu 6 machine >> running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X >> 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also >> point out that I am using PHP 5.2.9 to generate the chunked data. If >> there's a way of doing chunked output using "plain" Apache let me know >> and I can test that too if needed. > > Er, have you tested it with 2.2.x? The likelihood of us making any > non-security changes to the 2.0.x branch is extremely small. I have tested with 2.2.11 from MacPorts on my iMac and it also exhibits this behaviour. I'll try and do a 2.2.13 build to test with :) > Apache does chunked output by default if no content-length is > provided and the client says it is HTTP/1.1. > > Roy > >
Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.
2009/8/19 Roy T. Fielding : > On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote: > >> (This has been cross posted to us...@. I apologise if this mail isn't >> relevant to the dev list.) >> >> First some background. We use Apache HTTPD 2.0 over a high-latency, >> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL. >> We also use Transfer-Encoding: chunked sometimes. This is a machine >> monitoring application. We are using iframe streaming to push real >> time data to operators browsers. >> >> I have noticed after much faffing around with wireshark that httpd >> will transmit 3 Application Data fragments for each chunk in a chunked >> HTTP stream. The fragments are the HTTP chunk size, the chunk data, >> and then a CRLF. All fragments in an SSL session are rounded to the >> length of the ciphers block size. This results in the 4+2 bytes for >> the chunk frame ending up being 64 bytes of data over TCP. A waste of >> 58 bytes per chunk in this case. > > That's odd -- we don't do this with non-SSL writes (last I checked). > Perhaps we are relying on a TCP cork for the non-SSL case? > What is your operating system and platform? I initially discovered this issue on a fairly old Ubuntu 6 machine running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also point out that I am using PHP 5.2.9 to generate the chunked data. If there's a way of doing chunked output using "plain" Apache let me know and I can test that too if needed. >> I'm not aware of any reason for this to behave specifically in this >> way and clearly combining the entire chunk into a single SSL fragment >> would provide a significant bandwidth saving for HTTP streaming >> applications if not more mainstream ones. >> >> I've done a fair amount of poking through the httpd source today to >> try and isolate this but this is my first foray into the depths of >> httpd so I've not got far in the direction of a solution. I have >> identified that this 'problem' is due to the way the chunk_filter >> function adds buckets into the brigade which end up getting turned >> into their own fragments by mod_ssl. Creating a new bucket which has >> the extra data wrapped around it would presumably be far too >> inefficient for a general solution. I was considering using the FLUSH >> bucket type but am not fully aware of how it's used by the various >> filters. > > It should not be necessary to have multiple buckets -- they should > be written using a vector and not result in separate packets. However, > this may be limited by the SSL library's write interface. > FLUSH is the opposite of what you want. We should either be doing > the equivalent of a writev on SSL or add a buffering filter. > >> I'm not sure what the ideal way is to go about fixing this, or if it's >> even in fact an actual source code level problem rather than a >> configuration one, hence why this is posted to users for now. >> >> I can provide text dumps from tshark if that would be more >> illuminating of the patterns I'm seeing. >> >> Alex Stapleton > > Platform details would be helpful -- you've narrowed the cause well > enough that I doubt the text dumps would help. > > Roy >
mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.
(This has been cross posted to us...@. I apologise if this mail isn't relevant to the dev list.) First some background. We use Apache HTTPD 2.0 over a high-latency, high packet loss GPRS WAN. The cost per byte is tangible. We use SSL. We also use Transfer-Encoding: chunked sometimes. This is a machine monitoring application. We are using iframe streaming to push real time data to operators browsers. I have noticed after much faffing around with wireshark that httpd will transmit 3 Application Data fragments for each chunk in a chunked HTTP stream. The fragments are the HTTP chunk size, the chunk data, and then a CRLF. All fragments in an SSL session are rounded to the length of the ciphers block size. This results in the 4+2 bytes for the chunk frame ending up being 64 bytes of data over TCP. A waste of 58 bytes per chunk in this case. I'm not aware of any reason for this to behave specifically in this way and clearly combining the entire chunk into a single SSL fragment would provide a significant bandwidth saving for HTTP streaming applications if not more mainstream ones. I've done a fair amount of poking through the httpd source today to try and isolate this but this is my first foray into the depths of httpd so I've not got far in the direction of a solution. I have identified that this 'problem' is due to the way the chunk_filter function adds buckets into the brigade which end up getting turned into their own fragments by mod_ssl. Creating a new bucket which has the extra data wrapped around it would presumably be far too inefficient for a general solution. I was considering using the FLUSH bucket type but am not fully aware of how it's used by the various filters. I'm not sure what the ideal way is to go about fixing this, or if it's even in fact an actual source code level problem rather than a configuration one, hence why this is posted to users for now. I can provide text dumps from tshark if that would be more illuminating of the patterns I'm seeing. Alex Stapleton
mod_cache not caching some RFC valid cacheable requests
Hi there,
I ran into a weird case where *I think* mod_cache should be caching a
request that it is not. Thought I would try to fix it myself, but
would like to seek your feedback as it is my first httpd patch (thanks
to pquerna for the help/encouragement).
Caching a 302 is generally not valid (RFC2616 13.4), unless the
response headers includes an Expires or Cache Control header (section
13.4, last paragraph). This makes the fix a matter of messing with the
cacheability logic. I optimized for least amount of code change, but
there are surely different ways to do this. Feedback on the best
approach would be greatly appreciated!
Thanks,
-Alex
PS: I also filed a bug, if that is a better forum for this discussion:
https://issues.apache.org/bugzilla/show_bug.cgi?id=46346
--
Index: modules/cache/mod_cache.c
===
--- modules/cache/mod_cache.c (revision 723584)
+++ modules/cache/mod_cache.c (working copy)
@@ -438,8 +438,27 @@
* We include 304 Not Modified here too as this is the origin server
* telling us to serve the cached copy.
*/
-reason = apr_psprintf(p, "Response status %d", r->status);
+if (exps != NULL || cc_out != NULL) {
+/* We are also allowed to cache any response given that
it has a valid
+ * Expires or Cache Control header. If we find a either
of those here,
+ * we pass request through the rest of the tests. From the RFC:
+ *
+ * A response received with any other status code (e.g.
status codes 302
+ * and 307) MUST NOT be returned in a reply to a subsequent request
+ * unless there are cache-control directives or another
header(s) that
+ * explicitly allow it. For example, these include the
following: an
+ * Expires header (section 14.21); a "max-age", "s-maxage", "must-
+ * revalidate", "proxy-revalidate", "public" or "private"
cache-control
+ * directive (section 14.9).
+ */
+}
+else {
+reason = apr_psprintf(p, "Response status %d", r->status);
+}
}
+if (reason) {
+/* noop */
+}
else if (exps != NULL && exp == APR_DATE_BAD) {
/* if a broken Expires header is present, don't cache it */
reason = apr_pstrcat(p, "Broken expires header: ", exps, NULL);
Apache 2.2.0 support?
Hello all--I'm sure this has been a subject of ongoing discussion; but could someone perhaps fill me in on the timeline for adding mod_python support for Apache 2.2.0? I just put together mod_python 3.2.7 and Apache 2.2.0 with Python 2.4.2, and it doesn't really work. Any clues why not, or ideas as to when it might would be welcomed. Or if these questions have already been answered, is there an archive of this list I could search? Thanks much.
2.0.53 on Windows
Forgive me if this has already been mentioned, but I can't find anything in the archives. Is anybody planning on packaging a Windows version of the source files? Thanks, Alex.
Message: winnt_accept: Asynchronous AcceptEx failed.
Hi, We see a lot of WARNING messages in Windows Event Log: winnt_accept: Asynchronous AcceptEx failed. The actual error is always 995: ERROR_OPERATION_ABORTED - The I/O operation has been aborted because of either a thread exit or an application request. It's coming from server\mpm\winnt\child.c The call: GetOverlappedResult((HANDLE)context->accept_socket, &context->Overlapped, &BytesRead, FALSE) GetLastError() is not called there, instead: closesocket(context->accept_socket); context->accept_socket = INVALID_SOCKET; It all seems like normal condition to me - why log WARNING ? Is it OK to turn it off? Thanks, Alex
RE: Memory leak on Windows [Bugzilla #11427]
I believe that I have made some progress on this, but I'm not confident
that my solution is the correct one. There were several factors playing
into this that pulled my attention away from the real issue.
I originally reported that this problem was CGI specific; it turns out
that this is not the case. The way I was reproducing what appeared to be
a major leak was by sending a number of parallel requests to CGIs that
generate a lot of output. The result of this was that Apache needed a lot
of temporary buffers at the same time. After digging around for a while,
I discovered that Apache on Windows never calls
apr_allocator_max_free_set(), and the default behaviour is to hold on to
all memory ever allocated, and allocate memory future requests out of that
block. When I spawned the parallel requests, Apache grabbed a whole bunch
of memory, and then kept reusing it. After setting a limit here, I was
able to move on to the bigger problem.
After making the change above, I realized that every request was leaking a
little bit, not just the CGI requests. To confirm this, I made a simple
configuration file that redirects all requests to a static HTML file.
Sure enough, it was leaking.
The problem, as best as I can tell, is that mpm_winnt calls
apr_bucket_alloc_create() once for each thread, and registers it in the
pchild pool. This bucket allocator is then passed through to
core_create_conn(), and used for all apr_bucket_XXX() routines during the
request. The pchild pool is not cleared until the server shuts down, so
the memory used here grows and grows. To solve this problem I changed
mpm_winnt so that it creates the bucket allocator using the ptrans pool,
which gets cleared after every connection is finished.
After making this change, the system behaved much better. Just to check,
I then undid my first change related to the maximum memory to hold on to,
and the system continued to function corrrectly. So in the end, I only
needed the one fix.
The following patch shows the changes I made. My question now for the
experts is whether this will break anything.
Thanks,
Alex.
Index: server/mpm/winnt/child.c
===
RCS file: /home/cvspublic/httpd-2.0/server/mpm/winnt/child.c,v
retrieving revision 1.9
diff -u -u -r1.9 child.c
--- server/mpm/winnt/child.c14 Oct 2002 14:54:45 - 1.9
+++ server/mpm/winnt/child.c20 Dec 2002 01:02:38 -
@@ -122,6 +122,7 @@
if (context) {
apr_pool_clear(context->ptrans);
context->next = NULL;
+context->ba = apr_bucket_alloc_create(context->ptrans);
ResetEvent(context->Overlapped.hEvent);
apr_thread_mutex_lock(qlock);
if (qtail)
@@ -187,7 +188,7 @@
apr_pool_tag(context->ptrans, "ptrans");
context->accept_socket = INVALID_SOCKET;
-context->ba = apr_bucket_alloc_create(pchild);
+context->ba = apr_bucket_alloc_create(context->ptrans);
apr_atomic_inc(&num_completion_contexts);
}
@@ -416,7 +417,7 @@
context = apr_pcalloc(pchild, sizeof(COMP_CONTEXT));
apr_pool_create(&context->ptrans, pchild);
apr_pool_tag(context->ptrans, "ptrans");
-context->ba = apr_bucket_alloc_create(pchild);
+context->ba = apr_bucket_alloc_create(context->ptrans);
}
while (1) {
RE: Memory leak on Windows [Bugzilla #11427]
Hi Bill, Thank you for looking at this. I've just grabbed the httpd-2.0, apr, apr-util, and apr-iconv modules from CVS (-rHEAD). With the same configuration file as I was using in 2.0.43, I can still reproduce the problem. I then added 'KeepAlive off' to my configuration file. With this setting, the problem continues to display itself on my server. With 'KeepAlive on' and 'MaxKeepAliveRequests 10' in my configuration file, again the problem continues. I am running two different tests to reproduce this, although they may not be related. In the first test, I have a client machine running SilkPerformer which opens up several parallel requests to the server. In the second test, I use Internet Explorer and hold down the F5 key for a few seconds straight .. I think this triggers the client abort code inside Apache. Any other suggestions? My plan now is to walk through all the memory allocations in the call stack in the hopes that I find something using a pool that doesn't get released. I'm definitely not looking forward to this. Thanks, Alex. On Tue, 17 Dec 2002, Bill Stoddard wrote: > I am running 2.0.44-dev (latest snapshot in CVS from just a few minutes ago) and > I'm not able to recreate this problem. Have you tried the latest version of 2.0 > (CVS HEAD)? Just for fun, try disabling keep-alive connections or set > MaxKeepAliveRequests to a small value (maybe from the default of 100 to 10) and > report back the results. > > Bill > > > > > Hi, > > > > I'm still trying to track the cause of this problem down, and I'm hoping > > somebody around here can help me. > > > > To summarize, I'm seeing Apache's memory footprint grow abnormally large > > on Windows when using CGIs. The size of the growth seems to be > > proportional to the amount of data printed to stdout from the CGI. > > > > Some sample data: > > - With 16 threads and 1 meg sent to stdout, combined physical and virtual > >memory reaches about 70 megs after hammering the server for several > >minutes > > - If I increase the amount of stdout data to 2 megs, the process grows to > >about 130 megs within another few minutes. > > > > I've spent the last few days reviewing the code, and I'm a bit confused > > about the mpm_winnt pool cleanup code. I haven't spent a lot of time > > reading the code in the past, so there's a good chance that I've just > > missed something. As far as I can tell, ap_read_request() creates the > > request pool, but nothing explicitly cleans it up. Instead, it looks like > > mpm_recycle_completion_context() clears the ptrans pool the next time the > > thread handles a request. While this seems funny to me, I don't see why > > some of the memory would fail to be released. > > > > I've tried to simplify my httpd.conf file to reduce the test case: > > > > ServerRoot c:/varju/webct/webct/server > > ThreadsPerChild 16 > > LoadModule cgi_module modules/mod_cgi.so > > LoadModule alias_module modules/mod_alias.so > > Listen 80 > > ScriptAlias /test.pl c:/varju/webct/webct/webct/generic/public/test.pl > > > > Can anybody offer me suggestions of where to look next? > > > > Thanks, > > Alex. > >
Re: Memory leak on Windows [Bugzilla #11427]
Hi, I'm still trying to track the cause of this problem down, and I'm hoping somebody around here can help me. To summarize, I'm seeing Apache's memory footprint grow abnormally large on Windows when using CGIs. The size of the growth seems to be proportional to the amount of data printed to stdout from the CGI. Some sample data: - With 16 threads and 1 meg sent to stdout, combined physical and virtual memory reaches about 70 megs after hammering the server for several minutes - If I increase the amount of stdout data to 2 megs, the process grows to about 130 megs within another few minutes. I've spent the last few days reviewing the code, and I'm a bit confused about the mpm_winnt pool cleanup code. I haven't spent a lot of time reading the code in the past, so there's a good chance that I've just missed something. As far as I can tell, ap_read_request() creates the request pool, but nothing explicitly cleans it up. Instead, it looks like mpm_recycle_completion_context() clears the ptrans pool the next time the thread handles a request. While this seems funny to me, I don't see why some of the memory would fail to be released. I've tried to simplify my httpd.conf file to reduce the test case: ServerRoot c:/varju/webct/webct/server ThreadsPerChild 16 LoadModule cgi_module modules/mod_cgi.so LoadModule alias_module modules/mod_alias.so Listen 80 ScriptAlias /test.pl c:/varju/webct/webct/webct/generic/public/test.pl Can anybody offer me suggestions of where to look next? Thanks, Alex.
Memory leak on Windows [Bugzilla #11427]
Hi there, I am trying to track down the cause of this memory growth problem, but I'm not really sure where to start. In the hopes that I could at least figure out where the memory was first allocated, I've tried running Apache under Purify. Unfortunately I can't seem to get a debug version built that does not end up loading the non-debug msvcrt.dll. Has anybody successfully run the Windows version inside Purify? If so, are there any magic tricks that you could share with me? Thanks, Alex
Re: [VOTE] Location of aaa rewrite
On Tue, Sep 03, 2002 at 06:59:38PM -0400, [EMAIL PROTECTED] wrote: > > > Can I ask a stupid question? What have we actually broken since Apache > > > 2.0 went GA? Binary compatibility? How many functions? How many of > > > those were APR and not Apache? > > > > Sure, both source and binary compatibility were broken numerous times. > > Check the PHP bug database for umpteen reports on each breakage. > > Okay, but how is that different from early releases of 1.3? I would like to make a small point here that just because the same things are happening as happened before doesn't necessarily mean they're good things to happen (either now or then). I've heard people imply a few times now that since breaking things happened in the early releases of 1.3, it's ok to do it in 2.0 too. It seems to me this is more an argument that the process for 1.3 was probably not what it should have been, and we should try to do better in 2.x. Also, while I agree that 2.0 can be classified still as "early adopter" stage, I would like to point out that one of the biggest factors in determining exactly how much _longer_ it stays in the early-adopter stage is going to be how developmentally stable it's perceived to be by the rest of the world. Every time we break compatibility, we are likely pushing general adoption further into the future. (I'm not intending to address the aaa vote with these comments, BTW, as I'm not quite up enough on the issues to voice an opinion on that specific question. I'm just talking about general principles that I think people should keep in mind.) -alex
Re: Vote: mod_jk connector in /experimental
On Tue, Sep 03, 2002 at 09:51:20AM -0500, Jess M. Holle wrote: > It would be nicest of all to have builds of each version of the core for > each platform -- and pluggable binaries of all the extra modules for > each version/platform as well. Eergh.. this sounds like a maintenance nightmare. > This could be cranked out by automated > scripts as a release criteria/requirement, i.e. it's not a release until > everything builds on all platforms with the automated scripts (and > ideally passes some basic tests on all of them too). I can almost guarantee you this will translate to "we will never again have a release." There are still several significant official apache distribution modules from 1.3 which do not yet work under the current 2.0 line. Considering that we're talking about creating a repository which presumably will be containing not only all of this stuff but lots of third-party modules with various levels of maintenance and stability, requiring that they all compile and work before releasing a new version of httpd is, frankly, insane. Personally, what I would like to see is something along the following lines: 1. A core Apache distribution containing a minimal server. This would contain the core code and the few modules required to get the basic HTTPD behavior everybody expects from any server (serve files off a disk, run CGIs, and not much else). This would be useful for those wanting a "lean and mean" httpd server, or for those who want to build everything custom from the module repository. It would also make it easy to release core updates in a timely fashion, as new releases of this package could be made with a minimum of modules needing to be updated/tested. 2. An "enhanced" Apache distribution, containing everything from the minimal distribution, plus a bunch of really commonly used modules. This would be equivalent to what generally gets distributed now. Criteria for what modules get bundled into this should be based primarily on demand (only modules that lots of people out in the real world need and use), and of course there would be a requirement that any included modules must have people willing and able to actively develop and debug them in a timely fashion, so that if something breaks, it doesn't seriously slow down the release schedule (without good reason). It would be nice if releases of this package corresponded roughly to releases of the core package, but if a core change was made which required updating a lot of stuff, the core package could be released first, while work is still going on on updating all the other modules in this package to work with the new core before the enhanced package goes out the door. 3. A repository of all apache modules (including all the ones from the enhanced distribution, and from everybody else out there in the world) in a consistent, well-defined form with a modular build system for the core which you can just drop them into. Ideally, I would like to be able to download one of the above two distributions, unpack the source, cd into the source directory, and then unpack mod_foo.tar.gz and mod_bar.tar.gz (obtained from the repository), run configure/make, and get a server which includes the foo and bar modules just as if they'd been part of the initial distribution. With a well-defined module distribution file format and a build system which automagically supported modular-inclusions, this shouldn't be too hard to achieve. I don't think it's worth trying to do a global binary module repository (officially). Those responsible for building binary distributions for any given platform can obtain and build in all the modules from the repository which make sense and are well enough maintained to be feasable. Obviously, it would be good to compile things in such a way that third-party developers could also distribute their own binary modules, but I think any repositories/collections for that sort of thing would best be done on an as-needed, per-platform basis. -alex
Shared memory questions
Hello again, The module I'm in the middle of writing will have a need to share information between all the different threads/processes using it, and the cleanest way to do this that I see would be to take advantage of shared memory. I've been looking over the APR shared memory routines, and have a few questions, mainly regarding portability.. First the big question: How portable is shared memory use across the various platforms httpd-2.0 compiles on? The impression I get is that it's fairly well supported, but nothing I've found actually says. Are there platforms which don't support shared memory at all (through APR), and if so what are they? (I'm trying to decide whether my module needs to have some sort of a backup plan for sharing information) Second question: I want to make sure that the shared memory segment used by my module gets properly destroyed when everything is finished using it. What is the correct way to go about this? Is it safe/effective to call apr_shm_destroy when other processes may still be using the memory segment? if not, is there some way to tell whether the segment is still in use or not? -alex
Re: 2.0/2.1 split?
On Fri, Aug 30, 2002 at 11:10:27AM -0500, William A. Rowe, Jr. wrote: > I don't think we have enough -user- community to continue development > on any Apache 2.x. UNLESS we reconsider what we are doing wrong. > Breaking our users on every bugfix/point release would be a good start. > Seeing the successful completion of the PHP/Perl compatibility would > be a good finish. I have to throw my two cents in here. I completely agree with this. Speaking as a system administrator and web administration consultant for quite a few companies, I have to say that I do not currently, and will never, use 2.0 for more than trivial applications, until I feel that I can rely on its (developmental) stability. I still reccomend to all of my clients that they use 1.3, and I expect to continue to do this for quite a long time unless folks here can resolve this issue adequately. I cannot afford to use a server where any time I try to upgrade it for a bugfix I might have to rewrite my configuration files, tweak all my add-on modules, or otherwise spend potentially days figuring out what's changed and why my server doesn't work the way it used to anymore, when all I wanted was a security fix. I know that most new 2.0 releases aren't quite this drastic, but the problem is that there's always the _possibility_ lurking there, and I can't afford to take the gamble on the 40 or 50 servers I'm (directly or indirectly) responsible for. I don't really care (from a user perspective) whether there's a 2.1 or not. It really doesn't matter. I don't care about all the nifty new features people are wanting to add to the project, because they're all going into a product which I'm not gonna be able to use, so they don't matter either. As far as I'm concerned 2.0 doesn't even exist yet, and won't until this situation is fixed. Until then, 1.3 is still the only Apache server there is. I really hope this situation changes because I really would like to start using 2.0 more. -alex
Re: Questions about filter placement
On Thu, Aug 29, 2002 at 05:10:00PM -0700, Brian Pane wrote: > Another possibility would be to create a new metadata bucket type. > In a request-level hook or filter, insert a metadata bucket that > describes the appropriate bandwidth-throttling rules for the buckets > that follow. Then you can use a connection-level filter to do the > actual throttling; that filter, which won't otherwise have access > to request-level information, can look at the metadata buckets to > figure out what bandwidth limit to apply. Ooh! I like this.. this might also help with the problem of determining when a request is "done", too.. thanks for the suggestion, I'll look into this too. -alex
Re: Questions about filter placement
On Thu, Aug 29, 2002 at 04:55:57PM -0700, Ian Holsman wrote: > hmm > you might run into trouble on filetype/size (anything which you need the > response for) as there is no hook >after< the handler. Hmm, that's a little annoying.. I'm actually realizing that I probably can't do it cleanly based on size in any case (I was originally thinking of just looking at the file size of whatever file the request mapped to, but that doesn't do anything for non-file-backed requests, which means inconsistent behavior, which is bad. I'm not sure there would be a lot of demand for rate limiting based on response size anyway, tho, so I'll probably just leave that out for the time being. I would really like to be able to do limits based on content type, though. It sounds like this might actually require inserting an output filter for decision-making, so it would have access to that information. Is there a way for filters to remove themselves from the chain partway through serving a request? That way, at least, it could check things once on the initial call and then get out of the way for the rest of the output.. > I'm not sure if a EOS gets as far as the CORE-OUT if it does that is > what you'll need to check for. Yeah, that was one of the things I wasn't sure about.. I guess I'll just have to play it by ear.. Another option, I suppose, would just be to consider a request "done" when either another request comes through on the connection, or the connection is closed. That's kinda ugly, tho, because theoretically a client could hold a keepalive connection open long after the last request has finished, and bandwidth would still be being allocated for it.. hmph. -alex
Re: Questions about filter placement
On Fri, Aug 30, 2002 at 09:58:01AM +1000, Bojan Smojver wrote: > On Fri, 2002-08-30 at 09:46, [EMAIL PROTECTED] wrote: > > > I'll let people know when I have something people might actually want to use :) > > I don't know if you're planning to make this module free software, but > if you do - treat your users as developers and they will be the > developers :-) Oh, trust me, I'm a long-time open source advocate, and I've intended to release this under the Apache license from the beginning. What I meant by "something people might actually want to use" was really along the lines of "something that at least compiles".. Most of this code isn't even out of my head yet, give me a chance to scribble some of it down before you all try to test it! :) -alex
Re: Questions about filter placement
On Fri, Aug 30, 2002 at 01:31:04AM +0200, Werner Schalk wrote: > tell me more about your module. > Is it already available for testing > and apache2? Nope, as I mentioned, I've got the core decision-making code (and the rate-limiting theory) written, but I'm just starting to put together an apache module around it so it can actually be used with the apache server. I hope to have a basic prototype reasonably soon, tho, if I don't run into big problems figuring out how to write the apache code. It may be a bit longer before it supports all the options I mentioned in my description. I'll let people know when I have something people might actually want to use :) -alex > On Thu, Aug 29, 2002 at 08:06:45AM -0700, Ian Holsman wrote: > > your trying to limit traffic based on what kind of request/request > > path > > it has ? > > Yes, actually based on vhost, URI, directory, file type, size, user, > time of day, etc, etc.. pretty much anything you can think of. It also > supports multiple overlapping bandwidth restrictions and will restrict > traffic based on what other requests are currently being served to > ensure the cumulative rate for any given bandwidth limit is never > exceeded. > > I've got the core code working in a test harness, now I just need to put > it into an apache module..
Re: Questions about filter placement
On Thu, Aug 29, 2002 at 08:06:45AM -0700, Ian Holsman wrote: > your trying to limit traffic based on what kind of request/request path > it has ? Yes, actually based on vhost, URI, directory, file type, size, user, time of day, etc, etc.. pretty much anything you can think of. It also supports multiple overlapping bandwidth restrictions and will restrict traffic based on what other requests are currently being served to ensure the cumulative rate for any given bandwidth limit is never exceeded. I've got the core code working in a test harness, now I just need to put it into an apache module.. > > I could implement this as an AP_FTYPE_CONTENT_SET filter, which would make the > > most sense from a configuration and decision-making standpoint (since I have > > access to request information), but one of the questions I have about this is > > whether other buffering and such later in the filter chain (such as with > > transcode/connection/etc filters) would render any attempts at rate control at > > this level moot, or at least seriously degraded. > > yes, but from what I can see if you are trying to slow down the request > with your filter, this should not be a major drama. Well, my main concern is if there are things down the line which buffer large portions of data before sending them out, it would generate "bursty" network traffic, which I want to avoid. Part of the reason I'm doing this is because I want to have more smooth control of network utilization so it doesn't impact other services or requests.. I had seen some notes about the content-length filter, for example, setting aside the entire response until it got the end of it, which if my filter was before it would completely defeat the behavior of my rate limiting.. > if you are basing the rate limiting on something on the request I would > suggest you write a request hook, (somewhere after the request headers > have been read.. forget the name for the moment) and make it set a note > in the connection record. (or maybe use the apr_pool_userdata_set(pool) > call it's faster) Ah, thank you.. Yeah, I realize now I should have been thinking in terms of a request hook rather than a filter for the decision-making process, but aside from that little detail this is basically what I was envisioning. I wasn't aware of apr_pool_userdata_set or connection record notes, I'll go look into that. It sounds like it should do very much what I'm looking for. One last question: Because it keeps track of what other requests are currently being served, my implementation needs to know when serving a request has been completed, as well. Obviously, this could pose some problems with coordination between when request-processing is considered finished and when the data actually goes out over the net. What I would really like to do is consider a request "finished" once the last of its data goes out. Is there an appropriate hook or something for doing stuff when this happens, or should I just look for an EOS or something go through my limiting filter and do the processing there? I'm still getting the hang of a lot of this architecture, but I'd like to do things The Correct Way(tm) if possible :) > The only potential downside I can with implementing it this way is if > you have 2 small requests which get sent out together, you will get the > rate limit of the 2nd one. As long as they're small, I don't think anybody will care that much, so I can live with that. Thanks a lot for the help, -alex
Questions about filter placement
Greetings all, A little while back I started working on a module to control bandwidth utilization with rate limiting, employing fairly flexible configuration and some other nifty features.. At the time it became clear fairly early on that it would be much easier to do this properly within the 2.0 API than the then-current 1.3 server, but the 2.0 API was still somewhat in flux, so after looking into things a bit I decided to hold off for a little while and come back to things later. Now that things are somewhat more solidified I'm coming back and looking at this project again, but I'm still running into a few questions with the current 2.0 API that I was hoping some folks here could provide some insight/suggestions on.. >From what I've read about the 2.0 API, this application (rate limiting of server output) seems perfect for an output filter. However, one of my first questions centers around where in the filter chain behavior such as this should go. Obviously, in order to function properly, this filter should come after any content-modifying filters in the output chain. Given the fairly low-level nature of what it's doing (controlling network traffic) it seems to me that it really makes most sense to put this somewhere in the AP_FTYPE_CONNECTION, or possibly even AP_FTYPE_NETWORK part of things, however it's my understanding that connection and network filters do not have access to any information about the request, and since the bulk of deciding what rate limits are to be imposed for a given hunk of data is going to be determined by per-resource configuration, this presents a problem. I could implement this as an AP_FTYPE_CONTENT_SET filter, which would make the most sense from a configuration and decision-making standpoint (since I have access to request information), but one of the questions I have about this is whether other buffering and such later in the filter chain (such as with transcode/connection/etc filters) would render any attempts at rate control at this level moot, or at least seriously degraded. Ultimately, what seems like the most correct solution I can see would be to implement this in multiple stages, with one content_set filter used to determine the appropriate limits for a given set of data, and then a later filter actually controlling the output rate of that data when it's closer to going out over the network. The problem with this is that I can't see any obvious way for the earlier filter to communicate what it's decided about the given data to the later stage for acting upon. Is there any way to "mark" data going out from one filter with information that can be used by a later filter when it gets that data to work on? Anyway, does anybody out there have suggestions on the best approach to use for this application? Where should code to control data flow rate go in the filter chain, and if that place isn't in a request-related filter, is there an easy way to make request-level decisions related to that filtering? Thanks, -alex PS: As this subject seems to be a hotbutton for some people (at least it was the last time I brought it up on this list), please don't just respond to this by saying "why would you want to do that?" or "bandwidth limiting doesn't belong in the httpd server!" or "people have already made things that do that" or similar things. I've had all these discussions before, and I have good reasons for making this module that I'll be happy to explain to people in private if you really want to understand where I'm coming from with this, but ultimately it boils down to: Yes, this is required to do things which you really can't do other ways and is potentially very useful. No, nobody else has already done it with the capabilities I'm looking for (I've checked).
Re: Q1: Rollup Release Format - Score So Far...
Rodent of Unusual Size wrote: > Graham Leggett wrote: > >>But consensus has just been reached that there will be a >>single rollup release, so out of necessity there will >>have to be one version per release. >> > > That is a consensus that was built quite quickly, so it > is certainly non-binding if new data suggest it is not > the best alternative. > Just for clarification here, I would like to point out that I'm all in favor of the apparent consensus regarding the single rollup release, and nothing in my response was intended to change that in any way. It had appeared that, given the consensus on the "what" (single rollup), people had started to move on to the "how" (CVS tagging and rollup procedures), and I was responding to some of the details of that topic. My point was really that _given_ there's going to be a single rollup with a particular release number, that rollup release number doesn't necessarily have to be tied to the version numbers of the multiple components that are in it, and it might be easier if it wasn't. Sorry if there was any confusion.. -alex
Re: Q1: Rollup Release Format - Score So Far...
Graham Leggett wrote: > Alex Stewart wrote: >>There seems to be a big assumption here that "release" is the same as >>"version", which seems like an unnecessary restriction. >> >>Frankly, if these are separate subprojects we're talking about (which it >>seems pretty clear they're going to be evolving into, if they aren't >>already), they should have separate, independent versioning. >> > > But consensus has just been reached that there will be a single rollup > release, so out of necessity there will have to be one version per > release. Why? It's the necessity for the one-to-one mapping between versions and releases that I'm questioning. I don't see the requirement. My point is that there is (and should be) a difference between _release_ numbering and _version_ numbering. The same version of a module may go in multiple releases (if nothing's changed in that particular bit), so why change the module's version number just because it's being packaged again? Likewise, why restrict module versioning such that its version can't change unless there's another rollup (or worse yet, its version can't change unless there's a new httpd released)? >>Trying to >>coordinate the version numbers of umpteen different projects just >>because one of their possible distribution channels distributes them >>together is silly and a lot of unnecessary extra work. >> > > We are currently coordinating three different projects (httpd-core, apr, > apr-util) being released together and things are working fine. I don't > see how expanding this to 4 or 5 is such a problem? Well, your previous message demonstrated one reason: It requires a lot more coordination (the "enormous trumpet call") to make sure things are consistent at rollup time, and there's no advantage (that I see) gained from it. It also doesn't scale well at all. (As somebody who's designed and administrated a few different large-scale CVS-based software release systems, I'm speaking from personal experience on that bit.) In the short term, we may not be scaling to more than 4 or 5 projects, but I don't see why we should deliberately limit ourselves to that either, particularly since there's the potential for splitting this whole thing out into quite a few more groups (or bringing more things into the fold) later on if people decide it's worth it. >>I agree with the global tagging thing, but I don't see why this much >>effort has to be put into making everything ready concurrently just so >>it can be rolled together. Automatic coordination of this sort of thing >>is part of what CVS (and in particular CVS tags) is supposed to be good for. >> > > "Making everything ready" just means "make sure it's not currently > broken". This is exactly how we do things now, I don't think anything > should change. Except that you're going to get multiple semi-independent groups working on multiple internal timelines and all of a sudden you have to hold off the release of module A because module B's got a big problem that'll take a few days to fix, then by the time module B is fixed, module C has a problem, and when everything finally gets straightened out, something you could have gotten out the door in an hour has taken a week and a half. >>It seems to me that each subproject should attempt to maintain at all >>times a tag that says "current rollup-candidate", which isn't >>necessarily the latest-and-greatest, but is the latest version that's >>stable and without showstoppers. [Actually, I should have said "it's a _recent_ version that's stable and without showstoppers".] > I suggested this a while back - but after thinking about it some more I > realised this just means extra work. Instead of tagging it once when the > trumpet call is released, we must now update the latest-known-working > tag every time we make a commit - yuck. Umm, no. All it means is that each group maintains its own release schedule, and updates its "releasable" tag appropriately for their schedule. This doesn't have to be every commit, it could be every day, or every week, or whenever somebody feels like it (and it _can_ be that flexible, because each group doesn't have to drop everything and coordinate with everybody each time somebody wants to update things). -alex
Re: Q1: Rollup Release Format - Score So Far...
Graham Leggett wrote: > mod_foo wants to make a release, so they release v2.0.45.1 of the rollup > tree, containing 2.0.45 of core and 2.0.45.1 of mod_foo. But what about > mod_bar and the other modules? Will their tags need to be bumped up to > 2.0.45.1 also? I would imagine they would, which is a problem. There seems to be a big assumption here that "release" is the same as "version", which seems like an unnecessary restriction. Frankly, if these are separate subprojects we're talking about (which it seems pretty clear they're going to be evolving into, if they aren't already), they should have separate, independent versioning. Trying to coordinate the version numbers of umpteen different projects just because one of their possible distribution channels distributes them together is silly and a lot of unnecessary extra work. At the same time, saying that we can't have a specific bundle release number because all the contents have different versions is equally silly. The bundle release number reflects the number of the bundle, not necessarily the version of any of the contents. Well, ok, it makes sense that the rollup bundle of "httpd and friends" should reflect the version number of the httpd core that's in it (that's the one version number that most people on the outside would probably expect to be consistent). It also makes sense that there may be incremental bundles released between httpd version changes, so a sub-release identifier of some sort is needed. The number on the bundle, however, in no way has to have any relationship to any of the extra module version numbers: For example, apache-httpd-complete-2.0.1-12.tar.gz might contain: httpd version 2.0.1 (obviously) mod_foo version 2.0.1 mod_bar version 1.7 mod_baz version 18.7.3 apache-httpd-complete-2.0.1-13.tar.gz could contain exactly the same thing, except mod_bar is now at version 1.8, or whatever. Now, admittedly, you could do the same thing with a date stamp instead of a revision number, but for these purposes "12" works just as well as "20020423", and is arguably more readable/usable (for one thing, you can tell that "13" is the next release after "12", but who knows what "20020611" is). Anyway, the filenames we're looking at using are getting long enough already, IMO. > Ideally the rollup release should commence with an enormous trumpet > call, followed by the tagging of *all* the modules (including core) with > the same tag. At this point *all* modules (including core) have to fix > any showstoppers, and a release follows shortly afterwards to testers. > If the release works, woohoo - it's a release. If not, oh well, it's an > alpha. I agree with the global tagging thing, but I don't see why this much effort has to be put into making everything ready concurrently just so it can be rolled together. Automatic coordination of this sort of thing is part of what CVS (and in particular CVS tags) is supposed to be good for. It seems to me that each subproject should attempt to maintain at all times a tag that says "current rollup-candidate", which isn't necessarily the latest-and-greatest, but is the latest version that's stable and without showstoppers. At any point in time (any day of any week) and with no special warning, somebody should ideally be able to pull from all the appropriate CVS sources using that tag and get something that's appropriate to be made into a rollup tarball. When a subproject has an update worthy of a new rollup, they tag it with that tag in their tree, and ask whoever's in charge of rolling releases to do another run. At that point, a general notice might go out just so that everyone can do a quick double-check that what's tagged in their repositories is the stuff they really want going out, and then it gets pulled and rolled. No big fanfare or mad scrambling needed, though. Anyway, that's my $.02.. -alex
Re: cvs commit: httpd-2.0/server/mpm/worker worker.c
On a largely unrelated note, but something I found a little ironic given the nature of this list: Aaron Bannert wrote: > http://www.apachelabs.org/apache-mbox/199902.mbox/<[EMAIL PROTECTED]> Please note that the above is not a valid URL. Specifically, the "<" and ">" characters are technically not allowed in URLs and must be escaped. (I bring this up partly because in Mozilla, this message came up with two links, a http: link to 199902.mbox, and a mailto: link to [EMAIL PROTECTED], so I had to do some cutting and pasting to actually see the right document) Whoever does the software behind apache-mbox (I take it this is mod_mbox?) might want to take note that it's spitting out invalid URLs.. -alex
Re: another map_to_storage gotcha.
Slive, Joshua wrote: > 1. (Important) As OtherBill has been trying to point out, is > applied after . Therefore, > if you put these things in , lots of things in will > fail to work. People won't understand why > this doesn't deny access to anything: > > > Order allow,deny > allow from all > > > deny from all > And, IMO, this is just plain wrong, and needs to be fixed. It should never be possible for to override with looser access restrictions, just as it should not be possible for to override with looser permissions. In both cases, access should be determined by the most restrictive specification for a given resource. Doing anything else opens up lots of opportunities for accidental security holes and is just bad design. -alex
Re: [PATCH] performance patch for mod_log_config
Brian Pane wrote: > William A. Rowe, Jr. wrote: > >> Is this the right place to be caching, or should this become a >> straightforward >> optimization to apr's time.c functions? I'd think the advantages are >> many for >> keeping 15 current seconds in apr, and would pay off across the >> board. Within >> apr, we can always recalculate just the ms as well, for fun. >> > I think putting it in APR would work. The one limitation I can think of > is that adding the cache in apr_explode_localtime() itself wouldn't be a > win because we'd have to add the overhead of a gettimeofday() call to > check whether the supplied time was indeed current (and thus susceptible > to caching). I'm not that familiar with the usage patterns of this function in Apache et al, but it seems to me that a gettimeofday() call could be avoided reasonably well by simply tracking the previously-used time argument. If the current call is for a time which is later than the one of the last call (within a certain margin), cache it. This should be just about as effective as checking gettimeofday() (possibly moreso, as it would also match anything within a reasonable proximity of the current time). It might miss one or two if there are long periods without a call, but in that situation it's arguable that the speed of this function probably isn't the largest concern anyway. Just a thought.. -alex
Re: Bandwidth control
Charles Randall wrote: > You may want to look at the Zeus server to understand the features > that were product-worthy as one example. It appears that they've only > implemented this at the virtual server level. Good suggestion.. I don't think it'll be any harder with my current model to implement limits for directories, users, hosts, etc than it would be for just vhosts, though, so I'm not planning on limiting things unnecessarily (besides, part of what I really want for my own use are limits per directory and/or location).. >As you're probably aware, thttpd also provides flexible bandwidth >throttling, > >http://www.acme.com/software/thttpd/thttpd_man.html#THROTTLING Actually, no I wasn't, so thanks for the pointer. I'll definitely look at what they're doing. At first brief glance it looks like what I've got already is probably slightly more flexible, tho. Ian Holsman wrote: > you may want to contact the writers of the apache 1.3 module > (mod_bandwidth) and see what they are doing. > (http://www.cohprog.com/mod_bandwidth.html) > there is also a mod_throttle I think. Yeah, I've already looked at both mod_bandwidth and mod_throttle. mod_throttle (if I remember right) just inserts pauses between requests, so it's not the sort of thing I'm interested in anyway. mod_bandwidth does have actual rate-limiting, but their rate-per-request calculations appear to produce some odd results in many cases, and because it's implemented as a content handler, it can't be used with CGIs, server-parsed content, etc, so it won't work well for a lot of things. You're right that I should check with the authors to see if they're working on anything for 2.0, tho. I'll do that. -alex
Re: Bandwidth control
Charles Randall wrote: > Often times, this can be done easier at the OS level. What OS are you using? Linux, and I am aware of various kernel-level controls for traffic shaping, etc, however if you can tell me how to enforce different bandwidth limits for different name-based vhosts, different directories, etc, then I'm all ears, but I suspect I can also give you pretty good reasons why even if such decisions could somehow be done at the OS level, they really shouldn't be. -alex
Bandwidth control
Greetings all, I have recently been looking around for a means to control bandwidth utilization of my Apache server. I have in the process found three or four modules which attempt to do this in various ways, all of which seem to be less than ideal to me in one or more areas, and of course lots of general responses by various people suggesting using traffic shaping in the linux kernel, etc, which is also less-than-ideal for my applications.. I am therefore looking at writing my own implementation of general-purpose bandwidth control for Apache. I have noticed that the framework of the 1.3 server is apparently not well suited to generic output filters, which would mean that to do this in a comprehensive way there would probably require some hacks to the core of the server and wouldn't be possible just as a plain module, but I have also noticed that the 2.0 line apparently has much better support for exactly this sort of module, so I'm looking at starting with a module for 2.0 and then possibly backporting to 1.3.. For general information, I'm looking at making something with the following features: * Real-time rate-limiting of output (not just sticking delays between requests) * Hard (instantaneous rate) or soft (average rate over request) limiting. * Ability to control total bandwidth used by the whole server, individual vhosts, locations, or directories. * Support for limits based on source IP/domain, user, time of day, day of week, etc. * Multiple limits with varying criteria, all simultaneously enforced. * Minimum bandwidth setting which, when hit, maximum limits will be stretched to accomodate or requests will be turned away (configurable). I've already got a basic model for limit calculations working, and have a good idea how to go about most of the rest of it, I think, but suggestions are welcome.. Before I get deeply into this, however, I do have a couple of questions which I hope some folks here could help out with: 1. Are my analyses of the architecture issues with 1.3/2.0 basically correct, or am I missing something obvious? (I'm relatively new to all this code so I may be overlooking something) 2. Is there in general some reason why bandwidth control is not already built into Apache httpd (philosophical/technical/political issues), or is it just that nobody has gotten around to doing it? If the latter, would people be adverse to possibly including this stuff into the main tree down the line? 3. Are other people already working on this who I'm not aware of and would be duplicating effort with? 4. I've looked around the various apache web sites, source tarballs, etc, and found a bit of documentation, but noticed that there isn't a lot of developer docs for the 2.0 architecture lying around yet.. This is, understandable as I do realize it's still seriously under development, but I just wanted to throw out a brief query to see if anybody has any pointers to less-obvious sources of information (notes, scribbles, partially-finished whatevers) that might be helpful for a prospective 2.0 module developer.. Alternately, any big gotchas I should watch out for or general advice is welcome, too. 5. If anybody else out there is interested in this sort of thing and has specific features or things they'd like me to work into what I'm doing, I'm open to suggestions. Anyway, I basically wanted to pop my head up, say hi, let people know what I was contemplating, and get any input people out there might have before proceeding, so if anybody's got any thoughts on this, please let me know.. -alex
