Re: Session Benchmarks
On Fri, 17 Jun 2005, Nicolas Lehuen wrote: As for the MySQL implementation I'd stay away from anything vendor-specific in mod_python, because then the question becomes why not a postresql, why not oracle, etc. Grisha
Re: Session Benchmarks
Gregory (Grisha) Trubetskoy wrote: On Fri, 17 Jun 2005, Jim Gallacher wrote: I was thinking we'd still use the current global locking scheme, but keep the file open between requests. Not sure if this would be robust or just asking for dbm file corruption though. I'm pretty sure it won't work, I think you cannot have a dbm open by more than one process without getting it corrupted. Well of course not. Just a little brain fart on my part. Jim
Re: Session Benchmarks
Gregory (Grisha) Trubetskoy wrote: On Fri, 17 Jun 2005, Jim Gallacher wrote: Any objection to just a SqlSession base class? May be - it depends on how complex it becomes. Any attempts I've to generalize SQL/DB stuff tend to become a mess since there are no firm standards in this area, so this just may be something for the contrib (or whatever it ends up being), like Nicolas pointed out. I'm a little concerned about staying focused on closing the last bugs so that we get to a point where a release can be made, since there hasn't been one in such a long time... I was not actually proposing a SqlSession class for 3.2.0. I just want to finish off the last couple of issues of FileSession for this release, and catalog the SqlSession idea as a possible future feature. It really would be helpful if everyone could read minds. Maybe we could put a mind-reading feature in 3.3? :) Although that might push 3.3's release back to 2020. And could it be made thread safe... Regards, Jim
[ANNOUNCE] Apache-Test 1.25
The uploaded file
Apache-Test-1.25.tar.gz
has entered CPAN as
file: $CPAN/authors/id/G/GO/GOZER/Apache-Test-1.25.tar.gz
size: 145772 bytes
md5: 315998641c7ba5d3a315da96c10e64d1
Changes since 1.24:
provide $Apache::TestUtil::DEBUG_OUTPUT as target for t_debug()
statements, defaulting to STDOUT. this allows for changing
t_debug() to STDERR when using functions like t_write_file()
from within handler() server-side tests. [Geoffrey Young]
adjust need_module()/have_module() to not try to require a module if
it was explicitly passed with a .c extension. in certain cases this
prevents a fatal error (e.g. trying to call
Apache::Test::have_module('mod_alias.c') from the Perl
sections. [Stas]
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/ F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5
2.1.5 available for testing
Please test and vote on releasing 2.1.5 as -alpha. Available at: http://httpd.apache.org/dev/dist/ (might take up to 2 hours for the files to appear, due to the rsync delay) MD5 (httpd-2.1.5-alpha.tar.Z) = f9dea893723fe7ddb8e9c5a4225a276b MD5 (httpd-2.1.5-alpha.tar.bz2) = 8ec926f27c2768c7cbd24f42fa22108a MD5 (httpd-2.1.5-alpha.tar.gz) = 7fdc3d61deb72b3815e137bdae282f0e Thanks, Paul
Re: 2.1.5 available for testing
Paul Querna wrote: Please test and vote on releasing 2.1.5 as -alpha. +1 from me, Tested on FreeBSD 6.0-CURRENT and NetBSD 2.0. -Paul
[error] (88)Socket operation on non-socket:
Hi, On my old Suse (7.2 (i386)) httpd (2.1 head) is not running and in the error_log I have the following: +++ [Fri Jun 17 13:58:50 2005] [error] (88)Socket operation on non-socket: apr_socket_accept: (client socket 0) +++ This only appends when starting http via crontab. Does it make sense to open fd 0, 1 and 2 in httpd to prevent such problems (in apr_proc_detach() for example)? Cheers Jean-Frederic
Keepalives
Title: Keepalives Here's the problem: If you want to use keepalives, all of you workers (threads/procs/whatever) can become busy just waiting on another request on a keepalive connection. Raising MaxClients does not help. The Event MPM does not seems to really help this situation. It seems to only make each keepalive connection cheaper. It can still allow all workers to be blocking on keepalives. Short Term solution: This is what we did. We use worker MPM. We wrote a simple modules that keep track of how many keeapalive connections are active. When a threshold is reached, it does not allow anymore keepalives. (Basically sets r-connection-keepalive = AP_CONN_CLOSE). This works for us, but the limit is per process and only works for threaded MPM's. Long Term solution: Keep track of keepalives in the scoreboard (or somewhere else). Allow admin's to set a threshold for keepalives: MaxClients 1024 MaxConcurrentKeepalives 768 Or something like that. Thoughts? I am willing to write the code if this seems desirable. Should this just be another module or in the http core? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Keepalives
At 08:11 AM 6/17/2005, Akins, Brian wrote: If you want to use keepalives, all of you workers (threads/procs/whatever) can become busy just waiting on another request on a keepalive connection. Raising MaxClients does not help. No, it doesn't :) But lowering the keepalive threshold to three to five seconds does. We are lowering the 'example' keepalive timeout in the next releases. Keepalives as originally implemented were to help users with loading additional images (and now, css stylesheets.) And the time was set to allow a user to grab 'another page' if they quickly clicked through. But the later use is not a good use of child slots, and the former use case has a -much- lower window these days, as most browsers are quite fast to compose the base document and determine what images are required. With the relative disappearance of 1200baud dialup, 15 seconds for the client to sit and think about grabbing more documents to compose the current page is very silly :) The Event MPM does not seems to really help this situation. It seems to only make each keepalive connection cheaper. It can still allow all workers to be blocking on keepalives. Short Term solution: This is what we did. We use worker MPM. We wrote a simple modules that keep track of how many keeapalive connections are active. When a threshold is reached, it does not allow anymore keepalives. (Basically sets r-connection-keepalive = AP_CONN_CLOSE). This works for us, but the limit is per process and only works for threaded MPM's. Long Term solution: Keep track of keepalives in the scoreboard (or somewhere else). Allow admin's to set a threshold for keepalives: MaxClients 1024 MaxConcurrentKeepalives 768 Or something like that. Thoughts? I am willing to write the code if this seems desirable. Should this just be another module or in the http core? If you experiment with setting the keepalive window to 3 seconds or so, how does that affect your test? Also, I'd be very concerned about additional load - clients who are retrieving many gifs (with no pause at all) in a pipelined fashion will end up hurting the over resource usage if you force them back to HTTP/1.0 behavior. Bill
Re: Keepalives
William A. Rowe, Jr. wrote: No, it doesn't :) But lowering the keepalive threshold to three to five seconds does. For us, in heavy loads, that's 3-5 seconds that a thread cannot process a new client. Under normal circumstances, the 15 seconds is fine, but when we are stressed, we need to free threads as quickly as possible. Also, I'd be very concerned about additional load - clients who are retrieving many gifs (with no pause at all) in a pipelined fashion will end up hurting the over resource usage if you force them back to HTTP/1.0 behavior. Yes, but if all threads are waiting for x seconds for keepalives (even if it is 3-5 seconds), the server cannot service any new clients. I'm willing to take an overall resource hit (and inconvenience some clients) to maintain the overall availability of the server. Does that make any sense? It does to me, but I may not be explaining our problem well. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Keepalives
Akins, Brian wrote: Short Term solution: This is what we did. We use worker MPM. We wrote a simple modules that keep track of how many keeapalive connections are active. When a threshold is reached, it does not allow anymore keepalives. (Basically sets r-connection-keepalive = AP_CONN_CLOSE). This works for us, but the limit is per process and only works for threaded MPM's. Could that be done dynamically? As in, make the max keepalive time a function of how near the server is to running out of spare workers? Oh, and is the default still ridiculously high? ISTR it being 15 secs at one time - not sure if that ever got changed. Also, have you looked into making keepalive dependent on resource type? E.g. use them for HTML docs - which typically have inline contents - but not for other media types unless REFERER is a local HTML page? Long Term solution: Keep track of keepalives in the scoreboard (or somewhere else). Allow admin's to set a threshold for keepalives: MaxClients 1024 MaxConcurrentKeepalives 768 Or something like that. Thoughts? I am willing to write the code if this seems desirable. Should this just be another module or in the http core? Is that a candidate application for the monitor hook? Other things being equal, I'd make it a module. -- Nick Kew
Re: Keepalives
Akins, Brian wrote: Here's the problem: If you want to use keepalives, all of you workers (threads/procs/whatever) can become busy just waiting on another request on a keepalive connection. Raising MaxClients does not help. The Event MPM does not seems to really help this situation. It seems to only make each keepalive connection cheaper. It can still allow all workers to be blocking on keepalives. If the event MPM is working properly, then a worker thread should not be blocking waiting for the next ka request. You still have the overhead of the tcp connection and some storage used by httpd to manage connection events but both of those are small compared to a blocking thread. Short Term solution: This is what we did. We use worker MPM. We wrote a simple modules that keep track of how many keeapalive connections are active. When a threshold is reached, it does not allow anymore keepalives. (Basically sets r-connection-keepalive = AP_CONN_CLOSE). This works for us, but the limit is per process and only works for threaded MPM's. Long Term solution: Keep track of keepalives in the scoreboard (or somewhere else). Allow admin's to set a threshold for keepalives: MaxClients 1024 MaxConcurrentKeepalives 768 Or something like that. Thoughts? Both approaches sound pragmatic (+.5) although I would like to think the best long term solution is to completely decouple TCP connections from worker threads. The event MPM is an experiment in that direction but it still has a long ways to go. Earliest I could see this happening is in the v 2.4 timeframe. Bill
Re: Keepalives
Nick Kew wrote: Could that be done dynamically? As in, make the max keepalive time a function of how near the server is to running out of spare workers? Sure. I'd have to poke around a bit to see the best way to do it. Speed is of utmost concern for us. I guess I could dynamically change r-server-keep_alive_max or r-server-keep_alive_timeout? Maybe make the timeout a sliding timeout something like: /*calculate max_clients by querying mpm*/ /*is there a good, fast way to get idle workers?*/ /*store keep_alive_timeout somewhere*/ r-server-keep_alive_timeout = keepalive_timeout / (max_clients / idle_workers); Thoughts? Also, have you looked into making keepalive dependent on resource type? E.g. use them for HTML docs - which typically have inline contents - but not for other media types unless REFERER is a local HTML page? Sounds horribly slow... Also, in our case, HTML and other content come from separate server pools. But most pages are made up of a few HTML pages. (You have to look at the HTML source to see what I mean). Also, we have some app servers that often have all connections tied up in keepalive because the front ends open tons of keepalives (I have no direct control of them). I was hoping for a more generic solution that would maybe help others. I'm sure there are others with similar situations. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Keepalives
At 09:27 AM 6/17/2005, Brian Akins wrote: Also, I'd be very concerned about additional load - clients who are retrieving many gifs (with no pause at all) in a pipelined fashion will end up hurting the over resource usage if you force them back to HTTP/1.0 behavior. Yes, but if all threads are waiting for x seconds for keepalives (even if it is 3-5 seconds), the server cannot service any new clients. I'm willing to take an overall resource hit (and inconvenience some clients) to maintain the overall availability of the server. Does that make any sense? It does to me, but I may not be explaining our problem well. Yes it makes sense. But I'd encourage you to consider dropping that keepalive time and see if the problem isn't significantly mitigated. We have a schema today to create 'parallel' scoreboards, but perhaps in the core we should offer this is a public API to module authors, to keep it very simple? I believe keepalive-blocked read should be able to be determined from the scoreboard. As far as 'counting' states, that would be somewhat interesting. Right now, it does take cycles to walk the scoreboard to determine the number in a given state (and this is somewhat fuzzy since values are flipping as you walk along the list of workers.) Adding an indexed list of 'counts' would be very lightweight, and one atomic increment and decrement per state change. This would probably be more efficient than walking the entire list. In any case, I would simply extend counts for all registered request states in the scoreboard, rather than a one-off for every state someone becomes interested in. Bill
Re: 2.1.5 on Friday
Reply seemd to be missing :( So i'll post again. First i updated awk.exe, i seem to have had the same one as online, i updated anyway to be sure. This time i wanted to do a full build with zlib and ssl. First Problem: SSL build commands as described in the documenation didn't work. These commands however did: -- perl Configure --openssldir=C:/Apache21/bin VC-WIN32 ms\do_masm nmake -f ms\ntdll.mak -- once that was compiled i unpacked apr-iconv into the correct folder. I then opend apache.dsw in vs.net and let it convert it to a sln file. I then tried to build apache from cmd(using vs.net command promt). I got some errors on apr, aprutil and apriconv, after removing the ending , is the rc files for those library's it went past that stage so all was good. Untill i hit zlib, it didn't compile it kept complaining about zlib.lib. (i tried zlib 1.1.4 and zlib 1.2.1) I then went into srclib\zlib\projects\visualc6 and opened zlib.dsw. Then i build LIB Release it generated zlib.lib which i moved to to srclib\zlib. Now the building contineud on till the file copieing part. there it was stuck. It seems giving \apache21 as install dir didn't go to wel. once i made it c:\apache21 the build went fine! Now all seemed to be fine. The config files where created and no error's. I tried writing a simple perl script to delete all the pdb files in the install dir but my perl skill's aren't so great and I faild. So I manual delete them to save diskspace. I then edited the listen directive to be 8080 so I could test it a bit along side my other apache instance. It started without a problem. So asside from the problems whit the rc's of the apr projects and the zlib problems all seems to build fine out of the box without any problems(well windows users need to run lineends.pl first) I hope this information is usefull. I would be more than willing to do more testing if needed. Jorge --- In [EMAIL PROTECTED], William A. Rowe, Jr. [EMAIL PROTECTED] wrote: At 04:05 AM 6/14/2005, Paul Querna wrote: I am volunteering to RM 2.1.5 on this Friday, June 17. We have had several big changes and fixes from the last release of 2.1.3 on 2/22/2005. I am planning to bundle APR 1.1.1 and APR-Util 1.1.2. If the APR project has new releases, would you please consider those, instead? There is -alot- of activity on the bug-fixing side of that world, especially in network APIs. I would like to be able to put up win32 binaries, just to get a wider testing group. Would anyone who could do those be able to? Happy to. I know there are still some unresolved issues with APR-Iconv for win32 binaries, but there are so many other areas in httpd that have changed, just testing on the win32 platform would be helpful. These issues are addressed by a patch I'll be committing later today for apr (core). apr_dso_filepath_get() will track down that libapriconv-1.so/.dll/.dylib/.sl filepath we were hunting for. In any case, if this is an alpha, I have no hesitation in supporting another round of testing and review. I do have a huge issue if the project tags any flavor before addressing the HTTP Request flaws, but your Friday window gives folks an opportunity to investigate those. Bill
Re: Keepalives
Bill Stoddard wrote: If the event MPM is working properly, then a worker thread should not be blocking waiting for the next ka request. You still have the overhead of the tcp connection and some storage used by httpd to manage connection events but both of those are small compared to a blocking thread. Should there be an upper limit on how many connections to have in keepalive, even when using event? Say you have 100 worker threads, you wouldn't want to have 8192 keepalive connections. So you would want some limit. Both approaches sound pragmatic (+.5) although I would like to think the best long term solution is to completely decouple TCP connections from worker threads. I really like the event mpm, but I still think there has to be an upper limit on how many connections to allow to keepalive. is an experiment in that direction but it still has a long ways to go. Earliest I could see this happening is in the v 2.4 timeframe. We've been doing some testing with the current 2.1 implementation, and it works, it just currently doesn't offer much advantage over worker for us. If num keepalives == maxclients, you can't accept anymore connections. I want to be able to limit total number of keepalives. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Keepalives
William A. Rowe, Jr. wrote: Yes it makes sense. But I'd encourage you to consider dropping that keepalive time and see if the problem isn't significantly mitigated. It is mitigated somewhat, but we still hit maxclients without our hack in place. Right now, it does take cycles to walk the scoreboard to determine the number in a given state (and this is somewhat fuzzy since values are flipping as you walk along the list of workers.) I know the worker MPM, for example, keeps a count of idle workers internally. Maybe just an mpm query to retrieve that value would be good? all MPM's keep track of this in some fashion because they all know when maxclients is reached. Adding an indexed list of 'counts' would be very lightweight, and one atomic increment and decrement per state change. This would probably be more efficient than walking the entire list. Sounds good. Of course, when changing from on state to another you would always have to decrement the previous state and increment the new one. The way the core seems to be now, that would require some careful examination of the code to ensure all the state changes were covered. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: HTTPD 2.1 (Head) Build issues for NetWare...
At 11:25 AM 6/16/2005, Greg Marr wrote: At 12:01 PM 6/16/2005, William A. Rowe, Jr. wrote: Back to httpd land; the question is --- is this the right choice for *our tarballs*? Which may or may not be related to the question above. In any case; this is useful metadata even for end users who build the package for the reasons I mentioned; does anyone have a desire/justification to lose the commit dates and use the RM's checkout date? It seems to me that modification or commit date/time is the right choice for *tarballs*. Ok. Other folks have thoughts about this? Checkout date/time is generally the right choice for developers, because otherwise make doesn't always pick up when a file has changed. (I've been bit by the Visual SourceSafe modification time default enough times.) Although - heh - you are assuming the order of the checked out files happen to be stacked so that the dependency happens to be a fraction of a second older than the target :) To clarify, the last tarballs WERE checkout/export time stamped. Bill
Re: Session Benchmarks
2005/6/17, Jim Gallacher [EMAIL PROTECTED]:
Nicolas Lehuen wrote:
Hi Jim,
You've done a pretty impressive work here. What surprises me is the
O(n) behaviour on DBM and FS. This seems to mean that indexes (or
indices, if you prefer) ar not used.
ext2/ext3 uses a linked list to access files, hence O(n) when adding a file.
Duh... And they call that a filesystem :P. That's where ReiserFS, the
WinNT FS and other modern FS shine : they use more efficient data
storage (I think they use BTree indices).
For DBM, well, if BDB could not handle indexes, this would be big
news. Are you 100% sure that the Berkeley implementation is used ?
I used dbhash, which according to the python docs is the interface to
the bsddb module. The code is pretty much the same as in DbmSession.
Code snippet is at the bottom of this message. Running /usr/bin/file
bsd.dbm gives:
bsd.dbm: Berkeley DB (Hash, version 8, native byte-order)
Brain Wave
It just occured to me that the performance problem may be related to
opening and closing the dbm file for every record insertion. Adjusting
the test so that the file is only opened once, I get O(1), and a great
speed boost: 0.2 seconds / per 1000 records all the way up to 50,000
records. At that point I start to see period performance hits due to
disk syncing, but that is to be expected.
I have no idea what to do with this knowledge unless we can figure out a
way to keep the dbm file open across multiple requests. Ideas??
End of Wave
Well, we could keep a permanent reference to the opened dbm file at
the module level and make sure that locks are used so that everything
works correctly in a threaded MPM environment ?
For FS, I don't know about ext3, but in ReiserFS or the Win NT
filesystem, there are indexes that should speed up file lookups, and
should certainly not yield a O(n) performance.
Don't forget, I only benchmarked creating new session files. Reading, or
writing to existing files may be an entirely different matter. Certainly
one of the benefits of ReiserFS is that it can handle a large number
of small files in an efficient manner.
Anyway, implementing
FS2 instead of FS is not that difficult, and if it yields predictable
results even on ext3, then we should go for it.
Already done - it's just a couple of extra lines. Doing some testing today.
Are you replacing FS with FS2 or adding a new implementation ? I think
replacing is better, since I can't see any drawback to the FS2
approach.
As for the MySQL implementation, well, I've been promising it many
times, but I can provide a connection pool implementation that could
speed up applicative code as well as your session management code.
What I would need to do is to make it mod_python friendly, i.e. make
it configurable through PythonOption directives. Do you think it would
be a good idea to integrate it into mod_python ?
Connection pooling seems like a common request on the mailing list, so
I'd say yes.
I'll see what I can do. The weather is very promising for this
week-end, so I don't think I'll have much time to stay and write some
code (all that given the fact that it's part of my day job ;), but who
knows ?
Regards,
Nicolas
Code snippet from my benchmark script.
import dbhash
def create_bsd(test_dir, test_runs, number_of_files, do_sync=False):
if not os.path.exists(test_dir):
os.mkdir(test_dir)
dbmfile = %s/bsd.dbm % (test_dir)
dbmtype = dbhash
i = 0
timeout = 3600
count = 0
total_files = 0
results_file = '%s/bsd.%02d.results' % (test_dir,i)
results = open(results_file,'w')
write_header(results, 'dbhash', test_runs, number_of_files)
while count test_runs:
start_time = time.time()
i = 0
while i number_of_files:
sid = _new_sid()
data = {'_timeout': timeout,
'_accessed': time.time(),
'stuff': 'test test test test test'
}
# dbm file is opened and closed for each insertion
# which is the same as the current DbmSession
# implentation
dbm = dbmtype.open(dbmfile, 'c')
dbm[sid] = cPickle.dumps(data)
dbm.close()
i += 1
total_files += 1
count += 1
elapsed_time = time.time() - start_time
print_progress(count, i, elapsed_time)
write_result(results,count, i, elapsed_time, total_files)
results.flush()
if do_sync:
s
Re: Session Benchmarks
Nicolas Lehuen wrote: Anyway, implementing FS2 instead of FS is not that difficult, and if it yields predictable results even on ext3, then we should go for it. Already done - it's just a couple of extra lines. Doing some testing today. Are you replacing FS with FS2 or adding a new implementation ? I think replacing is better, since I can't see any drawback to the FS2 approach. You could say replacing, but I prefer to think of it as bug fixing. The changes for do_save() and do_load() look like this: - filename = os.path.join(self._sessdir, 'mp_sess_%s' % self._sid) + path = os.path.join(self._sessdir, self._sid[0:2]) + filename = os.path.join(path, self._sid) Regards, Jim
Re: Keepalives
. Snipping all the other issues, which are largely valid and do contain some good ideas Akins, Brian wrote: Here's the problem: If you want to use keepalives, all of you workers (threads/procs/whatever) can become busy just waiting on another request on a keepalive connection. Raising MaxClients does not help. The Event MPM does not seems to really help this situation. It seems to only make each keepalive connection cheaper. It can still allow all workers to be blocking on keepalives. Can you be more detailed on this? It really _should_ help, and my testing says it does. What are you seeing behavior wise? Any changes you would like made?
Re: 2.1.5 on Friday
Jorge, I've got lots of feedback and a few mea culpas below... At 09:46 AM 6/17/2005, Jorge Schrauwen wrote: Reply seemd to be missing :( So i'll post again. First i updated awk.exe, i seem to have had the same one as online, i updated anyway to be sure. This time i wanted to do a full build with zlib and ssl. First Problem: SSL build commands as described in the documenation didn't work. These commands however did: -- perl Configure --openssldir=C:/Apache21/bin VC-WIN32 ms\do_masm nmake -f ms\ntdll.mak -- I will review; note that the 'intended' target was srclib/openssl/, built in-tree. Note that the apache make install target should copy the required openssl dll. I'm guessing this is 0.9.7? once that was compiled i unpacked apr-iconv into the correct folder. When 2.1.5 is approved, I'll roll a zip which includes -iconv to save that hassle :) I then opend apache.dsw in vs.net and let it convert it to a sln file. I then tried to build apache from cmd(using vs.net command promt). I got some errors on apr, aprutil and apriconv, after removing the ending , is the rc files for those library's it went past that stage so all was good. Yes; known bug, I'll hotfix the distribution. Note that .zip files we distribute do include the .mak files, as well as the .rc files. Untill i hit zlib, it didn't compile it kept complaining about zlib.lib. (i tried zlib 1.1.4 and zlib 1.2.1) Zlib 1.2 is required, note 1.2.2 has some security fixes. This differs from httpd-2.0, which relied on zlib 1.1.4 (and which never suffered from the security issue.) I then went into srclib\zlib\projects\visualc6 and opened zlib.dsw. Then i build LIB Release it generated zlib.lib which i moved to to srclib\zlib. Like openssl, Win32's build expects this to be built in-tree, in srclib/zlib. Now the building contineud on till the file copieing part. there it was stuck. It seems giving \apache21 as install dir didn't go to wel. once i made it c:\apache21 the build went fine! Ewww... exactly which version of Windows? (You mention you were in a cmd shell, not the 'command.com' shell, right?) What's your COMSPEC variable set to? Now all seemed to be fine. The config files where created and no error's. whew :- I tried writing a simple perl script to delete all the pdb files in the install dir but my perl skill's aren't so great and I faild. So I manual delete them to save diskspace. Note; those are -very- useful to help figure out why Apache crashes, if it does! But skip perl... del *.pdb /s works very effectively :) I then edited the listen directive to be 8080 so I could test it a bit along side my other apache instance. If you add PORT=8080 to the install line, the makefile.win should do that for you :) It started without a problem. So asside from the problems whit the rc's of the apr projects and the zlib problems all seems to build fine out of the box without any problems(well windows users need to run lineends.pl first) I will examine the directions and bring them up-to-date w.r.t. the version 2.1. Yes, a few things changed, and I'll need to make sure that httpd 2.0 / 2.1 users are pointed at the correct flavor.
Re: Keepalives
At 10:12 AM 6/17/2005, Brian Akins wrote: Adding an indexed list of 'counts' would be very lightweight, and one atomic increment and decrement per state change. This would probably be more efficient than walking the entire list. Sounds good. Of course, when changing from on state to another you would always have to decrement the previous state and increment the new one. The way the core seems to be now, that would require some careful examination of the code to ensure all the state changes were covered. In that exact order :) Much better to 'under report' a given state, and consider that under reporting (if you sum during an update) to be a product of state changes. I think ++, -- would be much more misleading, since the server would be taking more actions than could possibly occur at once. Bill
BUG in apache?????
The problem shows itself with the following simple module.When the hook is of type ap_hook_post_read_request per dir configuration is not instantiated correctly and debug has always value -1. With other kinds of hooks the debug variable is correctly is instantiated with the various values found in the DIRECTORY LOCATION and FILES directives.
Is this a bug?Regards,Lucamodule AP_MODULE_DECLARE_DATA config_module;typedef struct __my { int debug;
} config_dir_t;
static const char *debug_cmd(cmd_parms *cmd, void *in_dc, const char *name){ config_dir_t *dc = in_dc;
dc-debug= atoi(name); return NULL;}
static void *create_dir_config_t(apr_pool_t *p,char *dummy){ config_dir_t *dc = (config_dir_t *)apr_pcalloc(p,sizeof(config_dir_t)); dc-debug = -1; return dc;}
static int find_code_page(request_rec *r){ config_dir_t *dc = ap_get_module_config(r-per_dir_config, config_module); fprintf(stderr,Config: %d \n,dc-debug);
fflush(stderr); return DECLINED;}
static const command_rec cmds[] ={ AP_INIT_TAKE1(debug, debug_cmd, NULL, OR_FILEINFO, debug config di ma),
{NULL}};
static void charset_register_hooks(apr_pool_t *p){ ap_hook_post_read_request(find_code_page, NULL, NULL, APR_HOOK_MIDDLE);}
module AP_MODULE_DECLARE_DATA config_module ={ STANDARD20_MODULE_STUFF, create_dir_config_t, NULL, NULL, NULL, cmds, charset_register_hooks};
Re: Bug in apache?
* luca regini wrote: The problem shows itself with the following simple module. When the hook is of type ap_hook_post_read_request per dir configuration is not instantiated correctly and debug has always value -1. With other kinds of hooks the debug variable is correctly is instantiated with the various values found in the DIRECTORY LOCATION and FILES directives. Is this a bug? No. This hook is run right after the virtual host was determined, before anything about files and directories is known (translate_name, which maps URLs to files, runs later). nd -- Umfassendes Werk (auch fuer Umsteiger vom Apache 1.3) -- aus einer Rezension http://pub.perlig.de/books.html#apache2
Re: Keepalives
Brian Akins wrote: Bill Stoddard wrote: If the event MPM is working properly, then a worker thread should not be blocking waiting for the next ka request. You still have the overhead of the tcp connection and some storage used by httpd to manage connection events but both of those are small compared to a blocking thread. Should there be an upper limit on how many connections to have in keepalive, even when using event? Say you have 100 worker threads, you wouldn't want to have 8192 keepalive connections. So you would want some limit. Both approaches sound pragmatic (+.5) although I would like to think the best long term solution is to completely decouple TCP connections from worker threads. I really like the event mpm, but I still think there has to be an upper limit on how many connections to allow to keepalive. is an experiment in that direction but it still has a long ways to go. Earliest I could see this happening is in the v 2.4 timeframe. We've been doing some testing with the current 2.1 implementation, and it works, it just currently doesn't offer much advantage over worker for us. If num keepalives == maxclients, you can't accept anymore connections. Interesting point; it's been a while since I looked at the event MPM but I thought (mistakenly) that maxclients accounting was adjusted to reflect max number of concurrently active worker threads rather than active tcp connections. I agree we need some kind of upper limit on the max number of TCP connections into a running instance of httpd, regardless of whether those connections are associated with a worker thread or not. Bill
Re: Session Benchmarks
Jim Gallacher wrote: It just occured to me that the performance problem may be related to opening and closing the dbm file for every record insertion. Adjusting the test so that the file is only opened once, I get O(1), and a great speed boost: 0.2 seconds / per 1000 records all the way up to 50,000 records. At that point I start to see period performance hits due to disk syncing, but that is to be expected. I have no idea what to do with this knowledge unless we can figure out a way to keep the dbm file open across multiple requests. Ideas?? Keeping the dbm file open over multiple requests is really no different than the connection pool problem. I did run into problems with corrupt databases when I kept the file open a very long time, but that was a couple years ago. This may not be a problem anymore. I'll have to test it and see for myself. As for the MySQL implementation, well, I've been promising it many times, but I can provide a connection pool implementation that could speed up applicative code as well as your session management code. What I would need to do is to make it mod_python friendly, i.e. make it configurable through PythonOption directives. Do you think it would be a good idea to integrate it into mod_python ? Connection pooling seems like a common request on the mailing list, so I'd say yes. There seem to be a lot of things that would be good to include in mod_python, as they are often asked questions on the mailing list, but I suggest a contrib or example dir to hold these things, since they're not likely to work for everybody in every situation without some tweaks. That way people can edit and use these things in their projects without having to worry about editing installed modules in site-packages/mod_python. Nick
Re: HTTPD 2.1 (Head) Build issues for NetWare...
At 10:42 AM 6/17/2005, William A. Rowe, Jr. wrote: Checkout date/time is generally the right choice for developers, because otherwise make doesn't always pick up when a file has changed. (I've been bit by the Visual SourceSafe modification time default enough times.) Although - heh - you are assuming the order of the checked out files happen to be stacked so that the dependency happens to be a fraction of a second older than the target :) Actually, I'm assuming that there are no checked in files that are dependents of other checked in files, (no generated files ever checked in) since that's also asking for trouble (been there, done that).
Re: BUG in apache?????
At 10:34 AM 6/17/2005, luca regini wrote: The problem shows itself with the following simple module. When the hook is of type ap_hook_post_read_request per dir configuration is not instantiated Not a bug; nobody has started to break down the incoming request yet to figure out any location. translate name and map to storage come afterwards. mod_security and so on would be useless. They would be unable to examine the unprocessed request and determine some value. FWIW - even if we did instantiate a value for the per dir config, it would be discarded soon after post read request. It's not the appropriate hook to look at that member of the request struct. Please review; http://httpd.apache.org/docs-2.1/developer/request.html and remember that post read request isn't even a request hook!!! It's a connection hook that happens before request processing. Bill
Re: Keepalives
Any interest/objections to added another MPM query AP_MPMQ_IDLE_WORKERS (or some other name) in worker.c, could just add this to ap_mpm_query: case AP_MPMQ_IDLE_WORKERS: *result = ap_idle_thread_count; return APR_SUCCESS; and in perform_idle_server_maintenance we would update ap_idle_thread_count. I can submit a patch if anyone thinks this has a chance of being committed. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: 2.1.5 available for testing
fredag 17 juni 2005 09.40 skrev Paul Querna: Please test and vote on releasing 2.1.5 as -alpha. Available at: http://httpd.apache.org/dev/dist/ (might take up to 2 hours for the files to appear, due to the rsync delay) MD5 (httpd-2.1.5-alpha.tar.Z) = f9dea893723fe7ddb8e9c5a4225a276b MD5 (httpd-2.1.5-alpha.tar.bz2) = 8ec926f27c2768c7cbd24f42fa22108a MD5 (httpd-2.1.5-alpha.tar.gz) = 7fdc3d61deb72b3815e137bdae282f0e Thanks, Paul +1 on Mandriva LE2005 -- Regards // Oden Eriksson Mandriva: http://www.mandriva.com NUX: http://nux.se
Re: 2.1.5 available for testing
+1 netware [EMAIL PROTECTED] Friday, June 17, 2005 1:40:50 AM Please test and vote on releasing 2.1.5 as -alpha. Available at: http://httpd.apache.org/dev/dist/ (might take up to 2 hours for the files to appear, due to the rsync delay) MD5 (httpd-2.1.5-alpha.tar.Z) = f9dea893723fe7ddb8e9c5a4225a276b MD5 (httpd-2.1.5-alpha.tar.bz2) = 8ec926f27c2768c7cbd24f42fa22108a MD5 (httpd-2.1.5-alpha.tar.gz) = 7fdc3d61deb72b3815e137bdae282f0e Thanks, Paul
Re: expat install libapreq2-2.0.5-dev
Glenn Gallien [EMAIL PROTECTED] writes: I'm having the same problem while installing libapreq2-2.0.5-dev on FreeBSD 5.4. Error: /usr/bin/ld: cannot find -lexpat Apache 2 and mod_perl are installed from souce. Libtool, autoconf and automake where installed from ports. I've tried creating links to libexpat.a, but it didn't help. Any suggestions? ./configure --help | grep expat -- Joe Schaefer
Re: 2.1.5 available for testing
-1 on Win32, caddr_t isn't sufficiently portable (fix committed). Also, LDAP TLS is altogether broken, my gut says disable it, but this may not be an issue in the previous flavor of apr-util. I'm testing HEAD - which is why my apr-util result might vary. And stumbled across an old install of apache.exe which might have tripped me up (instead of using httpd.exe). So I'll have to unwind exactly what's going on with apr_ldap. I'm in the process of a broad range of 1.3 / 2.0 / 2.1 proxy header introspection, so I'll have a conclusive +/-1 on RFC2616 misbehavior shortly. In light of the Watchfire analysis, it's irresponsible of us to release anything until such issues are fully addressed, and I don't care if you call it beta, alpha, or nadda. At the very least, jfclere just fixed a major snafu in the T-E/C-L tests, which renders 2.1.5 borked, in my mind. Bill At 02:40 AM 6/17/2005, Paul Querna wrote: Please test and vote on releasing 2.1.5 as -alpha. Available at: http://httpd.apache.org/dev/dist/ (might take up to 2 hours for the files to appear, due to the rsync delay) MD5 (httpd-2.1.5-alpha.tar.Z) = f9dea893723fe7ddb8e9c5a4225a276b MD5 (httpd-2.1.5-alpha.tar.bz2) = 8ec926f27c2768c7cbd24f42fa22108a MD5 (httpd-2.1.5-alpha.tar.gz) = 7fdc3d61deb72b3815e137bdae282f0e Thanks, Paul
Re: Keepalives
Brian Akins wrote: Bill Stoddard wrote: If the event MPM is working properly, then a worker thread should not be blocking waiting for the next ka request. You still have the overhead of the tcp connection and some storage used by httpd to manage connection events but both of those are small compared to a blocking thread. Should there be an upper limit on how many connections to have in keepalive, even when using event? Say you have 100 worker threads, you wouldn't want to have 8192 keepalive connections. So you would want some limit. I really like the event mpm, but I still think there has to be an upper limit on how many connections to allow to keepalive. I'm pleased to hear you've tried the event mpm. not sure why there has to be a limit. are you talking about connections per worker process? except for the size of the pollset, I didn't see a need to put a limit on the number of connections per worker process back when I was stress testing it with specweb99. when a worker process was saturated with active threads, the listener thread would block in ap_queue_info_wait_for_idler() until a worker thread freed up. in the mean time, other processes would grab the new connections. so it was sort of self balancing as far as distributing connections among processes. not sure if the current code still behaves that way. I plan to find out soon though. We've been doing some testing with the current 2.1 implementation, and it works, it just currently doesn't offer much advantage over worker for us. If num keepalives == maxclients, you can't accept anymore connections. that's a surprise, and it sounds like a bug. I'll investigate. it used to be that maxclients was really max worker threads and you could have far more connections than threads. thanks for the feedback. Greg
