[AOLSERVER] recent updates
Hey all, me again. For those of you who don't monitor the commits list I wanted to share a few changes I've recently made as well as a few I'm still thinking about. - implemented native decoding of strings in ns_returnfile. This allows filenames that are not utf-8 to be passed, similar to tcl core functions. - committed a long-outstanding patch to address a fastpath caching bug where incorrect file content could be delivered in a system when inodes are rapidly being reused. Added a new config option "cacheminage" to control this behavior. - implemented a suggestion to re-run pre-auth and post-auth filters on internal redirects. As this could break assumptions in unusual situations, there is a config option "filterredirect" to disable it. Among other things, this addresses an issue in openacs that made it complicated to handle 404 errors. Also fixed a server crash when there is an error in the configured error page. - added flags to insert a filter at the beginning of the list of registered filters instead of only at the end, and to order filters by priority. - exposed the gzip flag to tcl scripts (ns_conn gzip) - added another new filter, "pre-write" which runs after content has been returned but before it has been sent over the network. To make this useful, there is a new method to ns_conn to examine and set the response content that is about to be sent. This could be useful for implementing something like apache's mod_pagespeed, which rewrites content to do things like merge javascript and css files to reduce external requests, or just strip extraneous whitespace from a page. - added a tuning knob "maxaccept" to let the driver accept more than one connection per spin. The SF RFE for this referenced a paper that found this type of change could improve performance and reduce dropped connections under load; I unfortunately don't have a network capable of simulating enough load to stress this so I cannot claim any measured improvement. I also have a few other ideas I haven't gotten around to adding (or even necessarily thinking through completely) yet: - removing encoding and gzip compression from the normal response writing codepath (Ns_ConnFlush) and making them into pre-write filters. - adding an api to name C-coded filters and handler procs such that they could be used by the tcl ns_register_filter/ns_register_proc commands. This could eliminate a few special-purpose commands that do this same thing for particular operations (ns_register_fastpath and ns_register_filter_shortcut). - add support for chunked encoding for requests, digest authentication, and Expect: 100-continue. I've heard that these are already supported in NaviServer, so this may be a fairly simple cut&paste enhancement. - replace c-coded versions of many file operations with equivalent tcl-coded versions. Many of these functions (e.g., ns_mkdir, ns_cp) come from before tcl had equivalent functionality, but tcl now has that functionality, and it's better (native decoding). - replace ns_share with tcl version built around nsv, if not eliminating it entirely. - remove other really old code like tclx keylist support. I honestly don't know if this is still being used, perhaps it could be kept around in a "nscompat" module along with other things like ns_var and ns_share). I'm a bit reluctant to break compatibility without an easy switch to un-break it. - make content sending event based, by pushing it back to the driver thread, a task thread, or a completely new writer thread, rather than blocking a conn thread if the content is big and the network is slow. There is support for doing this at the tcl level, but it would be nice to have it implicitly happen everywhere. - let queue-waits be set up from a conn thread and the conn re-queued in the driver thread. This would let you do database operations before making network calls and still wait for those calls to complete without tying up conn threads. This would mean a rather esoteric coding style but I think it would make for a really powerful mashup engine. Some of these are pretty disruptive (especially the last two), and introducing incompatibilities (even ones with an easy off switch) is not something to be taken lightly; does anyone else think a new 4.5.2 release with the recent bugfixes and non-disruptive new features could be done, and then slightly disruptive stuff (i.e., moving c-coded commands to tcl) could be targeted at 4.6? Naturally, I'm also interested in any feedback on anything I've done or suggested also. Cheers, -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Hi Jeff congrats on all the great work. That all sounds super. A new AOLserver release would be amazing, but maybe seeing as 4.0 was first released 8 years ago, we could consider that it's about time for a move to 5.0? Not wanting to get too Firefox-y about it, but I think it would be good for the project. best wishes Brian From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of Jeff Rogers [dv...@diphi.com] Sent: 19 October 2011 06:45 To: AOLSERVER@LISTSERV.AOL.COM Subject: [AOLSERVER] recent updates Hey all, me again. For those of you who don't monitor the commits list I wanted to share a few changes I've recently made as well as a few I'm still thinking about. - implemented native decoding of strings in ns_returnfile. This allows filenames that are not utf-8 to be passed, similar to tcl core functions. - committed a long-outstanding patch to address a fastpath caching bug where incorrect file content could be delivered in a system when inodes are rapidly being reused. Added a new config option "cacheminage" to control this behavior. - implemented a suggestion to re-run pre-auth and post-auth filters on internal redirects. As this could break assumptions in unusual situations, there is a config option "filterredirect" to disable it. Among other things, this addresses an issue in openacs that made it complicated to handle 404 errors. Also fixed a server crash when there is an error in the configured error page. - added flags to insert a filter at the beginning of the list of registered filters instead of only at the end, and to order filters by priority. - exposed the gzip flag to tcl scripts (ns_conn gzip) - added another new filter, "pre-write" which runs after content has been returned but before it has been sent over the network. To make this useful, there is a new method to ns_conn to examine and set the response content that is about to be sent. This could be useful for implementing something like apache's mod_pagespeed, which rewrites content to do things like merge javascript and css files to reduce external requests, or just strip extraneous whitespace from a page. - added a tuning knob "maxaccept" to let the driver accept more than one connection per spin. The SF RFE for this referenced a paper that found this type of change could improve performance and reduce dropped connections under load; I unfortunately don't have a network capable of simulating enough load to stress this so I cannot claim any measured improvement. I also have a few other ideas I haven't gotten around to adding (or even necessarily thinking through completely) yet: - removing encoding and gzip compression from the normal response writing codepath (Ns_ConnFlush) and making them into pre-write filters. - adding an api to name C-coded filters and handler procs such that they could be used by the tcl ns_register_filter/ns_register_proc commands. This could eliminate a few special-purpose commands that do this same thing for particular operations (ns_register_fastpath and ns_register_filter_shortcut). - add support for chunked encoding for requests, digest authentication, and Expect: 100-continue. I've heard that these are already supported in NaviServer, so this may be a fairly simple cut&paste enhancement. - replace c-coded versions of many file operations with equivalent tcl-coded versions. Many of these functions (e.g., ns_mkdir, ns_cp) come from before tcl had equivalent functionality, but tcl now has that functionality, and it's better (native decoding). - replace ns_share with tcl version built around nsv, if not eliminating it entirely. - remove other really old code like tclx keylist support. I honestly don't know if this is still being used, perhaps it could be kept around in a "nscompat" module along with other things like ns_var and ns_share). I'm a bit reluctant to break compatibility without an easy switch to un-break it. - make content sending event based, by pushing it back to the driver thread, a task thread, or a completely new writer thread, rather than blocking a conn thread if the content is big and the network is slow. There is support for doing this at the tcl level, but it would be nice to have it implicitly happen everywhere. - let queue-waits be set up from a conn thread and the conn re-queued in the driver thread. This would let you do database operations before making network calls and still wait for those calls to complete without tying up conn threads. This would mean a rather esoteric coding style but I think it would make for a really powerful mashup engine. Some of these are pretty disruptive (especially the last two), and introducing incompatibilities (even ones with an easy off switch) is not something to be taken lightly; does anyone else think a new 4.5.2 release with the recent bugfixes and
Re: [AOLSERVER] recent updates
Hey Jeff, I'm about to start using aolserver for the first time in many, many years, so your post is very encouraging, and the changes look good. Your "exposed the gzip flag to tcl scripts (ns_conn gzip)" item reminded me that I believe I had promised some code a while back on that...and I forgot to upload it...my apologies to those expecting it. I'll grab the latest and greatest from SF and start messing around with your changes. Is SF the main repo for aolserver? What about the modules (specifically nspostgres)? RE: TclX...can't that be loaded as an extension, so that those who still need it can load it? Thanks for your effort. --brett From: Jeff Rogers To: AOLSERVER@LISTSERV.AOL.COM Sent: Tuesday, October 18, 2011 10:45 PM Subject: [AOLSERVER] recent updates Hey all, me again. For those of you who don't monitor the commits list I wanted to share a few changes I've recently made as well as a few I'm still thinking about. - implemented native decoding of strings in ns_returnfile. This allows filenames that are not utf-8 to be passed, similar to tcl core functions. - committed a long-outstanding patch to address a fastpath caching bug where incorrect file content could be delivered in a system when inodes are rapidly being reused. Added a new config option "cacheminage" to control this behavior. - implemented a suggestion to re-run pre-auth and post-auth filters on internal redirects. As this could break assumptions in unusual situations, there is a config option "filterredirect" to disable it. Among other things, this addresses an issue in openacs that made it complicated to handle 404 errors. Also fixed a server crash when there is an error in the configured error page. - added flags to insert a filter at the beginning of the list of registered filters instead of only at the end, and to order filters by priority. - exposed the gzip flag to tcl scripts (ns_conn gzip) - added another new filter, "pre-write" which runs after content has been returned but before it has been sent over the network. To make this useful, there is a new method to ns_conn to examine and set the response content that is about to be sent. This could be useful for implementing something like apache's mod_pagespeed, which rewrites content to do things like merge javascript and css files to reduce external requests, or just strip extraneous whitespace from a page. - added a tuning knob "maxaccept" to let the driver accept more than one connection per spin. The SF RFE for this referenced a paper that found this type of change could improve performance and reduce dropped connections under load; I unfortunately don't have a network capable of simulating enough load to stress this so I cannot claim any measured improvement. I also have a few other ideas I haven't gotten around to adding (or even necessarily thinking through completely) yet: - removing encoding and gzip compression from the normal response writing codepath (Ns_ConnFlush) and making them into pre-write filters. - adding an api to name C-coded filters and handler procs such that they could be used by the tcl ns_register_filter/ns_register_proc commands. This could eliminate a few special-purpose commands that do this same thing for particular operations (ns_register_fastpath and ns_register_filter_shortcut). - add support for chunked encoding for requests, digest authentication, and Expect: 100-continue. I've heard that these are already supported in NaviServer, so this may be a fairly simple cut&paste enhancement. - replace c-coded versions of many file operations with equivalent tcl-coded versions. Many of these functions (e.g., ns_mkdir, ns_cp) come from before tcl had equivalent functionality, but tcl now has that functionality, and it's better (native decoding). - replace ns_share with tcl version built around nsv, if not eliminating it entirely. - remove other really old code like tclx keylist support. I honestly don't know if this is still being used, perhaps it could be kept around in a "nscompat" module along with other things like ns_var and ns_share). I'm a bit reluctant to break compatibility without an easy switch to un-break it. - make content sending event based, by pushing it back to the driver thread, a task thread, or a completely new writer thread, rather than blocking a conn thread if the content is big and the network is slow. There is support for doing this at the tcl level, but it would be nice to have it implicitly happen everywhere. - let queue-waits be set up from a conn thread and the conn re-queued in the driver thread. This would let you do database operations before making network calls and still wait for those calls to complete without tying up conn threads. This would mean a rather esoteric
Re: [AOLSERVER] recent updates
On Wed, Oct 19, 2011 at 05:09:48PM +0100, Fenton, Brian wrote: > A new AOLserver release would be amazing, but maybe seeing as 4.0 > was first released 8 years ago, we could consider that it's about > time for a move to 5.0? I guess he should just bump the version number up to 9.0 then, that'd look really swell! When Jeff releases a new version he should pick the version number based on the usual proper criteria; how extensive the changes or new features are, whether backwardness compatibility is preserved or broken, etc. How many years the codebase sat there unchanged is irrelevant. -- Andrew Piskorski http://www.piskorski.com/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
On Oct 19, 2011, at 6:51 PM, Andrew Piskorski wrote: > On Wed, Oct 19, 2011 at 05:09:48PM +0100, Fenton, Brian wrote: > >> A new AOLserver release would be amazing, but maybe seeing as 4.0 >> was first released 8 years ago, we could consider that it's about >> time for a move to 5.0? > > I guess he should just bump the version number up to 9.0 then, that'd > look really swell! Also known as the firefox version numbering method. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Fenton, Brian wrote: Hi Jeff congrats on all the great work. That all sounds super. A new AOLserver release would be amazing, but maybe seeing as 4.0 was first released 8 years ago, we could consider that it's about time for a move to 5.0? Not wanting to get too Firefox-y about it, but I think it would be good for the project. I think that any new release would be benificial; but I'm a bit old-school about version numbers. The hairs on the back of my neck stand up every time I get the monthly popup message for a new major version of FireFox. Major versions = can break functionality in a non-backwards compatible way Minor versions = can have fairly sizable changes and new features, but things can generally be expected to keep working Point versions = small new features, bugfixes, no incompatible changes I realize at least one of my changes can violate that, I'm hoping it's enough of an edge case to slide. IMO, a 5.0 release is a chance to incorporate major changes, such as to the way that interps are iinitialized (there's some ideas from Jim on this somwhere). The stuff I've done so far I think is still in the point release realm. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Brett Schwarz wrote: I'll grab the latest and greatest from SF and start messing around with your changes. Is SF the main repo for aolserver? What about the modules (specifically nspostgres)? Yep, SF is still the place for aolserver and nspostgres (which looks like it has been updated to work with 9.x). RE: TclX...can't that be loaded as an extension, so that those who still need it can load it? Good question .. from what I can gather from old threads the main users of this was internal AOL users who are probably all gone now, or at least not upgrading any longer. So this might be a pretty unimportant thing to maintain compatibility on from several angles. -J Thanks for your effort. --brett -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Hi, Cool! Nice updates :) On the version # question a few days back, I agree this is 4.x update. For a 5.x release, in addition to what you listed below, I'd add: -- integrate SSL support directly (comm driver, api's) -- integrate the comm drivers -- figure out some better build environment for windows -- ensure 64bit works everywhere -- finally re-factor the Tcl management -- make aolserver really just a loaded Tcl module The last two would be disruptive but interesting or maybe just dumb on closer inspection. Basically, aolserver could use the ns_itcl stuff along with Tcl package to better leverage existing extensions. As a module overall, the basic startup and event loop stuff could be exposed as commands instead of special case startup code in a custom binary. I think that could get aolserver/tcl closer to command-line PHP type flexibility which enables, e.g., "crush" in drupal. May be too much effort but something to explore. -Jim On Oct 19, 2011, at 1:45 AM, Jeff Rogers wrote: > Hey all, me again. > > For those of you who don't monitor the commits list I wanted to share a few > changes I've recently made as well as a few I'm still thinking about. > > - implemented native decoding of strings in ns_returnfile. This allows > filenames that are not utf-8 to be passed, similar to tcl core functions. > > - committed a long-outstanding patch to address a fastpath caching bug where > incorrect file content could be delivered in a system when inodes are rapidly > being reused. Added a new config option "cacheminage" to control this > behavior. > > - implemented a suggestion to re-run pre-auth and post-auth filters on > internal redirects. As this could break assumptions in unusual situations, > there is a config option "filterredirect" to disable it. Among other things, > this addresses an issue in openacs that made it complicated to handle 404 > errors. Also fixed a server crash when there is an error in the configured > error page. > > - added flags to insert a filter at the beginning of the list of registered > filters instead of only at the end, and to order filters by priority. > > - exposed the gzip flag to tcl scripts (ns_conn gzip) > > - added another new filter, "pre-write" which runs after content has been > returned but before it has been sent over the network. To make this useful, > there is a new method to ns_conn to examine and set the response content that > is about to be sent. This could be useful for implementing something like > apache's mod_pagespeed, which rewrites content to do things like merge > javascript and css files to reduce external requests, or just strip > extraneous whitespace from a page. > > - added a tuning knob "maxaccept" to let the driver accept more than one > connection per spin. The SF RFE for this referenced a paper that found this > type of change could improve performance and reduce dropped connections under > load; I unfortunately don't have a network capable of simulating enough load > to stress this so I cannot claim any measured improvement. > > I also have a few other ideas I haven't gotten around to adding (or even > necessarily thinking through completely) yet: > > - removing encoding and gzip compression from the normal response writing > codepath (Ns_ConnFlush) and making them into pre-write filters. > > - adding an api to name C-coded filters and handler procs such that they > could be used by the tcl ns_register_filter/ns_register_proc commands. This > could eliminate a few special-purpose commands that do this same thing for > particular operations (ns_register_fastpath and ns_register_filter_shortcut). > > - add support for chunked encoding for requests, digest authentication, and > Expect: 100-continue. I've heard that these are already supported in > NaviServer, so this may be a fairly simple cut&paste enhancement. > > - replace c-coded versions of many file operations with equivalent tcl-coded > versions. Many of these functions (e.g., ns_mkdir, ns_cp) come from before > tcl had equivalent functionality, but tcl now has that functionality, and > it's better (native decoding). > > - replace ns_share with tcl version built around nsv, if not eliminating it > entirely. > > - remove other really old code like tclx keylist support. I honestly don't > know if this is still being used, perhaps it could be kept around in a > "nscompat" module along with other things like ns_var and ns_share). I'm a > bit reluctant to break compatibility without an easy switch to un-break it. > > - make content sending event based, by pushing it back to the driver thread, > a task thread, or a completely new writer thread, rather than blocking a conn > thread if the content is big and the network is slow. There is support for > doing this at the tcl level, but it would be nice to have it implicitly > happen everywhere. > > - let queue-waits be set up from a conn thread and the conn re-q
Re: [AOLSERVER] recent updates
Op Tue, 25 Oct 2011, schreef Jim Davidson: Hi, Cool! Nice updates :) On the version # question a few days back, I agree this is 4.x update. For a 5.x release, in addition to what you listed below, I'd add: -- integrate SSL support directly (comm driver, api's) -- integrate the comm drivers -- figure out some better build environment for windows -- ensure 64bit works everywhere -- finally re-factor the Tcl management -- make aolserver really just a loaded Tcl module The last two would be disruptive but interesting or maybe just dumb on closer inspection. Basically, aolserver could use the ns_itcl stuff along with Tcl package to better leverage existing extensions. As a module overall, the basic startup and event loop stuff could be exposed as commands instead of special case startup code in a custom binary. I think that could get aolserver/tcl closer to command-line PHP type flexibility which enables, e.g., "crush" in drupal. May be too much effort but something to explore. Maybe I could point again at the ipv6 patch I wrote in 2008. Back there was even not even a response on this list, but maybe today, now the world has run out of ipv4 addresses, the interrest in serving ipv6 is a bit higher. Patch still available for download here: http://www.freepascal.org/~daniel/aolserver-ipv6support.diff Best regards, Daniël Mantione -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Good catch -- trying to get IP6 working makes sense as well. -Jim On Oct 25, 2011, at 3:54 PM, Daniël Mantione wrote: > > > Op Tue, 25 Oct 2011, schreef Jim Davidson: > >> Hi, >> >> Cool! Nice updates :) >> >> On the version # question a few days back, I agree this is 4.x update. For >> a 5.x release, in addition to what you listed below, I'd add: >> >> -- integrate SSL support directly (comm driver, api's) >> -- integrate the comm drivers >> -- figure out some better build environment for windows >> -- ensure 64bit works everywhere >> -- finally re-factor the Tcl management >> -- make aolserver really just a loaded Tcl module >> >> The last two would be disruptive but interesting or maybe just dumb on >> closer inspection. Basically, aolserver could use the ns_itcl stuff along >> with Tcl package to better leverage existing extensions. As a module >> overall, the basic startup and event loop stuff could be exposed as commands >> instead of special case startup code in a custom binary. I think that could >> get aolserver/tcl closer to command-line PHP type flexibility which enables, >> e.g., "crush" in drupal. May be too much effort but something to explore. > > Maybe I could point again at the ipv6 patch I wrote in 2008. Back there was > even not even a response on this list, but maybe today, now the world has run > out of ipv4 addresses, the interrest in serving ipv6 is a bit higher. > > Patch still available for download here: > > http://www.freepascal.org/~daniel/aolserver-ipv6support.diff > > Best regards, > > Daniël Mantione > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Daniël Mantione wrote: Maybe I could point again at the ipv6 patch I wrote in 2008. Back there was even not even a response on this list, but maybe today, now the world has run out of ipv4 addresses, the interrest in serving ipv6 is a bit higher. Patch still available for download here: http://www.freepascal.org/~daniel/aolserver-ipv6support.diff Best regards, Daniël Mantione I'll take a swing at getting this integrated sometime soon (available time waxes and wanes, as I'm sure it does for all of us). This patch is a bigger piece of functionality that the others I put in; I think 4.6 is the right target for this. I think now is the right time to mark a 4.5.2 cvs branch; I'll try to remember how to do that and tag it. Sorry if I missed this (now as then); I was mostly going off the SF RFEs looking for stuff to add. Actually, that's another thing I'd like to see happen tho I can't do it - clean up the bug reports and RFEs. There's a bunch of stuff sitting around from 2004 or earlier that's no longer relevant and much as I dislike closing bugs as "old version" I think that's what's needed for many of them. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] recent updates
Op Tue, 25 Oct 2011, schreef Jeff Rogers: I'll take a swing at getting this integrated sometime soon (available time waxes and wanes, as I'm sure it does for all of us). This patch is a bigger piece of functionality that the others I put in; I think 4.6 is the right target for this. Yup, after the patch all ip-adresses internally in the webserver are ipv6 addresses, so it's really doing brain surgery :) On the other hand, I was surprised the patch is this small. I think now is the right time to mark a 4.5.2 cvs branch; I'll try to remember how to do that and tag it. Sorry if I missed this (now as then); Heheh... let's say we now have years of experience about the reliability :) At least my webservers reliably serving both ipv4 and ipv6. Daniël -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
