[AOLSERVER] code patch to enable default filename extension

[John Buckman]

I wanted to have a a default filename extension feature in aolserver,  
where http://localhost/foo fetches foo.adp, so that URLs in a web  
site don't need to show the .adp on every url.


I read in the archives that openacs has this feature via a filter  
written in tcl, but I wanted the implementation to be fast, minimal  
and relatively transparent, so I wrote my own patch.


What my code does is change a page fetch internally from:

http://localhost/foo
to
http://localhost/foo.adp

This .adp appending only occurs on URLs that have don't have an  
existing filename extension on them.  Note that this patch also works  
correctly with directories that have a period in them.


I'm pretty sure my code below is harmless and appears bug-free, but  
it's not generic enough as it currently stands to warrant putting  
into the core.  I could make it work of an

ns_param default_ext ".adp"

if the patch seems interesting to others and there were interest to  
put it into the core. Given that it's a very small amount of C code,  
the overhead should be minimal with this feature enabled.


To put this patch in, you should insert the code below in nsd/ 
request.c right above "request->url = ns_strdup(ds2.string);"  Once  
you're happy with the patch you can remove the Ns_Log line.


/* john buckman added 4/14/06 */
/* check if should add default filename extension of .adp */
/* only if no / on end of url which indicates a directory */
char * dotpos;
if (ds2.string[ds2.length - 1] != '/') {
/* if not . in the entire url, or if there is a dot before the  
final / (indicating a . in a
   directory name, which is ok, then add the default filename  
extension */

dotpos = strrchr(ds2.string, '.');
if ((dotpos == NULL) || (strchr(dotpos, '/') != NULL)) {
Ns_DStringAppend(&ds2, ".adp");
    Ns_Log(Notice, "added default extension to get '%s'",  
ds2.string);

}
}
/* end john buckman added */

request->url = ns_strdup(ds2.string);
===


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] code patch to enable default filename extension

[John Buckman]

On Apr 15, 2006, at 4:07 pm, dhogaza@PACIFIER.COM wrote:


I'm pretty sure my code below is harmless and appears bug-free...


Well, it would break OpenACS.  If such a feature is added, it  
should be

optional and configurable.


What I was suggesting was a new optional parameter in config.tcl:

ns_param default_ext ".adp"

which would, in the sample-config.tcl file, be commented out and thus  
have no effect.  Only if you enable it would this patch take effect.


What I'd like to know is if others would find this feature useful.

-john

ps:

One small bug in what I sent to the list. In case anyone tried to  
compile the change and you're using an older C compiler, you need to  
move the


char * dotpos;

to the top of the function, like so:

static void
SetUrl(Ns_Request * request, char *url)
{
Ns_DString  ds1, ds2;
char   *p;
char * dotpos;

otherwise you'll get a compile time error.

-john
 



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] code patch to enable default filename extension

[John Buckman]

One thing I'm not clear on (patch doesn't include enough detail) is  
if a
directory url has to come in with a / in order to be considered a  
request for
a directory. Usually a redirect should be issued if there is a  
directory
corresponding to the url ,  like you don't have to put the / on the  
end shell
directory commands.  In other words, you chould do a file command  
to see if

there is actually a directory corresponding to the request url.


My patch comes after all the URL logic, which rewrites
http://localhost/dirname
to
http://localhost/dirname/

before my code gets run, so my patch doesn't impact that. That was  
very much on purpose.


I also don't know where exactly this appears in the request cycle.  
If you
change the internal URL, you may mess up any registered filters or  
registered

procs which might be handling this url.


My patch happens way at the very beginning, during parsing of the  
actual socket data. To filters or anything downstream, it looks like  
the user asked for http://localhost/foo.asp rather than http:// 
localhost/foo and any register filters would need to take that into  
account.  But, since most existing code would assume the .asp ending  
on anything they were interested in (since that's how it works now)  
it's unlikely that anyone would be harmed.


My suggestion is to use a filter to do this as an add on module  
based upon

ns_rewriteurl. It may not be as fast, but it will be vastly more
configurable.


Well, I'd probably leave to that someone else then, because fast is  
what I need. I'm currently running an online record label  
(www.magnatune.com) and get a very heavy load.


But... that's also why I posted the patch to the list. If it doesn't  
make it's way into the core, at least someone else may find it useful  
and apply the patch the their source.




Test a redirect (triggered by a 'not found' internal redirect and a  
directory

proc) and info on rewriteurl:



(Note that you could also use the 404 page to do exactly what you  
want here,

which is to look for another url if the request doesn't exist.)


A 404 would display the .adp to the end-user, which I don't want to  
do.  Logical URLs such as http://localhost/buy are really nice,  
rather than http://localhost/buy.adp, which exposes a technical layer  
the end-user shouldn't need to see.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] init.tcl created commands aren't in ADP interpreters

[John Buckman]

I'm running into a problem where new Tcl commands that BerkeleyDB  
makes in init.tcl are not available in adp interpreters.


Specifically, if I do something like this in init.tcl:
> berkdb open -create -btree a.db

a new command "db0" is now in the tcl interpreter, and [info  
commands] shows it (but not [info procs])


However, [info commands] when run on and ADP page shows that the ADP  
pages don't contain this "db0" command: they don't inherit commands  
that were created in init.tcl, just procs.


It seems that only procs from init.tcl are inherited. Is this a bug?

Or is there some way to register commands created in init.tcl so that  
they're present in ADP interpreters?


-john

ps: I know about the aolserver/berkeley-db library, but want to use  
the native tcl library provided by BerkeleyDB instead.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] per-interpreter globals possible?

[John Buckman]

Is there any way in AOLSserver 4 to have per-tcl-interpreter global  
variables?


Alternatively, does each interpreter have a unique ID that can be  
queried within tcl?  If so, I could use nsv_set to store per- 
interpreter globals?


The reason I need this is that commands created in a tcl interpreter  
have global scope (ie, creating a berkeleydb handle creates a new  
command called db0, db1, etc..) and I need to know what the handle  
name is. nsv_set is global to all interpreters, so that won't work.  
ns_cache creates a per-thread global, which is not quite the same  
thing, as sometimes the same thread will use a different interpreter.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] per-interpreter globals possible?

[John Buckman]

On Apr 17, 2006, at 6:37 PM, Andrew Piskorski wrote:


On Mon, Apr 17, 2006 at 10:47:45AM +0100, John Buckman wrote:


ns_cache creates a per-thread global, which is not quite the same
thing, as sometimes the same thread will use a different interpreter.


Hm, under what circumstances have you seen one AOLserver thread use a
different Tcl interpreter?  I wasn't aware that ever happened in
AOLserver, at least not without taking explicit and unusual steps to
make it happen (e.g., perhaps via the "interp create" Tcl command).


No, I think I was wrong, and you're right. I was trying to make  
berkeleydb's tcl library work, and it was losing state within the  
driver.


Now, it turns out that berkeleydb's tcl library is not thread safe,  
and that was the cause of my problem.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] code patch to enable default filename extension

[John Buckman]

On Apr 17, 2006, at 3:43 PM, carl garland wrote:


Why can't you just do this in your config file:

ns_section "ns/server/${servername}/adp"
ns_param   map "*"   ;# Any extension can be mapped.

This should not interfere with deliverty of any registered mime types
and should be as fast if not faster than the C patch.


Yes, that works, I tried that earlier before I made my patch, but:

1) you have to save ADP documents without any filename extension on  
the disk, which is weird looking
2) I couldn't make a saved document "index" (rather than index.adp)  
work, for whatever reason, so I would have index.adp, but all the  
docs in the directory would be w/o an extension.

3) it seems like a big security hole to execute everything and anything

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] per-interpreter globals possible?

[John Buckman]

On Apr 17, 2006, at 2:46 PM, Andrew Piskorski wrote:


On Mon, Apr 17, 2006 at 10:47:45AM +0100, John Buckman wrote:

Is there any way in AOLSserver 4 to have per-tcl-interpreter global
variables?


You already do.  Tcl global variables are per-thread, which is also
per-interpreter.


I don't believe that's true any longer. I read somewhere that this  
used to be the case, but at some AOLserver version this was changed  
so that globals are cleaned up after each ADP page (I couldn't find  
the URL ref for this now).


I wrote some code to test this, something like:

global x
catch {adp_puts $x}
set x 1

it should display nothing on the first run, then display 1 on the 2nd  
and subsequent runs. It doesn't -- it always displays nothing.


HOWEVER, Jeremy wrote:
You could throw the handle name into a namespace variable.  They  
don't currently get cleaned up.


and he's absolutely right, as code like this:

catch {namespace eval space {}}
catch {adp_puts $space:x}
set space:x 1

works as expected. So, Jeremy's tip to put per-interpreter globals  
into a namespace is an elegant work-around.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] build nsd w/o locking?

[John Buckman]

Is it possible to build AOLServer in a single-threaded, lock-free  
manner?


Because BerkeleyDB's tcl interface isn't thread safe, I'm running  
aolserver in single-thread mode, with

ns_param maxthreads 1

ApacheBench shows me getting 440 page requests per second, on an ADP  
page that does a database find and read operation, which is really  
fast still, even when I set concurrency at 10 with ApacheBench.


But, since I'm running in single-threaded mode, is it possible to  
disable thread-safety locking in AOLServer, to further speed things up?


If it isn't today, I could simple #define
Ns_MutexLock
Ns_MutexUnlock

as no-ops and that should do it.

Is it the case that Mutexes are the only lock type that's used in  
AOLServer's thread safety? Seems like it...


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] build nsd w/o locking?

[John Buckman]

On Apr 18, 2006, at 10:32 am, Andrew Piskorski wrote:

On Tue, Apr 18, 2006 at 08:46:39AM +0100, John Buckman wrote:

Is it possible to build AOLServer in a single-threaded, lock-free
manner?


My guess is the only remotely plausible way, short of redesigning and
reimplementing everything from scratch, would be to convert AOLserver
from multi-threaded to multi-process with System V shared memory, and
perhaps write an emulation layer for the POSIX Threads APIs on top of
that.


That really wasn't my question, it really was "if AOLServer is  
running in single threaded mode, does it not need locking, if if not,  
is it possible to disable the locking". However, you answered that  
question later on:

When you start up AOLserver, it is ALWAYS using multiple threads, two
or three or so at the very minimum.  I would be quite surprised if it
continued to run reliably after you removed every single mutex lock...


so that's fine.


Because BerkeleyDB's tcl interface isn't thread safe, I'm running
aolserver in single-thread mode, with


In what way is it not thread safe?  Depending on just what the problem
is, you may able to make it thread safe (at least for your use cases)
with only a modest amount of work.  (Or if you're unlucky, it could be
a LOT of work, it all depends.)


Internally, they're using a global structure to convert Tcl db  
handles into a C struct, and those aren't thread safe and get  
corrupted. When Aolserver runs in 1 thread-max-mode, I don't have a  
problem.  What's frustrating is that I'm wrapping a mutex around my  
berkeleydb operations, and that doesn't seem to be enough to prevent  
corruption in all cases (but it helps a lot)



Incidentally, I know next nothing about BerkelyDB, but why are you
using it in the first place?  Are you forced to because some 3rd party
library you need to use uses it?  Or are you free to use something
else instead which is already thread-safe, like SQLite?


BerkeleyDB is (one of) the underlying databases to MySQL, and is  
psychotically fast, much faster than SQL is, and the web site I'm  
writing (a book sharing site) needs high scalability.  I also run  
www.magnatune.com, which uses MySQL, and run against scalability  
problems all the time.



ns_param maxthreads 1

ApacheBench shows me getting 440 page requests per second, on an ADP
page that does a database find and read operation, which is really
fast still, even when I set concurrency at 10 with ApacheBench.


440 hits/s to this one single ADP page isn't fast enough for you?
Just what is this page doing, such that you need it to go faster?


440 hits/s second is plenty fast, amazing even (fyi, compared to  
about 12 hits/s I get if I code the same page in MySQL), I was just  
wondering if I could remove locks to speed things up a bit more, just  
for giggles.



"ns_param maxthreads" sets the maximum number of connection threads,
not the number of ALL threads of any type in the AOLserver process.


Very helpful to know, thanks.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] build nsd w/o locking?

[John Buckman]

On Apr 18, 2006, at 9:55 am, Bas Scheffers wrote:

You need to look into "environments" in the Berkely DB system. The  
way I
see it, you should be able to have concurrent access to one  
database from

many threads in AOLserver.


The -threads command, which enables thread safety in Berkeley DB  
environments, is commented out in the tcl driver for building  
environments from Tcl, unless you build the test mode.  And, there's  
a comment in the C code explaining that the Tcl code isn't thread  
safe, which is why it's commented out as a command.  My own testing  
shows there to be a thread safety issue around the database handle to  
the internal info structure, and that gets corrupted under heavy  
multithreading load.


Now, under C, there's no problem with multithreading in Berkeley DB,  
just under the Tcl code 


I'm trying to get the ns_berkeleydb module working, because that does  
connection pooling, and talks to the BerkeleyDB C api, which is  
thread safe. But... I'm having build problem with that (it's core  
dumping) and the developer (Vlad) and I are exchanging email about that.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] per-interpreter globals possible?

[John Buckman]

How can you know exactly what variables to clean up in each  
namespace?  Even the cleanup for the global namespace doesn't  
completely wipe out all variables.
This is exactly the problem we ran into when thinking about how we  
could better support Tcl namespace variables in AOLserver 4.5. The  
short answer is that there doesn't necessarily seem to be a good  
way to do this automatically for you, and instead it will be up to  
each developer to delete the namespace variables they have created:


   ns_ictl trace deallocate {... your cleanup here ...}

In AOLserver 4.5 the callbacks have been cleaned up and now fire in  
the right order, lifo for cleanups, fifo for inits. Hope that helps!


I didn't know about this "trace" option for ns_ictl, it is not  
documented:

http://www.panoptic.com/wiki/aolserver/Ns_ictl

is it equivalent to:
ns_ictl oncleanup script
 Registers a script to be executed at Tcl interpreter cleanup time.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] ns_adp_parse

[John Buckman]

Overall, I'm fairly happy with the results of Parser BC. I would think it would be possible to gain even further benefits if it were implemented in C (like ns_adp_parse and ns_adp_include). Additionally, if they are not currently, I think all 3 of them would benefit from being implemented as object commands as well as generating object based command byte codes.  My apologies for being so long winded. I asked the questions originally, but I thought others might find my conclusions to be useful.This is incredibly useful stuff, thanks for putting it all together. I was wondering if you could benchmark a [subst ] implementation of including content, something like this so:===proc header {title time } {	return [subst {$titleThe current time is [clock format $time]}]}ns_adp_puts [header "test" 100]===this is uses the Tcl-standard format for inline code, rather than <% code %>, and unlike your example, which uses "append" statements to build a string. I don't know how smart Tcl is with its bytecode, whether it's able to retain the bytecode form of the html string with its embedded tcl command, or whether that's scanned every time.For that matter, you can use this [subst] technique on a file, like so:ns_adp_puts [read_file "header.tml"]where header.tml is an html file, with code in [brackets] and $variables sprinkled within.  -john

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[AOLSERVER] config file example of multihome config?

[John Buckman]

I'm struggling to configure AOLserver to support multiple web sites,  
with independent TCP/IP addresses.


Can someone attach a config file example of multihoming?

The config file seems to support it, but there are both server- 
specific and global mentions of TCP/IP address and port #, so it's  
quite unclear how to transfer the sample config file into one that's  
multihome capable.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] config file example of multihome config?

[John Buckman]

On Apr 30, 2006, at 12:57 AM, Wojciech Kocjan wrote:


Dnia 30-04-2006 o 00:42:30 John Buckman <[EMAIL PROTECTED]> napisał:
I'm struggling to configure AOLserver to support multiple web  
sites, with independent TCP/IP addresses.


Can someone attach a config file example of multihoming?

The config file seems to support it, but there are both server- 
specific and global mentions of TCP/IP address and port #, so it's  
quite unclear how to transfer the sample config file into one  
that's multihome capable.


Notice you need to specify nssock multiple times and you need to  
configure each of it's instances.


The config file below seems to be to get one $server instance to  
listen to multiple IP addresses.


What I'm looking to do is have multiple $servers in one aolserver  
process


Iie: host several unrelated web sites with one aolserver process.

Doable? Config example?

-john




ns_section  "ns/server/$server/modules"
ns_param"nssock1"   "nssock.so"
ns_param"nssock2"   "nssock.so"


ns_section  "ns/server/$server/module/nssock1"
ns_param"Port"  "80"
ns_param"Hostname"  "www1.something.com"
ns_param"Address"   "10.0.0.1"

ns_section  "ns/server/$server/module/nssock2"
ns_param"Port"  "80"
ns_param"Hostname"  "www2.something.com"
ns_param"Address"   "10.0.0.2"

Of course ports may also differ.

--
Wojciech Kocjan
[EMAIL PROTECTED]


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
<[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the  
Subject: field of your email blank.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] config file example of multihome config?

[John Buckman]

On Apr 30, 2006, at 10:46 AM, Tom Jackson wrote:


John,

Here is an example which covers virtual servers using multiple  
nssocks,

and the same nssock for multiple virtual servers.



There is a static config at the top of
,
generated from the example.


Perfect, thank you! Clearly, one of the key ideas is multiple sock  
modules, as you have:


ns_section ns/modules
ns_param nssock0 /web/m2/bin/nssock.so
ns_param nssock2 /web/m2/bin/nssock.so
ns_param nssock3 /web/m2/bin/nssock.so

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] No Makefile.module in cvs

[John Buckman]

On May 7, 2006, at 9:54 AM, Bas Scheffers wrote:

Just playing with with CVS version, but when trying to build  
nspostgres I noticed there is no Makefile.module anymore.


Is this a bug or a new feature the nspostgres driver hasn't been  
updated for yet?


I build aolserver from source a few days ago, and the current CVS  
seems to have the following problems:


1) Makefile.module and Makefile.global are missing
2) all the util/*.tcl files are missing a "#!/usr/local/bin/ 
tclsh" (or equivalent) first line, enabling them to run as tcl  
scripts, so the build process fails when these get run as shell  
scripts by the makefile.

3) fails to build on GCC 2.96, because _nsmayalias is undefined.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] No Makefile.module in cvs

[John Buckman]

I built aolserver from source a few days ago, and the current CVS  
seems to have the following problems:


1) Makefile.module and Makefile.global are missing
2) all the util/*.tcl files are missing a "#!/usr/local/bin/ 
tclsh" (or equivalent) first line, enabling them to run as tcl  
scripts, so the build process fails when these get run as shell  
scripts by the makefile.

3) fails to build on GCC 2.96, because _nsmayalias is undefined.
Are you referring to the head (4.5)? If so, there was a backwards  
compatible version of Makefile.module recently committed which  
calls the new ns.mak. We're going to be further cleaning all of  
this up over the next couple of weeks to go back to simple autoconf  
based configure. Sorry for all the confusion!


Yes, these problems occurred about 7 days ago for me, building from  
CVS head (v4.5).


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] socket errors in log with CVS HEAD

[John Buckman]

In running CVS head, I'm getting the following socket errors on both  
linux and osx:


08/May/2006:09:17:04][11946.42009600][-nssock:driver-] Error: 2705:  
recv failed3: Connection reset by peer
[08/May/2006:09:17:04][11946.42009600][-nssock:driver-] Error: 2706:  
recv failed3: Connection reset by peer
[08/May/2006:09:17:04][11946.42009600][-nssock:driver-] Error: 2707:  
recv failed3: Connection reset by peer
[08/May/2006:09:17:04][11946.42009600][-nssock:driver-] Error: 2709:  
recv failed3: Connection reset by peer
[08/May/2006:09:19:04][11946.42157568][-conn:109-] Notice: exiting:  
timeout waiting for connection
[08/May/2006:09:19:04][11946.42156544][-conn:111-] Notice: exiting:  
timeout waiting for connection
[08/May/2006:09:19:04][11946.42155520][-conn:110-] Notice: exiting:  
timeout waiting for connection


And when I use the Omniweb web browser, I also get these errors:

[08/May/2006:15:25:35][13534.41963008][-nssock:driver-] Error: 8:  
recv failed3: Unknown error: 0
[08/May/2006:15:25:35][13534.41963008][-nssock:driver-] Error: 9:  
recv failed3: Invalid argument
[08/May/2006:15:25:35][13534.41963008][-nssock:driver-] Error: 11:  
recv failed3: Invalid argument


These appear to be harmless warnings, as my server appears stable.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] replacing default connection unauthorized message

[John Buckman]

if my ADP page runs

<%
ns_returnunauthorized
ns_adp_abort ok
%>

AOLserver returns the default message:
> Access Denied
> The requested URL cannot be accessed because a valid username and  
password are required.


I'd like to replace this message with my own page, which would have  
helpful information about how to join, if you lost your password, etc...


Trouble is, that message is hard-coded into the C code function  
Ns_ConnReturnUnauthorized.  I could issue a redirect, but that's  
dangerous because some web browsers probe a page 1st to see if a  
password is needed, and I don't want to mistakenly redirect somebody.


return Ns_ConnReturnNotice(conn, 401, "Access Denied",
"The requested URL cannot be accessed because a "
"valid username and password are required.");

I'm tempted to add a C code patch here with a new config file  
parameter to redefine the contents of this message.  Anyone have a  
better suggestion?


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] replacing default connection unauthorized message

[John Buckman]

On Jun 6, 2006, at 5:45 pm, patrick o'leary wrote:


ns_return should do it for you
e.g.

ns_return 401 text/html "Sorry but you can't come in, please..."


Your suggestion wasn't working for me wasn't working for me, and then  
looking at the C code I realized you need to also add a WWW- 
Authenticate header, like so:


ns_set put [ns_conn outputheaders] "WWW-Authenticate" {Basic  
realm="BookMooch"}

ns_return 401 "text/html" [ns_adp_parse -file "/src/authfailed.adp"]

otherwise IE goes into death-spin if you just do the ns_return.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] 404 redirect

[John Buckman]

I'm trying to redirect "404 not found" pages to the home page.  The  
annotated config file shows this as:


ns_section "ns/server/${servername}/redirects"
ns_param   404 "/index.adp"  ;# Not Found error page

but this isn't having any effect for me.  Can someone suggest how to  
go about debugging the problem?  Alternatively, is there a Tcl  
command to do this?


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] 404 redirect

[John Buckman]

On Jun 21, 2006, at 10:53 am, Bas Scheffers wrote:


John Buckman said:

I'm trying to redirect "404 not found" pages to the home page.  The
annotated config file shows this as:

Are you sure you want to do this? This is very confusing for users! I
personally hate this when I come acros a site that does it. I much  
prefer

a 404 page with link to the homepage and also a link to support pages
where they might tell you of the problem.


Yes, I agree, I was just trying to make my example clear. I agree,  
that a custom-written "page not found" is better than a redirect to  
home.



but this isn't having any effect for me.  Can someone suggest how to
None at all? (ie: you still get the simple/default 404 message) Or  
does it

show the homepage but with the same URL as before? These are internal
redirects, so they should show the contents of index.adp, but the URL
doesn't change; the server does NOT send a 301/302 redirect to the
browser.


No effect at all, I don't get the main page, I just get the generic  
404 error.  I don't know if there's a higher debug level, or way to  
dump the config to screen, or some other debugging technique...


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] gzip compression

[John Buckman]

Having said that, the preferred future direction seems to be  
ns_adp_compress:


http://sourceforge.net/tracker/index.php? 
func=detail&group_id=3152&atid=353152&aid=1099613


Does this gzip option work with the AOLserver source in the current  
CVS tree?


I tried it, and added the recommended config section:

> ns_section "ns/server/${servername}/adp/gzip"
> ns_param enabled on
> ns_param level 4
> ns_param minsize 0

and couldn't get a compressed result back with:
> curl -H "Accept-Encoding: gzip" http://londonmini

Which I think should work.

Digging in the source, I see references to config options "gzipmin"  
and "gziplevel"


if (!Ns_ConfigGetBool(path, "gzip", &i) || i) {
servPtr->opts.flags |= SERV_GZIP;
}
if (!Ns_ConfigGetInt(path, "gzipmin", &i) || i <= 0) {
i = 4 * 1024;
}
servPtr->opts.gzipmin = i;
if (!Ns_ConfigGetInt(path, "gziplevel", &i) || i < 0 || i > 9) {
i = 4;
}
servPtr->opts.gziplevel = i;

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] build nsd w/o locking?

[John Buckman]

I think that there are developers who are not familiar or  
comfortable with SQL
and database management systems. This can lead to a number of bad  
choices if
you are trying to develop a system which needs to store, select and  
maintain
data. Also, using a DBMS never precludes the opportunity to create  
an in
memory array of semi-static data, write semi-static html files, or  
use any
other type of cache. Until traffic develops, it is hard to predict  
where the
slow points will show up, but putting data into a key-value system  
from the

beginning, when you need to provide for the usual insert/update/delete
functions, will probably lengthen development time.

Of course, if scalability is really the main problem, this should  
imply that
going for a higher initial investment in equipment and development  
would
ensure success. Scalability probably needs to include the  
possibility of

scaling beyond one (reasonably sized) machine.


I'm not really sure this is an AOLServer topic, other than relating  
to AOLServer scalability and the ns_bdb module (which works very well).


As to scalability in general, in the decade of web interface  
programming, I've found three major areas of poor performance:
1) scripting overhead, esp as the application matures and more and  
more dependent code gets included on each page load
2) SQL database speed, especially in high concurrency situations, but  
also when millions-of-rows-per-table gets reached.

3) full text searching

As far as scripting overhead, my own benchmarks of "hello world" find  
aolserver/tcl to be by far the fastest scripted web development  
platform, at about 1/2 the speed of serving 1k GIFs, but about 10x  
faster than PHP, 3x faster than lighthttpd-fastcgi.  So, AOLserver is  
a good platform as far as scripting scalability is concerned, as long  
as the developer takes care not to load too much dependent code per  
page.


SQL, in high concurrency situations, tends to not do well on the web,  
especially in cases where lots of write/reads are occurring at the  
end of the table, which is a common scenario.  In my experience, many  
applications that use SQL actually only need key-lookup capability  
(ie, "your membership settings, your purchase history") and if the db  
supports multiple-matching-keys, much can be done with that.  What's  
nice about Berkeley DB is that it runs in-process, with an aggressive  
cache, so that during db reads it delivers performance close to  
matching in-memory databases.  Also, Berkeley DB can scale to running  
on several synced databases, which is a highly unusual feature, and  
if you've ever tried to make that work on Oracle or MS SQL, you know  
that it's tricky to get right.  Google uses Berkeley DB for their  
universal login, for just this reason.


Thirdly, you'll notice many sites have very poor full text searching  
performance.  Lucene, a recently popularized full text search engine,  
appears to finally solve this problem. However, in my case I wanted  
both fast full text searching, and grouped-by-type search results,  
such as Amazon returns.  I used to run a directory web site in the  
late-90s that ran about 300,000 full text search requests per day on  
a 5 million page corpus, and I tried all the commercial and open  
source solutions, and nothing could keep up (ie, deliver less-than-5- 
seconds results under peak load).  Back then, I wrote a full text  
engine (simple inverted index) using AOLServer and dbm under Solaris/ 
Intel (running as an AOLServer C extension) and managed peak load  
search times under 1/3rd of a second, so AOLserver + key-lookup-db  
definitely could scale to very high levels.


I know that going with Berkeley DB is controversial, but in my  
opinion it's extremely difficult to scale up a SQL backed application  
that's failing in-the-field.  That's generally why Oracle gets all  
those dot-coms -- they promise that you'll scale if you go huge, if  
you spend enough money.  In the OSS world, I've put hundreds of hours  
into trying to scale Postgresql and MySQL, and it's very difficult to  
do on a completed product.  My experience is that both of those OSS  
dbs can scale to huge heights, but only if you're an expert in that  
db platform and do extensive up-front design work to that effect,  
which few people do.


Just my opinion... all I can say is that AOLServer+berkeley-db, if  
you can live with a key-lookup database only, is incredibly fast, at  
least 3x faster than anything else I've benchmarked.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] build nsd w/o locking?

[John Buckman]

How do you mean "included"? Actual code that is run? One of the things
where AOLserver completely blows PHP out of the water is that PHP  
has to

re-interpret everything on every page. So if you include a library and
only use one function from it... AOLserver's library Tcl code is just
there at no extra cost, other than at thread creation.


Exactly : "package require" hugely helps performance, but more  
generally, web pages slowly sprout features on each page, such as  
"people who also bought..." which add overhead to every page.  In my  
environment, I have a debug hook that does "package forget" on every  
page, to make debugging easy, and turn that off when in production.





platform, at about 1/2 the speed of serving 1k GIFs, but about 10x
faster than PHP, 3x faster than lighthttpd-fastcgi.  So, AOLserver is
How could that be, PHP is better than anything, its popular for a  
reason! ;-)


That's good info, though. I still plan to do my own benchmarks,  
with some
simulated application so the database and it's drivers get benched  
too.

Not sure if I will ever get around to it, though.


I did that too, and got around 40pgs/second on my macmini against a  
single-sql query page using mysql, and 600 pgs/second with a 4 query  
bdb page.  The "real" (ie, in production) full text search page,  
against a  100,000 item corpus, runs around 120pgs/second on my  
macmini (which is an incredibly underpowered machine)





a good platform as far as scripting scalability is concerned, as long
as the developer takes care not to load too much dependent code per

As I said becfore I am not clear about what you mean by this, can you
elaborate?


Really, all I mean is per-page feature creep, esp features that  
require a db query for each page load.  Most web sites slowly add these.






end of the table, which is a common scenario.  In my experience, many
applications that use SQL actually only need key-lookup capability
I guess the answer here is "it depends"; rendering news.bbc.co.uk  
could be

done from BDB. But people end up wanting to do queries like "how much
money does the average customer spend per month" that only an RDBMS  
can
provide easily. And the "R" in RDBMS is quite important too. I  
think the
best use for BerkeleyDB is as a cache; save to the RDBMS, then  
export to
Berkeley. Like saving all the fields in a blog entry, then  
rendering the

page and stuffing it in a BerkeleyDB for viewing.


For the occasional fancy queries, in my case (www.bookmooch.com)  
things like "what are the most popular book topics" I simply do a  
table scan and cache the result (I need to look at AOLServer's time- 
expiring cache mechanism, I thought I saw something that did that).


I'm not really a mysql table scan + tcl function for average is any  
slower than SQL, because that's exactly what the SQL engine is doing  
when you ask "average of field X for whole table" and compiled Tcl is  
awfully fast, and all a SQL query optimizer really is, is a p-code  
compiler (just like tcl).  On my PHP-backed site www.magnatune.com,  
the "average stats" pages are run nightly and cached to HTML, because  
against mysql they take about 30s to run.


Now, granted, there is more code to write to do an average on a table  
scan with Tcl+bdb, than doing it with SQL, but the code is drop-dead  
simple to write and not in the critical path, that I don't worry  
about it.




that it's tricky to get right.  Google uses Berkeley DB for their
universal login, for just this reason.

It's extremely usefull for that. Though many sites don't quite need to
scale to Google-esque proportions.


No, but on the other hand, most interactive web sites have response  
times over 3 seconds.  Amazon, for instance, now is frequently >10  
seconds for a book page load.  Yahoo, with it's page times  
consistently under 1 second, is a pleasure to use.


Not to put too fine a point on it, but if you run a web-based  
business and may want to sell your company, one of the first  
questions you will hear is "how do you scale". Engineering for  
scalability way beyond your current needs really will help sell your  
business (I've sold two web-based businesses, and have been through  
discussion this quite a few times)





Thirdly, you'll notice many sites have very poor full text searching
performance.  Lucene, a recently popularized full text search engine,
appears to finally solve this problem. However, in my case I wanted
But Lucene is probably hard to integrate unless you use Java, isn't  
it? I
had a play with http://xapian.org/ two years ago. Seems very good,  
it'd be

nice to have an AOLserver module for it. Some day, when I have time...


There's a C client-side module for Lucene, which I assume could be  
fairly easily integrated into a Tcl module. However, I believe the  
server-side is Java, which many are biased against.






I know that going with Berkeley DB is controversial, but in my
opinion it's extremely difficult to scale up a

Re: [AOLSERVER] Scaling at the high end

[John Buckman]

that once you cross it, the characteristics of the problem set changes
and the optimal (and sometimes only) solutions become very  
specialized.
It's at this end of the spectrum where AOLserver truly shines, but  
it's

a very small set.


In general, my experience is that the simplest tool for the job is  
likely the most scalable, if for no other reason that it's easy to  
swap out well-understood low-functionality components.  AOLServer's  
easy escalation from Tcl to C for web pages makes for fast initial  
development, and a straightforward development path to very high  
performance.  My full text index for tile.net (which I sold to  
internet.com back in 1999) was STL-based C++ wrapped in an AOLServer  
module, and a memmap-based read-only dbm file sitting mostly in  
memory.  The sort of architecture just isn't possible in most web  
servers.  And, I was able to develop the whole thing in Tcl  
initially, with less performance critical parts (such as the indexing  
engine) left in Tcl.


AOLServer's internal model is very simple, basically being a C Engine  
-> C modules -> Embedded Tcl interpreter, with none of the fancy  
"process watcher" and "shared memory" stuff that Apache has.


John, it's fantastic to see you're actively looking at and working  
with

AOLserver.


I used to actively use it in the closed-source days of AOLServer.


From the list archives, it looks like your first message
came through around 15 Apr 2006.  I'm hoping that your interest in
AOLserver might mean that the upcoming version of Lyris may replace
tclhttpd with AOLserver?


Sadly, Lyris definitely not being moving to AOLServer, as I sold  
Lyris a year ago and no longer run the software development dept. The  
Lyris web interface is all based on tclhttpd. It'd probably just be a  
week's worth of work to port everything from tclhttpd to aolserver,  
but they'd need to learn a lot about AOLserver, and list server  
performance isn't really the hot-button it was in the dot-com era.   
Way back when Lyris was moving from a mod_perl interface to a tcl- 
backed one, I wanted to use AOLServer, but the closed-source license  
was unclear about bundling in a for profit product, and then when  
things went open source windows server support was not available, so  
we couldn't use it then.


An odd side-note: in my benchmarking of "hello world", tclhttpd gave  
terrible results vs AOLServer, but a simple all-tcl web server gave  
amazing results, almost 2x what AOLServer gave:


"Hello world" dynamic page on a slow mac-mini:
lighttpd-cgi: 15/second
tclhttpd: 32/s
aolserver: between 640/s and 750/s
trivial all-tcl-web-server http://wiki.tcl.tk/15244 : 1162/s


Or are you working with AOLserver for
something different (i.e., Magnatune)?


Magnatune is sadly, all mostly PHP, with back-end Tcl code.  I have a  
new project (www.bookmooch.com) -- a community for used book  
exchange, which is all AOLServer.


And once that launches (in a week or two), I plan on making the  
german/french versions of the Magnatune web site in AOLServer.


One of the reasons I want to use Berkeley DB is that I'd like every  
web page string to be a BDB database lookup, allowing wiki-style  
correcting of strings on a web page (ie, anyone can correct, on the  
spot, any translated text on any page).  SQL just wouldn't work with  
that design, with a dozen or so db lookups per page.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER]

[John Buckman]

Now...  if I visit a bogus url, I *still* receive the default  
aolserver not found page.  If I visit /404.adp directly, I  
successfully get the "sorry not found" message.  So, the redirects  
section seems to have no affect at server startup.  I've tried a  
few other methods with the same results.


I've gone back and verified that all of this works correctly under  
4.0.10.  I guess either this is a bug, or just a change in how to  
handle this.


Has anyone else run into this problem?  I'd appreciate if anybody  
could give a bit of time to sorting this out.


Yes, I reported this problem a few days ago, and should have  
mentioned that I was running the CVS head of 4.5, so it looks to me  
like it may be a genuine 4.5 bug.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] cvs head build problem

[John Buckman]

Tried to build CVS head today on two linux machines, and had to make  
the following patches to get it to build:


inserted
#!/usr/local/bin/tclsh

to
util/nsmakeall.tcl
util/nsremove.tcl
util/nsinstall.tcl

all 3 tcl files were missing a stated interpreter to use and caused  
the build to fail. I had to "chmod 777" them, too.


On the machine with gcc 2.96 I had to
# define _nsmayalias
otherwise the C code wouldn't build.  Gcc 4.1.0 didn't require this.

To build the nsdbdb module I had to copy from another build
include/Makefile.global
include/Makefile.module

as these no longer seem to be included in the current AOLserver.

The modifications to util/*.tcl stated above seem critical and the  
CVS head should be so patched.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] cvs head build problem

[John Buckman]

On Aug 7, 2006, at 6:10 PM, Dossy Shiobara wrote:


On 2006.08.07, John Buckman <[EMAIL PROTECTED]> wrote:

The modifications to util/*.tcl stated above seem critical and the
CVS head should be so patched.


The build is still kinda shaky.  In theory, you could build like this
(on Debian):

$ ./configure TCLSH=/usr/bin/tclsh --with-tcl=/usr/lib/tcl8.4
$ make
$ make install


I seem to remember one of the scripts pointing directly to /usr/local/ 
bin/tclsh as well.




And it'll use the path specified in TCLSH as the executable to use to
execute the .tcl scripts.
I'm not sure I like this: for one, we should probably use autoconf to
auto-detect the tclsh binary using the TCL_EXEC_PREFIX from  
tclConfig.sh

or some other magic as a default.  That would probably eliminate the
most FAQ build issue with 4.5.0 that you ran up against.


Sure, that'd be better, but as a hack, at least you could make
TCLSH=/usr/bin/tclsh --with-tcl=/usr/lib/tcl8.4
the default, that can be over-ridden.  That's not inelegant, I think.



The pre-3.3 gcc error you encountered I created a patch for and folks
have verified it fixes the problem, so I'll commit that now.

Thanks again for the feedback and congratulations on BookMooch.com, by
the way.  I love the Server: header.  :-)


 Thanks!

johnmac:/b/lib johnbuckman$ curl -I http://bookmooch.com
HTTP/1.1 200 OK
MIME-Version: 1.0
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Date: Tue, 08 Aug 2006 03:12:07 GMT
Server: MoochServer/4.5.0
Content-Type: text/html; charset=iso-8859-1
Content-Length: 6336
Connection: close

---

btw, that P3P header is pretty much a requirement if you want cookies  
to stick around. IE's default w/o that header is to toss them when  
the browser exits.


Also, I had a pretty rough day with AOLServer hanging all day after  
10 mins of running.


I can't prove it yet, but there's some situation that causes ns_db  
handles to not get cleaned up. It only happens in production, not  
under an "ab" test harness. I solved it by manually releasing all my  
db handles and wrapping catch{} statements around everything. It  
might be "adp flush failed" "abort exception raised" -- or maybe some  
other exception.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] cvs head build problem

[John Buckman]

Cookies require a P3P header now?


Yes, the default security prefs as of IE6 are to discard cookies at  
exit that don't have a compact privacy declaration indicating that  
you won't share their information with anyone.


Ever wonder why some sites seem to forget your cookies, and some are  
incredible at remembering them? The P3P header is the reasons.




I can't prove it yet, but there's some situation that causes ns_db
handles to not get cleaned up. It only happens in production, not
under an "ab" test harness. I solved it by manually releasing all my
db handles and wrapping catch{} statements around everything. It
might be "adp flush failed" "abort exception raised" -- or maybe some
other exception.


Which database are you using? I seem to remember that not all  
database drivers

reset after a conn closes.


Vlad's Berkeley DB driver nsdbbd. Stable under stress testing with  
"ab" but in the real world, I'm getting handle-not-released problems.




I have my own db api which uses an ns_atclose to cleanup/rollback  
anything not
released. In fact, I never release a handle in a script, that is  
what the

ns_atclose call is for.


Interesting that you use ns_atclose, why not use the default  
mechanism, which is registered with ns_db ? Like so:


static Ns_DbProc dbProcs[] = {
{DbFn_ServerInit, DbServerInit},
{DbFn_Name,   DbName},
{DbFn_DbType, DbDbType},
{DbFn_OpenDb, DbOpenDb},
{DbFn_CloseDb,DbCloseDb},
{DbFn_DML,DbDML},
{DbFn_GetRow, DbGetRow},
{DbFn_Flush,  DbFlush},
{DbFn_Cancel, DbCancel},
{DbFn_Exec,   DbExec},
{DbFn_BindRow,DbBindRow},
{DbFn_ResetHandle,DbResetHandle},
{0,   NULL}
};

or is it just the case that you're not using ns_db at all, and wrote  
your own?



Catch ends up being a much worse problem than anything else I've  
ever used in
tcl. It effectively removes most/all error information, so you  
cannot track
down a bug. Also, if a catch is around code which doesn't compile,  
the error

message you get is usually completely unrelated to the actual error.


Yeah, I definitely don't like manually releasing all my handles, but  
it solves the problem, and when you're in production...


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] ADP flush failed

[John Buckman]

I'm regularly getting "ADP flush failed" errors on my aolserver console.

I assume these appear because the html browser disconnected before  
the adp page was able to complete.


This error doesn't seem like anything so important that it should be  
displayed on the console screen.  Is there a way to disable it?


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] ns_cache size-limited by default?

[John Buckman]

It seems like ns_cache is -size limited by default -- is that  
correct? This isn't documented, that I can tell.


Am I right that the default is:
size = 1024 * 1000;

It seemed to me, from the docs, that an ns_cache was not size-limited  
by default, and thus could expand indefinitely.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] question about libnsd loading in tclsh

[John Buckman]

In the 4.5 release docs, it was announced:

libnsd:
The AOLserver library now includes an entry point suitable
for loading into an ordinary, thread-enabled, tclsh, e.g.,:

# tclsh
% load /usr/local/aolserver/lib/libnsd.so
% ns_time

Does this loaded libnsd.so share locks, such as ndb_handles and  
semaphores with an already running nsd?


I assume not, but thought I'd ask...

What I'm trying to do is run command-line tcl commands that would  
access my berkeleydb database while aolserver is still running, and  
if two separate processes are writing the same db file, it gets  
corrupted.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] cron (ie, timed) jobs in aolserver

[John Buckman]

Anyone have any advice on how to run tasks on a scheduled basis  
within the aolserver process?


Currently, I use cron to run "lynx http://bookmooch.com/ 
nightlyjobs.adp", and that works fine, but was wondering if there was  
an all-aolserver way, that was more elegant.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] Tcl interpreter bloating

[John Buckman]

On my bookmooch.com site, I'm noticing that my nsd process adds about  
300mb of memory usage every day, requiring a restart once a week  
after I approach my 2gb memory limit.  I'm trying to fix this.


One theory I have is that the Tcl interpreters are slowly bloating.

A simple test.adp page I made with a bunch of tcllib package  
requires, followed by package forgets, slowly bloats up.


Apache mod_perl deals with this kind of problem by occasionally  
killing an interpreter and reloading it. Is there an automated way to  
do the same thing in Aolserver?


I've written this function below, which uses ns_markfordelete and a  
namespace global to kill the current tcl interpreter every 100 runs.


namespace eval runtimes {}
proc check_if_should_cleanup_interp {} {
global runtimes::runcount
if {[info exists runcount] != 1} {
set runcount 0
}
incr runcount
puts "Tcl Interp run count: $runcount"
if {$runcount > 100} {
puts "marked tcl interpreter for deletion"
ns_markfordelete
}
}


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

Via private email, Rusty pointed me toward the connsperthread, which  
does just what I need for my tcl bloating.


It's discussed at
http://www.mail-archive.com/aolserver@listserv.aol.com/msg01525.html

Can I suggest changing the default comment on the default config  
file, which is currently this:
#ns_param   connsperthread  0 ;# Normally there's one conn  
per thread


the "Normally there's one conn per thread" comment seems incorrect to  
me and makes this parameter sound like it's about one connection  
feeding many web pages at once, which is not what it is at all, but  
rather an anti-tcl-bloat feature.


Note also that many modern OSes don't return memory to a process  
until a thread exits, in order to minimize semaphore locking around  
malloc() (Windows does this, and I think Solaris has added it too)


-john




On Aug 31, 2006, at 10:55 AM, John Buckman wrote:


On my bookmooch.com site, I'm noticing that my nsd process adds  
about 300mb of memory usage every day, requiring a restart once a  
week after I approach my 2gb memory limit.  I'm trying to fix this.


One theory I have is that the Tcl interpreters are slowly bloating.

A simple test.adp page I made with a bunch of tcllib package  
requires, followed by package forgets, slowly bloats up.


Apache mod_perl deals with this kind of problem by occasionally  
killing an interpreter and reloading it. Is there an automated way  
to do the same thing in Aolserver?


I've written this function below, which uses ns_markfordelete and a  
namespace global to kill the current tcl interpreter every 100 runs.


namespace eval runtimes {}
proc check_if_should_cleanup_interp {} {
global runtimes::runcount
if {[info exists runcount] != 1} {
set runcount 0
}
incr runcount
puts "Tcl Interp run count: $runcount"
if {$runcount > 100} {
puts "marked tcl interpreter for deletion"
ns_markfordelete
}
}





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] nsdb dropping handles?

[John Buckman]

Has anyone ever had any problems with nsdb slowly losing database  
handles, so that eventually "ns_db gethandle" never returns?


I'm working with Vlad to track down that kind of problem, that we're  
assuming is in his BerkeleyDB/nsdb driver, but I thought I'd ask if  
anyone's had that problem with other drivers.


If I manually "ns_db releasehandle" after every handle usage, I have  
no problems (but the app is slower), it's only when I rely on the  
auto-cleanup of handles when the ADP page closes that there's a problem.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] AOLserver's documentation woes and its future

[John Buckman]

I know that the sad state of the documentation has been a big problem
for a long time.  I'd really like to hear suggestions from everyone as
to how we might go about solving it.


I want to chime in with a different viewpoint:

1) the documentation is actually excellent, it's just scattered  
between old and new, it's just recent features are poorly  
documented.  A few common tasks need how-tos. Most open source  
projects have little or no docs, only the the massive Apache-sized  
stuff is a lot better than what aolserver has.


2) there is lots of good competition - everything from Ruby, Python  
and Zope to LightHttpd is in the same kind of mind space --  
alternatives to Apache that have cool ideas in them.


3) the name is a small problem -- I have to explain it to people, but  
then they get excited by the realization "wow, if aol uses it, it  
must be pretty good", so while aol is not known for open source  
projects, it does serve as one hell of a reference customer


4) tcl is more of a problem, people just don't want to learn a language.

5) if there's one thing I think aolserver could do to get more people  
to use it, it would be to put php in as a standard language, not as  
cgi, but mod_php style, AND take all the groovy aolserver apis and  
make them available to php commands.  I know I can run php from cgi,  
but it's in no way better than apache at that task.  But... I'll bet  
that aolserver running php as a primary language would be faster than  
apache.


6) pick on apache's weak spots, most specifically, languages that are  
cool but run very slowly in apache.  Ruby is what comes to mind --  
I'm seeing reports of massive scalability problems with Ruby under  
apache due to the cgi architecture, and since languages are so easy  
to glue into aolserver, that seems like a no-brainer.


7) in general, if aolserver was a great place to run the common web  
languages in a fast, multithreaded way, and did well on the  
benchmarks vs apache, I think the language zealots would begin  
pushing aolserver as the best platform


8) BTW, I convinced my ex-company Lyris to give aolserver a try,  
moving from tclhttpd, and Rusty (who's on this list) is doing a test- 
port to aolserver right now.  So, there definitely are companies  
willing to experiment with aolserver.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] AOLserver's documentation woes and its future

[John Buckman]

Perl thread safety has never been properly debugged. In fact, that  
was the reason I moved from Perl to Tcl many years back. I had  
assumed that Python had fixed the global semaphore thing from when I  
looked at it 8 years ago, but no.


Ok, Dossy, I buy your argument that the other pop languages aren't  
going to perform better because of their thread safety issues.


Your server-side javascript argument is *very* interesting.  It could  
potentially make AJAX programming quite a bit easier and there are  
some excellent debugging tools for Javascript.


One thing I really miss in the Aolserver/Tcl world is a good debugger  
-- "puts" isn't such a good alternative :D.  Javascript, I believe,  
has nice development environments and debuggers available.  It'd also  
be nice to have a wider world of source code available to us, as Tcl  
has a lot, but nothing compared to the Perl/PHP world.



IMHO, this change could put AOLserver back in the beauty pageant.

As always, I welcome any comments or criticisms or flaming of my
half-baked off-the-cuff email-to-the-whole-list ideas I presented  
here.

I know there's plenty of them ... :-)



I think you nailed it. It's the obvious solution, a very strong  
direction for aolserver to go.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] AOLserver's documentation woes and its future

[John Buckman]

On Sep 5, 2006, at 6:00 PM, Jeff Hobbs wrote:

One thing I really miss in the Aolserver/Tcl world is a good debugger
-- "puts" isn't such a good alternative :D.  Javascript, I believe,   
has nice development environments and debuggers available.  It'd also

be nice to have a wider world of source code available to us, as Tcl
has a lot, but nothing compared to the Perl/PHP world.


The Tcl Dev Kit Debugger should be able to be inserted into the  
AOLServer
environment for effective debugging.  You currently couldn't do  
that with
Komodo, but if there was demand, we could make some modifications.   
However,
TDK can handle multiple simultaneous debugging sessions, so it  
should really

be adept at the debugging needs of something like AOLServer.


Jeff, I agree, BUT feel free to express my vote for a Mac OSX version  
of Tcl Dev Kit.  


Komodo is on OSX, but as you indicated, the Komodo Tcl debugger won't  
work with aolserver (I worked with your activestate tech support  
folks and they said it wasn't supported)


As to "why a debugger for aolserver"?

Any large web-based application shares many of the same complexity  
problems of traditional applications, and from my C++ days, I learned  
that I should never leave code in that I hadn't stepped through at  
least once.


Lots of the bugs on BookMooch (22,000 lines of Tcl procs at launch)  
have been logic errors, and in the beta and first month of launching  
BookMooch, my bug database reports 414 separate bugs fixed.  That's a  
lot of code, and a proper debugger is nice.


One nice thing tclhttpd has is a debug mode that gives you a stack  
track, and a few other introspection features when a Tcl error  
occurs. Might be worth borrowing the concept.


If it weren't for Komodo doing syntax checks as I type my Tcl code,  
I'm sure I'd have many more bugs (that's an amazing feature, folks,  
if you don't know about it)


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] connected socket pool

[John Buckman]

I'd like to keep a permanently connected socket connection to another  
machine, with each aolserver thread having a socket that is already  
connected to that other app.


I tried using the namespace trick to keep a global around, like so:

namespace eval sbuff {}
proc mysockget {} {
global sbuff:mysock
if {[info exists mysock] == 1} {
return $mysock
}
set mysock [socket localhost ]
}

but while the socket descriptor stays around with this trick,  
aolserver automatically closes the socket itself.  This must be some  
sort of cleanup code in aolserver, normally a good thing.


Is there a way to either:
1) have my socket NOT be cleaned up by aolserver
2) have a connected socket pool, like nsdb, but w/o any functionality  
other than a connected socket


I'm not alone with this need, I know that Rusty needs to do this too,  
and lots of tcpip socket protocols have login steps, that cause a new- 
connection-per-adp-page strategy to be inefficient.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

So did setting connsperthread take care of the bloating? I keep  
seeing assertions that memory use by AOLserver should level off at  
some steady state - perhaps controlled by this parameter. But I  
have yet to see any one come back and say "I changed X to Y and now  
my server doesn't behave like it has a memory leak".


FYI, BookMooch.com bloats to 2gigs within 4 days of use and needs to  
be restarted.


I'm hoping that the problem is with the berkeleydb driver AOLserver  
driver, and am going to dry another database connectivity technique  
(RMI) to see if that fixes it.  If not...


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

Your problem is most likely the Tcl memory allocator, and  
fragmentation that can occur within the pools. There have been some  
posts detailing these issues a few months ago I think.


Ah, ok.

One thing to try is building Tcl so that it doesn't use the  
threaded-allocator, and to instead try something like Hoard or  
Google's tcmalloc. Now all that aside, is it really a big deal to  
do rolling restarts?


Right restarting is not something I like to doing, because most of  
the time when I ctrl-c aolserver, it exists uncleanly with one of  
these errors:


21597 aborted
or
called TclFindHashEntry on deleted table
or
open database file handle at exit: (db name)

and side-effects of this unclean exit are:
1) incomplete database operations (and I don't use transactions, so I  
get dangling records) that need to get cleaned up (ie, referential  
integrity is broken)
2) the possibility of database corruption (since I'm using  
berkeleydb) if I shut down hard in the middle of a database write,  
which has occurred a few times, and required an export to text/ 
reimport to fix.

3) anxiety

All these things *may* be caused by bugs in the berkeleydb/aolserver  
driver, I realize this.  So, I'm going to a simple Tcl RPC mechanism,  
which will be slower (18x, by my benchmarking), but safer and then  
I'll know if the bloat & stability problems are due to berkeley or  
aolserver/tcl.  Also, I'll be able to do scheduled restarts if I  
separate aolserver from the database server, rather than having them  
both be in process.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

FYI, Berkeley DB's Tcl binding isn't truly thread-safe:

http://dossy.org/archives/000307.html


I'm not using their Tcl binding, as it's not even close to thread  
safe. I'm using Vlad's BerkeleyDB/aolserver nsdb driver.  However,  
that may be leaky, so what I was going to do is make a simple tcl rpc  
server that uses the Berkeley DB tcl binding, and have aolserver talk  
tcpip rpc to that. It'll be slow, but it'll show me if the bloat is  
the berkeley db nsdb driver or not.




I'm not sure if this can be "fixed" by implementing a BDB nsdb driver
for AOLserver that uses the BDB C API or not.


That *is* what I'm using currently, and Vlad's driver works very  
well, but I do have bloat and I don't know where it comes from. It's  
quite bad now, going to 1.3g of memory use in 24h.




What you can consider doing is using the new nsproxy module to use the
BDB Tcl binding from nstclsh in separate processes in a nsproxy pool.
Then, you're relying on BDB's "concurrency" (multi-process locking)
being implemented correctly.


Yes, I prototyped exactly that, and it worked, and yield only a 4x  
slowdown in database fetch time, rather than the 18x slowdown that  
the RPC mechanism yielded.  However, I probably won't initially use  
that technique, as I first want to see where the bloat problem is.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

I still have been unable to get nsproxy to work.  Did you try that  
thing I sent you a few weeks back?  (Basically nsproxy would work  
for a few minutes and then crash the server)


FYI, nsproxy worked fine for me under macosx and linux.  I believe  
Rusty may be running it under windows (can you confirm?).


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

This morning my server complained:
"unable to alloc 2169111 bytes"

and thus exited.  So, I'm thinking that malloc fragmentation may very  
well be the problem.


I read recently that the
connsperthread
parameter no longer works in aolserver.

I believe this parameter destroys a Tcl interpreter and recreates it,  
yes?.


This technique seems to me a much better way to deal with tcl memory  
fragmentation than experimenting with a different malloc (ie, it's  
parallel to the way Apache deals with memory bloat in mod_perl).


I'm trying (right now on the live server) to do a manual version of  
connsperthread, using ns_markfordelete and a per-interpreter counter,  
cleaning up an interpreter after 1000 uses.  I'll be curious to see  
if that fixes the bloat problem.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

Yes, I prototyped exactly that, and it worked, and yield only a 4x
slowdown in database fetch time, rather than the 18x slowdown that
the RPC mechanism yielded.  However, I probably won't initially use
that technique, as I first want to see where the bloat problem is.


Uh, if you use nsproxy to push the BDB access out to a proxy pool,  
it'll

help you isolate the problem, won't it?  nsproxy pools are pools of
separate processes.  If you push your BDB accesses out to a nsproxy
pool, but your nsd memory footprint still grows, then you know the  
issue

is elsewhere in your code that's still running in your main nsd.


Right, good catch, I forgot that nxproxy pools run as separate  
processes, so yes, that would help catch it.


However nsd exits hard, then the nsproxy pools exit hard too, risking  
database corruption.


Is there a way to run an nsproxy pool started by a separate process  
from nsd, so that if nsd crashes the nsproxy pool keeps running.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Tcl interpreter bloating

[John Buckman]

On Oct 2, 2006, at 11:10 PM, Nathan Folkman wrote:


[EMAIL PROTECTED] wrote:

This morning my server complained:
"unable to alloc 2169111 bytes"

How large was the nsd process when it crashed?


Not sure how big, I wasn't around.

But, nsd regularly grows to 2gb, and I usually shut it down around  
1.5gb.


I've got nsd shutting down with ns_shutdown every 24h now, and auto- 
restarting. That's a temporary fix that will work fine for now, and  
is pretty clean. It also gives me an oppty to do a database backup  
every 24h, which I like.


Using ns_markfordelete every 100 interp passes, had no effect on  
stopping my nsd bloating, which makes me think it's NOT the tcl  
memory allocator fragmenting, because it seems to be that a tcl  
interp exiting would free its memory back to the process for reuse.


So, next on my agenda is moving off the berkeleydb/aolserver nsdb  
driver, to see if that fixes the bloating. If so, then I'll worry  
about performance.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] gzip output compression

[John Buckman]

Hi, you can take a look at some of the included documentation for  
the new "nszlib" module:


http://aolserver.cvs.sourceforge.net/aolserver/aolserver/nszlib/ 
README?view=markup
http://aolserver.cvs.sourceforge.net/aolserver/aolserver/nszlib/ 
example.tcl?view=markup


Hmm.. I thought I had read something about the zlib compression  
kicking in automatically at a larger than X size document being  
returned.  Or is gzip support in AS different than zlib support  
(since gzip uses zlib internally, nay? http://www.gzip.org/)


I posted about this in June:
http://www.mail-archive.com/aolserver@listserv.aol.com/msg09964.html

-john

---



Re: [AOLSERVER] gzip compression
John Buckman
Mon, 26 Jun 2006 00:49:53 -0700

Having said that, the preferred future direction seems to be  
ns_adp_compress:


http://sourceforge.net/tracker/index.php?  
func=detail&group_id=3152&atid=353152&aid=1099613


Does this gzip option work with the AOLserver source in the current  
CVS tree?

I tried it, and added the recommended config section:

> ns_section "ns/server/${servername}/adp/gzip"
> ns_param enabled on
> ns_param level 4
> ns_param minsize 0

and couldn't get a compressed result back with:
> curl -H "Accept-Encoding: gzip" http://londonmini

Which I think should work.

Digging in the source, I see references to config options "gzipmin"  
and "gziplevel"

if (!Ns_ConfigGetBool(path, "gzip", &i) || i) {
servPtr->opts.flags |= SERV_GZIP;
}
if (!Ns_ConfigGetInt(path, "gzipmin", &i) || i <= 0) {
i = 4 * 1024;
}
servPtr->opts.gzipmin = i;
if (!Ns_ConfigGetInt(path, "gziplevel", &i) || i < 0 || i > 9) {
i = 4;
}
servPtr->opts.gziplevel = i;

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] build problem with extension

[John Buckman]

I'm having a problem loading the berkeley db extension, built on  
MacOSX/Intel, that I haven't had on MacOSX/PowerPC, gcc 4.01.
The reason I'm inquiring here (and not with Vlad, who doesn't know  
either) is it seems like a general compiler/platform issue


Aolserver, on load displays this error:

> Warning: modload: could not find Ns_DbDriverInit in /usr/local/ 
aolserver/bin/nsdbbdb.so

> Error: dbdrv: failed to load driver 'berkeleydb'

yet Ns_DbDriverInit is defined in the .c source code
> NS_EXPORT int Ns_DbDriverInit(char *hModule, char *configPath)

This looks to me like a simple problem of the function not being  
exported correctly.  Or is it a problem of gcc in c++ vs C mode? Or  
some compile time flag magic that's needed...


Any suggestions?

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] build problem with extension

[John Buckman]

In case anyone needs this info in the future, I fixed this problem  
below.


Evidently, on osx, a dynamic library and a static library are  
created, and at least in my case, the default aolserver module  
makefiles were not listing the module's .o file in the link list for  
the static library, only for the dynamic library.


I was able to determine the fault with the "nm" command, run on  
the .o and .dylib files gave me a happy list of exposed functions,  
while the .so was stripped.


So, I modified the default build command for the module, from this:
gcc -pipe -bundle -o nsdbbdb.so  -L. -lnsdbbdb -lnsd -lnsthread - 
L/usr/local/lib -ltcl8.4   -lpthread -framework CoreFoundation


to now list nsdbbdb.o in the link list. Which then gave me some link  
errors, solved by adding BerkeleyDB, and ns_db's libraries in the  
link list:


-L/usr/local/aolserver/lib -L/usr/local/BerkeleyDB.4.4/lib/ -lbdb -lnsdb

and then aolserver ran happily with my module, and "nm" shows all the  
functions correctly exposed.


-john


On Oct 10, 2006, at 6:30 PM, John Buckman wrote:

I'm having a problem loading the berkeley db extension, built on  
MacOSX/Intel, that I haven't had on MacOSX/PowerPC, gcc 4.01.
The reason I'm inquiring here (and not with Vlad, who doesn't know  
either) is it seems like a general compiler/platform issue


Aolserver, on load displays this error:

> Warning: modload: could not find Ns_DbDriverInit in /usr/local/ 
aolserver/bin/nsdbbdb.so

> Error: dbdrv: failed to load driver 'berkeleydb'

yet Ns_DbDriverInit is defined in the .c source code
> NS_EXPORT int Ns_DbDriverInit(char *hModule, char *configPath)

This looks to me like a simple problem of the function not being  
exported correctly.  Or is it a problem of gcc in c++ vs C mode? Or  
some compile time flag magic that's needed...


Any suggestions?

-john




--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] mutex try lock?

[John Buckman]

I'm looking to have mutexes with timeouts, and I see support in the C  
code for this but none carried over to Tcl.


In the C code, there's a Ns_MutexTryLock() function, but no tcl  
function for calling it.


Ns_MutexLock calls Ns_MutexTryLock() and there appears to be timeout  
support:

if (!NsLockTry(mutexPtr->lock)) {
return NS_TIMEOUT;
}

but I don't see any way of setting the mutex timeout seconds.

It's trivially simple to modify NsTclMutexObjCmd (in tclthread.c) to  
support "ns_mutex try" and I'm wondering if there's a reason this  
hasn't been done.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] mutex try lock?

[John Buckman]

Forgot to mention that I know about Dossy's comment in the wiki that  
nsv_incr can be used to this effect, and I'm doing that.

http://panoptic.com/wiki/aolserver/Nsv_incr

but I was wondering why "try" wasn't part of ns_mutex when the C code  
is there for it.


-john



On Feb 22, 2007, at 3:09 pm, John Buckman wrote:

I'm looking to have mutexes with timeouts, and I see support in the  
C code for this but none carried over to Tcl.


In the C code, there's a Ns_MutexTryLock() function, but no tcl  
function for calling it.


Ns_MutexLock calls Ns_MutexTryLock() and there appears to be  
timeout support:

if (!NsLockTry(mutexPtr->lock)) {
return NS_TIMEOUT;
}

but I don't see any way of setting the mutex timeout seconds.

It's trivially simple to modify NsTclMutexObjCmd (in tclthread.c)  
to support "ns_mutex try" and I'm wondering if there's a reason  
this hasn't been done.





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] mutex try lock?

[John Buckman]

On Feb 23, 2007, at 8:31 AM, John Buckman wrote:

Should also get around to implementing "ns_mutex eval $mutex
$script"--acquire the mutex, evaluate the script, release the mutex.
Catch any Tcl errors and re-throw them.  This guarantees that the  
mutex

is unlocked, which avoids the trivial deadlock case:


mutex eval would be very handy, and "trylock" is fine by me, though  
"tryandlock" would be handy, where the lock is obtained if it can be,  
and if it can't be the function returns immediately with an error.   
The C code for mutex lock appears to already support this (an E_BUSY  
flat, if memory serves) but I didn' see a way to set a pthread to  
return failure if a lock isn't possible.


To get around the error-causes-mutex-to-not-be-unlocked problem, I  
created a "lock object" that automatically unlocks when the function  
that created it goes out of scope.


You create a lock like this:
proc foo {
  mooch_lock aLock
  stuff...
}
(mutex unlocked automatically when function exits)

and this is the lock/unlock code:

proc mooch_lock {name} {
set mutex [nsv_get . $name]
	uplevel 1 "set $name \"$mutex\";trace add variable bar unset  
\"mooch_unlock $name\""

puts "locking $name / $mutex"
ns_mutex lock $mutex
}

proc mooch_unlock {name args} {
set mutex [nsv_get . $name]
puts "unlocking $name / $mutex"
ns_mutex unlock $mutex
return
}

that way, you don't need to use "eval" around code that is mutex  
protected.  The "trace" function handle mutex unlocking for you.


None-the-less, I still occasionally get a deadlock on my site, so  
either this mechanism above isn't as robust as I think it should be,  
or I have double-lock depending code somewhere on my web site.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] mutex try lock?

[John Buckman]

On Feb 22, 2007, at 8:21 PM, Nathan Folkman wrote:
For John. Just curious as to what problem the "try lock" was needed  
for.


I'm trying to block the "double submit" problem on web pages, where a  
page submit takes too long (because something else is taking up the  
server), so the user hits cancel and resubmits.


When the original taking-up-the-server code completes, now the two  
submissions occur at exactly the same moment, and the submission is  
processed twice.  I currently use a database (check the database to  
make sure this operation hasn't previously run) call to prevent this,  
but if the two pages occur at the exact same moment, the database  
call isn't atomic enough to prevent the double-submit.


I previously used a simple mutex to prevent the double-submit  
problem, but would occasionally get a deadlock when deployed in the  
field.  The deadlock is probably due to some operations needing two  
locks, and thus locking against each other, and is debuggable, but  
with some work and with quite negative consequences on a live server,  
since I have to shut down the server to unlock the jammed threads.


Instead, to solve the "double submit" problem, I planned a more  
simple strategy:

1) try to grab a mutex, so I'm the only one doing this
2) if I can't grab a mutex, wait 1 second, and try again
3) after 5 seconds, give up on the mutex, and proceed anyway, since  
if there is a double-submit, chances are the two threads aren't  
running at the same moment any more. And, display an error, so that  
the programmer can investigate and see if there is a deadlock-causing  
bug in this code.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] mutex try lock?

[John Buckman]

So, the question becomes: is it better to add "-timeout time" to  
all the

blocking subcommands, or to add "ns_mutex trylock"?  In theory,
"-timeout 0" should perform the trylock equivalent.


I'd vote for -timeout on all lock requests, as that way I get both  
the "see if lock is available" functionality, as well as "lock it if  
it's available, but give up after a few seconds if it isn't"


Generally, with any code I put into production, I try to put a sanity  
check in so that spectacular unexpected failures cause LOUD log  
warnings.,For normal semaphore locks, I might want to give up after 1  
minute and display a warning in the log that there's likely a coding  
error in there (and I can even show a tcl stack trace to help find  
it).  Otherwise, it's down to gdb and a core file to figure out what  
semaphores were blocking, and that's kind of ugly.


I'm currently using nsv_incr to emulate this behavior, and it works  
well enough.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] tdom leak fix

[John Buckman]

FYI, on the tdom leak, I found that my app (BookMooch) was leaking as  
well (thanks for mentioning the leak problem you're having, it helped  
me find mine), and the reason is that tdom NEVER automatically  
releases memory for the documents it creates (the docs say " the  
document object command has to be explicitly deleted ").


Therefore, I highly recommend adding a
$doc delete

to manually delete any tdom objects you create.

Below are the relevant docs for tdom for this.

-john



dom parse ?options? ?data?
Parses the XML information and builds up the DOM tree in memory  
providing a Tcl object command to this DOM document object. Example:

dom parse $xml doc
$doc documentElement root
parses the XML in the variable xml, creates the DOM tree in memory,  
make a reference to the document object, visible in Tcl as a document  
object command, and assigns this new object name to the variable doc.  
When doc gets freed, the DOM tree and the associated Tcl command  
object (document and all node objects) are freed automatically.


set document [dom parse $xml]
set root [$document documentElement]
parses the XML in the variable xml, creates the DOM tree in memory,  
make a reference to the document object, visible in Tcl as a document  
object command, and returns this new object name, which is then  
stored in document. To free the underlying DOM tree and the  
associative Tcl object commands (document + nodes + fragment nodes)  
the document object command has to be explicitly deleted by:


$document delete
or
rename $document ""


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] unclean shutdown

[John Buckman]

My shutdowns of aolserver are rarely clean, nsd usually crashes on  
shutdown.


I've started running nsd inside gdb, so I can see where the crash is.

Here is a stack trace on a ctrl-c, using the latest cvs source.  If  
there's anything else I can do to help track down these unclean  
shutdowns, let me know.


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0004
[Switching to process 4464 thread 0x4613]
0x0a00c254 in Tcl_AsyncDelete ()
(gdb) where
#0  0x0a00c254 in Tcl_AsyncDelete ()
#1  0x00649500 in SignalCmdCleanUp ()
#2  0x0a00f35e in DeleteInterpProc ()
#3  0x0a00cad5 in Tcl_DeleteInterp ()
#4  0x0007da11 in DeleteData ()
#5  0x793a in NsCleanupTls ()
#6  0x891f in FreeThread ()
#7  0x90024f60 in _pthread_tsd_cleanup ()
#8  0x90024ae8 in pthread_exit ()
#9  0x8203 in Ns_ThreadExit ()
#10 0x0006f6b6 in NsConnThread ()
#11 0x88ee in ThreadMain ()
#12 0x90023d87 in _pthread_body ()


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] unclean shutdown

[John Buckman]

On Mar 16, 2007, at 6:13 AM, Dossy Shiobara wrote:


On 2007.03.15, John Buckman <[EMAIL PROTECTED]> wrote:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0004
[Switching to process 4464 thread 0x4613]
0x0a00c254 in Tcl_AsyncDelete ()
(gdb) where
#0  0x0a00c254 in Tcl_AsyncDelete ()
#1  0x00649500 in SignalCmdCleanUp ()
#2  0x0a00f35e in DeleteInterpProc ()
#3  0x0a00cad5 in Tcl_DeleteInterp ()

...

What patchlevel of Tcl?


8.4.14 - current release.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] unclean shutdown

[John Buckman]

On Mar 16, 2007, at 2:50 PM, Dossy Shiobara wrote:


On 2007.03.16, John Buckman <[EMAIL PROTECTED]> wrote:

What patchlevel of Tcl?


8.4.14 - current release.


What modules are you loading?  Are you using any of the Tcl async.  
stuff

(introduced in 4.5) in your application code?


I don't think I'm using any async stuff, at least not in any code  
I've written, hopefully no library uses it automatically.


Here's a list of modules I'm using.  nsd is totally stable, running  
for weeks w/o a reboot, it's just at shutdown that I have a problem,  
and I should mention that it's not every time.  About 75% of the time  
I do get a clean shutdown.


package require ncgi
package require struct::list
package require tdom
package require csv
package require crc32 1.3
package require http
package require html 1.3
package require Tclx

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] unclean shutdown

[John Buckman]

On Mar 17, 2007, at 5:24 AM, Dossy Shiobara wrote:


On 2007.03.16, John Buckman <[EMAIL PROTECTED]> wrote:

package require Tclx


Good, that confirms my guess.  From your previous stacktrace:


(gdb) where
#0  0x0a00c254 in Tcl_AsyncDelete ()
#1  0x00649500 in SignalCmdCleanUp ()
#2  0x0a00f35e in DeleteInterpProc ()

...

That crash is happening in Tclx's thread cleanup code:

http://tclx.cvs.sourceforge.net/tclx/tclx/generic/tclXsignal.c

I'm surprised this doesn't cause instability at runtime, but I'm
guessing you don't routinely send the nsd signals.


Right, I don't use the nsd signals.  Is there a patch for that TclX  
problem?  There hasn't been a TclX update in 16 months (http:// 
sourceforge.net/project/showfiles.php?group_id=13247) but I haven't  
checked their CVS tree.




When you shut down, are you doing it by issuing a "ns_shutdown" via  
the

control port (nscp), or by sending it a signal?


ns_shutdown

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] easy way to tell all tcl interps to exit next time?

[John Buckman]

Can someone suggest an easy way to tell aolserver to reload its tcl  
interps at the next opportunity?


What I basically need is a way to run ns_markfordelete for every tcl  
interpreter nsd has open.

http://panoptic.com/wiki/aolserver/Ns_markfordelete

When I push out a new version of BookMooch (via cvs) to the live  
server, I currently restart nsd on the production server.  What I'd  
like to do instead, is to have all my interps exit at their next  
opportunity, so that they use the new tcl libraries that are loaded  
with "package require".


I can think of a few ways to do it, but they're all somewhat  
complicated, so I was wondering if anyone had an easy way to do it?


ie, I could use "nsv_set exit_interp_now 1" that each interp would  
check at the end of each page render, but with many tcl interps in  
separate threads, it's tricky to make sure each interp exits exactly  
one time (ie, how to tell whether an interp that has exited once to  
not exit again when it reloads).


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] easy way to tell all tcl interps to exit next time?

[John Buckman]

On Apr 4, 2007, at 3:01 PM, Rusty Brooks wrote:


would
ns_eval [ns_markfordelete]
work?  ns_eval runs a bit of tcl code in each thread.


See, I knew there ought to be an easy way! Thanks!

Thanks Dossy & Rusty!

Personally, I don't use packages/package require, I have an init  
script that loads everything.  Additionally, all my pieces of code  
are broken up into modules.  So I can say something like  
"module_load /module1" and it'll run the init for that module,  
updating all the existing threads.  If I say "module_load /" then  
it'll load all the modules (they are hierarchically organized)


P.S. the first and only time I've used packages in aolserver was a  
port of Listmanager, which you're obviously familiar with.  When I  
want to reload stuff I actually do a ns_eval myinit where myinit  
just reloads everything in custom and htdocs/libtml.  It's crude  
but it works for my purposes.


I kind of do the same thing, in that I have an init proc that does a  
"package require" of everything my aolserver app needs, and if rerun  
it first does a "package forget" on all of them.  While in  
development, every page load causes a "package forget" so that I  
reload everything, so that every code change becomes immediately  
apparent.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] profiling an aolserver site

[John Buckman]

I'm wondering if anyone on this list has written code to "profile" a  
web site running under Aolserver.


By this, I mean, timing the start/stop time of every page, logging  
it, and then running a bit of analysis to find out what pages are the  
slowest running and which pages are the most commonly loaded, then  
multiplying the two (ie execution time x requests per day=total  
machine load per page per day)


In traditional systems programming, profiling is a common tool used  
to determine what code should be optimized.   I'd like to do the same  
inside aolserver.


One efficient alternative I was thinking about would be to patch  
ns_log to include both the start request time, and the time the page  
was returned, in the log.  That could be done if ns_log is called  
after the page is rendered, and I don't know if that's the case.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] profiling an aolserver site

[John Buckman]

On Apr 11, 2007, at 9:00 AM, patrick o'leary wrote:


Something that might be of use is the tcl-lib profiler
http://www.tcl.tk/community/tcl2004/Tcl2003papers/kupries-doctools/ 
tcllib.doc/profiler/profiler.html


Which gives a little insight into proc calls. Might be of use to  
you, obviously use with care and don't enable it

on all production requests.


The tcllib profiler doesn't like living inside aolserver, because it  
redefines "proc" and some other core things and aolserver doesn't  
like that. I've tried to use it in the past, but not with success.   
Instead, I make sure the adp page calls a tcl proc, and then I  
profile the tcl proc on the tclsh command line, which works ok.


-john




--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] profiling an aolserver site

[John Buckman]

I added server-side logging of ADP page execution a while back.  
Maybe that would help you?


ns_section "ns/server/server1/module/nslog"
ns_param logreqtime true


Yes, that's great.

Will it also log ns_register_proc pages?

Have you written anything to crunch the numbers? If not, no problem,  
I can do that.


-john




[MacBook-Pro:~] nfolkman% tail -f /usr/local/aolserver/servers/ 
server1/modules/nslog/access.log
192.168.1.129 - - [11/Apr/2007:10:46:28 -0400] "GET /flash/ HTTP/ 
1.1" 200 191 "" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US;  
rv: 1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0.106342
192.168.1.129 - - [11/Apr/2007:10:46:28 -0400] "GET /favicon.ico  
HTTP/1.1" 404 534 "" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en- 
US; rv: 1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0.000361


On 4/11/07, John Buckman < [EMAIL PROTECTED]> wrote:
I'm wondering if anyone on this list has written code to "profile" a
web site running under Aolserver.

By this, I mean, timing the start/stop time of every page, logging
it, and then running a bit of analysis to find out what pages are the
slowest running and which pages are the most commonly loaded, then
multiplying the two (ie execution time x requests per day=total
machine load per page per day)

In traditional systems programming, profiling is a common tool used
to determine what code should be optimized.   I'd like to do the same
inside aolserver.

One efficient alternative I was thinking about would be to patch
ns_log to include both the start request time, and the time the page
was returned, in the log.  That could be done if ns_log is called
after the page is rendered, and I don't know if that's the case.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <  
[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the  
Subject: field of your email blank.




--
Nathan Folkman
[EMAIL PROTECTED]

--
AOLserver - http://www.aolserver.com/


To Remove yourself from this list, simply send an email to  
<[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the  
Subject: field of your email blank.




--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] ns_zlib compress

[John Buckman]

On Apr 19, 2007, at 10:11 AM, Rusty Brooks wrote:


I don't believe you can use ns_return with binary data.

By the way, are there any plans to change that?  Or make a  
ns_return_binary or something?  I have a lot of code that writes to  
a file and then does ns_returnfile


ns_write is what you should use for binary data:

set h [subst {HTTP/1.1 200 OK
MIME-Version: 1.0
Server: MoochServer/4.5.0a
Content-Type: image/jpeg
Content-Length: [string length $d]
Connection: close

$d}]
ns_write $conn $h
return TCL_OK

so I guess I should do that for my ns_compress code, but anyhow,  
that's not the problem


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] ns_zlib compress

[John Buckman]

Below is a ns_return_gzipped proc that replaces ns_return and sends  
back gzipped output.


It worked fine everywhere, *except* with a debugging http proxy I  
used (in ActiveState's Komodo) where pages stalled.  I was thinking  
perhaps the problem is that "Connection: close" wasn't being honored  
by the proxy, but it makes me nervous, because the builtin ADP gzip  
compression works fine with the proxy.


Is there a way to force a connection closed in a ns_register_proc  
page, after ns_write? I can't use "ns_conn close" in this context,  
since that's an ADP command.


-john


proc ns_return_gzipped {conn html} {

set zl [ns_zlib gzip $html]

set h [subst {HTTP/1.0 200 OK
MIME-Version: 1.0
Server: AolServer/4.5.0a
Connection: close
Content-Type: text/html; charset=iso-8859-1
Content-Length: [string length $zl]
Content-Encoding: gzip

$zl}]

ns_write $conn $h
   return TCL_OK
}


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] ns_zlib compress

[John Buckman]

I have ns_param gzip on  , but do i still need to set some headers ?

I don't know.

How do i know its gzipping anything ?

telnet www.yourwebserver.com 80
And send a GET request.


That won't invoke compression unless you include the appropriate  
Accept-encoding header.  You need to sent a complete request like


If you’d like to see whether a page is compressed, use Firefox and  
install the “Live Headers” add-on from http:// 
livehttpheaders.mozdev.org/.


With this add-on, when you choose “info” on a page, a new tab titled  
“headers” is now available. Click on that tab, and look for “Content- 
Encoding: gzip” on the bottom window (i.e. the “Response Headers”).  
If there’s no “Content-Encoding” then gzip is not being used on that  
page. You’ll also see the transferred size of the file (2022 bytes in  
this case) and if you save the file to disk, you can see what the  
real size is (12k in this case) and thus what the speedup was.


from: http://blog.bookmooch.com/2007/04/20/gzip-compression-of- 
bookmooch-pages-faster-faster/


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] ns_zlib compress

[John Buckman]

How do you enable gzip page compression on ns_register_proc pages?

The adp gzip compression occurs in C code, in Ns_ConnFlush(), which  
checks for "Accept-Encoding: gzip"


Since a "ns_register_proc" uses "ns_return", like this:
ns_return $conn 200 text/html $html

I believe it's bypassing this gzip code, and always returning html.

I wrote a ns_return_gzipped() proc (see below), but it doesn't seem  
to work, so I was wondering if anyone's done this already, and how.


FYI I switched BookMooch to use adp gzip compression, and the results  
are impressive.  200k html files go to 35k, greatly speeding up the  
user experience.


-john



proc ns_return_gzipped {conn html} {

if {[string first {gzip} [ns_set get [ns_conn headers] {Accept- 
Encoding}]] != -1} {

set zl [ns_zlib gzip $html]
ns_set put [ns_conn outputheaders] Content-Encoding gzip
ns_return $conn 200 text/html $zl
} else {
ns_return $conn 200 text/html $html
}

}


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] learning from naviserver

[John Buckman]

So, in the spirit of open source software meritocracies: please place
your money where your mouth is.  Come up with a list of actionable
changes you'd make if you were king.  Lets hear it--and if everyone
agrees to a particular change, we'll declare it made.  (Note:  
declaring

anything to a volunteer-driven organization doesn't guarantee that
anyone will actually do it.)


In no way do I want to be king, and in an effort to calm things down  
a bit, let me say that I'm really, really happy with aolserver, and  
the last major release had amazing things in it. My only real nit is  
that so many cool things are not-so-well-documented, but hell, it's  
open source. I just spot things in the text files, and then figure it  
out by reading the C code, which is often commented, but always clean  
and readable.


On the subject of cool things and not-so-well-documented, I'd like to  
bring up Naviserver (you can find it on sourceforge, it's an  
independent fork of aolsever, at http://naviserver.sourceforge.net).   
Their fork of aolserver has an insane number of changes to it, and  
lots of great ideas (look at http://naviserver.cvs.sf.net/naviserver/ 
naviserver/ChangeLog?view=markup).  There's a handful of developers  
working on it, and it seems like a real hotbed of innovation.


I've not used naviserver, though I evaluated it seriously, because it  
has so many differences to aolserver, almost all undocumented, that  
it was really hard for me to get up to speed to, and I found it less  
reliable than aolserver, probably because of all those innovations.   
I kind of like the slower pace of aolserver, I can actually run a  
production web site on it. This is not meant as an insult to Vlad and  
the other Naviserver developers, I'm just pointing out how the two  
development communities differ.


However, I was wondering if we should perhaps look at merging back  
some of the best things in naviserver, back into aolserver.  In fact,  
maybe we should treat the aolserver/naviserver split like ubuntu  
treats its two releases, and recognize that naviserver as an  
innovative, highly chaotic playground, and merging back in the best  
ideas from it back into aolserver after a long delay (6 months to a  
year) once each feature has settled down a bit and we can evaluate  
whether, in hindsight, it really was a good idea and the way it was  
implemented turned out well.


I'd be game to go through the naviserver changelog in the future, and  
be part of a discussion of what is in there that we might want to  
merge back into aolserver.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] configured minthreads, maxthreads doesnt show up with [ns_server threads] command

[John Buckman]

On Aug 3, 2007, at 3:39 PM, [EMAIL PROTECTED] wrote:


Please watch the vulgar language - there's simply no need for it.


And precisely who are you to say so?


He's someone who is trying to keep the discussion civil and  
productive, and trying to be helpful.


Really, I think it's best to express yourself in a non- 
confrontational way, you're much more likely to have the other side  
listen to you.


Folks can we tone down the flamage on this list?  We're a small  
community, and nobody will be served well by angering each other.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver focus

[John Buckman]

I'm wondering -- does it make sense to just try to close the gap  
with LAMPP as a model, driving to the "batteries-included" distro  
Dossy's been talking about for years?  That seems to me like a  
project tons of folks could contribute too -- from docs to  
extensions to installers, etc.


On new aolserver installations, I install the ActiveState "batteries  
included" tcl version, and then copy over all the libraries it has  
(which is a *lot*) into aolserver's tcl directory (in my case /usr/ 
local/), which makes for an extremely capable AolServer/tcl distro.  
hmm.. it might actually be possible to build aolserver against the  
activestate distro directly to accomplish this.


The ActiveState "batteries included" tcl version competes really  
well, IMHO, with PHP and Perl.  My only issue has been that  
apparently TclX doesn't play nice with AOLserver and can cause  
unclean shutdowns (I think Dossy said this), otherwise I have a wide  
variety of libraries, pretty much all the same stuff as PHP/Perl.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver focus

[John Buckman]

On new aolserver installations, I install the ActiveState  
"batteries included" tcl version, and then copy over all the  
libraries it has (which is a *lot*) into aolserver's tcl directory  
(in my case /usr/local/), which makes for an extremely capable  
AolServer/tcl distro. hmm.. it might actually be possible to build  
aolserver against the activestate distro directly to accomplish this.
The ActiveState "batteries included" tcl version competes really  
well, IMHO, with PHP and Perl.  My only issue has been that  
apparently TclX doesn't play nice with AOLserver and can cause  
unclean shutdowns (I think Dossy said this), otherwise I have a  
wide variety of libraries, pretty much all the same stuff as PHP/ 
Perl.


I've said before that AOLServer would best be served as a series of  
modules that leverage standard distributions.  In this way it would  
become more like Ruby on Rails or similar Python frameworks.  A  
more elegant coupling will probably improve things all around.  To  
that end, I can assist with the de/recoupling, although my time  
resources for me are unfortunately not in abundance.


Would it be even possible to have AOLServer as a tcl module, and thus  
automatically distributed with the batteries included releases?




As to the TclX problem ... I wonder if that has to do with mt  
issues.  I don't know if TclX has been fully vetted for thread- 
safety (and you should most definitely never fork() with threads).


Yeah, that would make sense, though it is completely stable for  
except during shutdown.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver focus

[John Buckman]

On Aug 3, 2007, at 8:41 PM, Rick Cobb wrote:

John, does this distro's Http package work well within AOLServer?  
We did
a cURL integration for AOLServer 3.4.2, but have been holding off  
on any

contribution because it's very intertwined with our C++ stuff -- and
figured that one reason the AOLServer community did all its changes  
for

4.0 & 4.5 was to be able to use packages like that.  We haven't gotten
far enough in our port to 4.5 to have tested the Http package (we've
been focused on C/C++ integration, not Tcl, so far).


Yes, the http package works perfectly inside AOLServer, I use it for  
Amazon API XML lookups on BookMooch, which is a heavily used feature,  
so definitely no problems.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] two OSX bugs

[John Buckman]

I've had these two bugs for a while on OSX, thought I'd report them.  
I'm running today's current CVS.


--
Bug #1: nszlib causes "abort trap" on OSX.

007:23:52:46][16387.2684407808][-main-] Notice: modload: loading '/ 
usr/local/aolserver/bin/nszlib.so'
[06/Aug/2007:23:52:48][16387.2684407808][-main-] Notice: conf: [ns/ 
server/BookMooch]enabletclpages = 0
[06/Aug/2007:23:52:48][16387.2684407808][-main-] Notice: default  
thread pool: minthreads 0 maxthreads 10 idle 0 current 0 maxconns 0  
queued 0 timeout 0

alloc: invalid block: 0x27e000: ef ef 80
./aol: line 4: 16387 Abort trap  sudo bin/nsd -ft  
config.tcl -u john -b 127.0.0.1:81


if I don't use nszlib, I have no such problems. Also, nszlib works  
fine for me on Linux. I haven't reported the problem before, because  
I develop on OSX, but my servers are Linux, so not having zlib during  
debugging isn't a big deal.


Program received signal SIGABRT, Aborted.
0x9003d66c in kill ()
(gdb) where
#0  0x9003d66c in kill ()
#1  0x9010e8cf in raise ()
#2  0x9010d422 in abort ()
#3  0x0a05788e in Tcl_PanicVA ()
#4  0x0a0578a9 in Tcl_Panic ()
#5  0x0a064fdb in TclpFree ()
#6  0x0a00e9c2 in Tcl_DeleteCommandFromToken ()
#7  0x0a0526fc in TclTeardownNamespace ()
#8  0x0a00f172 in DeleteInterpProc ()
#9  0x0a00c9b1 in Tcl_DeleteInterp ()
#10 0x0007d60c in PushInterp ()
#11 0x0007467b in NsInitServer ()
#12 0x0006d85e in Ns_Main ()
#13 0x2a7e in main ()
--

Bug #2: occasional crash on OSX, when threads give notice of timeout  
waiting for connection


The gdb stack trace is a bit more useful in this case:

[07/Aug/2007:00:05:15][16497.25228800][-nssock:driver-] Notice:  
nssock: listening on 127.0.0.1:81
[07/Aug/2007:00:05:24][16497.25229824][-conn:0-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25203712][-conn:5-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25230848][-conn:2-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25229824][-conn:1-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25202688][-conn:4-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25201664][-conn:3-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25203712][-conn:6-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25229824][-conn:7-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25202688][-conn:9-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25204736][-conn:8-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25201664][-conn:10-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25203712][-conn:11-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25202688][-conn:12-] Notice: exiting:  
timeout waiting for connection


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0004
[Switching to process 16497 thread 0x3a0b]
0x0a00c130 in Tcl_AsyncDelete ()
(gdb) where
#0  0x0a00c130 in Tcl_AsyncDelete ()
#1  0x00655500 in SignalCmdCleanUp ()
#2  0x0a00f23a in DeleteInterpProc ()
#3  0x0a00c9b1 in Tcl_DeleteInterp ()
#4  0x0007da21 in DeleteData ()
#5  0x793a in NsCleanupTls ()
#6  0x891f in FreeThread ()
#7  0x90025400 in _pthread_tsd_cleanup ()
#8  0x90024f88 in pthread_exit ()
#9  0x8203 in Ns_ThreadExit ()
#10 0x0006f756 in NsConnThread ()
#11 0x88ee in ThreadMain ()
#12 0x90024227 in _pthread_body ()
(gdb)

This doesn't consistently occur, but it's often enough that I run nsd  
in a re-starting shell script while running on OSX.


--

I wish I could be more helpful and actually give a C code patch for  
these, but they're not aggravating enough for me to spend the ramp-up  
time figuring how to fix this sort of thing, but I thought at least I  
should report them so they could be logged.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] two OSX bugs

[John Buckman]

I banged on this thread-time-crash-on-OSX bug a bit, and found that  
setting

ns_param   threadtimeout   1   ;# Idle threads die at this rate

gets rid of this problem. While setting
ns_param   threadtimeout   0   ;# Idle threads die at this rate

or
ns_param   threadtimeout  1   ;# Idle threads die at this rate

causes the problem to come back. So, it seems the problem is  
generally with idle threads being shut down.


-john


Bug #2: occasional crash on OSX, when threads give notice of  
timeout waiting for connection


The gdb stack trace is a bit more useful in this case:

[07/Aug/2007:00:05:15][16497.25228800][-nssock:driver-] Notice:  
nssock: listening on 127.0.0.1:81
[07/Aug/2007:00:05:24][16497.25229824][-conn:0-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25203712][-conn:5-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25230848][-conn:2-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25229824][-conn:1-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25202688][-conn:4-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:24][16497.25201664][-conn:3-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25203712][-conn:6-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25229824][-conn:7-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25202688][-conn:9-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25204736][-conn:8-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25201664][-conn:10-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25203712][-conn:11-] Notice: exiting:  
timeout waiting for connection
[07/Aug/2007:00:05:25][16497.25202688][-conn:12-] Notice: exiting:  
timeout waiting for connection


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0004
[Switching to process 16497 thread 0x3a0b]
0x0a00c130 in Tcl_AsyncDelete ()
(gdb) where
#0  0x0a00c130 in Tcl_AsyncDelete ()
#1  0x00655500 in SignalCmdCleanUp ()
#2  0x0a00f23a in DeleteInterpProc ()
#3  0x0a00c9b1 in Tcl_DeleteInterp ()
#4  0x0007da21 in DeleteData ()
#5  0x793a in NsCleanupTls ()
#6  0x891f in FreeThread ()
#7  0x90025400 in _pthread_tsd_cleanup ()
#8  0x90024f88 in pthread_exit ()
#9  0x8203 in Ns_ThreadExit ()
#10 0x0006f756 in NsConnThread ()
#11 0x88ee in ThreadMain ()
#12 0x90024227 in _pthread_body ()
(gdb)

This doesn't consistently occur, but it's often enough that I run  
nsd in a re-starting shell script while running on OSX.


--

I wish I could be more helpful and actually give a C code patch for  
these, but they're not aggravating enough for me to spend the ramp- 
up time figuring how to fix this sort of thing, but I thought at  
least I should report them so they could be logged.


-john





--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver focus

[John Buckman]

Idle curiosity - I wonder if anyone is running a system with both  
apache and
aolserver listening on port 80 on different ifs/ips.  Should be  
possible and

not even difficult, tho probably of limited utility.


Yes, I have setups like this and is the best solution to this problem.
However, in many situations, multiple ip-addresses are unavailable and
Apache and AOLserver compete for port 80.


I use AOLserver + lighthttpd on the same machine, different IPs, so  
that mp3s and GIF/JPEGs are fed off lighthttpd, which is stupid and  
very fast (it's wikipedia's media server)


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver focus

[John Buckman]

On 2007.08.07, John Buckman <[EMAIL PROTECTED]> wrote:

I use AOLserver + lighthttpd on the same machine, different IPs, so
that mp3s and GIF/JPEGs are fed off lighthttpd, which is stupid and
very fast (it's wikipedia's media server)


Out of curiousity--have you benchmarked lighttpd vs. AOLserver (since
you already have both set up)?  AOL replaced their "ArtBlaster" (very
thin, lightweight, single-threaded HTTP server for static assets) a
while back and replaced it with AOLserver 4.0, since it was "fast
enough" ... if lighttpd's benchmarks are significantly better, I'd  
like

to try and understand why/how and start tuning AOLserver to match.

I'd just like to know if folks are running lighttpd because it  
truly is

faster, or simply because it isn't AOLserver ...


Yes, two yeasr ago I did 3 benchmarks, with CGI, small GIF image, and  
large mp3.  I also compared aolserver to a simple all-tcl single  
threaded web server which I obtained off the tcl wiki. All benchmarks  
below were on a powerpc mac mini, two years ago, with "ab" and 10  
simultaneous queries.


900 byte image fetch benchmark:
---
apache img fetch: 593 /s
aolserver img fetch: between 1019 and 1267/s
lighthttp img fetch: 1089/s
mathopd img fetch: 900/s
tclhttpd: 69/s
trivial http w/image cache: 1127/s

This big surprises were how well aolserver did on small images, as  
well as the all-tcl web server.


requests per second handled with trivial tcl dynamic web page ("hello  
world"):

---
lighttpd-cgi: 15/second
tclhttpd: 32/s
apache/php: 120/s
aolserver: between 640/s and 750/s
trivial all-tcl-web-server http://wiki.tcl.tk/15244: 1162/s

Unfortunately, I don't have any notes on the large image benchmark,  
but let me put some numbers in from memory:


1mb image fetch benchmark:
---
apache img fetch: 33 /s
aolserver img fetch: around 400/s
mathopd img fetch: 750/s
lighthttp img fetch: 900/s
tclhttpd: 3/s

--

as far as what lighthttpd and mathopd are doing to get better speeds,  
is that they both are not multithreaded, they are just a single async  
loop, serving static files.   I remember that this was an option in  
Aolserver v2, but I believe it went away in v3.  The async sending  
feature really only kicks the performance up with large files that  
require many writes to a socket.  For small files that fit inside 4k,  
aolserver is really fast.


What my benchmarks found is that except for possibly java-based web  
servers, nothing comes close to aolserver's performance for web- 
application building.  For small images, aolserver is competitive  
with lighthttpd.  It's only with large files that the async-event- 
loop approach yields better results.


One other note, is that mathopd/lighthttp use less than 5% of the CPU  
running the large image benchmark.  Mathopd also had an option to use  
SendFile() (which is no longer available on linux) which offloaded  
most of the work to the kernel.


I'm including my trivial all-tcl http server, just for reference.   
What surprised me was how badly performing tclhttpd was in all  
benchmarks, vs this all-tcl server design.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.


ht.tcl
Description: Binary data

Re: [AOLSERVER] aolserver focus

[John Buckman]

Or as an alternate answer: use apache itself as the proxy.  The  
poor saps who subject themselves to PHP will be happy and the OACS  
users can have a real system to work with.


Are there any caching proxy plugins for aolserver?  I have cheap  
bandwidth in other countries, which I'd like to load balance media  
file serving to.


I've used squid in the past, but its algorithm is impenetrable, and I  
didn't see the bandwidth savings I thought I should.


Something like this:
a) receive http request
b) is file available locally?
   b1) YES: return the file
   b2) NO: return HTTP redirect, and get the file from the source,  
so it's available for the next request

c) are we out of disk space? if so, delete the least used file(s)

Obviously, a very easy algorithm to write, and I was thinking of  
using aolserver to do this


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] async background delivery

[John Buckman]

as far as what lighthttpd and mathopd are doing to get better speeds,
is that they both are not multithreaded, they are just a single async
loop, serving static files.  I remember that this was an option in
Aolserver v2, but I believe it went away in v3.


Gustaf Neumann of WU-Wien patched AOLserver to do asynchronous
background delivery, and has been using the feature heavily since
2005:

  http://openacs.org/xowiki/weblog-portlet?ptag=asynchronous
  http://openacs.org/forums/message-view?message_id=482221


Very nice. The only thing I'd change would be to have this not  
require Tcl, but rather to have registered file types that are  
automatically sent via the async method (ie mp3/zip)


So, what's involved with having the patch applied to cvs?

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Speaking of nsdci...

[John Buckman]

On Aug 9, 2007, at 2:07 AM, Nathan Folkman wrote:


A variant is used in production, although it's not exactly the same.


Can you say more about variant?

There's only a couple of us working on updating this code at the  
moment, but I'd be more then happy to add anyone to the project who  
is interested in contributing. There are a bunch of sample  
configuration files, which honestly is the most confusing part of  
this code base. Contact me off list if interested in pitching in,  
and I'll get you set up.


Ok, so it's messy right now, so probably not good for merging in.

Anyhow, I'm not that concerned about this feature right now, but  
Dossy mentioned wanting to know why I used lighthttpd for large files  
instead of aolserver, and that's how this async issue came up.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] found a bug in nsd shutdown routine

[John Buckman]

I believe I've found a bug in queue.c which occasionally causes a  
core dump at exit.


The bug is caused by this order of events at the end of NsConnThread:

poolPtr->threads.idle--;
poolPtr->threads.current--;
if (poolPtr->threads.current == 0) {
Ns_CondBroadcast(&poolPtr->cond);
}
Ns_ThreadExit(dataPtr);

the Ns_ThreadExit doesn't always get to complete, because poolPtr- 
>threads.current can be set to zero before Ns_ThreadExit concludes,  
and thus nsd exits (since the server is waiting threads.current to  
reach zero) while the thread exit code is still running.


I believe the fix to this is to move the thread exit call  
(Ns_ThreadExit) to immediately *before* the decrementing of the  
current threads count, ie like so:


Ns_ThreadExit(dataPtr);

poolPtr->threads.idle--;
poolPtr->threads.current--;
if (poolPtr->threads.current == 0) {
Ns_CondBroadcast(&poolPtr->cond);
}

in my stress tests, this led to a successful nsd shutdown, even while  
the server was being pounded by http requests, whereas before I  
regularly crashed.


I found this bug by adding these log statements around the thread exit:

Ns_Log(Notice, "starting thread exit");
Ns_ThreadExit(dataPtr);
Ns_Log(Notice, "ending thread exit");

if you do this, and try to exit nsd during a stress test, you will  
find that several connection threads display the starting message,  
but never display the ending message. Sometimes this causes a crash,  
sometimes not, but it's always bad form.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] Switching to Trac to manage the AOLserver project?

[John Buckman]

After looking more closely at Trac [1], I'm really thinking it  
might be a

good idea to use it to manage the AOLserver project.  Does anyone have
any strong feelings about this, particularly against it?


I think this is a great idea.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] 64bit crashes right away

[John Buckman]

Is there any special trick to getting aolserver to work on 64bit linux?

A stock nsd crashes right away (strace below).

I'm using
Linux www64 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007  
x86_64 GNU/Linux


The only change I made from the cvs tree was the previously mentioned  
-nostartfiles mod:

CC  = $(PURIFY) gcc -nostartfiles

I'm crashing running "bin/nsd -h"

-john


strace of bin/nsd -h

...
munmap(0x2b0c9308a000, 51575)   = 0
open("/etc/host.conf", O_RDONLY)= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,  
0) = 0x2b0c9308a000

read(4, "# The \"order\" line is only used "..., 4096) = 92
read(4, "", 4096)   = 0
close(4)= 0
munmap(0x2b0c9308a000, 4096)= 0
futex(0x2b0c93c96ca0, FUTEX_WAKE, 2147483647) = 0
open("/etc/hosts", O_RDONLY)= 4
fcntl(4, F_GETFD)   = 0
fcntl(4, F_SETFD, FD_CLOEXEC)   = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=284, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,  
0) = 0x2b0c9308a000

read(4, "127.0.0.1\tlocalhost\n#127.0.1.1\tu"..., 4096) = 284
read(4, "", 4096)   = 0
close(4)= 0
munmap(0x2b0c9308a000, 4096)= 0
dup2(3, 0)  = 0
write(2, "\nError: required -t  opt"..., 50
Error: required -t  option not specified
) = 50
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 26160 detached

and normal startup:

...
munmap(0x2b2a082a7000, 4096)= 0
open("/usr/local/aolserver/lib/libnss_files.so.2", O_RDONLY) = -1  
ENOENT (No such file or directory)

open("/lib/libnss_files.so.2", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \0\0\0"...,  
832) = 832

fstat(4, {st_mode=S_IFREG|0644, st_size=43440, ...}) = 0
mmap(NULL, 2139464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,  
4, 0) = 0x2b2a082a7000

mprotect(0x2b2a082b1000, 2093056, PROT_NONE) = 0
mmap(0x2b2a084b, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE| 
MAP_FIXED|MAP_DENYWRITE, 4, 0x9000) = 0x2b2a084b

close(4)= 0
open("/etc/host.conf", O_RDONLY)= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,  
0) = 0x2b2a084b2000

read(4, "# The \"order\" line is only used "..., 4096) = 92
read(4, "", 4096)   = 0
close(4)= 0
munmap(0x2b2a084b2000, 4096)= 0
futex(0x2b2a082a2ca0, FUTEX_WAKE, 2147483647) = 0
open("/etc/hosts", O_RDONLY)= 4
fcntl(4, F_GETFD)   = 0
fcntl(4, F_SETFD, FD_CLOEXEC)   = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=283, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,  
0) = 0x2b2a084b2000

read(4, "127.0.0.1\tlocalhost\n127.0.1.1\tub"..., 4096) = 283
read(4, "", 4096)   = 0
close(4)= 0
munmap(0x2b2a084b2000, 4096)= 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 26091 detached




--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] 64bit crashes right away

[John Buckman]

Thanks for the tip. That gets me more-or-less working, though now  
document over a few kb "stick".


Here is what I did for a configure statement to stop crashing:

./configure --prefix=/usr/local/aol3 --enable-64bit --enable-threads  
--enable-shared --with-tcl=/usr/local/lib


I also did:
export CC="gcc -m64"

and also added -nostartfiles to get around the _init redefinition  
failure:

ns.mak:CFLAGS_OPTIMIZE = -O2 -nostartfiles

that gets nsd more-or-less working.

However, nsd now "sticks" on pages more than a few kb large, and a  
large JPG only loads 1/4 of the way.


Any tips?

-john

On Sep 11, 2007, at 5:21 PM, Tom Jackson wrote:


My laptop is 64 bit:
Linux localhost 2.6.18-gentoo-r4 #1 SMP Mon Nov 20 16:49:16 UTC  
2006 x86_64

AMD Turion(tm) 64 X2 Mobile Technology TL-52 AuthenticAMD GNU/Linux

For Tcl I do:
./configure --prefix=/web/nsd45 \
  --enable-64bit \
  --enable-threads \
  --enable-shared

But when it builds I get the impression it isn't 64 bit.  Don't  
know how to

check it.

AOLserver builds and runs as normal.

tom jackson



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] 64bit crashes right away

[John Buckman]

On 2007.09.11, John Buckman <[EMAIL PROTECTED]> wrote:
Is there any special trick to getting aolserver to work on 64bit  
linux?

Which flavor of Linux?  Debian?  Redhat?  Something else?
Which version of gcc?  And glibc?


Ubuntu server edition 7.04

Note in my previous followup message that using the older ./configure  
got me no longer crashing, but now large web pages stick mid- 
download.  Using nsconfig creates an nsd that crashes on "bin/nsd -h"


Machine Details:

uname -a
Linux www64 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007  
x86_64 GNU/Linux


 gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c+ 
+,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with- 
system-zlib --libexecdir=/usr/lib --without-included-gettext --enable- 
threads=posix --enable-nls --program-suffix=-4.1 --enable- 
__cxa_atexit --enable-clocale=gnu --enable-libstdcxx-debug --enable- 
mpfr --enable-checking=release x86_64-linux-gnu

Thread model: posix
gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)

[EMAIL PROTECTED]:~#  /lib/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
Compiled by GNU CC version 4.1.2 (Ubuntu 4.1.2-0ubuntu4).
Compiled on a Linux >>2.6.15.7<< system on 2007-04-04.
Available extensions:
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
GNU libio by Per Bothner
NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
Thread-local storage support included.


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] 64bit crashes right away

[John Buckman]

On Sep 12, 2007, at 9:47 PM, Tom Jackson wrote:

Below is my  configure and lib information.
On Wednesday 12 September 2007 07:42, Dossy Shiobara wrote:

On 2007.09.12, John Buckman <[EMAIL PROTECTED]> wrote:

[EMAIL PROTECTED]:~#  /lib/libc.so.6

What about /lib64/libc.so.6?
And, what's "ldd nsd" say you're actually linked to?


Here is mine:

The nsconfig.tcl built one:

[EMAIL PROTECTED]:/usr/local/aolserver.bak# bin/nsd -h
Segmentation fault
[EMAIL PROTECTED]:/usr/local/aolserver.bak# ldd bin/nsd
libnsd.so => /usr/local/aolserver/lib/libnsd.so  
(0x2b2ccebf9000)
libnsthread.so => /usr/local/aolserver/lib/libnsthread.so  
(0x2b2ccee81000)
libtcl8.4.so => /usr/local/aolserver/lib/libtcl8.4.so  
(0x2b2ccf08c000)

libdl.so.2 => /lib/libdl.so.2 (0x2b2ccf354000)
libpthread.so.0 => /lib/libpthread.so.0 (0x2b2ccf559000)
libm.so.6 => /lib/libm.so.6 (0x2b2ccf774000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2b2ccf9f6000)
libc.so.6 => /lib/libc.so.6 (0x2b2ccfc05000)
/lib64/ld-linux-x86-64.so.2 (0x2b2cce9dc000)
[EMAIL PROTECTED]:/usr/local/aolserver.bak#

and the working-but-sticks-on-large-pages from Tom's ./configure hack:

[EMAIL PROTECTED]:/usr/local/aolserver# ldd bin/nsd
libnsd.so => /usr/local/aol3/lib/libnsd.so (0x2b8ce3b39000)
libnsthread.so => /usr/local/aol3/lib/libnsthread.so  
(0x2b8ce3dc1000)
libtcl8.4.so => /usr/local/aol3/lib/libtcl8.4.so  
(0x2b8ce3fcc000)

libdl.so.2 => /lib/libdl.so.2 (0x2b8ce4294000)
libpthread.so.0 => /lib/libpthread.so.0 (0x2b8ce4499000)
libm.so.6 => /lib/libm.so.6 (0x2b8ce46b4000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2b8ce4936000)
libc.so.6 => /lib/libc.so.6 (0x2b8ce4b45000)
/lib64/ld-linux-x86-64.so.2 (0x2b8ce391c000)

appears identical to me.


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] AOLserver on 64-bit Linux

[John Buckman]

I'm not sure if my post is off-topic, but when I did compile  
Aolserver in my x86-64 machine I had problems with TCL, tDOM (not  
Aolserver).
You can see my problem in: http://openacs.org/forums/message-view? 
message_id=369867
It was a problem I had installing Aolserver. I think it doesn't  
matter here but perhaps it is.


I use tDOM on the 64 bit machine, and instead of installign tDOM from  
a tea file, I installed from the tarfile, with the command:


../configure --enable-threads --with-tcl=/usr/local/lib/  --enable-64bit

and didn't have any problems with tDOM under 64 bit aolserver.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] aolserver vs lighthttpd, benchmarks

[John Buckman]

I did some benchmarks of aolserver vs lighthttpd on plain files of  
various sizes, on my 8cpu 64 bit server.


For 3 runs, with a 4k text file, the lighthttpd stats were 15103.87/ 
s,  14845.20/s and 15307.17/s, vs (as Dossy reports http://dossy.org/ 
archives/000517.html) aolserver's 15237.00/s.


Result: Aolserver is performing virtually identically to lighthttpd  
with a small 4k file.


With a 44k JPG (the BookMooch home page illustration) aolserver is  
slightly slower:

aolserver: 8164.54/s, 8283.93/s
lighthttpd: 10281.01/s, 9969.91/s

With 1,414k sized zip, aolserver is about 1/2 as fast:
aolserver: 406.36/s,  421.10/s
lighthttpd: 844.05/s,  807.83/s

with a 128mb file, aolserver is 40% slower:
aolserver: 5.42/s
lighthttpd: 8.88/s

And just FYI "hello world" in tcl (<% ns_adp_puts "hello" %>):
aolserver: 950.85/s

I haven't done a "hello world" in C: I assume it'd be close to the  
16k/s speed


-john


www64:/b# ab -c 5 -n 5 http://images.bookmooch.com/x.txt
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking images.bookmooch.com (be patient)
Completed 5000 requests
Completed 1 requests
Completed 15000 requests
Completed 2 requests
Completed 25000 requests
Completed 3 requests
Completed 35000 requests
Completed 4 requests
Completed 45000 requests
Finished 5 requests


Server Software:lighttpd/1.4.18
Server Hostname:images.bookmooch.com
Server Port:80

Document Path:  /x.txt
Document Length:4070 bytes

Concurrency Level:  5
Time taken for tests:   3.266443 seconds
Complete requests:  5
Failed requests:0
Write errors:   0
Total transferred:  21895 bytes
HTML transferred:   20350 bytes
Requests per second:15307.17 [#/sec] (mean)
Time per request:   0.327 [ms] (mean)
Time per request:   0.065 [ms] (mean, across all concurrent  
requests)

Transfer rate:  65458.97 [Kbytes/sec] received

Connection Times (ms)
  min  mean[+/-sd] median   max
Connect:00   0.0  0   0
Processing: 00   0.1  0  12
Waiting:00   0.1  0  12
Total:  00   0.1  0  12

Percentage of the requests served within a certain time (ms)
  50%  0
  66%  0
  75%  0
  80%  0
  90%  0
  95%  0
  98%  0
  99%  0
100% 12 (longest request)


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks

[John Buckman]

www64:/b# ab -c 5 -n 5 http://images.bookmooch.com/x.txt


Since your machine is a 2-CPU 8-core box, could you try runs with -c 8
and -c 16?  It may have no effect, but it'd be nice to know that for
sure.


I get more-or-less the same numbers at -c 8, -c 16 and -c 32, varying  
about 100 requests/second between runs.


I'm not sure why the tcl "hello world" test yields speeds so much  
slower, with

ab -c 8 -n 5000 http://bookmooch.com/test.adp

yielding:
Requests per second:971.45 [#/sec] (mean)

On the positive side, a real world function, namely a registered tcl  
proc for returning a member's uploaded photo from the database:

ab -c 8 -n 5000 http://bookmooch.com/photo/johnbuckman.jpg

(I know it looks like an image url, but it's actually tcl) yields:
Requests per second:892.18 [#/sec] (mean)

which is a very good real-world speed (fyi, the jpg is 6831 bytes long).

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks

[John Buckman]

Hmm--I'm not sure what you're thinking of or referring to, but the
common "optimization" is to use sendfile(2), which seems to be
alive and well in the 2.6 tree.


Whoops, you're right, I just remembered a sysadmin email about  
sendfile not existing on the kernel we're using. Hmm...



Sendfile requires that the data to be written come from a file
descriptor.  The assumption is that the program opens a file, then  
wants
to pass it off to have it sent out another file descriptor,  
typically a

network connection.  But, what if you want to avoid disk I/O and cache
that data in memory?  sendfile() isn't necessarily workable, there.

Perhaps you could mmap() to a fd and use that--should test the
performance of such an implementation.


I've done exactly that in the past, and found virtually no speed  
improvement, because Unix has a very aggressive disk cache, so adding  
your own cache on top of that just skips a few system calls.



So, I don't see any point in worrying about improving aolserver's
plain-file-sending, at least until we get start getting 10 gigabit
network cards (!)


I am concerned about your "hello world" dynamic request benchmark,
though.  I would have expected at least 4k req/sec--not the sub-1k
req/sec you saw.  I have a feeling it has to do with the default
ns_pools/ns_limits settings, which naturally are NOT tuned for a 8- 
core

CPU box.

I bet with just a few minutes of tweaking and tuning, we can get  
between

4k-8k simple dynamic req/sec out of your hardware.


I have an 8 cpu Mac I, but it's not on the net (it's at my home), so  
I can't give you ssh access. If you can suggest a thing or two to  
try, I'll give it a whirl.


I benchmarked the "/helloworld" C module, and got about 16k responses  
per second vs 900/s with the tcl equivalent.


I've been looking at C-caching of Tcl dynamic content, with "dirty  
cache" support.  For example, replacing the Tcl code that returns a  
user's uploaded photo with C code.


I wrote C code to do this, and got 14k/second vs 240/s for the same  
tcl function.  So, C really is a good replacement for Tcl when a  
particular URL needs to be very fast.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] caching tcl pages with tcl access to mark the cache as dirty

[John Buckman]

Does a module exist that does this:

- register a Tcl function for a given URL, ie /photo/*
- and specifying an ns_cache for the resulting rendered page
- so that C code completely handles successive requests for that URL
- and so that if the back-end data changes, the ns_cache can be  
marked as "dirty" by another tcl function (ie, if the user uploads a  
new photo)


If not, I'm happy to write it, it's not hard, but I thought I'd ask.

-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] UTF/asian charset in urls

[John Buckman]

Does anyone know what these "$map(%E7%A7%8B)" things are in URLs when  
asian-UTF characters are used?


Ie, typing into a search form on aolserver (BookMooch) for the  
japanese character "秋" converts it to


http://bookmooch.com/m/s/$map(%E7%A7%8B)

That $map looks like an array lookup, but I've never seen any  
documentation about this, or how it might be re-converted back into  
UTF.  I can't find any mention of $map in the aolserver or tcl source  
code, nor online.


The other possibility is that the hex codes: %E7%A7%8B are just 3- 
byte representations of each character, in linear order, that needs  
to be reconverted back into UTF somehow.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks

[John Buckman]

On Sep 28, 2007, at 8:43 AM, John Buckman wrote:
My solution to that problem was simply caching in the filesystem  
and serving static files. The way this works in a multi-server  
environment is that the custom 404 handler figures out the request  
was for "/photo/123/axbcgsfdt.jpg" and just grabs it from the  
database, caches it and redirects back to it.
That way I get the convenience of Tcl code and the speed of static  
files - save for the first request.


The 404 handler approach is very clever, that'll do exactly what I  
need, thanks!


Dossy: not sure if it's even worth benchmarking this, as this  
approach will yield the static-file-speeds, which are amazing, except  
in the rare no-static-file-available case, where it's back to 900/s.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

[AOLSERVER] code for 404 pattern (aka static cache)

[John Buckman]

Bas Scheffers was kind to share his 404-pattern code with me (ie, a  
custom 404 handler to enable static caching of frequently requested  
files), which I used to write my own.


It's not rocket science, but since I asked the question, I thought  
I'd share my code, in case anyone else finds it helpful.


-john

from config.tcl:
ns_section "ns/server/$server1/redirects"
ns_param   404   /404.adp

$ cat 404.adp
<% handle_404 %>


proc handle_404 {} {
set myurl [myurl]
set parts [split $myurl /]
set root [lindex $parts 1]

if {$root == "photo"} {
return [404_photo $parts]
} else {
ns_returnredirect "[myhost]/"
return
}

}

proc 404_photo {parts} {

set userid [file rootname [lindex $parts 2]]
set d [user_photo_data $userid]
if {$d == ""} {
set fn "/b/photo/unknown.jpg"
} else {
set fn "/b/photo/${userid}.jpg"
write_binary_file $fn $d
}
ns_returnfile 200 "image/jpeg" $fn
}

proc myurl {} {
return [lindex [split [ns_conn request]] 1] 
}

proc photo_cache_dirty {userid} {
set fn "/b/photo/${userid}.jpg"
file delete $fn
}


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks

[John Buckman]

When comparing lighthttpd vs aolserver, notice that aolserver only  
does worse than lighthttpd for large files, and on the same file  
system/hardware. Thus, the difference in benchmarks is not likely to  
be the access logs or disk.


Lighthttpd is *not* using the system call to send a file to a socket  
(I forget the name) as this call was taken out of the Linux kernel, I  
believe with 2.4.  I remember reading a note about this from Linus,  
that the performance for that system call was terrible, so they were  
taking it out.


Based on my own experience with sending large files over tcpip, the  
difference is that aolserver uses a thread-based approach vs a  
lighthttpd's single-thread async approach.


But really, I'm not sure aolserver's performance is really an issue  
anyhow, because aolserver only slows down with large files (100mb+)  
and at that point, you're completely saturating your network  
connection (in my 128mb benchmark below, aolserver is sending 695mb  
per second!)


So, I don't see any point in worrying about improving aolserver's  
plain-file-sending, at least until we get start getting 10 gigabit  
network cards (!)


-john




On Sep 27, 2007, at 4:53 PM, Tom Jackson wrote:


I was looking at lighttpd performance
(at http://trac.lighttpd.net/trac/wiki/Docs%3APerformance )

Considering how well AOLserver stands up to lighttpd, my question  
was why does

lighttpd do better?

For sending small files AOLserver is slightly better, but then  
performance

goes down.

One reason might be that lighttpd uses a syscall that sends the  
file directly
to the network adapter, bypassing lighttpd. I wonder if AOLserver  
does this,
or if it is possible to detect errors in the webserver if the whole  
file

isn't sent.

Another reason might be that lighttpd doesn't write an access.log  
by default.
I wonder if you can turn this off in AOLserver? Or, if not, can the  
benchmark

be run for lighttpd with the access.log enabled:

http://trac.lighttpd.net/trac/wiki/Docs%3AModAccessLog

With multi-cpus and lighttpd processes, lighttpd might mess up the  
access.log
writing, but if these writes are taken out of the performance  
equation, it

sure gives lighttpd a headstart.

Are there any hints on how to setup the benchmarking? I can run one  
here on a

1cpu/2core/64bit laptop.

If we can establish a benchmarking suite, it would help test  
various setting

for ns_limits and ns_pools.

It would also be interesting to see benchmarks under misbehaving  
clients.


One other factor is number of disks. Lighttpd recommends 2 x number  
of disks
to determine processes, although it seems like this wouldn't be  
such an easy

choice.

Ideas?

tom jackson

On Tuesday 25 September 2007 00:50, John Buckman wrote:

I did some benchmarks of aolserver vs lighthttpd on plain files of
various sizes, on my 8cpu 64 bit server.

For 3 runs, with a 4k text file, the lighthttpd stats were 15103.87/
s,  14845.20/s and 15307.17/s, vs (as Dossy reports http://dossy.org/
archives/000517.html) aolserver's 15237.00/s.

Result: Aolserver is performing virtually identically to lighthttpd
with a small 4k file.

With a 44k JPG (the BookMooch home page illustration) aolserver is
slightly slower:
aolserver: 8164.54/s, 8283.93/s
lighthttpd: 10281.01/s, 9969.91/s

With 1,414k sized zip, aolserver is about 1/2 as fast:
aolserver: 406.36/s,  421.10/s
lighthttpd: 844.05/s,  807.83/s

with a 128mb file, aolserver is 40% slower:
aolserver: 5.42/s
lighthttpd: 8.88/s

And just FYI "hello world" in tcl (<% ns_adp_puts "hello" %>):
aolserver: 950.85/s

I haven't done a "hello world" in C: I assume it'd be close to the
16k/s speed



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to  
<[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the  
Subject: field of your email blank.



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] code for 404 pattern (aka static cache)

[John Buckman]

On Sep 29, 2007, at 11:00 PM, Jeff Rogers wrote:


John Buckman wrote:
Bas Scheffers was kind to share his 404-pattern code with me (ie,  
a custom 404 handler to enable static caching of frequently  
requested files), which I used to write my own.
It's not rocket science, but since I asked the question, I thought  
I'd share my code, in case anyone else finds it helpful.


I've taken the liberty of adding this to the wiki, on the  
"Cookbook" page: http://panoptic.com/wiki/aolserver/AOLserver_Cookbook


Thanks for reminding us of this wiki page.  I'll endeavor to put my  
"bits" in there in the future.


-john


--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.

Re: [AOLSERVER] UTF/asian charset in urls

[John Buckman]

On Oct 1, 2007, at 8:52 AM, Hossein Sharifi wrote:

I don't see this problem in 4.0 or 4.5.
It looks like searching for 秋 on bookmooch first goes to
http://www.bookmooch.com/search?w=%E7%A7%8B&search.x=14&search.y=13
(which looks fine - that's the correct URL encoding of the UTF-8  
representation of that character)
but that page immediately redirects to a cleaner search URL  
( http://bookmooch.com/m/s/...) which contains the $map(...).   
Maybe it's related to the code that builds the cleaner URL?


Thanks Hossein, for the insight.

You were right, my problem was caused by a bug in the ncgi url  
encoding function.


ncgi builds a $map() array of character conversions, and puts $map 
(character) around anything that isn't A-Za-Z0-9, then a [subst - 
nocommand $string] around all that.


The higher-UTF characters cause problems with the ncgi function, and  
they emerge from it with $map() wrapped around them.  A simple regexp  
to remove $map() from what ncgi can't encode, and now it works  
perfectly.


I have to say that the aolserver handling of UTF is really well done.

At Lyris, we never did quite get the UTF handling in TclHttpd done  
perfectly, there were still some fringe cases that caused garbling.


With aolserver, except for this ncgi problem, and figuring out that I  
needed to switch to utf-8 as the default (from the default iso  
8859-1), non-english character sets have worked perfectly.


-john



--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> 
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: 
field of your email blank.