On Jun 30, 8:27 am, Gustaf Neumann <neum...@wu-wien.ac.at> wrote: > Am 30.06.10 00:23, schrieb Sep Ng:> Basically we had aolservers running and > while serving pages, it's also > > doing some heavy load processing from a ton of scheduled custom > > written procedures. Aolserver crashes and segmentation faults are > > fairly frequent and the logs at the time pointed to these running > > threads as a probable cause. Possibly a configuration issue, but I > > remember we tried fiddling with those numbers and it never really > > helped. > > From my experience, the two most common causes for crashes in aolserver > are > a) c-based extensions (tcl extensions and/or aolserver modules) which > are not thread > safe (e.g. using non thread-safe libraries) or which have bugs > (e.g. double frees of > memory), or > b) running out of memory. The aolserver threads are typically > heavy-weight, > the zippy memory allocator over-allocates memory and does not > free it. All threads use the same memory block, which is limited > on most > 32bit machines to 2GB. When you hit the limit, the server crashes. > > A solution for (b) is to compile everything (tcl, aolserver, modules) > with 64 bit. That's a good point, but I'm not entirely sure that aolserver ever really hit the 2GB limit. I guess I couldn't really speculate on it.
> If you are using just Tcl + Aolserver commands, the server should never > crash > with recent versions of aolserver and tcl (e.g. tcl 8.4.19, 8.5.8). > What C extensions or aolserver modules are you using? > I think we tend to use a lot of aolserver modules. And I would suppose a little haphazardly too. These are the extra ones that I remember. nsoracle nscache nsclamav nsaspell (not sure) nssha1 (not sure) nszlib nsopenssl (not sure) and nsreturnz on 4.0.10 - basically this was a port I found sometime back but I know we're removing it when we fully move to 4.5.1. How can I tell which ones are thread safe? This sounds like something I will need to look into before I start writing code. > The following heuristic might help in the search of the problem. > If your crashes are random and independent of the load (e.g. happen with > already > a little load) then bugs in the C code of the extensions are likely. If > crashes > happens during thread end, bugs in the memory management of the > c-extensions > are likely (tcl deletes an interpreter with all associated memory). If > the crashes > happen under heavy load, thread-safety of some component is likely to > be the problem (e.g. we had a long time problems with our aolserver, which > ran stable up to about 600 concurrent users, then it started to crash. > One of > the problem was the kerberos library, which was not reentrant, although > it claimed to be so).> So basically, what I'm currently beating my head over > is to > > build a much cleaner and better way of handling all the load but in so > > doing, I'm not entirely sure whether or not to spawn a lot of threads > > for the jobs, or basically keep it to a minimum. > > > Judging from Andrew's post, it would likely be better to reuse threads > > but I'm not entirely sure how that happens. I mean, everytime you'd > > invoke ns_thread begin/begindetached, you are creating a new thread > > already, no? How do you reuse them? > > In general, you have two options: you can use aolserver-threads and threads > created by the tcl thread library (libthread), which can be loaded as a > module > into the aolserver (when compiled with the appropriate flags). > I prefer libthread, since all such threads run in an event loop. > This makes it very easy to send to these threads commands which are > implemented via event-loops. One can as well to implement this way > easy communication between the threads. It is essentially the same > programming model that can be used by using tcl+libthread outside > of the aolserver. I don't think I've ever heard of this on Aolserver... I always thought Aolserver's threads would eventually end up using the tcl+libthread but it seems that there's a real difference in this. > This is probably a stupid question, but is there a distinction between > > the threads and connection threads in aolserver? I know the > > connection threads are probably the connections to the database (I > > think). > > no, the connection threads handle incoming HTTP requests, they have a > connection > to the web-client. These are the threads controlled in the config file via > e.g, maxthreads. Typically, a connection thread is created on demand > (incoming > request) and lives, until either it times out (it received no requests > for a > certain time) or it has served a maximum number of requests (all > configurable > in the config file). When it reaches the end of its live-cycle, it is > terminated > (and maybe recreated afterwards by new demand). > > The threads of aolserver are typically heavyweight, since every thread > contains a full tcl interpreter. The "weight" depends on the size of the > "blueprint", which is determined essentially by all tcl-procs available > at runtime. If you are using e.g. OpenACS with many OpenACS > packages defined, these might be several thousand procs. When a > thread starts, it reads the blue-print with all procs, which might take > a while. If you are using just Aolserver with a few tcl-modules, > the cost of starting a new thread (essentially loading the blueprint) > might not be so bad, and you might start for every jobs a new thread. > But this certainly depends on your needs and the kind of performance > you might want to reach. > So typically the config file has no connection whatsoever to the threads of aolserver and that it only pertains to the connection threads, or am I confusing this even further? The config file seems to deal with concurrency in terms of how many users / connections the server is supposed to expect and manage. It looks like the best way to handle this is to start a thread for handling the jobs and then basically submitting the jobs into the running thread. I guess, typically... having to > -gustaf neumann > > > > -- > AOLserver -http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <lists...@listserv.aol.com> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <lists...@listserv.aol.com> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
