[AOLSERVER] Lengthy tcl interp initialization
Calling all gurus - any advice is greatly appreciated We have a server that loads in a great deal of tcl - so much so that the resulting init script is 6.6M in size and contains over 9900 procs. In going from AOLserver 3.5.5 (tcl 8.4.4) to AOLserver 4.0 (tcl 8.4.4) we have seen two unexpected effects: 1. The startup of an individual thread has more than doubled in time (to ~4 seconds) 2. The slowdown gets even worse with multiple threads. On server initialization, we start a number of threads simultaneously (~30-40) and have seen the overall server start time seriously degrade (to ~15 minutes) Item number one is still a bit of a mystery. The init script is not dramatically different between our 3.5 server and our 4.0 server. A somewhat unscientific attempt to load the 4.0 init script into a 3.5 server showed a similar slowdown, so we do not currently believe that this slowdown is due to a change in the AOLserver core thread initialization code between 3.5 and 4.0 (but we haven't totally ruled that out). The one difference we do know of is the use of tcl packages, but its not yet clear to us how that might dramatically affect the initialization time. Based on data from various tools (quantify, pstack, etc.) - item number two seems to be due to a huge amount of malloc contention as the init scripts are simultaneously evaluated. Since, at the time of the processing, the thread is being initialized, the thread memory allocator does not help avoid this contention since it must malloc to populate the pool. We have implemented a less-than-ideal workaround by adding the optional 'tclinitlock' ns parameter that forces serialization of the thread initialization (thus reducing the malloc thrashing and cutting our initialization time from 15 minutes to 6) - but this is obviously not an ideal solution. Call stacks of the two most prevalent areas of contention are included at the end of this e-mail. For grins, we have tried some experiments with running without the thread memory allocator, and running with hoard - but this has just made the performance worse. We are then left with two problems to solve: 1. Is there any way to reduce the time to initialize an interp? (besides the obvious, but not necessarily feasible, option of reducing the procs that are loaded). Has anyone seen similar behavior and have some insight into it? 2. Is there any way to avoid the malloc contention, but still allow concurrent processing? Our investigation is continuing, but this is really the last known issue before we feel we can declare 4.0 ready for prime time, so any help is greatly appreciated. Thanks, -Elizabeth Call Stacks from malloc contention: ff099ae0 lwp_sema_wait (f818be78) feecb288 _park (f818bdc0, f818be78, 0, feef8a6c, f458bdc0, 0) + 10c feecacc4 _swtch (5, feeec9ac, f818be54, f818be50, f818be4c, f818be48) + 128 feecc814 _mutex_adaptive_lock (ff0b9fc8, 4c00, feeec9ac, 1, 4d58, fffe) + 120 feecc5c4 _cmutex_lock (ff0b9fc8, ff, feeec9ac, ff045724, 0, 0) + 50 ff045724 malloc (3f9c, 0, fc, 8, f818b30c, f818b308) + 18 ff2612e0 GetBlocks (6ecb7a8, 8, e0, 0, 1c, 0) + 484 ff25fd48 TclpAlloc (a25, 1b3, feeec9ac, f818adec, 0, ff00) + 124 ff1bc6ec Tcl_Alloc (a25, a25, f818ace4, f818ac08, f818ac88, 2dae8a78) + 18 ff25cfbc Tcl_NewStringObj (1c776170, a24, ff2a857c, ff21500c, ff283ec0, 0) + bc ff254884 TclCreateProc (6e449e8, 3da7c88, 164637e8, 6ffef70, 2247b4c0, f818af44) + bc ff2544b4 Tcl_ProcObjCmd (0, 6e449e8, 4, 6eaba74, ff254260, 0) + 254 ff1b4e14 TclEvalObjvInternal (6e449e8, 4, 6eaba74, 0, 0, 0) + 4dc ff1fb95c TclExecuteByteCode (6e449e8, 297f3a38, ff1e7c7c, 1000, f818b30c, f818b308) + 13e0 ff1fa478 TclCompEvalObj (6e449e8, 715c690, ff2420a8, 3a00, f818b4d0, f818b4d4) + 224 ff1b6654 Tcl_EvalObjEx (6e449e8, 715c690, 0, 0, ff28b768, 0) + 13c ff240de4 NamespaceEvalCmd (0, 6e449e8, 4, f818b728, 0, f818b5e4) + 170 ff24019c Tcl_NamespaceObjCmd (0, 6e449e8, 4, f818b728, ff23fffc, f818b674) + 1a0 ff1b4e14 TclEvalObjvInternal (6e449e8, 4, f818b728, 1672e112, 644455, 0) + 4dc ff1b5d58 Tcl_EvalEx (6e449e8, 1672de38, 64c979, 2, 0, 734c5c5) + 318 ff34d01c UpdateInterp (734c490, 1, ff34d06c, 734c490, ff2a8a1c, 0) + 68 ff34cf58 InitInterp (6e449e8, 406e0, f818ba64, , fff8, 734ca20) + 1b8 ff34bc8c Ns_TclAllocateInterp (34c98, ff38620c, ff38620c, f818bbc8, 64, f818bcb8) + b0 ff34b914 Ns_TclEval (f818bb68, 34c98, 7447540, 332d, 332d, 7438170) + 1c ff35be74 NsTclThread (7447538, 7438158, ff35be08, ff2e7d10, 0, 0) + 6c ff2e4ee0 NsThreadMain (7438158, 0, 0, 0, 0, 0) + 8c ff2e76ac ThreadMain (7438158, 0, 1, f000, 1, feeec9ac) + c feedbc7c _thread_start (7438158, 0, 0, 0, 0, 0) + 40 ff099ae0 lwp_sema_wait (e798be78) feecb288 _park (e798bdc0, e798be78, 0, feef8a6c, ec18bdc0, 0) + 10c feecacc4 _swtch (5, feeec9ac, e798be54, e798be50, e798be4c, e798be48) + 128 feecc814 _mutex_adaptive_lock (ff0b9fc8, 4c00, feeec9ac, 1, 4d58, fffe) + 120 feecc5c4 _cmutex_lo
Re: [AOLSERVER] Lengthy tcl interp initialization
It appears that my original send of this e-mail (sent from the new AOL communicator) was eaten by some spam filters (for reasons only partially understood). I am attempting to resend using AOL 8.0, so if people have problems with this copy as well, please let me know. Meanwhile, I apologize for the redundant send. -Elizabeth --- Calling all gurus - any advice is greatly appreciated We have a server that loads in a great deal of tcl - so much so that the resulting init script is 6.6M in size and contains over 9900 procs. In going from AOLserver 3.5.5 (tcl 8.4.4) to AOLserver 4.0 (tcl 8.4.4) we have seen two unexpected effects: 1. The startup of an individual thread has more than doubled in time (to ~4 seconds) 2. The slowdown gets even worse with multiple threads. On server initialization, we start a number of threads simultaneously (~30-40) and have seen the overall server start time seriously degrade (to ~15 minutes) Item number one is still a bit of a mystery. The init script is not dramatically different between our 3.5 server and our 4.0 server. A somewhat unscientific attempt to load the 4.0 init script into a 3.5 server showed a similar slowdown, so we do not currently believe that this slowdown is due to a change in the AOLserver core thread initialization code between 3.5 and 4.0 (but we haven't totally ruled that out). The one difference we do know of is the use of tcl packages, but its not yet clear to us how that might dramatically affect the initialization time. Based on data from various tools (quantify, pstack, etc.) - item number two seems to be due to a huge amount of malloc contention as the init scripts are simultaneously evaluated. Since, at the time of the processing, the thread is being initialized, the thread memory allocator does not help avoid this contention since it must malloc to populate the pool. We have implemented a less-than-ideal workaround by adding the optional 'tclinitlock' ns parameter that forces serialization of the thread initialization (thus reducing the malloc thrashing and cutting our initialization time from 15 minutes to 6) - but this is obviously not an ideal solution. Call stacks of the two most prevalent areas of contention are included at the end of this e-mail. For grins, we have tried some experiments with running without the thread memory allocator, and running with hoard - but this has just made the performance worse. We are then left with two problems to solve: 1. Is there any way to reduce the time to initialize an interp? (besides the obvious, but not necessarily feasible, option of reducing the procs that are loaded). Has anyone seen similar behavior and have some insight into it? 2. Is there any way to avoid the malloc contention, but still allow concurrent processing? Our investigation is continuing, but this is really the last known issue before we feel we can declare 4.0 ready for prime time, so any help is greatly appreciated. Thanks, -Elizabeth Call Stacks from malloc contention: ff099ae0 lwp_sema_wait (f818be78) feecb288 _park (f818bdc0, f818be78, 0, feef8a6c, f458bdc0, 0) + 10c feecacc4 _swtch (5, feeec9ac, f818be54, f818be50, f818be4c, f818be48) + 128 feecc814 _mutex_adaptive_lock (ff0b9fc8, 4c00, feeec9ac, 1, 4d58, fffe) + 120 feecc5c4 _cmutex_lock (ff0b9fc8, ff, feeec9ac, ff045724, 0, 0) + 50 ff045724 malloc (3f9c, 0, fc, 8, f818b30c, f818b308) + 18 ff2612e0 GetBlocks (6ecb7a8, 8, e0, 0, 1c, 0) + 484 ff25fd48 TclpAlloc (a25, 1b3, feeec9ac, f818adec, 0, ff00) + 124 ff1bc6ec Tcl_Alloc (a25, a25, f818ace4, f818ac08, f818ac88, 2dae8a78) + 18 ff25cfbc Tcl_NewStringObj (1c776170, a24, ff2a857c, ff21500c, ff283ec0, 0) + bc ff254884 TclCreateProc (6e449e8, 3da7c88, 164637e8, 6ffef70, 2247b4c0, f818af44) + bc ff2544b4 Tcl_ProcObjCmd (0, 6e449e8, 4, 6eaba74, ff254260, 0) + 254 ff1b4e14 TclEvalObjvInternal (6e449e8, 4, 6eaba74, 0, 0, 0) + 4dc ff1fb95c TclExecuteByteCode (6e449e8, 297f3a38, ff1e7c7c, 1000, f818b30c, f818b308) + 13e0 ff1fa478 TclCompEvalObj (6e449e8, 715c690, ff2420a8, 3a00, f818b4d0, f818b4d4) + 224 ff1b6654 Tcl_EvalObjEx (6e449e8, 715c690, 0, 0, ff28b768, 0) + 13c ff240de4 NamespaceEvalCmd (0, 6e449e8, 4, f818b728, 0, f818b5e4) + 170 ff24019c Tcl_NamespaceObjCmd (0, 6e449e8, 4, f818b728, ff23fffc, f818b674) + 1a0 ff1b4e14 TclEvalObjvInternal (6e449e8, 4, f818b728, 1672e112, 644455, 0) + 4dc ff1b5d58 Tcl_EvalEx (6e449e8, 1672de38, 64c979, 2, 0, 734c5c5) + 318 ff34d01c UpdateInterp (734c490, 1, ff34d06c, 734c490, ff2a8a1c, 0) + 68 ff34cf58 InitInterp (6e449e8, 406e0, f818ba64, , fff8, 734ca20) + 1b8 ff34bc8c Ns_TclAllocateInterp (34c98, ff38620c, ff38620c, f818bbc8, 64, f818bcb8) + b0 ff34b914 Ns_TclEval (f818bb68, 34c98, 7447540, 332d, 332d, 7438170) + 1c ff35be74 NsTclThread (7447538, 7438158, ff35be08, ff2e7d10, 0, 0) + 6c ff2e4ee0 NsThreadMain (7438158, 0, 0, 0,
Re: [AOLSERVER] Lengthy tcl interp initialization
This was our working theory for awhile; however, it has not so far proven to be the source of the slowdown. (although we have found that setting the TCL_LIBRARY environment variable is a very good idea). Thanks! -Elizabeth Brett Schwarz wrote: > Item number one is still a bit of a mystery. The init > script is notdramatically different between our 3.5 > server and our 4.0 server. Asomewhat unscientific > attempt to load the 4.0 init script into a 3.5server > showed a similar slowdown, so we do not currently > believe thatthis slowdown is due to a change in the > AOLserver core threadinitialization code between 3.5 > and 4.0 (but we haven't totally ruledthat out). The > one difference we do know of is the use of tcl > packages,but its not yet clear to us how that might > dramatically affect theinitialization time. > > * If auto_path has a lot of directories to check, then > it will take a while to search through the filesystem. > This *may* be the problem you see with packages. When > you do a [package require] it searches for that > package based on the auto_path variable. So, if there > are alot of directories, then the longer it will take > to search. You could try an experiment, and try using > source/load instead of [package require], or see what > auto_path is set as, to see if it is too long. If it > is long, try trimming that to just what is actually > needed. > > --brett > > > __ > Do you Yahoo!? > Yahoo! SiteBuilder - Free, easy-to-use web site design software > http://sitebuilder.yahoo.com > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[EMAIL PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On Wednesday 13 August 2003 17:16, Elizabeth Thomas wrote: > 1. Is there any way to reduce the time to initialize an interp? (besides > the obvious, but not necessarily feasible, option of reducing the procs > that are loaded). Has anyone seen similar behavior and have some insight > into it? IMHO, unless somebody implements: int Tcl_CloneInterp(Tcl_Interp *interp) at the C-level (a non-mt-safe implementation is lurking somewhere in the Tcl CVS branches, done by Jeff, Andreas and co at ActiveState), you will have little luck with this issue. Although I doubt that this kind of function is possible to implement generically in mt-safe fashion, one may get a good approximation which would (could) be satisfactory for the AS usage. So, the question is: who would dare to take this step? Zoran -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
+-- On Aug 13, Zoran Vasiljevic said: | Although I doubt that this kind of function is possible to implement | generically in mt-safe fashion, one may get a good approximation | which would (could) be satisfactory for the AS usage. AOLserver could keep an initialized "donor" interpreter around (for each VS) that is never used to execute code after it executes the init script. Initializing another interpreter becomes a matter of cloning the donor interpreter. Serializing the cloning operation between threads might be necessary, but shouldn't have a significant performance impact. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On Wednesday 13 August 2003 17:51, Rob Mayoff wrote: > +-- On Aug 13, Zoran Vasiljevic said: > | Although I doubt that this kind of function is possible to implement > | generically in mt-safe fashion, one may get a good approximation > | which would (could) be satisfactory for the AS usage. > > AOLserver could keep an initialized "donor" interpreter around (for > each VS) that is never used to execute code after it executes the init > script. Initializing another interpreter becomes a matter of cloning the > donor interpreter. Serializing the cloning operation between threads > might be necessary, but shouldn't have a significant performance impact. > This is correct. One may also try to load from bytecodes instead of the string (TclPro compiler, right?). This may reduce the time to get the donor/master up. But still, I'd like to get some initiative to get Tcl_CloneInterp() in place because this is *very* sexy. Zoran -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
Item number one is still a bit of a mystery. The init script is notdramatically different between our 3.5 server and our 4.0 server. Asomewhat unscientific attempt to load the 4.0 init script into a 3.5server showed a similar slowdown, so we do not currently believe thatthis slowdown is due to a change in the AOLserver core threadinitialization code between 3.5 and 4.0 (but we haven't totally ruledthat out). The one difference we do know of is the use of tcl packages,but its not yet clear to us how that might dramatically affect theinitialization time. * If auto_path has a lot of directories to check, then it will take a while to search through the filesystem. This *may* be the problem you see with packages. When you do a [package require] it searches for that package based on the auto_path variable. So, if there are alot of directories, then the longer it will take to search. You could try an experiment, and try using source/load instead of [package require], or see what auto_path is set as, to see if it is too long. If it is long, try trimming that to just what is actually needed. --brett __ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On 2003.08.13, Elizabeth Thomas <[EMAIL PROTECTED]> wrote: > We have a server that loads in a great deal of tcl - so much so that the > resulting init script is 6.6M in size and contains over 9900 procs. > [...] > > 1. Is there any way to reduce the time to initialize an interp? > (besides the obvious, but not necessarily feasible, option of reducing > the procs that are loaded). Has anyone seen similar behavior and have > some insight into it? Is there any chance to do further profiling? Where are we losing most of our time? I imagine it has a lot to do with allocating 6.6M of memory for the init script, filling it with the script, then telling Tcl to go parse and execute it. I think the biggest win would be to try and figure out how to get Tcl to bytecode compile the init script, then push that bytecode from the master interp into the new slave interps as they get created. Cut out the entire parse/bytecode compile steps. Perhaps someone who knows and understand those intricate details of Tcl 8.4 can speak up? -- Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
We are currently pursuing a very encouraging approach of adding an optional 'lazy proc definition' capability, capitalizing on the 'unknown' processing of tcl. (Thanks to Jeff Hobbes for putting us on this path). Since most of our threads use a relatively small subset of all available procs, we hope to achieve significant performance and memory consumption wins by only loading procs in the interpeter that are actually needed. More details to come early next week. -Elizabeth Dossy wrote: > On 2003.08.13, Elizabeth Thomas <[EMAIL PROTECTED]> wrote: > > We have a server that loads in a great deal of tcl - so much so that > the > > resulting init script is 6.6M in size and contains over 9900 procs. > > [...] > > > > 1. Is there any way to reduce the time to initialize an interp? > > (besides the obvious, but not necessarily feasible, option of reducing > > the procs that are loaded). Has anyone seen similar behavior and have > > some insight into it? > > Is there any chance to do further profiling? Where are we losing most > of our time? > > I imagine it has a lot to do with allocating 6.6M of memory for the init > script, filling it with the script, then telling Tcl to go parse and > execute it. > > I think the biggest win would be to try and figure out how to get Tcl to > bytecode compile the init script, then push that bytecode from the > master interp into the new slave interps as they get created. Cut out > the entire parse/bytecode compile steps. Perhaps someone who knows and > understand those intricate details of Tcl 8.4 can speak up? > > -- Dossy > > -- > Dossy Shiobara mail: [EMAIL PROTECTED] > Panoptic Computer Network web: http://www.panoptic.com/ > "He realized the fastest way to change is to laugh at your own > folly -- then you can let go and quickly move on." (p. 70) > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[EMAIL PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On Fri, 15 Aug 2003, Elizabeth Thomas wrote: > We are currently pursuing a very encouraging approach of adding an > optional 'lazy proc definition' capability, capitalizing on the > 'unknown' processing of tcl. (Thanks to Jeff Hobbes for putting us on > this path). Since most of our threads use a relatively small subset of > all available procs, we hope to achieve significant performance and > memory consumption wins by only loading procs in the interpeter that are > actually needed. An approach like this was discussed on this list a couple of years ago, and Rob Mayoff had particular suggestions for handling namespaces. At the time, if I recall, some AOLserver mods were necessary to support proper handlikng of namespaces, but those are probably obsolete now. Also, at the time, Jim felt the approach was not necessary for AOL, so it didn't make it into the default thread initialization. The old notes are probably in the list archives. Pete. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On 2003.08.15, Elizabeth Thomas <[EMAIL PROTECTED]> wrote:
> We are currently pursuing a very encouraging approach of adding an
> optional 'lazy proc definition' capability, capitalizing on the
> 'unknown' processing of tcl. (Thanks to Jeff Hobbes for putting us on
> this path). Since most of our threads use a relatively small subset of
> all available procs, we hope to achieve significant performance and
> memory consumption wins by only loading procs in the interpeter that are
> actually needed.
>
> More details to come early next week.
This works for procs, but what about the rest of the interp init code?
Suppose you have this in your interp init script:
foreach procName [list a b c d] {
proc foo_$procName args "
eval some_proc $procName \$args
"
}
Pretty simple, you'd only worry about foo_a through foo_d, regardless of
how they came to be defined.
But, what about:
global x
set x 0
foreach x [list 1 2 3 4 5] {
proc foo_$x args "
global x
format {the last proc defined is %u} \$x
"
}
I really wish I had /real/ code to quote here, but the point is: there
CAN be code that doesn't live within a proc definition in your interp
scripts that have side-effects that matter.
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of
your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On Friday 15 August 2003 17:44, you wrote: > > This works for procs, but what about the rest of the interp init code? > The blueprint script can be modified to store plain proc definitions (Part1) apart from any other non-proc code (Part2) Then load the Part2 always and use unknown to fish out proc definitions from Part1. The defs can be retrieved from the master interp easily during runtime on the fly. Would require some internal changes to the Tcl init code and one or two new ns_ commands but it may work. On the first glance, I like the idea. One may also make use of nsv to avoid changing internals. Zoran -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
Just as an alternative approach have you thought of breaking your 9900 procs into separate nsds that tackle separate problems and have a proxy server out front of your nsd that route the call to the appropriate nsd. For example if your app has a message board part . only load the common procs and the message board procs for that server. Then configure the proxy/load balancer to direct all call to the message boards to that nsd. I'm sure you are already multiple machines, right? Many load balancers now can direct the request by Url in addition to Round Robin, Weighted, Least Connection, etc. Just an idea, Carl Garland _ Add photos to your messages with MSN 8. Get 2 months FREE*. http://join.msn.com/?page=features/featuredemail -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
More information on our progress on this issue. We can discuss in further detail at today's chat if desired. (For purposes of this discussion, it should be assumed that we do not have the option of reducing/distributing the size/number of procs that our server must have access to. - but that doesn't mean it isn't also continually evaluated :) ) We are experimenting with the introduction of an optional 'lazy proc definition' capability. This would be a new configuration parameter for AOLserver 4.0. At this moment it is at the process level. (depending on time permitting will dictate whether it gets moved to the virtual server level) The default value is false. If the value is false, server behavior is exactly as it was prior to the introduction of this value. ns/parameters ns_param lazyprocdef true When AOLserver initializes, it creates its 'init script' which contains all the variables, procs, etc. which will be loaded into any tcl interpreter. On one of our servers, where over 9900 procs are sourced, this init script is over 6.5M in size. The result is that every thread requires the allocation of this memory (and then some), plus the evaluation of all that tcl. This led to lengthy thread initialization times as well as memory allocation lock contention (which further degraded performance). When 'lazyprocdef' is set to true, we do not put the procs in the init script (all other tcl commands (e.g. variables) will still be there). Instead, we stash the procs in a centralized store. When tcl tries to execute a command that it doesn't know about - it calls the tcl 'unknown' command. We have wrapped the tcl unknown command with our own which will first look for the command in our central store and load it (if it isn't there, the tcl 'unknown' command is processed as before). This allows the threads to only load the procs they actually use - improving performance and reducing memory consumption. The results have been dramatic. Thread initialization times dropped from 4-5 seconds to ~100 milliseconds. The overall process size is a 1/3 of what it was. The effect on request times is not yet known (but we will be looking very closely at that). In theory, there could be a degradation on the first request to a connection thread (vs. a request to a connection thread that was 'warmed up' at server startup time). As well as locking contention on the central store. The hardest part is dealing with namespaces, the 'info' commands, and ns_eval. At this point, we *think* we have them addressed. For 'info', we wrap the info command. On info commands or info procs, we merge the results of our stored procs and the results of tcl info. For commands against a specific proc (e.g. info args), we load the proc, and then just call tcl 'info'. To handle ns_eval (which is indeed evil), we must create a copy of the nsv arrays for each ictl epoch. To do proper garbage collection we need to keep track of reference pointers so we can dump the older ones once they are no longer referenced. At this time, we've got the whole thing coded in nsd/init.tcl using nsv arrays as the proc store. Future optimizations could move some of this into more efficient C implementations. If all goes well, we hope to have this ready for incorporation into a beta12 release in the next few days... Cheers, -Elizabeth Elizabeth Thomas wrote: > We are currently pursuing a very encouraging approach of adding an > optional 'lazy proc definition' capability, capitalizing on the > 'unknown' processing of tcl. (Thanks to Jeff Hobbes for putting us on > this path). Since most of our threads use a relatively small subset of > all available procs, we hope to achieve significant performance and > memory consumption wins by only loading procs in the interpeter that are > actually needed. > > More details to come early next week. > > -Elizabeth > > Dossy wrote: > > > On 2003.08.13, Elizabeth Thomas <[EMAIL PROTECTED]> wrote: > > > We have a server that loads in a great deal of tcl - so much so that > > the > > > resulting init script is 6.6M in size and contains over 9900 procs. > > > [...] > > > > > > 1. Is there any way to reduce the time to initialize an interp? > > > (besides the obvious, but not necessarily feasible, option of > reducing > > > the procs that are loaded). Has anyone seen similar behavior and have > > > some insight into it? > > > > Is there any chance to do further profiling? Where are we losing most > > of our time? > > > > I imagine it has a lot to do with allocating 6.6M of memory for the > init > > script, filling it with the script, then telling Tcl to go parse and > > execute it. > > > > I think the biggest win would be to try and figure out how to get > Tcl to > > bytecode compile the init script, then push that bytecode from the > > master interp into the new slave interps as they get created. Cut out > > the entire parse/bytecode compile steps. Perhaps someone who knows and > >
Re: [AOLSERVER] Lengthy tcl interp initialization
On 2003.08.21, Elizabeth Thomas <[EMAIL PROTECTED]> wrote:
>
> To handle ns_eval (which is indeed evil), we must create a copy of the
> nsv arrays for each ictl epoch. To do proper garbage collection we need
> to keep track of reference pointers so we can dump the older ones once
> they are no longer referenced.
On ns_eval, do we "rename procName {}" away all procs defined in the nsv
store, so that the next call to the proc pulls out the fresh definition
from the nsv store?
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of
your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
Only new or 'refreshed' interps will get the new values (since they will
have a pristine interp that is associated with the new nsv array).
Existing interps will use purely the old values (since they have an old
interp that is associated with the old nsv array). In the interp
performing the ns_eval, it will evaluate the file, so the new
definitions take effect there, and no 'unknown' processing kicks in for
them. Does these cover the scenario you are envisioning?
-E
P.S. I'm sure more questions will be answered (and probably even more
asked) when I publish the proposed changes - but they need some final
touches :)
Dossy wrote:
> On 2003.08.21, Elizabeth Thomas <[EMAIL PROTECTED]> wrote:
> >
> > To handle ns_eval (which is indeed evil), we must create a copy of the
> > nsv arrays for each ictl epoch. To do proper garbage collection we need
> > to keep track of reference pointers so we can dump the older ones once
> > they are no longer referenced.
>
> On ns_eval, do we "rename procName {}" away all procs defined in the nsv
> store, so that the next call to the proc pulls out the fresh definition
> from the nsv store?
>
> -- Dossy
>
> --
> Dossy Shiobara mail: [EMAIL PROTECTED]
> Panoptic Computer Network web: http://www.panoptic.com/
> "He realized the fastest way to change is to laugh at your own
> folly -- then you can let go and quickly move on." (p. 70)
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <[EMAIL PROTECTED]> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of
your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
On 2003.08.21, Elizabeth Thomas <[EMAIL PROTECTED]> wrote:
> Only new or 'refreshed' interps will get the new values (since they will
> have a pristine interp that is associated with the new nsv array).
> Existing interps will use purely the old values (since they have an old
> interp that is associated with the old nsv array). In the interp
> performing the ns_eval, it will evaluate the file, so the new
> definitions take effect there, and no 'unknown' processing kicks in for
> them. Does these cover the scenario you are envisioning?
Here's a sample scenario:
In page code, suppose I have the snippet "<% foo %>". In my Tcl lib
dir, I have foo.tcl and inside it, is:
proc foo {} {
ns_adp_puts foo
}
I hit the page, and with lazyprocdef turned on, assuming a new thread
and interp had to be created to serve this request, attempting to invoke
the "foo" proc, which doesn't exist in the interp as a proc yet, causes
the unknown proc to get invoked. That in turn creates the proc as it's
defined in the nsv store. Then, the actual foo proc is invoked.
Now, what happens if I issue "ns_eval proc foo { ns_adp_puts bar }" from
the nscp port?
New interps will see the new foo def, but the interp that served the
previous request now has a real proc foo defined, and thus won't hit the
unknown proc or see the new def in the nsv store.
IMHO, if this is what's happening, it's a problem. It breaks the
contract that ns_eval is supposed to fulfill.
Ideally, when ns_eval happens, it should instruct all the interps get a
list of all the procs that are defined in the nsv store, and rename
$proc {} them away -- so that the next attempt to invoke a proc goes
through the unknown proc and refreshes the proc def from the nsv store.
Is the lazyprocdef stuff checked into CVS now?
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of
your email blank.
Re: [AOLSERVER] Lengthy tcl interp initialization
Ah - I see now. Excellent question. Yes, its the 'refreshed' interps
that get us into trouble. When an interp is re-used, it checks for the
epoch change, and if so, re-runs the init script (which does not contain
the lazy procs and will not remove any procs that are already defined)
This is the same behavior that keeps ns_eval from effectively removing
procs from existing interps (with or without lazy proc). Will address
this :)
-Elizabeth
Dossy wrote:
> On 2003.08.21, Elizabeth Thomas <[EMAIL PROTECTED]> wrote:
> > Only new or 'refreshed' interps will get the new values (since they
> will
> > have a pristine interp that is associated with the new nsv array).
> > Existing interps will use purely the old values (since they have an old
> > interp that is associated with the old nsv array). In the interp
> > performing the ns_eval, it will evaluate the file, so the new
> > definitions take effect there, and no 'unknown' processing kicks in for
> > them. Does these cover the scenario you are envisioning?
>
> Here's a sample scenario:
>
> In page code, suppose I have the snippet "<% foo %>". In my Tcl lib
> dir, I have foo.tcl and inside it, is:
>
> proc foo {} {
> ns_adp_puts foo
> }
>
> I hit the page, and with lazyprocdef turned on, assuming a new thread
> and interp had to be created to serve this request, attempting to invoke
> the "foo" proc, which doesn't exist in the interp as a proc yet, causes
> the unknown proc to get invoked. That in turn creates the proc as it's
> defined in the nsv store. Then, the actual foo proc is invoked.
>
> Now, what happens if I issue "ns_eval proc foo { ns_adp_puts bar }" from
> the nscp port?
>
> New interps will see the new foo def, but the interp that served the
> previous request now has a real proc foo defined, and thus won't hit the
> unknown proc or see the new def in the nsv store.
>
> IMHO, if this is what's happening, it's a problem. It breaks the
> contract that ns_eval is supposed to fulfill.
>
> Ideally, when ns_eval happens, it should instruct all the interps get a
> list of all the procs that are defined in the nsv store, and rename
> $proc {} them away -- so that the next attempt to invoke a proc goes
> through the unknown proc and refreshes the proc def from the nsv store.
>
> Is the lazyprocdef stuff checked into CVS now?
>
> -- Dossy
>
>
> --
> Dossy Shiobara mail: [EMAIL PROTECTED]
> Panoptic Computer Network web: http://www.panoptic.com/
> "He realized the fastest way to change is to laugh at your own
> folly -- then you can let go and quickly move on." (p. 70)
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <[EMAIL PROTECTED]> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of
your email blank.
