Re: [AOLSERVER] are nsv_* functions freeing memory ?
> +-- On Feb 18, Jean-Fabrice RABAUTE said: > > There is no impact on the process memory ! It seems to not work at all. > > It won't release memory back to the OS; it just makes it available for > nsd to use for other things again. This is what's considered the "high-water mark" approach. The allocator is designed to allocate memory, but not free it (this is an allocator dependent thing, but that's what the one that AOLServer uses). The idea is that if you allocated mem for 20MB worth of array elements, then you will likely need that much again. When you free those 20MB, they are marked reusable in AOLServer though, so you should always see a "high-water mark", rather than constantly increasing memory. For those who worry about "but I only need 20MB once, and then 1MB all the rest of the time", any decent OS with a good VM system (pretty much all nowadays) will just swap those unused pages out anyway. There are ways to change this behavior. Tcl's built-in mem allocator is a high-water mark allocator, but it was only used on Windows and Mac (back when it was faster than their internal ones). In 8.3.4 I changed to use the system allocator on Windows, so it's no longer high-water there. AOLServer uses a special allocator to support thread and obj allocation efficiently, but that could still be modified (perhaps at a performance penalty). In any case - I would like to hear more detail about Jean-Fabrice's problem, because there shouldn't be a clear leak. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] File Uploading
... > If each part of a multi-part form had a Content-Length, there would be no > reason to parse for the boundary strings. There would, in fact, be little > need for boundary strings at all. I'm not entirely clear on why it was > done with boundaries instead of lengths in the first place. But > anyway... even if IE does return a Content-Length, netscape (at least, > 4.76) does not so we still need to handle boundary strings. ... > I'm going to code something up in C and see how it works. I'll try to > make it a drop-in replacement for form.tcl, and post it back here, get > some comments. This is correct. However, the form.tcl is not optimal code. You don't need to go through and search each line. You start at the beginning and break into parts by doing string first \n$bound\n $string You can either do this by reading the whole file in memory, or if you are mem-conscious, do it in 512KB chunks. It's a fairly simple technique. Before you do and spend a lot of time coding form.tcl in C though, I would recommend seeing what the AOLServer head is doing. I didn't examine it closely, but it's only about 20 lines of Tcl now. Jeff
Re: [AOLSERVER] building AOLServer with Tcl 8.4a4
Thomas, The Tcl sources require a minor change before you can have it work with AOLServer. The guys at AOL developed a special allocator designed for much better performance with threads. I do plan on putting this into the core (now that license issues are resolved), but it needs a bit more time because the current version (in the AOLServer head) only works on Unix. Once that is addressed, 8.4 (and likely a backport for 8.3.5) will work fine "out-of-the-box". I guess it would actually work now, but without the enhanced allocator, the performance would be unsatisfactory. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/ > Has anybody built AOLServer using Tcl8.4a4? If so, any > comments/problems? > > I'm interested in trying this out. I was just wondering if anybody has > already modified the Makefiles and header files, before I started > hacking them together myself. > > If nobody has done it, I'd be happy to try it out and post my > modifications. Please let me know. > > Best regards, > thomas park
Re: [AOLSERVER] shared object Tcl Library
... > * If I recall, you can load it in a tcl script by doing > > ns_eval [list load /path/to/file.so] > > one time, and it will exist in all interpreters. Of course, you'll > probably want to wrap this up so that it doesn't get loaded every time. FWIW, Tcl's load command actually caches pathnames of files that it loads to ensure it is only loaded once. However, the Pkg_Init call will be called as many times as you "load" something. This is similar to Windows' LoadLibrary caching behavior - it will load it twice if you use different pathnames (like through different mount points), as they could be separate libraries. This is just to say you are somewhat safe from having the library loaded 1000 times in memory. You still want to use the correct ns_ calls to ensure that you only init the library in the interp as few times as possible (ideally once per interp of course). Jeff
Re: [AOLSERVER] new AOL-based thread-happy mem allocator in 8.4
> On Tuesday 23 April 2002 08:00, you wrote: > > I have finally commited the improved memory allocator from the AOLServer > > Tcl mods to the 8.4 branch. ... > This may also be the point to reconsider improving the > AOLserver's way of dealing with Tcl-only extensions and > copyiny command-sets from startup to connection threads. > There are couple of design issues there which require > AOLserver core patch to get some extensions (like XOTcl) > to get loaded and used properly. > I have the patch for that but I'd like to see wether it > can be introduced in the core server, since it will > ease the task of using arbitrary (thread-safe) Tcl > extensions within AOLserver. AFAIK, what I commited takes care of all the known patches that AOLServer had against the Tcl core. I believe you should be able to use the AOLServer head with the Tcl core head without any mods to the Tcl core, or any funny bugs and the like. However, if there are some "design" level issues that may need improvement, do tell me. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] Preliminary AOLserver 3.3ad13 and 3.3ad13tcl8.4 Debian packages and tests with AOLserver+Tcl 8.4
> The Tcl 8.4 version was a bit faster, but it also crashed a couple times. > I didn't investigate further as to why. The Tcl 8.4 version was using the > default generic/tclAlloc.c shipped _with Tcl_ (not the AOLserver one). > > Tcl 8.4 was _not_ compiled with --enable-threads (a configure flag that > apparently doesn't exist on Tcl 8.3.2). I'd love to hear comments from > folks more knowledgeable (Jim, Kriston, Jeff Hobbs, Zoran, etc.) I assume this was 8.4b1, or the most recent CVS? You have to compile that with --enable-threads, and then you will get the new threaded allocator which is based off the one from AOL. 8.3.2 should also have had --enable-threads ... but perhaps if that was the special AOL version, that aspect was hard-coded on because it was required. That might also indicate why 8.4 would crash where 8.3 doesn't, because IIUC, AOLserver *requires* a threaded Tcl. You can find out by starting Tcl and doing 'parray tcl_platform'. If the 'thread' key shows up == 1, then it is threaded. I would make sure that all that is kosher and no crashes are occuring before analyzing the tests (which seem so close together as to be statistically insignificant). Jeff
Re: [AOLSERVER] ns_get_multipart_formdata
> > Just curious, 'file tail' did not work for you? I thought it worked
> > cross platform...
>
> It would if the backslashes were converted to forward slashes. (It ought
> to work with backslashes under Windows.)
>
> % set tcl_platform(os)
> Linux
> % file tail {c:\windows\foo.bar}
> c:\windows\foo.bar
> % file tail [string map [list \\ /] {c:\windows\foo.bar}]
> foo.bar
(hobbs) 50 % file tail {c:\windows\foo.bar}
foo.bar
(hobbs) 50 % info patchlevel
8.4b1
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] Building Tcl 8.4.0 and AOLserver 3.5.0 for Debian
> The 3.5 work is a branch based on the 3.4.2 version. The idea was to > release an interim version which was essentially identical to 3.4.2, > but had all of the Tcl changes removed in favor off Tcl 8.4. It was > released to help those, like ourselves, who are interested in more > easily migrating from 3.4.2 to the upcoming 4.0 release. I'd be interested in hearing from those playing with 3.5 what their general experiences are on using Tcl off the core trunk, with respect to ease-of-use (integration), stability (hopefully just as good or better than before) and performance (8.4 *should* be a big jump over 8.3 - when used "right"). Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] Building Tcl 8.4.0 and AOLserver 3.5.0 for Debian
> > and performance (8.4 *should* be a big jump over 8.3 - when used > > "right"). > > Can you point us to the appropriate places in the changelogs for 8.4 > that talk about what "right" means? > > What were specific things where we might see measurable improvement? See http://wiki.tcl.tk/1611. 8.4 is the fastest release of Tcl ever, even beating 8.0 (although now we have full thread-safety and unicode). Note that the above numbers are all non-threaded builds, but the special threaded mem allocator from AOL was integrated into the Tcl core in 8.4 as well. For those who made it to the recent Tcl conference, we went in depth into a few main areas that have been improved (or where you can see improved results). 1. Existing code getting a boost through new byte-compiled commands. append, list, lindex, llength, lappend, string compare, string equal, return, ... All these are now byte-compiled in 8.4. 2. Existing code getting a boost through internal optimizations. string match and string map are two examples of commands that got algorithmic internal boosts. regexp's are now analyzed to turn simple regexp's into straight string match or string equal instructions in byte-code where possible. Simple static regsubs use the better string map style substitution when possible (checked at runtime). All loops should also be faster, and there is a true proc noop instruction. 3. Use of new commands to improve code. This is true for those that learned Tcl starting at 8.0 or earlier. 8.2+ have had many feature enhancements (more added in 8.4) at the Tcl command level that can really improve coding. regexp -all -inline. string equal (new in 8.2) or the 'eq' and 'ne' expr operators (new in 8.4). string map (new in 8.2) vs. multiple regsub's. lset (new in 8.4). This is a topic that one could go on for quite a while. Of course to get the most advantage, you want to use code in procs that get reused as much as possible. Toplevel, run-once code really doesn't get boosted (except perhaps with algorithmic changes mentioned in 2). > Also, now that AOLserver uses Tcl out of the box, does this mean > namespace support in Tcl under AOLserver is fully functional? Errr, uh ... yes. ? I'm not quite sure that I know what intricate details that question implies. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] How to remove unsafe commands?
...
> Thinking about it I think it is reasonably secure, the only command that I
> don't know how to deal with so far is "info", as it is called with "info
> procs", "info body" (...) in "namespace.tcl" and I need the "info exists"
> part by myself.
You can override any command you want, or have the user code eval
in a slave interpreter, where you may not require 'info' to be
present at all (although that would be limiting).
> Btw: Would it be possible and/or make sense to byte-compile _everything_?
Just
> thought about it when examining the line
>
> #catch { set fh [open init_interp_code.tcl w] ; puts $fh $init ;
> close $fh }
>
> in namespace.tcl, proc _ns_getinit.
> Somebody ever tried this?
What do you mean? What exactly are you trying to achieve?
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] How to remove unsafe commands?
> Is it a bad idea, maybe before use on the production system, to let AOLserver > create the file, byte compile the procs, adding "load libtbcload..." and > sourcing this file instead of running through the process every time? procomp/tbcload is designed for obfuscation, not speed. It will not speed anything up, unless you have procs that take a very long time to compile. We (ActiveState) actually just did some work recently on a Tcl_CloneInterp implementation, which could possibly benefit something like AOLServer. Getting it into the core will be touchy though, because it breaks the existing bytecode structures, which would mean all existing tbcload'able code. > Would that help speed up things? No, the aspect of obfuscation prevents any speed gain. > Would it "hide" all procs from the info vars/procs/body... command to make it > (more) difficult to inspect vars, procs and environment at runtime? Very much so, that is the exact purpose. Users would still be able to see the global variables and procs with 'info', but 'info body' would not return anything. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] gzip compression
> > In any case, I did want to mention that since 8.2 Tcl has had stacked > > channels in the core. Extensions like Trf, memchan and TLS use this > > to do compression and/or encryption on channels transparent to the > > user. > I create a new channel type in tclcmds.c in the nsopenssl module to > allow Tcl's gets, puts and friends to work with an SSL "socket", though > I didn't use channels for the C API portion of the process. > > Are there any performance issues with stacking channels? That's a good question ... I've never actually done performance analysis on the stacked channel stuff, but it is fairly efficient. It just passes buffers from one stack to the next as they are consumed. Special purpose filters may do a better job as they can suck in large amounts of data as they operate, but I'm not sure that you'd win if you have to have extra exec overhead. What would be a good test? Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] Weird expr error
> My aolserver returns an error when I do e. g. a [expr 12.0 + 1]. > [expr 12 + 1] works fine. The log says: syntax error in expression "12.0 + > 1". It seems that it can't handle floats anymore. As soon as there is a > decimal it returns the error. As you are somewhere in Europe, you can get locale-sensitive. Tcl explicitly sets LC_NUM to "C" locale (American) so that numbers like "12.0" can be used always, not "12,0" as would be "European". Some other extension may be modifying your locale, which it shouldn't. You can easily test this assumption by trying "12,0" in the expr above. Jeff Hobbs The Tcl Guy Senior Developer http://www.ActiveState.com/
Re: [AOLSERVER] Weird expr error
> I know what you mean, here in europe we encounter some strange behaviour in > different layers too often (Try to return umlauts with a vanlilla aolserver > 3.4). Strange here is that the server has installed exactly the same linux > distribution and exactly the same aolserver and works fine, as did the test > machine in the past. The only thing I have changed is the nsjava module, but > that doesn't touch the aolserver sources. That's the problem with fiddling with locale, and why extension code should never do it - it modifies the locale for an entire process. Jeff
Re: [AOLSERVER] ns_register_proc
I missed the beginning on this conversation (too much email while
I was away), but here are my relevant comments to the last message:
> > During my lunch break, I wrote a small program to
> > compare the execution of Tcl_StringMatch() (what
> > ns_register_filter uses to match URLs) and
> > Tcl_RegExpExec() (what I propose
> > ns_register_filter should have the option of using).
It all depends on what you are actually matching, and at what
level (C or Tcl). I will show some Tcl-based examples only,
to explain what opts occur for the Tcl level. Note that in C
you would best your Tcl_StringCaseMatch (which is all that
Tcl_StringMatch calls). Also in C, if possible, storing the
RE as an obj and using the obj-based C RE functions is best.
In Tcl, you will see that string match is better whether you
use no special chars or with:
1 % time {eval string match foo bar} 1000
39 microseconds per iteration
2 % time {eval regexp foo bar} 1000
50 microseconds per iteration
3 % time {eval string match b* bar} 1000
38 microseconds per iteration
4 % time {eval regexp b.* bar} 1000
53 microseconds per iteration
REs have the advantage that the pattern can get more complex,
but if that isn't necessary, they will always be slower. I
rewrote the string match algorithm in 8.4 for more speed. The
above tests are 8.4.2 BTW. Also, I have to use eval due to
magic that occurs with the bytecode compiler which truly makes
things faster:
5 % time {string match foo bar} 1000
2 microseconds per iteration
6 % time {regexp foo bar} 1000
2 microseconds per iteration
and it isn't just static string checks, which do occur in part,
but sometimes more can be done with one command versus another:
[a is foo, b is bar]
7 % time {string match $a $b} 1000
3 microseconds per iteration
8 % time {regexp $a $b} 1000
16 microseconds per iteration
9 % time {regexp foo $b} 1000
3 microseconds per iteration
In the above you see the difference when I can't see whether I
can turn the regexp into a string equal or string match at
compile time. BTW, the above gets even more blurry since some
of the cases become string equal under the covers.
> I believe in Tcl 8.4 that if regexp sees a "simple" pattern (i.e. one
> that string match could use), then it will use string match to process
> it.
Yes, it is not an exhaustive check, but the when compiling a
regexp I do checks for static strings, anchored strings and
some handling for ".*" -> "*". Here is an example of where
regexp is never regexp under the covers, it is either string
match or string equal:
67 % time {regexp {^foo$} $b} 1000
2 microseconds per iteration
68 % time {string match {*foo*} $b} 1000
3 microseconds per iteration
69 % time {regexp foo $b} 1000
3 microseconds per iteration
70 % time {string equal foo $b} 1000
2 microseconds per iteration
All that said, it depends on what the ns_register_proc is doing.
Is it all C code, or are we talking about using regexp vs.
string commands at the Tcl level?
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list: http://www.aolserver.com/listserv.html
List information and options: http://listserv.aol.com/
