Re: [naviserver-devel] Mercurial at Bitbucket is ending. What next?

[Jeff Rogers]

Sourceforge supports mercurial repos as well as git.   It's struck me as 
odd that the source was hosted on bitbucket while the "main page" and 
distributions are on SF;  is there history for why the source repo isn't 
on SF as well?


-J


On 08/29/2019 03:06 AM, Zoran Vasiljevic wrote:

On Thu, 29 Aug 2019 09:13:17 + (UTC)
Roderick  wrote:


After reading something about mercurial only for cloning Naviservers
Repo, I read this:

https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket

It would be nice to have NaviServer as fossil repo. :)

I believe most logical choice is git. I'm not fan of any of
the systems, to be honest. Just, the chance that we will have
to switch again (cvs -> mercurial -> ?) is less if the ? = git.



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel





___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] select/poll (was: Re: Tcl nonblocking file IO in NaviServer?)

[Jeff Rogers]

Recent / upcoming releases of Tcl include support for poll and epoll to
get past the limits of select.  Is this going to conflict with anything
in naviserver?

https://core.tcl-lang.org/tips/doc/trunk/tip/458.md

-J


On 06/06/2019 12:23 PM, Gustaf Neumann wrote:


I am rather planing to reduce the usage of the Tcl-based
AsyncDiskWriter, since this depends on select() in current Tcl
implementations, which will run into problems when there are
more than 1024 fds are open. It is however quite easy to
provide a Tcl Interface to NsAsyncWrite() in NaviServer,
to implement this when needed, sine the base infrastructure is
already there.




___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] keepalive performance issue

[Jeff Rogers]

Thanks!   This was puzzling me;  I had found a few references to Nagle in
searching but it didn't seem to make sense.  TCP_CORK appears to be already
used which I thought was in essence manual management of the packet
delays, but I guess not.  Yay, linux?

In any case, setting the option correctly makes the server do just what 
I was

expecting it to do in the first place.

-J

On 12/04/2018 12:48 AM, Gustaf Neumann wrote:
> Hi Jeff, > > I found the problem: One has just to tell Linux to turn 
the delay > off. > > What sounds as a joke can be done in the config 
file as shown below, >  which turns off the good old Nagle algorithm for 
incoming packages > on the keep-alive socket. This reduces the delay 
seen from poll() > substantially. > > We should probably change the 
default for "nodelay" on nssock and > nsssl from "false" to "true". It 
seems, that e.g. mozilla has done > this ~8 years ago. > > all the best 
-g > > PS: setting minthreads is limited to the current value of 
maxthreads. > so, when setting only minthreads to a value of 20 (as in 
your > example) the effective value is 10 (the default for maxthreads). 
So, > probably we should align the value of maxthreads when setting the 
> value of minthreads above maxthreads, and vice versa when setting > 
maxthreads to a value lower than minthreads. > > > ns_section  
"ns/servers" ns_param default Naviserver > > ns_section  
"ns/server/default" ns_param maxthreads 20 ns_param > minthreads 20 > > 
ns_section  "ns/server/default/modules" ns_param nssock > 
nssock.so ns_param nslog   nslog.so > > ns_section  
"ns/server/default/module/nssock" ns_param port 8080 >  ns_param nodelay 
true > >> On 30.11.18 21:17, Jeff Rogers wrote: >>> Ok, thickening the 
plot a little bit - if I enable adp parsing >>> and serve the exact same 
file as adp, the delay on localhost >>> goes away.   So, something weird 
with plain file handling on >>> loopback? >> i tested now with 
debian-sid and can confirm the behavior, which >> happens on linux, but 
not on macOS. > > > > > ___ 
naviserver-devel > mailing list naviserver-devel@lists.sourceforge.net > 
https://lists.sourceforge.net/lists/listinfo/naviserver-devel





___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] keepalive performance issue

[Jeff Rogers]

Ok, thickening the plot a little bit - if I enable adp parsing and serve 
the exact same file as adp, the delay on localhost goes away.   So, 
something weird with plain file handling on loopback?


-J


On 11/30/2018 11:51 AM, Jeff Rogers wrote:


So really, this appears to be NOT a naviserver issue...




___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] keepalive performance issue

[Jeff Rogers]

Hi Gustaf,

The results are completely stable, from 5 to 5000 requests, and with 
concurrency from 1 to 10ish.   I'm running on linux, haven't had any 
issues with ab before.   I actually initially noticed the issue using 
jmeter.


The timing is stable enough that it feels like a timeout or artificial 
delay, but no idea what that would be.   Here is a snippet from 'strace 
-r' output that highlights a delay of about the right amount of time:


[pid 19394]  0.12 write(2, 
"[30/Nov/2018:10:24:11][19386.7fd"..., 108 
[30/Nov/2018:10:24:11][19386.7fdb0b086700][-conn:default:1:1-] 
Debug(request): end GET /hello.html HTTP/1.0
[pid 19395]  0.08 poll([{fd=5, events=POLLIN}, {fd=7, 
events=POLLIN}, {fd=8, events=POLLIN}], 3, 5001 

[pid 19394]  0.16 <... write resumed> ) = 108
[pid 19394]  0.21 write(2, 
"[30/Nov/2018:10:24:11][19386.7fd"..., 
269[30/Nov/2018:10:24:11][19386.7fdb0b086700][-conn:default:1:1-] Debug: 
[1] end of job, waiting 0 current 2 idle 1 ncons 9998 fromQueue 0 start 
1543602251.352459 1543602251.352459 accept 0.00 queue 0.000491 
filter 0.51 run 0.000971 netrun 0.000920 total 0.001462

) = 269
[pid 19394]  0.39 futex(0x174, 
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5, {1543602371, 
353964000},  
[pid 19395]  0.039269 <... poll resumed> ) = 1 ([{fd=8, 
revents=POLLIN}])
[pid 19395]  0.58 write(2, 
"[30/Nov/2018:10:24:11][19386.7fd"..., 
118[30/Nov/2018:10:24:11][19386.7fdb0a885700][-driver:nssock:0-] 
Debug(ns:driver): === PollWait returned 1, trigger[0] 0

) = 118
[pid 19395]  0.65 write(2, 
"[30/Nov/2018:10:24:11][19386.7fd"..., 
108[30/Nov/2018:10:24:11][19386.7fdb0a885700][-driver:nssock:0-] 
Debug(ns:driver): RequestNew reuses a Request

) = 108

Right around 39ms in "... poll resumed".

I tried running ab from a different host, and there, things work as 
expected.   I wonder if there is some localhost network delay being 
injected here (delayed ack or nagle?).   Strangely tho, tclhttpd isn't 
affected; the keepalive and non-keepalive times are both in the 1 
ms/request range.


So really, this appears to be NOT a naviserver issue...

-J

On 11/30/2018 01:11 AM, Gustaf Neumann wrote:

Hi Jeff,

30 requests are not a lot. Are these results stable?
When I tried to repeat your results, I started the server
with your configuration:

/usr/local/ns/bin/nsd -u nsadmin -t ~/scripts/aol/jeff-conf.tcl -f

When running "ab" without -k" and 1000 requests:
ab -n 1000 -c 1http://localhost:8080/hello.html
...
Time per request:   0.168 [ms] (mean)
When running  "ab" with -k" and 1000 requests:
ab -k -n 1000 -c 1http://localhost:8080/hello.html
...
Time per request:   0.098 [ms] (mean)
The results of "ab" are most probably platform dependent.
My results are on a notebook with macOS Darwin 17.7.0,
NaviServer 4.99.17, some snapshot of Tcl 8.7,
most components without C-level optimization.

The reported times show that the keepalive option
improves the results, which is an expected result.

With high numbers for "-c", "ab" hangs sometimes, but that
might be a problem with "ab" on the mac (Apple version);
i remember that i had to fix "ab" in the past in order to get results
on the mac at all.

all the best
-gn

On 29.11.18 23:48, Jeff Rogers wrote:

Hi all,

I am running a test using ab against a local naviserver instance with 
a vanilla config, and I am seeing requests with http keepalive 
enabled having a delay of ~38ms per request compared to 
non-keepalive.   This rather surprised me;  is there a config setting 
to avoid that delay, or is it otherwise to be expected?


My config:

ns_section  "ns/servers"
ns_param default Naviserver

ns_section  "ns/server/default"
ns_param minthreads 20

ns_section  "ns/server/default/modules"
ns_param nscpnscp.so
ns_param nssock  nssock.so
ns_param nslog   nslog.so

ns_section  "ns/server/default/module/nssock"
ns_param port 8080


Tests:

$ ab -n 30 -c 1 http://localhost:8080/hello.html
...

Time per request:   0.170 [ms] (mean)

$ ab -k -n 30 -c 1 http://localhost:8080/hello.html
...

Time per request:   38.666 [ms] (mean) 






___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] keepalive performance issue

[Jeff Rogers]

Hi all,

I am running a test using ab against a local naviserver instance with a 
vanilla config, and I am seeing requests with http keepalive enabled 
having a delay of ~38ms per request compared to non-keepalive.   This 
rather surprised me;  is there a config setting to avoid that delay, or 
is it otherwise to be expected?


My config:

ns_section  "ns/servers"
ns_param default Naviserver

ns_section  "ns/server/default"
ns_param minthreads 20

ns_section  "ns/server/default/modules"
ns_param nscpnscp.so
ns_param nssock  nssock.so
ns_param nslog   nslog.so

ns_section  "ns/server/default/module/nssock"
ns_param port 8080


Tests:

$ ab -n 30 -c 1  http://localhost:8080/hello.html
...

Time per request:   0.170 [ms] (mean)

$ ab -k -n 30 -c 1  http://localhost:8080/hello.html
...

Time per request:   38.666 [ms] (mean)



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] commit/naviserver: 3 new changesets

[Jeff Rogers]

Bitbucket wrote:
> 3 new commits in naviserver:

> https://bitbucket.org/naviserver/naviserver/commits/2758b9f8392f/
> Changeset:   2758b9f8392f
> User:gustafn
> Date:2015-11-18 19:35:25+00:00
> Summary: - The new command "ns_urlspace" that allows to attach data to 
> URLs in
>a hierarchical manner form the Tcl level. The command allows e.g. to
>write ns-perm like access control in Tcl without much hassle.

This is very cool; it's something I've wanted to expose as a tcl api for 
a long time.

A thought for a further extension to this is to make the separator 
character changeable instead of only "/", so strings represented as 
"a.b.c.d" or "a:b:c:d" could be handled too.  However, that would 
probably add a good deal of complexity that is unjustified lacking an 
actual use case.

I was also thinking that the trie structure could be applicable to 
logging configurations: enabling logging for "debug/sql" and "info/*" 
but not "debug/*" would be really nice.  The "/" as a separator 
character seems odd at first, but ultimately unimportant as it conveys 
the same hierarchical information as "." or ":" would.

-J

--
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Nsssl SNI

[Jeff Rogers]

I was just considering this exact same thing, and it appears the answer 
is no.  You should be able to serve multiple vhosts on different ip 
addresses (or ports) by running nsssl multiple times with different cert 
configs, but that isn't particularly helpful.

I haven't explored this completely, but to add SNI support to nsssl I 
think the cleanest approach config-wise would be to add a "servers" and 
"certs" section underneath nsssl to map hostnames to certificates as 
well as servers, ala nssock.

For example:
ns_section ns/module/nsssl/servers
ns_param server1 www.example.com
ns_param server2 www.example2.com

ns_section ns/module/nsssl/certs
ns_param www.example.com /usr/local/ssl/certs/server1.pem
ns_param www.exmaple2.com /usr/local/ssl/certs/server2.pem

Dynamic vhosts could perhaps be supported by defining the cert file for 
a given domain to be a standard name under a "certs" subdirectory in the 
vhost tree (i.e., servers/${servername}/host.com/certs/host.com.pem). 
I would address the explicit configuration above first, however.

This SO post points at the implementation strategy: 
http://stackoverflow.com/questions/511/how-to-implement-server-name-indication-sni

Implementing this is not on my immediate to-do list (we're using ELB for 
termination) but it may become a concern sometime soon.

-J


David Osborne wrote:
> Hi there,
>
> Is there any way to replicate the behaviour of SNI aware https servers
> using naviserver nsssl?
> Namely, where different certificates can be presented on the same ssl
> port depending on the servername sent by the TLS client
>
> https://www.domain.com -> nsssl.server.com:443 
><- www.domain.com  cert
>
> https://sub.domain.com -> nsssl.server.com:443 
><- sub.domain.com  cert
>
> (I don't think SNI is supported by nsssl - please correct me if I'm wrong)



--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] logging

[Jeff Rogers]

Hi all,

I ran into a bug trying to change the loglevel for nsdb:
"ns_db verbose" crashes with an assertion failure.  It looks like this 
was recently changed to change the log level for the entire ns_db module 
rather than just for one handle (a change that IMHO makes sense), but 
this part doesn't work right.

Related, it would be exceedingly useful to set messages at a particular 
loglevel to go to a logfile other than the default server log.  However 
it doesn't appear you can do this.  The feature isn't supported by the 
default logger, and logging filters don't provide a mechanism (e.g., 
ns_break) to indicate that the message has been successfully logged and 
further filters should be skipped.

Log filters also process everything, there's no matching as in request 
filters.  This could be worked around easily enough, but it would make 
things a little more structured.  The usage I'm envisioning would be 
something like:
ns_logctl filter Debug(sql) ns:logtofile /tmp/sqldebug.log

Lastly, would it be reasonable to allow ns_log calls to give a loglevel 
that is not already defined (which would default to being disabled)?  Or 
maybe that would make sense only when the loglevel is of the form
"Level(subsystem)"  (e.g., Debug(sql)) and the "Level" is already known. 
  Then I could sprinkle module-specific ns_log debug statements 
everywhere without needing to pre-declare all the levels I might use.

Cheers,
-J

--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] tcl modules

[Jeff Rogers]

Gustaf,

Thanks.  I also just pushed up a commit to fix the handling of ns_module 
private/shared, so that while loading a module they return the correct 
path.  (They were reversed).

-J

Gustaf Neumann wrote:
> Dear all,
>
> i've commited a change for that issue, that
> William Jordan sent to me. I hope, this fixes the
> issue.
>
> -g
>
> Am 14.12.14 10:27, schrieb Gustaf Neumann:
>> Loading the same module twice is not intentional.
>> Loading modules is probably more complicated as it
>> has to be, but it has to care about global
>> modules (for all servers) and per-server modules, and that
>> network modules are loaded at the end.
>>
>> -g
>>
>> Am 10.12.14 02:54, schrieb Jeff Rogers:
>>> Hi all,
>>>
>>> It looks like a tcl-only module declared in the conf file will get
>>> loaded twice by init.tcl (as a network and a non-network module).
>>> There's a mention in init.tcl about needing to do a 2-phase loading of
>>> the network modules, but loading modules twice seems wrong.
>>>
>>> 
>>> ns_section ns/server/server1/modules
>>>
>>> ns_param mymodule tcl
>>> 
>>>
>>> Before I go ahead and fix this, is there a particular reason for the
>>> behavior as it is?
>>>
>>> -J
>
> --
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Performance comparison of base64 encoders

[Jeff Rogers]

I just checked, and ns_base64encode is also faster than [binary encode 
base64] by about a factor of 2.

-J

Gustaf Neumann wrote:
> Dear friends,
>
> Maybe, someone finds this information useful:
>
> just now, i did a quick comparison of the various
> options for base64 codecs. It shows that NaviServer's
> built-in ns_base64encode  is very fast:
> it is up to a factor of 1.000 faster than the Tcl-only
> version of base64::encode, and even a factor of 10
> faster than base64::encode based on Trf (Trf 2.1.4).
> While a big difference relative to the plain Tcl version
> is expected, the improvement over the C-based Trf
> is remarkable.
>
> The results are measured with the latest version of
> NaviServer from bitbucket under Mac OS X 10.9.5
>
> Using ns_base64encode reduces the external
> dependencies and is by far the fastest version.
>
> all the best
> -g
>
>
>
> size=100
>   tcllib:   134.713824 microseconds
>   tcllib+trf: 4.561536 microseconds
>   ns: 0.541651 microseconds
>
> size=1000
>   tcllib:  1281.819696 microseconds
>   tcllib+trf:16.93683  microseconds
>   ns: 1.517516 microseconds
>   
> size=1
>   tcllib: 12638.85578 microseconds
>   tcllib+trf:   143.11120 microseconds
>   ns:11.08016 microseconds
>
> size=10
>   tcllib:127344.794553 microseconds
>   tcllib+trf:  1423.021879 microseconds
>   ns:   113.504248 microseconds
>
> --
>
> foreach i {10 100 1000 1} {
> set s [string repeat 0123456789 $i]
> lappend _ size=[string length $s]
> lappend _ tcllib:[time {base64::encode $s} 1000]
> lappend _ ns:[time {ns_base64encode $s} 1000]
> }
> join $_ \n
>
>
>
> --
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>
>
>
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] tcl modules

[Jeff Rogers]

Hi all,

It looks like a tcl-only module declared in the conf file will get 
loaded twice by init.tcl (as a network and a non-network module). 
There's a mention in init.tcl about needing to do a 2-phase loading of 
the network modules, but loading modules twice seems wrong.


ns_section ns/server/server1/modules

ns_param mymodule tcl


Before I go ahead and fix this, is there a particular reason for the 
behavior as it is?

-J

--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Naviserver hangs on Windows

[Jeff Rogers]

Gustaf Neumann wrote:
> Am 06.10.14 06:46, schrieb Jeff Rogers:
>>
>> This struck me as an interesting optimization question, so I wrote a
>> quick program to test it (attached).

> This is not an interesting optimization question, since Tcl itself uses
> gettimeofday()
> (plus jumping around which might have a bad cache and locality
> influence, and some
> more copying).

I think my underlying point was missed in the measurements.

If there is not significant performance advantage to the 
platform-specific optimized version of Ns_GetTime over the 
platform-independent Tcl_GetTime, is it worth it to keep the optimized 
version?

The penalty to be paid is in problems like the current one, where the 
windows build is for some reason having problems.  On windows the 
function is delegating to Tcl_GetTime, but the system-specific bits 
confuse the issue.

Your measurements are consistent and show the relative results I 
expected.  Such is the benefit of real hardware :)  They show a small 
advantage to Ns_GetTime over Tcl_GetTime, about 1.2%.  This *is* a very 
frequently called function, but it is also pretty fast in any case. 
That's 1.2% of perhaps 5% (??  totally random guess) of overall request 
processing time.  To me, that adds up to not a lot.

A different takeaway from this benchmark is that if you are trying to 
wring every last bit of performance out of this call, it might be 
advantageous to define it as a macro (or use gcc/c99 inline declarations 
so it can be inlined and avoid the function call overhead and get to 
within a few instructions of the raw gettimeofday call (presumably 
you'll still need to copy the elements to the Ns_Time struct).

-J


>
> One cannot expect a big difference. For better reliability,
> i've increased the count. Below are 4 runs on openacs.org.
> The native call is from 8% to 20% faster (the latter one most
> probably due to external influences). The machine is a bare metal.
> Numbers will vary depending on the OS. The faster the
> system function gettimeofday() is, the bigger is the percentage
> difference.
>
> -g
>
> gustafn@openacs:~$ ./tt
> count: 1
> Tcl_GetTime:3858686 usec 0.04 per
> Ns_GetTime:3811593 usec 0.04 per
> gettimeofday:3561367 usec 0.04 per
> gustafn@openacs:~$ ./tt
> count: 1
> Tcl_GetTime:3858657 usec 0.04 per
> Ns_GetTime:3824398 usec 0.04 per
> gettimeofday:3563398 usec 0.04 per
> gustafn@openacs:~$ ./tt
> count: 1
> Tcl_GetTime:4351765 usec 0.04 per
> Ns_GetTime:4024717 usec 0.04 per
> gettimeofday:3594736 usec 0.04 per
> gustafn@openacs:~$ ./tt
> count: 1
> Tcl_GetTime:3864511 usec 0.04 per
> Ns_GetTime:3831866 usec 0.04 per
> gettimeofday:3541932 usec 0.04 per
>
>
>
> --
> Slashdot TV.  Videos for Nerds.  Stuff that Matters.
> http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Slashdot TV.  Videos for Nerds.  Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Naviserver hangs on Windows

[Jeff Rogers]

Gustaf Neumann wrote:


What i did was to extend "configure" to look, if there is gettimeofday()
available
on the system. If it is, it bypasses the call to Tcl_GetTime() to get
the timestamp
via system call. On busy systems, Ns_GetTime() is one of the most
frequent calls, used e.g. for mutex timings, so this makes a difference.

The advantage of using Tcl_GetTime() is to delegate system dependencies
to Tcl.


This struck me as an interesting optimization question, so I wrote a 
quick program to test it (attached).


The only test environment at the moment I have is a linux VPS, so 
there's bound to be random influences from virtualization and whatever 
else happens to be running on the host at the time, so the noise is 
high.  Still, I was not able to see any clear difference between 
gettimeofday(), Ns_GetTime, and Tcl_GetTime - on any given run any of 
the three was equally likely to be fastest.


time-int the program reports typically 2:1 system time:user time.

What this suggests to me is that - at least in my environment - the 
probable few hundred cycles difference in the implementations is 
completely lost to syscall and timeslice overhead or possibly cache line 
flushes, and it's not worth spending too much time worrying about it.


I'd be interested to hear anyone else's results and interpretations, or 
suggestions about how to better measure the differences in these.


-J
/*
 * test timing 
 * timing of raw gettimeofday vs Ns_GetTime and Tcl_GetTime
 * compile with: 
 * cc -O2 -I include -I /usr/include/tcl8.5 -o tt tt.c -ltcl8.5 -L lib -lnsd -lnsthread -lpthread
 */
 
#include "ns.h"
#include "tcl.h"
#include 
#include 

void showdiff(char *fn, int count, struct timeval s, struct timeval e) {
int diff=(e.tv_sec-s.tv_sec)*100 + (e.tv_usec-s.tv_usec);
fprintf(stderr,"%s:\t%d usec %4.2f per\n",fn,diff,(double)diff/count);
}

int main(int argc, char** argv) {
struct timeval s,e;
struct timeval d;
int diff,c;
int diff_s,diff_us;
int count=1000;
Tcl_Time timePtr;
Ns_Time ntimePtr;

if (argc == 2) {
count=strtol(argv[1],NULL,10);
}
fprintf(stderr,"count: %d\n",count);

/* warmup */
for (c=0;c<100;c++) {
gettimeofday(&d,NULL);
Tcl_GetTime(&timePtr);
Ns_GetTime(&ntimePtr);
}

gettimeofday(&s,NULL);
for(c=0;c--
Slashdot TV.  Videos for Nerds.  Stuff that Matters.
http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] default filename extension

[Jeff Rogers]

John Buckman wrote:
>
> On Jul 9, 2014, at 3:00 PM, Jeff Rogers  <mailto:dv...@diphi.com>> wrote:
>
>> There's no builtin way I know of to define a default extension.  I would
>> do this with a preauth filter, something like this:
>
> The preauth filter is a nice way to do it, albeit with a bit of file
> existence checking overhead for every request that I'd prefer avoid.
 >
 > The way my C code worked (see previous msg) was to look to see if the
 > request url has no extension, and no trailing / on it either, and then
 > to append .adp in that case.  That has the advantage of being very low
 > overhead.

I certainly understand the desire to minimize overhead.  The tcl filter 
I posted could easily enough be changed to only look at the string 
values and not worry about checking for file existence.

There is still the overhead of the filter call and the tcl code, however 
(my tests show this overhead in the 10-12 microsecond range).  You can 
eliminate most of this overhead by implementing a filter in C rather 
than tcl, and the naviserver C api makes this easy.  That would provide 
most of the performance boost you're looking for while being lots more 
flexible and easier to manage than by changing the guts of SetUrl. In 
particular, since it's a filter you can easily limit the functionality 
to one subdirectory.

I created a simple extension to show the idea; it's at
https://bitbucket.org/jeffr/nsdefext/src

To build, clone it to a subfolder of your naviserver build folder, "cd 
nsdefext", and "make install".

To use it, just add it to your modules in your conf file:

ns_section  "ns/server/default/modules"
ns_param nssock  nssock.so
ns_param nslog   nslog.so
ns_param nsdefext   nsdefext.so

I haven't tested the code extensively, but the main body is about 10 
lines, so it should be pretty simple to understand and modify.

Admittedly it's been a while since I've written an apache extension, but 
I recall it being much more complex than this :)

-J

>
> As you noted, Apache has this with MultiViews, and I use that feature
> extensively on Magnatune.com <http://Magnatune.com> so that URLs look
> nice and clean, ie: http://magnatune.com/artists/linda_wood
>
> So yes, I'd vote for this to be a config level feature, since it's
> already in Apache that way.
>
> Looking at request.c it looks to me like the action happens in the
> "SetUrl" function.
>
> Specifically, there already is logic in here to deal with trailing
> slashes, so a default filename extension feature would slot in well at
> this point:
>
>  /*
>   * Append a trailing slash to the normalized URL if the original URL
>   * ended in slash that wasn't also the leading slash.
>   */
>
>  while (*url == '/') {
>  ++url;
>  }
>  if (*url != '\0' && url[strlen(url) - 1] == '/') {
>  Ns_DStringAppend(&ds2, "/");
>  }
>
> /*
>   john - check to see if a default_extension has been set in the config
> file, and if so, append it now if appropriate.
> */
>
>  request->url = ns_strdup(ds2.string);
>  Ns_DStringFree(&ds2);
>


--
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] possible range bug?

[Jeff Rogers]

Gustaf Neumann wrote:
> Am 09.07.14 21:18, schrieb Jeff Rogers:
>> If I understand correctly, this codepath if used for a non-file based
>> return, so e.g., "ns_return -binary" should get here.
> The code paths are more complex and depend also on the settings of the
> configuration file (e.g. caching, mmap, ...);
>
> when i was working on the async writer i created the call-graph [1]
> below (manually, might contain errors; just did if for the parts i was
> interested in). ReturnRange() is reached for file-descriptor based and
> for data based deliveries

I started to make a similar diagram, but mine's only on a whiteboard so 
far and not as complete.  What did you use for this diagram (I'm hoping 
LibreOffice draw), and can you share the source file?


>> Also, it looks like ranges aren't supported at all for character data
>> (e.g., adp responses or non-binary ns_return).  Is this intentional and
>> desirable?  It seems reasonable at first glance, since the most useful
>> use case for ranges is large binaries, but it seems a bit inconsistent.

> as mentioned a while earlier on the list, the call graph of the data
> delivery logic
> is quite complex and might be simplified. I've simplified it in some
> steps when
> working on the async writer, but still, more can be done.

Agreed here.  Maybe it's time to get rid of some of the old C apis, and 
fix some of the naming to make others clearer?  The use of "Send" vs 
"Write" is confusing,  especially in the case of Ns_ConnSend* vs 
Ns_ConnSend.

> yes, currently range requests are ignored for e.g. .adp requests, but they
> are -supposed to be  - fully supported on file-requests, which are in
> practice
> the most important cases (e.g. various pdf readers depend on this, i think
> i have seen this as well range requests on video formats).
>
> Seems that so far, no-one had needs for range requests on dynamic content,
> which is often useless (e.g. language settings lead to different message
> strings
> on systems like e.g. OpenACS), but there might certainly be use-cases
> for that.

I agree with the sentiment here.  I could come up with a use case but 
I'm not aware of any currently in use.  Remember too, that even dynamic 
content is frequently mostly static, or static for long enough periods 
of time to allow static treatment (i.e., caching) to be useful.

-J

>
> the best start is usually to add a test case. I've added a test case
> showing that
> range requests are ignored on .adp requests.
>
> -gustaf neumann
>
> [1] http://openacs.org/xowiki/file/writer.png
>
>
>
> --
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] default filename extension

[Jeff Rogers]

There's no builtin way I know of to define a default extension.  I would 
do this with a preauth filter, something like this:

=== default_extension.tcl ===
ns_log notice "Loading default extension"

proc default_extension {why} {
 set url [ns_conn url]
 # set loglvl notice
 set loglvl debug
 ns_log $loglvl "default_extension url: $url"
 set file [ns_url2file $url]
 ns_log $loglvl "default_extension file: $file"
 if {![file exists $file]} {
 if {[file exists $file.adp]} {
 ns_log $loglvl "adp extension for $file found"
 ns_internalredirect $url.adp
 } elseif {[file exists $file.html]} {
 ns_log $loglvl "html extension for $file found"
 ns_internalredirect $url.html
 } else {
 ns_log $loglvl "no default extension for $file found"
# will just fall through
 }
 }
 return filter_ok
}

ns_register_filter preauth * * default_extension

ns_log notice "Loaded default extension"

=== cut here ===


You could also implement similar logic in a url2file handler.

Maybe this is something generally useful enough (something like apache 
MultiViews) to include as a default module, although in that case it 
would make sense to make it configurable from the config file.

-J


John Buckman wrote:
> I was wondering if there was simple of of defining the default filename
> extension naviserver looks for, like there is for
> "ns_param directoryfile" but for any url where the filename extension
> isn't specified in the url.
>
> For example, if you go to:
> http://localhost/about
>
> I want "about.adp" to be the file that is loaded.
>
> I've done this in aolserver two ways:
>
> 1) with a 404 handler
> 2) or with a code path in request.c (see code sample below)
>
> -john
>
>
>
> 
> /* john buckman added 4/14/06 */
> /* check if should add default filename extension of .adp */
> /* only if no / on end of url which indicates a directory */
> char * dotpos;
> if (ds2.string[ds2.length - 1] != '/') {
>  /* if not . in the entire url, or if there is a dot before the
> final / (indicating a . in a
> directory name, which is ok, then add the default filename
> extension */
>  dotpos = strrchr(ds2.string, '.');
>  if ((dotpos == NULL) || (strchr(dotpos, '/') != NULL)) {
>  Ns_DStringAppend(&ds2, ".adp");
>  /* Ns_Log(Notice, "added default extension to get '%s'",
> ds2.string); */
>  }
> }
> /* end john buckman added */
>
> request->url = ns_strdup(ds2.string);
> ===
>
>
> --
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
>
>
>
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] possible range bug?

[Jeff Rogers]

Hi all,

As part of porting a feature to naviserver, I'm trying to understand all 
the code paths that outgoing data can take. I think I found a bug in the 
range handling code, but I can't figure out how to tickle it, which 
makes me wonder if there's something less-obvious going on.

This code from return.c:ReturnRange (~lines 889-893) looks wrong to me:

for (i = 0; i < rangeCount; i++) {
vbuf[0].iov_base = (void *)(intptr_t)bufs[0].offset;
vbuf[0].iov_len  = bufs[0].length;
len += bufs[0].length;
}

If I understand correctly, this codepath if used for a non-file based 
return, so e.g., "ns_return -binary" should get here.

It's looks to be looping over the ranges set from Ns_ConnParseRange, but 
using index 0 instead of index i.  I would expect the result to be that 
the first requested range is returned multiple times, but it seems to 
return multiple ranges just fine.  I was able to construct a failing 
byterange test with an incorrect length (if the multiple ranges are 
different sizes rather than all the same size), but the content is still 
correct, which is puzzling.

Also, it looks like ranges aren't supported at all for character data 
(e.g., adp responses or non-binary ns_return).  Is this intentional and 
desirable?  It seems reasonable at first glance, since the most useful 
use case for ranges is large binaries, but it seems a bit inconsistent.

Cheers,
-J


--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Losing interp alias

[Jeff Rogers]

Hi David,

This is a known deficiency - the introspection script that creates the 
tcl initialization script doesn't capture interp aliases.  I don't think 
it's difficult to add, just hasn't been done yet.

-J

David Osborne wrote:
> Hi,
>
> Wondering if you can help with a problem.
>
> I am attempting to preload the trf accelerated version of tcllib's
> ::md5::md5 in our naviserver config.
>
> As a simplified testcase to demonstrate what's happening I can do the
> following:
>
> - use a default configuration of naviserver, e.g. in /usr/local/ns
> - copy osborne-procs.tcl to/usr/local/ns/modules/tcl/osborne-procs.tcl
> where osborne-procs.tcl contains:
>
> package require Trf
> package require md5
>
> - use the sample config as distributed with naviserver
> /usr/local/ns/bin/nsd -c -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl
>
> I get the following result:
>
> % ::md5::md5 -hex teststring
> invalid command name "Hex"
>
> This seems to be because the tcllib md5x.tcl code, checks for the Trf
> package, and if present sets up aninterp alias for the::md5::Hex
> command. And this seems to be getting lost somewhere.
>
> line 502:
> http://tcllib.cvs.sourceforge.net/viewvc/tcllib/tcllib/modules/md5/md5x.tcl?revision=1.19&view=markup
>
> If I subsequently add the alias manually everything is fine:
> % interp alias {} ::md5::Hex {} ::hex -mode encode --
> ::md5::Hex
> % ::md5::md5 -hex teststring
> D67C5CBF5B01C9F91932E3B8DEF5E5F8
>
> Can anyone shed any light as to what's going on here?
>
>
> # info patchlevel
> 8.5.8
>
> tcllib:
>Installed: 1.12-dfsg-2
> tcl-trf:
>Installed: 2.1.4-dfsg-2
>
> # /usr/local/ns/bin/nsd -V
> NaviServer/4.99.6
> Tag: 16f91bafc825+ default tip
> Built:   Aug 19 2013 at 11:26:14
> Tcl version: 8.5
> Platform:linux
>
>
> --
> David
>



--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Clock.tcl repeat initialisation after ns_eval

[Jeff Rogers]

Gustaf Neumann wrote:

> also the definition of proc "clock", which redefines itself in terms of
> an ensemble. This makes it sensitive to the definition order.


>> I notice the same test case doesn't raise an error in Aolserver. Seems
>> after a ns_eval the ::tcl::clock:: vars left not populated so
>> ::tcl::clock::Initialize runs cleanly. Couldn't ascertain from my
>> limited understanding of the internals why it should be different.
> aolserver does essentially the same thing as naviserver. my guess is
> that this is sheer luck in the order of definitions. Furthermore, a
> single call to lock "at the right time" can heal everything or lead to
> the error, since clock is kind of a "self modifying" code...

Sheer luck is definitely a possibility, but also naviserver is missing 
the ensemble serialization that aolserver has, so things defined (or 
redefined) as ensembles would get lost.

I've copied this code over from aolserver.  I don't know if it will help 
the clock problem, although it is possible clock was the motivating 
issue for that code in the first place.

In aolserver the code is wrapped to prevent it from executing on pre-8.5 
tcl.  Is that still needed, or is 8.4 dead enough to have 8.5 as a 
default requirement?

-J

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] gzipped content delivery for static files

[Jeff Rogers]

Gustaf Neumann wrote:

> When the gzip_cmd is configured, NaviServer keeps track of
> updating the .gz file for the cases the source file is updated.
> There is no burden for the admin. The gzip command is
> called via Tcl "exec", which can in turn be executed via
> nsproxy (see e.g. OpenACS). I took this approach
> over an in-memory variant to avoid memory bloats in
> case huge gzip files should be compressed. Note that
> the compression overhead is typically just a one-time
> operation. The website maintained defines, what
> files should be sent gzipped by compressing these
> once.

I think there's a potential security hole here.  I didn't come up with a 
proper exploit, but if a user can get control of a filename (e.g., if 
there is an ability to upload files), then an arbitrary string could get 
passed to the exec command, including but not limited to [] (which would 
let tcl do commmand expansion) or spaces (which could cause the filename 
to be interpreted as multiple words and hijack the exec behavior).

Using Tcl_DStringAppendElement instead of Tcl_DStringAppend should 
prevent this, as it will force the filename to be a proper list element.

Alternately, it would be more flexible to change the definition of the 
zipCmd to be a tcl command that is passed the filename and zipfile name, 
e.g.,  "gzip_cmd file file.gz", with the tcl definition of gzip_cmd 
choosing how to handle it, whether by exec or compressing in-process 
(e.g., with 'zlib compress'), or choosing based on the file size.

-J

--
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] startup delays

[Jeff Rogers]

Hi all,

This message:
http://openacs.org/forums/message-view?message_id=4027670
tapped into something I've been thinking about for a while.

I have a server that takes around 5 minutes to start or restart.  The 
major reason for this is that it's a slow $2/mo openvz VPS and you get 
what you pay for, but it seems that other people have similar issues 
although not as bad.

In any case, the underlying problem is that the server can't start 
serving requests until after startup is complete and since startup can 
do arbitrary things, it can take arbitrarily long.

My ideas here aren't completely baked, so I'd like to bounce the 
concepts off people before I attempt to implement them.

One thought I had is to have a sort of "pre-startup" server running that 
can answer http requests and just respond "server starting, please 
wait".  That would be an improvement over the current situation where 
you get a connection refused. (or it may stay in a "connecting" state 
until the server actually starts if the listen socket is prebound) 
However, restarts would still lead to a period of time where no content 
is being served.

Another thought is to enhance the pre-bind mechanism to tell the server 
to not just bind particular sockets before dropping privileges, but to 
pass in already-opened descriptors and tell the server what they are. 
Then when restarting, the currently running server would pass its own 
open listen socket fds to the new server to prebind and then keep 
running until the new server signals that it is ready to start (e.g., by 
a signal).  So the timeline looks like;

parent server: child server:
start
serve requests ...
ns_server_restart
   fork()
   exec("nsd -b 0.0.0.0:80:@5") starts, notes that fd 5 is its listen
signal(SIGUSR2, cleanup and exit)   socket
keeps running  runs startup scripts
serve requests ... ...
...kill(getppid(), SIGUSR2)
cleanup and exit   serve requests


There's various ways this could be made to work, but the core idea is to 
keep the old server running until the new one is ready.  This approach 
wouldn't work so well with daemontools where the server is typically 
restarted by just having the current one exit, but it would be a 
possible alternative.

Does anyone else think this is worth pursuing?

-J

--
Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET
Get 100% visibility into your production application - at no cost.
Code-level diagnostics for performance bottlenecks with <2% overhead
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap1
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] removing obsolete functions

[Jeff Rogers]

These functions are all part of the "public" api, so they could 
conceivably be used in C extensions.  How many of them actually are used 
is another story of course, and it's far from clear if anyone is 
developing C extensions.

In general, I think it's a good thing to have a rich API, including 
convenience functions (e.g., Ns_CacheSetValue) and plain old utility 
packages, like the Ns_List functions that aren't used but aren't hurting 
anything either.  Duplicating functionality just makes for a confusing 
API, so the deprecated functions (which specifically do have a preferred 
replacement) can be eliminated, perhaps as part of a version bump.

Some of these should have tcl apis around them, such as the Cls 
functions.  AOLserver has 'ns_cls' tcl functions for these.  Of course, 
these aren't needed without other capabilities like queue-wait or 
pre-write callbacks as otherwise conn=interp=conn.

The APIs that should be more closely examined are those for actually 
using the platform, like the ConnSend and ConnWrite and WriteConn 
functions - just what is the difference between these?  There should be 
one general way (including convenience wrappers) to specify data to be 
sent, but it seems like there's a lot more than that.

-J

Gustaf Neumann wrote:
> Dear elders of the naviserver list,
>
> below is a list of about 120 C-functions, which are not used by the
> current naviserver. A few (about 5) were obsoleted by me, but the
> majority is jsut sitting around. Since these functions are not called by
> naviserver this functions are not covered by the regression test. i
> think it is time for a major cleanup.
>
> - most of the deprecated functions were deprecated 2005, a few 2007.
> Are there arguments against dropping these now?
>
> - what should we do with the unused functions?
> Delete these as well?
> Mark these as deprecated?
> Discuss these in more detail?
>
> Even when the functions are deleted, they are not lost, reviving these
> in the case someome needs it could be done quickly.
>
> -gustaf neumann
>
>
> Deprecated:
>
> Ns_BindSock
> Ns_ConnFlushHeaders
> Ns_ConnInit
> Ns_ConnLocation
> Ns_ConnQueueHeaders
> Ns_ConnResetReturn
> Ns_ConnSetRequiredHeaders
> Ns_ConnWrite
> Ns_DStringPop
> Ns_DStringPush
> Ns_DecodeUrlCharset
> Ns_DecodeUrlWithEncoding
> Ns_EncodeUrlCharset
> Ns_EncodeUrlWithEncoding
> Ns_FreeConnInterp
> Ns_GetEncoding
> Ns_PageRoot
> Ns_SetLocationProc
> Ns_SetLogFlushProc
> Ns_SetNsLogProc
> Ns_TclInitInterps
> Ns_TclLogErrorRequest
> Ns_TclRegisterAtCleanup
> Ns_TclRegisterAtCreate
> Ns_TclRegisterAtDelete
> Ns_TclRegisterDeferred
> Ns_WriteCharConn
> Ns_WriteConn
>
>
> Not used:
>
> Ns_AbsoluteUrl
> Ns_AdpAppend
> Ns_AdpFlush
> Ns_AdpGetOutput
> Ns_AdpRequestEx
> Ns_AuthorizeUser
> Ns_CacheSetValue
> Ns_CacheSignal
> Ns_CacheTryLock
> Ns_CacheWait
> Ns_ClearSockErrno
> Ns_ClsAlloc
> Ns_ClsGet
> Ns_ClsSet
> Ns_CompressGzip
> Ns_ConnCopyToDString
> Ns_ConnCopyToFd
> Ns_ConnCopyToFile
> Ns_ConnFlushContent
> Ns_ConnGets
> Ns_ConnPuts
> Ns_ConnReadHeaders
> Ns_ConnReturnAdminNotice
> Ns_ConnReturnHtml
> Ns_ConnReturnNoResponse
> Ns_ConnReturnNotImplemented
> Ns_ConnReturnOk
> Ns_ConnReturnOpenFile
> Ns_ConnSendDString
> Ns_ConnSendFd
> Ns_ConnWriteChars
> Ns_ConnWriteData
> Ns_DriverInit
> Ns_FreeRequest
> Ns_GetAllAddrByHost
> Ns_GetRequest
> Ns_GetSockErrno
> Ns_GetThreadServer
> Ns_GetUserHome
> Ns_IndexDel
> Ns_IndexDup
> Ns_IndexFindInf
> Ns_IndexIntInit
> Ns_IndexStringAppend
> Ns_IndexStringDestroy
> Ns_IndexStringDup
> Ns_IndexStringInit
> Ns_IndexStringTrunc
> Ns_IntPrint
> Ns_LibPath
> Ns_ListCopy
> Ns_ListDeleteDuplicates
> Ns_ListDeleteIf
> Ns_ListDeleteLowElements
> Ns_ListFree
> Ns_ListLast
> Ns_ListLength
> Ns_ListMapcar
> Ns_ListNmapcar
> Ns_ListPrint
> Ns_NextWord
> Ns_ObjvDouble
> Ns_ObjvEval
> Ns_ObjvIndex
> Ns_ObjvLong
> Ns_RelativeUrl
> Ns_SetListFree
> Ns_SetThreadServer
> Ns_SkipUrl
> Ns_SockPipe
> Ns_SockPortBound
> Ns_SockRecv
> Ns_SockRecvBufs
> Ns_SockSend
> Ns_SockSendFileBufs
> Ns_SockStrError
> Ns_SockTimedConnect
> Ns_SockWait
> Ns_StrNSt
> Ns_StringArgProc
> Ns_UrlIsDir
> Ns_UrlIsFile
> Ns_UrlSpecificGetExact
> Ns_UrlSpecificGetFast
> Ns_VarAppend
> Ns_VarExists
> Ns_VarGet
> Ns_VarIncr
> Ns_VarSet
> Ns_VarUnset
> Ns_WaitProcess
> ns_closeonexec
> ns_duphigh
> ns_socknbclose
>
>
> --
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. SALE $99.99 this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122912
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


-

Re: [naviserver-devel] ns_info / ns_server

[Jeff Rogers]

Gustaf Neumann wrote:

> In naviserver, we have essentially the two commands "ns_info" and
> "ns_server" to obtain information about the "nds" process, about a
> single (virtual) server and about (connection) pools.

> I would propose to make the following changes to address these problems:

The changes you suggest are sensible, giving better overall introspection.

Does it make sense to have 2 different information-gathering commands, 
tho?  Why not consolidate all these functions into "ns_info", giving 
appropriate subcommands -server and/or -pool flags as necessary?

As for the pool subcommands, do the existing subcommands handle any new 
data points that may be available?  For example, if there were nested 
pools you would want parents and children, or if they were dynamic you 
would want to know when they were created.  Or more concretely, can they 
provide info on how many connections in the pool are being handled with 
background delivery?

> Some of the variants with the specified ?-server ...? might have
> concurrency issues, but maybe it is not necessary to support for every

I recall some comments in the past that ns_server doesn't do all the 
necessary locking and so is not particularly safe, being intended for 
occasional use on development servers rather than frequent use on 
heavily loaded servers.  That may have been about older or 
aolserver-specific code.

-J


--
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] naviserver with connection thread queue

[Jeff Rogers]

Gustaf Neumann wrote:
> Funny enough, the delivery of the error message blocked the connection
> thread longer than the delivery of the image when it is above the
> writersize.
>
> I will reduce the writersize further, but still a slow delivery can even
> slow down the delivery of the headers, which happens still in the
> connection thread. Most likely, i won't address this now.  however, i
> will check the effects of fastpath cache, which was deactivated so far...

Hi Gustaf,

This has been some really interesting bits you've been sharing.  I keep 
meaning to add in my thoughts but I haven't had the time.

One quick idea on the writer thread is to, regardless of size always 
make one write attempt in the conn thread, and if less than the complete 
buffer was written, then pass the remainder off to the writer thread. 
This would get the best of both worlds - fast/small requests don't incur 
the overhead of moving the duplicating the buffers, while large/slow 
requests wouldn't block the conn thread.

-J

--
Keep yourself connected to Go Parallel: 
VERIFY Test and improve your parallel project with help from experts 
and peers. http://goparallel.sourceforge.net
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] stacksize on linux

[Jeff Rogers]

Ok, this must be something weird about my build.  I'll ignore it for now.

Thanks
-J

Gustaf Neumann wrote:
> Jeff, are you sure, you are setting the parameter correctly in you
> config file? It seems to work for me (Linux FC 17).
>
> clone(child_stack=0x7f912f2b6fb0, 
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>  parent_tidptr=0x7f912f2b79d0, tls=0x7f912f2b7700, 
> child_tidptr=0x7f912f2b79d0) = 11862
> mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, 
> -1, 0) = 0x7f9116212000
> mprotect(0x7f9116212000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f9116231fb0, 
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>  parent_tidptr=0x7f91162329d0, tls=0x7f9116232700, 
> child_tidptr=0x7f91162329d0) = 11863
> mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, 
> -1, 0) = 0x7f9115f94000
> mprotect(0x7f9115f94000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f9115fb3fb0, 
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>  parent_tidptr=0x7f9115fb49d0, tls=0x7f9115fb4700, 
> child_tidptr=0x7f9115fb49d0) = 11864
>
> -gustaf
>
> Am 05.11.12 22:56, schrieb Jeff Rogers:
>> Hi all,
>>
>> I've been trying to track down why my naviserver processes have such a
>> large memory footprint compared to similarly configured aolserver
>> processes.  One culprit I already found is the pre-allocation of
>> per-conn compression streams (the fixes to this have already been
>> mentioned, tho not yet committed).
>>
>> The other culprit is strange.  It appears that naviserver is completely
>> ignoring the stacksize parameter, and starting all threads with the
>> default size, which on linux is the maximum stack size at process start
>> time, 8Mb by default.This isn't a case of the wrong parameter being
>> set, it appears to be a behavior of the nsthread library.  nsthreadtest
>> shows the same behavior, even after it calls
>>
>> Ns_ThreadStackSize(81920);
>>
>> I've added a debug printf to pthread.c to double-check what is being
>> passed as the stacksize parameter, and it's the expected value in all
>> cases, it's just not used for some reason.  I've been using strace to
>> peek at what's going on, and nsthreadtest in naviserver looks like
>>
>> mmap2(NULL, 8392704, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xb4bfa000
>> mprotect(0xb4bfa000, 4096, PROT_NONE)   = 0
>> clone(child_stack=0xb53fa494, ...
>>
>> while in aolserver it's
>>
>> mmap2(NULL, 90112, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xb748b000
>> mprotect(0xb748b000, 4096, PROT_NONE)   = 0
>> clone(child_stack=0xb74a0494, ...
>>
>>
>> As you can see it's an ~8M vs 90k (81920 + guard pages) allocation.
>>
>> Does anyone else on linux see the same behavior, or should I be looking
>> at local differences?
>>
>> Second, the naviserver and aolserver nsthread libraries aren't
>> identical, but they aren't very different.  Any ideas what would be
>> causing this behavior?
>>
>> -J
>>
>> --
>> LogMeIn Central: Instant, anywhere, Remote PC access and management.
>> Stay in control, update software, and manage PCs from one command center
>> Diagnose problems and improve visibility into emerging IT issues
>> Automate, monitor and manage. Do more in less time with Central
>> http://p.sf.net/sfu/logmein12331_d2d
>> ___
>> naviserver-devel mailing list
>> naviserver-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>


--
LogMeIn Central: Instant, anywhere, Remote PC access and management.
Stay in control, update software, and manage PCs from one command center
Diagnose problems and improve visibility into emerging IT issues
Automate, monitor and manage. Do more in less time with Central
http://p.sf.net/sfu/logmein12331_d2d
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] stacksize on linux

[Jeff Rogers]

Hi all,

I've been trying to track down why my naviserver processes have such a 
large memory footprint compared to similarly configured aolserver 
processes.  One culprit I already found is the pre-allocation of 
per-conn compression streams (the fixes to this have already been 
mentioned, tho not yet committed).

The other culprit is strange.  It appears that naviserver is completely 
ignoring the stacksize parameter, and starting all threads with the 
default size, which on linux is the maximum stack size at process start 
time, 8Mb by default.This isn't a case of the wrong parameter being 
set, it appears to be a behavior of the nsthread library.  nsthreadtest 
shows the same behavior, even after it calls

  Ns_ThreadStackSize(81920);

I've added a debug printf to pthread.c to double-check what is being 
passed as the stacksize parameter, and it's the expected value in all 
cases, it's just not used for some reason.  I've been using strace to 
peek at what's going on, and nsthreadtest in naviserver looks like

mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xb4bfa000
mprotect(0xb4bfa000, 4096, PROT_NONE)   = 0
clone(child_stack=0xb53fa494, ...

while in aolserver it's

mmap2(NULL, 90112, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xb748b000
mprotect(0xb748b000, 4096, PROT_NONE)   = 0
clone(child_stack=0xb74a0494, ...


As you can see it's an ~8M vs 90k (81920 + guard pages) allocation.

Does anyone else on linux see the same behavior, or should I be looking 
at local differences?

Second, the naviserver and aolserver nsthread libraries aren't 
identical, but they aren't very different.  Any ideas what would be 
causing this behavior?

-J

--
LogMeIn Central: Instant, anywhere, Remote PC access and management.
Stay in control, update software, and manage PCs from one command center
Diagnose problems and improve visibility into emerging IT issues
Automate, monitor and manage. Do more in less time with Central
http://p.sf.net/sfu/logmein12331_d2d
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Changing pageroot

[Jeff Rogers]

It's "pagedir" rather than "pageroot", and it's relative to the server 
root for virtual hosts (it can still be absolute).  The documentation 
appears to be not up to date on this item.

-J


David Osborne wrote:
> Hi again,
>
> In Aolserver we used to change pageroot in the config as so:
> ns_param pageroot /new/page/root
>
> Should the same work in Naviserver?
> ns_info pageroot is returning /usr/local/ns/pages no matter what I put
> in the config for pageroot (built from the latest version in bitbucket).
> Trying to work out if this is a config error or a code difference... any
> pointers?
>
> Thanks,
> - David
>
>
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
>
>
>
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>


--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff Rogers]

Andrew Piskorski wrote:
> On Fri, Oct 26, 2012 at 08:30:26PM +0100, Stephen Deasey wrote:
>
>> I was thinking it could work something like this:
>>
>> - driver acquires lock, takes first conn thread off queue, releases lock
>
> What if there are no conn threads waiting in the queue?
>

Same as currently I'd think: the driver holds on to them as waiting 
sockets.  I think the handling of this is a bit less efficient than 
putting them on the conn queue tho, as it creates more work for the 
driver to do on every spin and it needs to get woken up once threads are 
available.

-J

--
WINDOWS 8 is here. 
Millions of people.  Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff Rogers]

Stephen Deasey wrote:
> Interesting, but I wonder if we're not thinking this through
> correctly. My suggestion, and your here, and Gustaf's recet work are
> all aimed at refining the model as it currently is, but I wonder if
> we're even attempting to do the right thing?

Do we even know what the right thing is?  It could be any of
- maximize performance at any cost
- minimize resource usage
- adapt to dynamically changing workload
- minimize admin workload

And so forth.  I think there is no one-size-fits-all answer, but it 
should be possible, and hopefully easy, to get something that fits 
closely enough.

>> So I'm assuming that the available processing power - the number of
>> threads - should correlate to how busy the server is.  A server that is
>> 50% busy should have 50% of its full capacity working.
>
> But what is busy, CPU?  There needs to be an appropriate max number of
> threads to handle the max expected load, considering the capabilities
> of the machine. Too many and the machine will run slower. But why kill
> them when we're no longer busy?

I don't know how to precisely define, let alone measure busy, which is 
why I'm picking something that is readily measurable.

A thread is busy in the sense that it is unavailable to process new 
requests, but there are different reasons why it might be unavailable - 
either it's cpu bound, or it's blocking on something like a database 
call or i/o.  Different reasons for being busy might suggest different 
responses.

> - naviserver conn threads use a relatively large amount of memory
> because there tends to be one or more tcl interps associated with each
> one

Different requests could have different memory/resource requirements, 
which is a really nice thing about pools.  I'm hypothesizing that there 
are several different categories that requests fall into, based on the 
server resources needed to serve them and the time needed to complete them.

Server resources (memory) is either 'small' for requests that do not 
need a tcl interp (although tcl filters could tend to make this a 
nonexistent set), or 'big' for those that do.  Time is either slow or 
fast, by some arbitrary measure.

So a small/fast pool could be set up to serve static resources, a 
big/fast pool for non-database scripts, and a big/slow pool for database 
stuff.  I'm not sure what could be small/slow, maybe a c-coded proxy 
server or very large static files being delivered over slow connections.

The small/fast pool would only need a small number of threads with a 
high maxconnsperthread, while the large/slow pool might have many 
threads as most of those will be blocking on database access at any 
given time.

The important question in all of this is if a complex segmented setup 
like this works better in practice than a single large-enough pool of 
equal threads.  To which I don't have a good answer.

> - killing threads kills interps which frees memory
>
> But this is only useful if you can use the memory more profitably some
> where else, and I'm not sure you can.

Not only that, but memory isn't always released back to the system when 
free()d.  (vtamlloc is supposed to be able to, but I haven't had too 
much success with it so far.)  So freeing memory by shutting down 
threads won't necessarily make that available to your database.

However, memory that is not used by a process could be swapped out, 
making more physical ram available for other processes.  Having 20 
threads all used a little bit could keep them all in memory while having 
just 1 used a lot would keep a smaller working set.

This is a much bigger concern for low-resource systems.  Big systems 
nowadays have more physical memory than you can shake a stick at and 
swapping seems almost quaint.

> I think it might be better to drop min/max conn threads and just have
> n conn threads, always:

I've heard this recommendation before, in the context of tuning apache 
for high workloads - set maxservers=minservers=startservers.

I think it would make tuning easier for a lot of people if these was a 
basic "systemsize" parameter that is small/medium/large that set various 
other parameters to preset values.  As to what those values should be, 
that would take some thinking and experimentation.

> Thread pools are used throughout the server: multiple pools of conn
> threads, driver spool threads, scheduled proc threads, job threads,
> etc. so one clean way to tackle this might be to create a new
> nsd/pools.c which implements a very simple generic thread pool which
> has n threads, fifo ordering for requests, a tcl interface for
> dynamically setting the number of threads, and thread recycling after
> n requests. Then try to implement conn threads in terms of it.

I was thinking the exact same thing.

Sorry for the rambling/scattered thoughts, having a long commute does 
that :/

-J

--
WINDOWS 8 is here. 
Millions of people.  Your app in 30

[naviserver-devel] compression

[Jeff Rogers]

It looks like we're enabling compression for all http/1.1 requests 
regardless of whether it was specified in the request header, or even 
specifically disallowed.  This seems incorrect, but the code has been in 
place for several years (connio.c:CheckCompress) ;  is there a reason 
not to change that?

Also on compression, a separate compression stream is pre-allocated and 
initialized for each pre-allocated Conn, whether or not compression is 
even enabled for the server.  The causes a fairly large initial memory 
footprint.  I think this pre-initialization could be made optional, and 
bypassed entirely when compression isn't enabled.  Any thoughts?

-J

--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs: conn threads

[Jeff Rogers]

I started working on some related ideas on a fork I created 
(naviserver-queues).  The main thing I'm trying to improve is how the 
state-managing code is spread out across several functions and different 
threads.

My approach is to have a separate monitor thread for each server that 
checks all the threads in each pool to see that there are enough threads 
running, that threads die when they have been around too long (services 
too many requests), and that threads die when no longer needed.  The 
driver thread still just queues the requests, but now there's an extra 
layer of control, and yes, overhead.

The question of how many threads are needed is an interesting one: is it 
better to create threads quickly in response to traffic, or to create 
them more slowly in case the traffic is very bursty?  The answer is of 
course, it depends.

So I'm assuming that the available processing power - the number of 
threads - should correlate to how busy the server is.  A server that is 
50% busy should have 50% of its full capacity working.  I'm using the 
wait queue length as a proxy for business:  if the wait queue is 50% 
full, then there should be 50% of maxthreads running.  But this seems 
like it unnecessarily underpowers the server, I added in an adjustable 
correction factor to scale this, so that you could tune for eager thread 
creation so you will be close to maxthreads when the server is 20% busy, 
or you could wait until the server is 80% busy before spawning more than 
a few threads.  On the idle end it works similarly;  if you tuned for 
eager thread creation then the threads wait longer to idle out.

I expect that this approach would lead to a sort of self-balancing: if 
the queue gets bigger then it starts getting serviced faster and stops 
growing, while if it shrinks then it gets serviced slower and stops 
shrinking.  There's room for experimentation here on exactly how to tune 
it; in my revision this logic is all in one place so it's simple.

My initial testing shows that the server handles the same throughput 
(total req/s about the same) and is a bit more equitable (smaller 
difference between slowest and fastest request) but slightly less 
responsive (which is expected, since the requests inherently spend 
longer in the wait queue.)

I'm still cleaning it up and it's definitely not ready for prime-time, 
but I'd be interested to hear what others think.

-J

Gustaf Neumann wrote:
> I don't think, that a major problem comes from the "racy"
> notification of queuing  events to the connection threads.
> This has advantages (make os responsible, which does this
> very efficiently, less mutex requirements) and disadvantages
> (little control).
>
> While the current architecture with the cond-broadcast is
> certainly responsible for the problem of simultaneous dieing
> threads (the OS determines, which thread receives sees the
> condition first, therefore round robin), a list of the
> linked connection threads does not help to determine on how
> many threads are actually needed, how bursty thread creation
> should be, how to handle short resource quenches (e.g.
> caused by locks, etc.). By having a conn-thread-queue, the
> threads have to update this queue with their status
> information (being created, warming up, free, busy,
> will-die) which requires some overhead and more mutex locks
> on the driver. The thread-status-handling is done currently
> automatically, a "busy" request ignores currently the
> condition, etc.
>
> On the good side, we would have more control over the
> threads. When a dieing thread notifies the
> conn-thread-queue, one can control thread-creation via this
> hook the same way as on situations, where requests are
> queued. Another good aspect is, that the thread-idle-timeout
> starts to makes sense again on busy sites. Currently, the
> thread-reduction works via counter, since unneeded threads
> die and won't be recreated unless the traffic requires it
> (which works in practice quite well). For busy sites, the
> thread-idle timeout is not needed this way.
>
> currently we have a one-way communication from the driver to
> the conn-threads. with the conn-thread-list (or array), one
> has a two way communication, ... at least, how i understand
> this for now.

>> I think this is racy because all conn threads block on a single
>> condition variable. The driver thread and conn threads must cooperate
>> to manage the whole life cycle and the code to manage the state is
>> spread around.
>>
>> If instead all conn thread were in a queue, each with it's own
>> condition variable, the driver thread could have sole responsibility
>> for choosing which conn thread to run by signalling it directly,
>> probably in LIFO order rather than the current semi-round-robin order
>> which tends to cause all conn threads to expire at once. Conn threads
>> would return to the front of the queue, unless wishing to expire in
>> which case they'd go on the back of the queue, and the driver would
>>

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Jeff Rogers]

Maurizio Martignano wrote:
> OK
>
> Simple reason: Visual Studio complains about that stuff and it is annoying.
> I changed that everywhere. It doesn't do any arm, and for you it is free.
> Why you just do not accept it?

If the problem is that VS is emitting a meaningless warning, I would 
first check to see if there's a way to disable that particular warning.

The code change does do harm - it adds in noise - and it's not free - 
the stylistic change must be maintained on all future code changes.  As 
I said before tho, I speak for myself only, and a stylistic change like 
this should be agreed upon (or discounted) by the community.

> And the same goes for the keywords "new" and "delete". On the contrary the
> keyword "bool" as I already mentioned interferes with macro definition on
> "bool" in the C201X STD.
>
> The K&R STD function style was not only in the file I mentioned, but as I
> mentioned is present also one or two more places in the codebase, so feel
> free to check my archive and do a diff.

I didn't review all your code changes, I'm just commenting on what you 
said you changed.  I agreed with you on the K&R style definitions.  That 
is a far less controversial change.

> If all this C/C++ compatibility is so "crazy" and "non-sense" how come
> there's an option in gcc checking for a common subset?

I don't recall calling it crazy or nonsense.  I said I think it's not a 
good idea.

> I do not know if you are aware of the MISRA-C rules[...]

I'm not familiar with them beyond a few minutes of web searching.  But 
from those few minutes it's pretty clear that as/ns could not possibly 
conform to those rules, at least the 1998 version.  (break, continue, 
and function pointers are deal-breakers)

But those rules appear to be targeted at embedded and/or realtime 
systems; tho we might like as/ns to be realtime, it isn't :)

> Anyhow. All of this in not really important. I do not want to be annoyed by
> Visual Studio. I will use my version of the codebase. Want to use it? Please
> do, you are very welcome. Want to keep the original version, it is ok too.

You asked for comments, and I offered mine.  I expect others will chime 
in too, and they may well agree with you.  Either way, we can still find 
some way to work together.

-J

>
> -Original Message-
> From: Jeff Rogers [mailto:dv...@diphi.com]
> Sent: 17 October 2012 07:26
> To: Maurizio Martignano
> Cc: naviserver-devel@lists.sourceforge.net
> Subject: Re: [AOLSERVER] Naviserver Win-64 Sources
>
> Hi Maurizio,
>
> I appreciate your efforts on the windows portability and build front; but on
> the issue of c++ acceptability, I think you are trying to do something that
> really doesn't need doing.
>
> naviserver/aolserver is written in C.  C is not C++.  C++ is not "new C"
> or "a better C" - it is C++.  Making the code look a little more palatable
> to programmers who know C++ but not C is asking for more trouble.  I would
> expect an enthusiastic young C++ programmer to see all those casts from
> ns_malloc and ask "why is this using old-fashioned
> casts and malloc, when it should be using type-safe new?"   And the
> answer is that C is not C++; in ansi C casting from a void * to another data
> pointer type is explicitly a safe operation.  The cast is unnecessary noise.
> Many C++ programmers will tell you that casts are a sign of poor design.  In
> C++ that may be true. C is not C++.
>
> Similarly, aolserver/naviserver is not hello.c.  I don't mean to turn away
> new programmers - on the contrary, I think the code is very clean and
> understandable, but it is still complicated;  if a programmer is going to
> get confused by seeing "new" as a variable name instead of a reserved word
> then they're going to be utterly baffled trying to understand race
> conditions in rapid thread creation or the lifecycle of conn threads.
>
> So what this really amounts to is a suggestion to change the coding
> standards and build/warning flags for the project.  That is a valid topic of
> discussion.  I personally don't think they need changing, but I speak for me
> and not for anyone else involved, and as I'm sure you're well aware coding
> standards are things that holy wars are made of so there's likely to be no
> shortage of opinion on it.  If you wish to change the coding standards, it
> should be brought up as such, and not under the guise of windows
> portability, which it has nothing to do with.
>
> -J
>
> PS: re: tclxkeylist.c - ok, perhaps not so surprising there.  I suspect that
> code was imported verbatim around 1998 and updated very little since then.
>

Re: [naviserver-devel] [AOLSERVER] Naviserver Win-64 Sources

[Jeff Rogers]

Hi Maurizio,

I appreciate your efforts on the windows portability and build front; 
but on the issue of c++ acceptability, I think you are trying to do 
something that really doesn't need doing.

naviserver/aolserver is written in C.  C is not C++.  C++ is not "new C" 
or "a better C" - it is C++.  Making the code look a little more 
palatable to programmers who know C++ but not C is asking for more 
trouble.  I would expect an enthusiastic young C++ programmer to see all 
those casts from ns_malloc and ask "why is this using old-fashioned 
casts and malloc, when it should be using type-safe new?"   And the 
answer is that C is not C++; in ansi C casting from a void * to another 
data pointer type is explicitly a safe operation.  The cast is 
unnecessary noise.  Many C++ programmers will tell you that casts are a 
sign of poor design.  In C++ that may be true. C is not C++.

Similarly, aolserver/naviserver is not hello.c.  I don't mean to turn 
away new programmers - on the contrary, I think the code is very clean 
and understandable, but it is still complicated;  if a programmer is 
going to get confused by seeing "new" as a variable name instead of a 
reserved word then they're going to be utterly baffled trying to 
understand race conditions in rapid thread creation or the lifecycle of 
conn threads.

So what this really amounts to is a suggestion to change the coding 
standards and build/warning flags for the project.  That is a valid 
topic of discussion.  I personally don't think they need changing, but I 
speak for me and not for anyone else involved, and as I'm sure you're 
well aware coding standards are things that holy wars are made of so 
there's likely to be no shortage of opinion on it.  If you wish to 
change the coding standards, it should be brought up as such, and not 
under the guise of windows portability, which it has nothing to do with.

-J

PS: re: tclxkeylist.c - ok, perhaps not so surprising there.  I suspect 
that code was imported verbatim around 1998 and updated very little 
since then.  I wonder how much this functionality is still used, or if 
it would make sense to move it into a separate loadable module (could 
users just use a complete tclx, or would that interact badly with signal 
handling or other bits?), since I would expect most new code to use dict 
instead.


Maurizio Martignano wrote:
> Dear Jeff,
>   Thank you very much for your remarks. I would like to share what I
> think.
>
> 2.a
> Of course we are talking about C files and not C++ files, so any compiler
> will accept the sources as they are (at least till now). But the point here
> I believe is making the codebase easy to read and maintain. Nowadays
> programmers (unlike  us I am afraid - I am 52 years old) are used to C++
> more than C; so trying to use the common subset between ISO C e C++ standard
> make sense. If I take gcc and I compile the code with the option
> "-Wc++-compat" I get a warning for every ISO C construct that is outside of
> the common subset of ISO C and ISO C++, e.g. request for implicit conversion
> from void * to a pointer to non-void type.
> So that is an effort in making the code easily readable by programmers (not
> compilers) with C++ background.
>
> 2.b
> Shocking? Have a look at the file "tclXkeylist.c"... The same construct is
> used  here and there in other places (look at the diffs).
>
> 2.c
> Well the new STD 201X defines as keyword (see section 6.4.1) "_Bool", but
> there's a macro definition converting "bool" to "_Bool" . For "new" and
> "delete" the point is once again avoiding mismatches between C and C++,
> causing confusion to a reader/programmer/maintainer with a C++ mindset.
>
> Summing up:
> 1. I have done a static code analysis of all the codebase.
> 2. I have identified these discrepancies and corrected all of them
> 3. This makes the overall system easier to read and to maintain
>
> BTW: 90% of my current business (bread and butter) comes from doing this
> type of analysis on various systems. Most of the time it is for embedded
> systems used in avionics applications. There the language is mostly Ada and
> not C or C++.
>
> Hope it clarifies my point,
> Maurizio
>
>
> -Original Message-
> From: Jeff Rogers [mailto:dv...@diphi.com]
> Sent: 16 October 2012 01:38
> To: Maurizio Martignano
> Cc: aolserver-t...@lists.sourceforge.net
> Subject: Re: [AOLSERVER] Naviserver Win-64 Sources
>
> Maurizio Martignano wrote:
>
>> 2.A set of necessary cosmetics/make up changes to the overall code
>> base to make it more compliant with nowadays C STDs, and therefore
>> more "acceptable" to nowadays C compilers, they are:
>>
>> a.I

Re: [naviserver-devel] lurking bugs

[Jeff Rogers]

Gustaf Neumann wrote:
> On 11.10.12 21:01, Jeff Rogers wrote:
>> Gustaf Neumann wrote:
>>> Am 11.10.12 19:42, schrieb Jeff Rogers:
>>>> I'll clean up my testcases and add them.
>>> great!
>> Hrm.  I have a completely reproducible case,
> good test case. The frequency in which the script sends new
> requests
> is much higher than from programs like "siege" and the like,
> since the
> tcl script just fires locally the http-requests
> asynchronously without waiting
> for the replies.

It's somewhat similar to running 'ab' with a high concurrency (500 in 
this case).  siege I've always gotten frustrated with.  I've often found 
the need to write my own testing scripts for stuff like this, because 
the standards don't do what I want.  ab is a good example; it will tell 
you simply and easily that your server can handle 100 requests/second. 
But since it waits for each request to finish before sending the next 
one, it won't tell you and can't simulate what happens when requests 
actually arrive faster than the server can handle them.


> less than the number of queued requests, there will be threads
> left in the queue when the last connection thread exits.
>
> I have done some refactoring and added a liveliness test to
> the driver.
> it is set up currently with a timeout of 1 second.

I have a different solution which seems simpler, but I'm not sure if it 
has non-obvious implications.  What I did is to check when the conn 
thread exits if it exited because it hit maxconns and there are still 
conections queued, and if so spawn a replacement thread before exiting. 
  The subtlety I'm concerned about is that this changes the parent of a 
conn thread from the driver thread to a no-longer-existing conn thread. 
  I don't think this is a problem, because conn threads aren't joined by 
the driver thread, they're joined by other conn threads (or the main 
thread on server shutdown).  But I'm not positive, so I haven't checked 
in my changes.

-J


--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs

[Jeff Rogers]

Gustaf Neumann wrote:
> Am 11.10.12 19:42, schrieb Jeff Rogers:
>> I'll clean up my testcases and add them.
> great!

Hrm.  I have a completely reproducible case, but I'm not sure how to 
actually write a .test file for it.  Maybe you can give me some 
pointers.  Here's the setup:

config file includes:
ns_section  "ns/server/default"
ns_parammaxthreads 2
ns_parammaxconnections 1000
ns_paramconnsperthread  50

ns_section "ns/server/default/adp"
ns_parammap "/*.adp"

The pools settings greatly simplify reproduction, but I'm fairly certain 
are not strictly necessary.  I'm otherwise using simple-config.tcl

In server root are 2 sample pages:
slow.adp
---
<% after 500 %>
ok
---

fast.txt -
---
ok
---


test script "dorequests.tcl"
---
package require http 2

set slow_url "http://127.0.0.1:8080/slow.adp";
set slow_count 4
set fast_url "http://127.0.0.1:8080/fast.txt";
set fast_count 500

proc cleanup {tok} {
 puts "$tok completed"
 http::cleanup $tok
}

for {set c 0} {$c < $slow_count} {incr c} {
 http::geturl $slow_url -command cleanup
}
puts "started $slow_count slow requests"
for {set c 0} {$c < $fast_count} {incr c} {
 http::geturl $fast_url -command cleanup
}
puts "started $fast_count fast requests"

vwait forever
---

Start the server, then run "tclsh dorequests.tcl".  The "slow" requests 
will bind up the running conn threads for a bit, then the "fast" 
requests will fill up the connection queue.  The slow requests will 
eventually complete and the workers will each process cpt + spread 
requests then exit, leaving some 300 or so requests sitting in the 
queue, with nothing to process them.  A new incoming request will start 
one worker, which will process the next cpt+spread queued requests and 
exit, and so forth.

Limiting maxconnections helps some, because it ensures that any new 
thread created as a result of a new incoming connection will be able to 
process the entire queue.  However, since the spread is random, it's 
unlikely but possible in certain configurations for all threads to 
finish their overtime work with connections still in the queue.

> i looked a little into the request burst problem and got some better
> understanding. The problem actually happens when the server runs
> out of threads and more than maxconn requests arrive constantly
> in the timespan of a thread creation.

I think it's when many requests arrive in less than the timespan of a 
thread lifetime, not just creation.

> This served us well over the last
> two years with busy real-world traffic. However, under
> benchmark-situations, the server can be flooded so fast, that the
> queuing during the serialized thread creation causes the problem.

Since new connections will respawn threads, it's nearly impossible to 
run into the problem on a busy server.

> So, the behavior should be at least parameterized, and tailorable.
> i have added a proposal to hg, which is just a few lines of code,
> that allows parallel thread creation above a certain threshold.
> the threshold should become a parameter. If the threshold
> is set large enough, the thread serialization happens, if it is
> small, parallel thread creates are allowed. Ideas for a
> good name of the parameter are very welcome.
>
> Please test, if this helps as well for your test cases...

Thanks, I'll take a look.

-J


--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] lurking bugs

[Jeff Rogers]

Gustaf Neumann wrote:
> Dear Jeff,
>
> Is it possible to add test cases showing the problems to the
> regression test?
> This way it is easier to pinpoint the problem.

I'll clean up my testcases and add them.

> for the connection queuing example, it is harder to write a
> test.
> yes, it is true to run in this situation in pathological
> cases such
> as benchmarks and "bad configurations". While i tend to agree
> to this diagnosis, i am not so sure about the prosed cure.

After giving it some thought, my proposed solution won't work.  It may 
more effectively mask the problem, but it doesn't fix it.


> this is not a desirable solution: connsperthread might be
> set to 1 by a user
> who wants always fresh threads, but setting "maxconnections"
> to 1 would
> lead under this rule to a single connection structure
> (maxconnection == 1)
> which implies sequential connection  processing.

Yes, this would be a problem too.

> Wouldn't be
> a timeout on
> the queue a better solution, functioning similar to incoming
> request but
> handling queued requests if there are these?

I'm not sure how that would work - the problem is that there is nothing 
running to time out, unless there was a pool watchdog thread to start up 
new conn threads if the pool drops below minthreads running.

-J



--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] lurking bugs

[Jeff Rogers]

Hi,

I've been browsing the naviserver code, learning the differences from 
the aolserver code that I'm more familiar with, and checking for a few 
bugs that I've found and/or fixed previously.  Here's what I've come 
across so far.  These are all pretty unusual cases with straightforward 
workarounds, but they're still bugs.

fastpath stale cache - with fastpath caching enabled, it is possible to 
serve incorrect data with 'ns_returnfile' if a constant filename is 
being used and overwritten repeatedly.  The problem is much less serious 
than in aolserver where the file inode is used as the cache key instead 
of the file name, meaning that using [ns_tmpnam] or [ns_mktemp] for a 
temporary file instead of a constant name completely avoids the issue; 
but it is still present and the fix of not caching files with 
ctime==time() works.

ns_returnfile doesn't decode passed filenames from tcl correctly.  As a 
result, [file exists $file] could report that a file does exist but 
[ns_returnfile 200 text/html $file] could give a "not found" error.  The 
circumstances needed to make this happen are uncommon (changing tcl's 
system encoding, actually having files in strange encodings) but it is 
possible.  Fix should be to pass the argument to ns_returnfile through 
Tcl_UtfToExternalDString to get a 'native' file name.

ADP parsing doesn't handle nested <% %> sequences correctly.  There is a 
fix available.

It is possible to get into a situation where there are connections 
queued but no conn threads running to handle them, meaning nothing 
happens until a new connection comes in.  When this happens the server 
will also not shut down cleanly.  As far as I can figure, this can only 
happen if the connection queue is larger than connsperthread and the 
load is very bursty (i.e., a load test);  all the existing conn threads 
can hit their cpt and exit, but a new conn thread only starts when a new 
connection is queued.  I think the solution here is to limit 
maxconnections to no more than connsperthread.  Doing so exposes a less 
severe problem where connections waiting in the driver thread don't get 
queued for some time; it's less of a problem because there is a timeout 
and the dirver thread will typically wake up on a closing socket fairly 
soon, but it can still result in a simple request taking ~3s to 
complete.  I don't know how to fix this latter problem.

Obsolete commands in manpages - several still have examples with 
ns_share - ns_mutex, ns_register, and ns_register_filter.  Examples in 
ns_mutex and ns_sockcallback refer to 'detach', which I've never heard 
of but from the usage I'm guessing is the predecessor to ns_chan.

-J

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] building with tcl8.6

[Jeff Rogers]

Hi all,

An advisory about building with tcl8.6: all exported symbols in shared 
libraries need to be declared as NS_EXPORT.  I checked in the necessary 
changes to the header files, but all modules are affected too.

So for modules,
int Ns_ModuleVersion = 1;
needs to change to
NS_EXPORT int Ns_ModuleVersion = 1;

and
int
Ns_ModuleInit(char *server, char *module)

becomes
NS_EXPORT int
Ns_ModuleInit(char *server, char *module)

Other than that, I haven't noticed any issues.

-J

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_set

[Jeff Rogers]

Stephen Deasey wrote:

> How are you planing to use these key/value structures?

[...]

> Or maybe you're trying to recreate PHP's do-everything hash-list
> container... :-)

Honestly, it's just something that has bugged me - a data structure with 
O(n) performance?!  This cannot stand!  But as a practical matter, in 
the typical use cases, the constant factor is low enough that it's 
simply not an issue.  Saving 30 microseconds per request isn't going to 
make a difference in any real-world application.

I'll stash my half-written code away for another day, in case someone 
ever loses a microsecond or two :)

-J


--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] positioning naviserver

[Jeff Rogers]

Gustaf Neumann wrote:

> Forward-looking does certainly not mean that we do not want
> to keep compatibility (we have all large code bases using
> naviserver) but i would certainly hate to see e.g. the
> useless $conn argument to be re-introduced in naviserver
> just because some 10+ year old code expects it.
>
> The general rule of change should be
>  major-versions-are-allowed-to-break-backward-compatibility
> and
>  minor-versions-should-keep-backward-compatbility
> where we are talking about intra-naviserver compatibility.

This is pretty standard and sensible for a project supported by volunteers.

However, what I'd like to see - when reasonable - is an option for 
backwards compatibility, either by setting a config flag or loading a 
compatibility module, as nsshare appears to be shaping up into.  Adding 
in a version of ns_register_filter that passes aolserver-style args 
would not be a stretch.

This I think could give the best of both worlds - improvements in 
functionality without alienation of user base.

-J

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_set ... -persistent

[Jeff Rogers]

Zoran Vasiljevic wrote:
>
> On 10.10.2012, at 12:11, Stephen Deasey wrote:
>
>> I was thinking that the 2 or 3 people in the world that need backward
>> compatibility would load the nsshare module and then:
>>
>>   rename _ns_set ns_set
>>
>> You wouldn't use them both, you just want your old code to work the
>> way it did in 1999.
>>
>
> I find this to be a good compromise.

I agree, this is a good compromise.  But could that renaming of the core 
tcl command be done as part of the module?  So the instructions would be 
just "load this module in your config" rather than "load this module in 
your config and add this line somewhere in your library code".  The goal 
being to make backwards compatibility settable just in the server config.

-J

> The nsshare module we can deliver
> as-is and declare unsupported and
> deprecated. People that really depend
> on it can then take the responsibility
> of keeping it up-to-date. And since it
> is not in the main server code, we can
> (mostly) forget about it.
>
> Cheers,
> Zoran


--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_set ... -persistent

[Jeff Rogers]

Zoran Vasiljevic wrote:
>
> On 09.10.2012, at 20:49, Jeff Rogers wrote:
>
>> creating a hashtable mapping the keys to indexes
>
> I assume you used Tcl hash-tables with TCL_STRING_KEYS?
> You know that ns_set keys can be case-insensitive?

Yes - I've only half-implemented it so far, but the full implementation 
would need to create two hashtables, one with case preserved and one 
with all keys tolower-ed.  I expect the costs and benefits would be similar.

-J

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_set ... -persistent

[Jeff Rogers]

I don't know that people love it so much as that there is no exact 
replacement for the functionality.

I'm a big proponent of compatibility but shared sets as they previously 
existed are problematic.  I wouldn't be at all surprised if they are 
thread-unsafe to the point of being able to crash the server.

An alternative might be to provide equivalent functionality 
(order-preserving, multiple entries for keys, access by index or key or 
case-insensitive key) as a new nsv subcommand, perhaps 'nsv_multiset'.

Separately, I was experimenting with a performance enhancement to 
ns_sets, creating a hashtable mapping the keys to indexes.  There is a 
slight cost in memory and on the first lookup, but it should get to 
breakeven after only around 3 lookups on average, with decreasing 
amortized cost after that.  The difference is measurable and can be 
significant, as much as 4x faster on a large set.  However, that's 
measured on a very large set, 1000 keys;  and the real savings is pretty 
small, around 15 microseconds per lookup on average hardware.  Is that 
kind of micro-optimization worth the additional complexity?

-J

Gustaf Neumann wrote:
> Stephen,
>
> do you remember why you took out the -shared flag from ns_set?
>
> https://bitbucket.org/naviserver/naviserver/changeset/1cbaf1acc09436f2a1c56102269a8b7fab0be168
>
>
> it seems that some people love it. We have either to take it
> out of
> the documentation (and give sensible explanation) or
> reintroduce it in the code...
>
> -gustaf


--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Greetings

[Jeff Rogers]

Hi all,

I've exchanged correspondence with a number of people here but I'd like 
to extend a formal introduction - well, as formal as mailing lists get.

I suspect most of the people on this list also read the aolserver list 
and have seen the recent interest in re-merging aolserver and naviserver 
development and community.  I'm happy to be a part of this effort.  I've 
only kept up with developments in naviserver intermittently, so I'm just 
now starting to learn and explore the differences.  From what I can tell 
a lot of what I'd like to see in aolserver has already been done in 
naviserver.

I have worked with aolserver for a while, mostly as a user and 
occasional patch-submitter / bugfixer, but only recently as an active 
developer.  Other than what I've been doing, AOLserver has been seeing 
very little development.  This, along with other unrelated 
considerations has led me to adopt some sloppy practices, notably a 
tendency to commit working but incomplete (i.e., poor style, 
documentation) changes earlier and clean up on later commits.

While this has worked ok for me in the past, naviserver is a different, 
more active community and development style.  I've already made a few 
missteps and been advised of commit guidelines, but I'm still learning 
what the preferred development style is (and also getting used to the 
quirks of mercurial).   So then - what is the preferred style?  Propose 
changes on the development list and then make changes to the main 
branch,  make changes on a dev branch and merge after discussion, or 
create a private forked repository and merge changes in as patches/pull 
requests, or something else entirely.  I'm happy to follow whatever the 
convention is, tho I certainly prefer to avoid additional administrative 
overhead and making changes in multiple places.

I'm looking forward to working with everyone.

-J

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Migrating aol.com from proprietary platform (AOLserver) to Open Source

[Jeff Rogers]

Stephen Deasey wrote:
> Some pretty funny miss-statements in this history of AOLserver running 
> aol.com:
> 
> http://velocityconference.blip.tv/file/2286110/

Wow, the hatred for tcl is palpable.  This is something I've never 
understood.

-J

--
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] How to listen on 2 ports?

[Jeff Rogers]

Vasiljevic Zoran wrote:
> He you socket/driver gurus out there!
> 
> What are my options if I want to have NS listen
> to more than one port?
> 
> We have the setup with just one virtual servers
> and listen on 0.0.0.0 address (all interfaces).
> What we would like to do is to listen on 1 more
> port but if possible from the same virtual server.
> I would not like to complicate things by introducing
> a second virtual server, it at all possible.
> 
> Ideas?

Naviserver may have changed enough underneath to be different here, but 
in aolserver, you can load the nssock module twice under different names 
and configure them separately.  Kind of like how the ssl driver is 
loaded separately from the plain socket driver.

I've used a config like this to accomplish it:

[ns/server/s/modules]
ns_param nssock nssock.so
ns_param nssock2 nssock.so

[ns/server/s/module/nssock]
ns_param Address bla.example.com
ns_param Port 80

[ns/server/s/module/nssock2[
ns_param Address bla.example.com
ns_param Port 8822

-J

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Bug in fastpath cache causes info leak

[Jeff Rogers]

Brett Schwarz wrote:
> Wow, I am amazed at the different reactions to this issue between
> this list and the aolserver list. Makes me want to switch to
> naviserver even more now...

90% of the difference is that the issue was introduced here by someone
saying "I contributed a patch."  That in itself gets a better reaction 
than hyperbole and insisting that everyone else doesn't know what 
they're talking about.  On the flipside, some of the reactions were 
overly harsh and dismissive.

BTW, the solution in the patch of adding the filename to the hash key
was rejected by the OP.

After all the discussion there, I am generally in agreement that there 
is no real problem other than insufficient documentation, caveats, and 
examples.  But some options to make the cache less aggressive when 
necessary can be a good thing.  Other than adding in the filename (which 
causes more memory to be used) a way to avoid the caching in the first 
place could be good; either a -nocache flag to ns_returnfile or a 
different command (to still get the performance of mmap-ing) or my other 
suggestion of skipping caching for files matching certain patterns 
(e.g., under a particular directory like /tmp, /var/tmp, or whatever) 
would avoid uselessly using up the cache memory in the first place.

As far as the performance impact (of mmap or fastpath in the first 
place) numbers speak louder than speculation, and testing it isn't that 
hard.


-J

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns as loadable extension

[Jeff Rogers]

Andrew Piskorski wrote:
> On Wed, May 07, 2008 at 05:05:13PM +0200, Vasiljevic Zoran wrote:
> 
>> Does anybody have any interest in making us being a
>> yet-another Tcl loadable extension?
> 
> Yes.  From his past comments on the AOLserver list, Jeff Hobbs would
> probably be interested too.

Rather than the entire server being a loadable extension, what about 
breaking out the major c-coded functional pieces into a group of 
loadable extensions?  The modules could include adp (a fast/extensible 
template engine), urlspace (a trie oriented around urls), driver (a 
threaded alternate to the standard tcl event loop, oriented around 
creating events on one thread to be dispatched on others), and of 
course, db for database access.  Some other pieces of functionality, 
like scheduled jobs ("after") , ns_sets (dicts?), nsv shared variables 
(thread package tsvs), http, and sockets seem to be already sufficiently 
covered in the core or standard tcl library.

-J

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_config read/write

[Jeff Rogers]

Vasiljevic Zoran wrote:

> Do you have some other idea how (if) we should build the chnageable
> config? It seems pretty silly to me that we need to restart the server
> to change some marginal parameter... In the 21. century...

However this gets resolved, I think there should also be a way to easily 
write out a config file based on the current dynamic configuration, so 
that if the server does get restarted the configuration can stay the same.

-J

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Proposal: just an idea about config syntax

[Jeff Rogers]

Vlad Seryakov wrote:

I like the general look and that it can be implemented in pure tcl 
(heck, even by just sourcing the implementation at the top of your 
current config).

> it may look like this (still Tcl)
> 
> section "fastpath" {
>  cache   false
>  cachemaxsize512
>  cachemaxentry   512000
>  mmapfalse
> }
> 
> section "server/${servername}" {
>  connsperthread  0
>  flushcontentfalse
>  maxconnections  100
>  maxthreads  10
>  minthreads  0
>  threadtimeout   120
> }

Alot of the section names are heirarchical too so could this concept be 
taken further?  e.g.,

server "server1" {
 directoryfile index.html
 pageroot  /root
 section "tcl" {
 library   /root/server1/lib/tcl
 }
 section "modules" {
 nslog  ${bindir}/nslog.so
 }
}



> As you can see this is Tcl command section
> 
> proc section { name args } {
> 
> ns_section ns/$name
> foreach { key val } $args {
>ns_param $key $val
> }
> }

It would need to be uplevel-ed if variable expansion is to take place 
but that's pretty minor.

-J

> 
> But config looks like more 21 century and more structured and easier to 
> read. Very similar to lighttpd by the way.
> 
> 


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] aolserver incompatibilities

[Jeff Rogers]

Is there a simple, concise, and comprehensive[*] list of specific 
incompatibilities between the current versions of ns and aolserver?
I'm considering moving my aolserver code to naviserver, I want to know 
what pitfalls I can expect.

Thanks,
-J

[*] - irony intentional :)

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_cache pruning, when, how?

[Jeff Rogers]

Stephen Deasey wrote:


Hit rate is not necessarily the best measure of effectiveness. Perhaps
it's more expensive to fill one cache than it is to fill another?  A
single, shared LRU seemed like the simplest solution.  Ideas
appreciated.


What if 'ns_cache eval' stored with the cached value an estimate of how 
much work (i.e., time) it took to create the cache entry, and used that 
as a hint for expiring or flushing values?  An operation that is used 
1/10 as often but takes 50x longer to compute may be worth keeping 
around longer.


And as long as I'm musing on caching, 2 situations dealing with 
expensive cache calculations come to mind.  One is where there is no 
current value for an entry and several threads request it simultaneously 
(or it is requested multiple times before the first request has time to 
complete evaluation of the script).  Do the second and subsequent 
accesses start their own evaluation of the script, or do they block 
until the first completes and then all return the single value?  Without 
digging into the code or running tests, my guess is that this works 
correctly, since the key should be locked with a mutex and reads would 
block the thread until the mutex is released.


A second situation is where there is an expired entry and multiple 
threads access it at the same time.  It is probably handled as above, as 
if there was no entry.  But it might be desirable in certain situations 
to return a stale entry immediately and start a separate thread to run 
the script.


These solutions may be too clever for their own good and apply only to a 
very few marginal cases.


-J

Re: [naviserver-devel] Quest for malloc

[Jeff Rogers]

Gustaf Neumann wrote:

This is most probably the best variabt so far, and not complicated, such a
optimizer can do "the right thing" easily. sorry for the many versions.. 


-gustaf


{ unsigned register int s = (size-1) >> 3;
  while (s>1) { s >>= 1; bucket++; }
}

  if (bucket > NBUCKETS) {
bucket = NBUCKETS;
  }


I don't think anyone has pointed this out yet, but this is a logarithm 
in base 2 (log2), and there are a fair number of implementations of this 
available; for maximum performance there are assembly implementations 
using 'bsr' on x86 architectures, such as this one from google's tcmalloc:


// Return floor(log2(n)) for n > 0.
#if (defined __i386__ || defined __x86_64__) && defined __GNUC__
static inline int LgFloor(size_t n) {
  // "ro" for the input spec means the input can come from either a
  // register ("r") or offsetable memory ("o").
  size_t result;
  __asm__("bsr  %1, %0"
  : "=r" (result)   // Output spec
  : "ro" (n)// Input spec
  : "cc"// Clobbers condition-codes
  );
  return result;
}
#else
// Note: the following only works for "n"s that fit in 32-bits, but
// that is fine since we only use it for small sizes.
static inline int LgFloor(size_t n) {
  int log = 0;
  for (int i = 4; i >= 0; --i) {
int shift = (1 << i);
size_t x = n >> shift;
if (x != 0) {
  n = x;
  log += shift;
}
  }
  ASSERT(n == 1);
  return log;
}
#endif

(Disclaimer - this comment is based on my explorations of zippy, not vt, 
so the logic may be entirely different)  If this log2(requested_size) is 
used to translate directly index into the bucket table that necessarily 
restricts you to having power-of-2 bucket sizes, meaning you allocate on 
average nearly 50% more than requested (i.e., nearly 33% of allocated 
memory is overhead/wasted).  Adding more, closer-spaced buckets adds to 
the base footprint but possibly reduces the max usage by dropping the 
wasted space.  I believe tcmalloc uses buckets spaced so that the 
average waste is only 12.5%.


-J

Re: [naviserver-devel] Directory structure

[Jeff Rogers]

Vlad Seryakov wrote:

Hi,

I am proposing new directory structure for /usr/local/ns, current 
installation of naviserver comes from aolserver multi-server environment 
which is not very usual. For beginners it is very confusing, for 
advanced users it does not matter because they customize it as they want.


I'n new to naviserver but I've been working with aolserver for a while.

The simpler sample-config and directory structure is good, but please 
include a more complex example too - a working setup with multiple 
servers (virtual hosts) would be a good thing; setting up vhosts in 
aolserver 4 is very flexible but not at all clear to the novice (or even 
intermediate) developer.  For vhosts, better still would be a 
question/answer script to generate the appropriate part of the config file.


-J