I'm re-sending this on the list as my first answer went only to Karl.
On Wed, 19 Jan 2011 18:18:55 +0000, Karl Lehenbauer wrote
> Greetings, Programs!
>
> Here are a couple-three things I think would make a big difference for
> using Rivet to build and deploy big websites.
> * Named separate virtual interpreters.
> We use separate virtual interpreters for development (for each developer
> we diddle auto_path to give them source-controlled private copies of all
> of our packages) and it's really badass. The problem comes in the
> all-or-nothing approach of SVI. We set up a virtual host for each
> developer, for both port 80 (well really 8080 + varnish on 80) and 443.
> This results in 18 interpreters in each httpd process on our development
> machine, making for very large httpd processes and slow startup time
> after a graceful.
>
> If we could name the virtual interpreters, we could cut the number of
> interpreters from 18 to 9 in this case.
>
If I understand you would keep a vhost for each developer but both the
80 and 443 ports would have the same interpreter by changing the naming
scheme in Rivet_InitTclStuff
for (sr = s; sr; sr = sr->next)
{
....
if (sr != s) /* not the first one */
{
if (rsc->separate_virtual_interps != 0) {
char *slavename = (char*) apr_psprintf (p, "%s_%d_%d",
sr->server_hostname,
sr->port,
interpCount++);
/* Separate virtual interps. */
myrsc->server_interp = Tcl_CreateSlave(interp, slavename, 0);
if (myrsc->server_interp == NULL) {
ap_log_error( APLOG_MARK, APLOG_ERR, APR_EGENERAL, s,
"slave interp create failed: %s",
Tcl_GetStringResult(interp) );
exit(1);
}
Rivet_PerInterpInit(s, myrsc, p);
} else {
myrsc->server_interp = rsc->server_interp;
}
......
}
...
}
it doesn't seem impossible, probably just creating a temporary hash table
mapping server_hostname to a slave interpreter and reassigning the same
interpreter down in the chain of virtual host records.
Did I get it right?
> 2. Virtual interpreter restarts without a graceful
>
> Right now if one of our developers changes a package, private to them,
> they still have to do an apachectl graceful to pick up the change.
> This restarts all of the httpd processes and reinitializes all of the
> interpreters. Our interpreter initialization is intense.
> Each FlightAware httpd process loads 468 packages.
>
> I'd like to be able to cause only one vhost's Tcl interpreters to be
> reloaded by a Tcl_DeleteInterp / Tcl_CreateInterp / Rivet initialization
> process. Instead of a graceful, you'd be able to specify something
> like a trigger file for each vhost. Every time a vhost (with separate
> virtual interpreters) serves a page, it gets the mtime of the trigger
> file. If the mtime of the trigger file has changed since the last time
> the interpreter served a page, Rivet deletes the virtual host's
> interpreter, creates and initializes a new one, and then handles the
> page. [I tried to write this but kind of lost control of it and
> was not successful.]
>
> This way, developers could totally reload all their libraries
> without any httpd processes being stopped or started. Also this
> will lower overall overhead because a lot of times a httpd process
> won't have ever handled a page
> in its lifetime for many to most of the virtual interpreters.
>
> An additional improvement would be to create the ability to not even
> initialize a vhost's separate virtual interpreter until the first time
> it is needed.
I think this is doable and in a way overlaps with what I'm doing
with my ServerInitScript. I was trying to isolate the interpreter
initialization function in order to make it reusable in different
contexts. It would be nice if we could came up with an ipc scheme
that could help also in the application context. Apache people
say that stuff like shared memory is something one has to keep
off when doing web programming, but I always liked system
programming....
>
> 3. Apache children inheriting a preloaded interpreter from the
> parent Apache process
>
> This is unrelated to separate virtual interpreters -- this is for the
> production website. When we graceful a production webserver we take
> it out of the load balancer first. When we start 200 httpd processes
> and each one loads
> 468 packages, it's not pretty. There is a lot of lock contention in
> FreeBSD while these processes all bang on the same package directories,
> something we don't totally understand. (This is much faster with ZFS,
> once tuned, btw.)
> It takes minutes to settle down before our stuff can put the
> server back into the webserver. What would be incredibly cool would
> be to be able to load up the 468 packages in the parent httpd process
> one time and then have each child process use the same Tcl interpreter
> already loaded with the packages. This would be a way more than
> 200-fold improvement in Apache startup time for us (because it
> would eliminate all the contention.) [I don't even know if this
> is possible.]
>
> I offer this because it's like a possible direction to take Rivet
> development. I can help, definitely, and provide production load :-),
> if anyone is interested in taking any of these ideas to further
> design, code, etc, please let me know.
>
so slave interpreters don't inherit packages loaded in their parent
interp? Getting the answer should be easy...
> Karl
-- Massimo
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
