Re: mod_fcgid: different instances of the same program

[Danny Sadinoff]

On Tue, Nov 10, 2009 at 1:47 AM, Danny Sadinoff  wrote:

> On Tue, Nov 10, 2009 at 12:53 AM, Jeff Trawick  wrote:
> >
> > On Mon, Nov 9, 2009 at 5:16 PM, Danny Sadinoff 
> wrote:
> > > 2) Virtual hosts
> > > The above item holds true even across virtual hosts.   So while
> > > it's possible to adjust the FcgidInitialEnv items on a per-vhost
> > > basis, this is a recipe for disaster if two vhosts point at the same
> > > fcgi executable, because the resulting processes with potentially
> > > different Environments will be inserted into the same pool.  Once that
> > > occurs, we may expect that a server spawned with config defined in
> > > vhost A will be parcelled out to vhost B.
> >
> > Where does this occur?  Entries in the process table are distinguished
> > by virtual host.  (I think the implementation of this check is broken,
> > in that it requires that ServerName is set in the virtual hosts.  Are
> > you using a simple test config that doesn't have ServerName set?)
>
> My case is not yet simple.  I'll get back to you.
>

It turns out that the problem was that I was using mod_fcgid-2.2   Upgrading
to mod_fcigd-2.3.4 fixed the problem. I apologize for the noise.

Should this item (vhost independence) be added to the upgrade notes section?
Are new features in mod_fcgid going to be manifest in the documentation in
general?

-- 
Danny Sadinoff
da...@sadinoff.com

balancer-manager and server-status feature request.

[Mark Watts]

The statistics one gets from both /balancer-manager and mod_status are
useful but of course only exist until httpd is restarted.

It would be nice if they could be configured to periodically write some
lines to the error log (at LogLevel info or so) with these statistics so
the data can be preserved.
This would make it easy to parse by log monitoring tools and also allow
for analysis if desired.

XML output of both would be the icing on the cake :)


Mark.

-- 
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, Managed Services Manpower
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg


signature.asc
Description: This is a digitally signed message part

Re: balancer-manager and server-status feature request.

[Mladen Turk]

On 16/11/09 11:33, Mark Watts wrote:


The statistics one gets from both /balancer-manager and mod_status are
useful but of course only exist until httpd is restarted.

It would be nice if they could be configured to periodically write some
lines to the error log (at LogLevel info or so) with these statistics so
the data can be preserved.


This could be done using watchdog module that would fire some
balancer log module.
Regarding xml data, it is my long standing wish to create
log output filter sub module system where the log lines would
go trough a VFS filter capable of writing to xml, database, etc
(depending on the VFS implementation).


If you came up with such a module I'd be happy to review it :)

Regards
--
^TM

mod_proxy_fcgi changes SCRIPT_FILENAME?

[Dong Wang]

I am trying to use apache-2.3's mod_proxy_fcgi in 2.2.13. It seems to be
built and run all right.

But I have noticed that the SCRIPT_FILENAME has been changed to
"proxy:balancer://xx", it cann't be recognized by  the remote PHP
backend. So the request failed.
In my opinion, the remote PHP backend use the SCRIPT_FILENAME to find the
script file. But the PHP and the Apache may be in different computer, so the
Document Root may be different. is that means the PHP shouldn't rely on the
SCRIPT_FILENAME? then how can the PHP know which script file to execute?

Or, can the mod_proxy_fcgi make the SCRIPT_FILENAME to be the real file
location? then the PHP backend in the same computer will find the file
correctly.

Re: mod_rewrite and mod_fcgid pass wrong fcgi request

[Jeff Trawick]

On Mon, Nov 16, 2009 at 1:04 AM, Felipe Alcacibar  wrote:
> ...
>> When comparing modes of PHP execution:
>>
>> - CGI and FastCGI are directly comparable because the information that
>> Apache needs to pass to PHP is the same.  (In fact, mod_cgi[d] and
>> mod_fcgid use the same core Apache code to build almost all of that
>> information.)
>>
>> You should be able to take your CGI configuration and change the
>> handler from cgi-script to fcgid-script and have PHP work as FastCGI.
>> One nuance is that if you're relying on ScriptAlias you'll need to use
>> both Alias and Options +ExecCGI in the FastCGI equivalent.
>>
>> - mod_php and mod_fcgid configurations are not comparable.
>>
>
> As you sayed it i made a similar environment (the difference is the
> FcgidWrapper line)

PHP can be configured via Action instead of FcgidWrapper with
mod_fcgid too.  That is the most direct comparison.


>  from mod_cgid to mod_fcgid and the result is the
> same, in the mod_cgid environment all runs good and in the mod_fcgid
> persisst the problem...
>
> rohan counter # cat /etc/apache2/modules.d/70_php5_cgi.conf
> 
>        ScriptAlias /.httpd/cgi-interpreters /var/cgi-interpreters/
>        
>                AllowOverride None
>                Options None
>                Options +ExecCGI
>
>                Order allow,deny
>                Allow from all
>        
>
>        
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .php
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .php2
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .php3
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .php4
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .php5
>        FcgidWrapper "/var/cgi-interpreters/php-cgi" .phtml
>
>        FcgidInitialEnv PHP_FCGI_CHILDREN "12"
>        FcgidInitialEnv PHP_FCGI_MAX_REQUESTS "1"

BTW, check the mod_fcgid docs for comments on these two PHP
environment variables.

>
>        AddHandler fcgid-script .php .php3 .php4 .php5 .php6 .phtml
>        
>
>        
>                AddHandler php-script .php .php3 .php4 .php5 .php6 .phtml
>                Action php-script "/.httpd/cgi-interpreters/php-cgi"
>        
>
>        
>                Options +ExecCGI
>        
>
>        DirectoryIndex index.php index.php3 index.php4 index.php5 index.phtml
> 
>
>
> Any suggestion or step to do??...

Use Action with mod_fcgid too so that mod_fcgid processing works
through the same mechanisms as with your mod_cgid setup.

I'll try to find some time today to play with that as well.
Eventually I'd like the documentation to show an Action-based example,
with support for easily switching between CGI and FastCGI for
debugging or other purposes.

Re: mod_fcgid: different instances of the same program

[Jeff Trawick]

On Mon, Nov 16, 2009 at 5:03 AM, Danny Sadinoff  wrote:
> On Tue, Nov 10, 2009 at 1:47 AM, Danny Sadinoff  wrote:
>>
>> On Tue, Nov 10, 2009 at 12:53 AM, Jeff Trawick  wrote:
>> >
>> > On Mon, Nov 9, 2009 at 5:16 PM, Danny Sadinoff
>> >  wrote:
>> > > 2) Virtual hosts
>> > > The above item holds true even across virtual hosts.   So while
>> > > it's possible to adjust the FcgidInitialEnv items on a per-vhost
>> > > basis, this is a recipe for disaster if two vhosts point at the same
>> > > fcgi executable, because the resulting processes with potentially
>> > > different Environments will be inserted into the same pool.  Once that
>> > > occurs, we may expect that a server spawned with config defined in
>> > > vhost A will be parcelled out to vhost B.
>> >
>> > Where does this occur?  Entries in the process table are distinguished
>> > by virtual host.  (I think the implementation of this check is broken,
>> > in that it requires that ServerName is set in the virtual hosts.  Are
>> > you using a simple test config that doesn't have ServerName set?)
>>
>> My case is not yet simple.  I'll get back to you.
>
> It turns out that the problem was that I was using mod_fcgid-2.2   Upgrading
> to mod_fcigd-2.3.4 fixed the problem. I apologize for the noise.

great/np

> Should this item (vhost independence) be added to the upgrade notes section?

Unfortunately, I'm still in the dark about vhost independence and how
exactly the share_grp_id construct in the source code can change the
process association without more code written to tie it to the
configuration.  ISTR that Rainer made the same observation --
share_grp_id isn't completely implemented.  (Maybe it once worked and
some code was lost?)

> Are new features in mod_fcgid going to be manifest in the documentation in
> general?

definitely

Re: TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Jean-Marc Desperrier]

Stefan Fritsch wrote:

On Tuesday 10 November 2009, Jean-Marc Desperrier wrote:

[ Apache + openssl 0.9.8l = TLS renegotiation fully disabled ]
First there's the short SSLSessionCacheTimeout problem :
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c23
[...] If they actually are renegotiation caused by SSLSessionCacheTimeout,

>> [...], this means this was

  already broken in some way before, but it used to be of little
  consequences and will now be a huge problem.

Second there's the SSLVerifyClient optional problem :
https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c21
[...]  what this comment report is that simply having SSLVerifyClient
 optional set, [...], will cause renegotiation to happen
 and therefore sites to break when TLS renegotiation is disabled.


I cannot reproduce the problems. With an openssl that rejects all
renegotiations, both reconnections after ssl session timeout and
connections to a host with sslverifyclient optional work fine (with
openssl s_client).


Thank you for your interest on that problem.

One thing still : Everyone who uses client certificate authentication 
knows that they are many apache configurations around that will force 
the user to repeatedly reauthenticate himself for apparently no good reason.


It's hard to believe the explanation is only that all of the concerned 
sites forgot to activate the "session resume" option.
SSLVerifyClient and SSLSessionCacheTimeout forcing unnecessary 
renegotiation did seem like a very plausible alternative explanation.


This fact is the very reason why the vilified "remember client 
certificate" option is there in Firefox 3.5 (wasn't there in 3.0), 
there's a large number of bugs opened on the subject in their bugzilla :

https://bugzilla.mozilla.org/show_bug.cgi?id=510820
https://bugzilla.mozilla.org/show_bug.cgi?id=453802
https://bugzilla.mozilla.org/show_bug.cgi?id=428744
https://bugzilla.mozilla.org/show_bug.cgi?id=474445
https://bugzilla.mozilla.org/show_bug.cgi?id=395399
https://bugzilla.mozilla.org/show_bug.cgi?id=32010

I'll try to find out more about this, with so many users reporting that 
problem, there should be a way to get some more detailed info about what 
causes it, if it's related with erroneous renegotiation or not.

A fundamentally secure Apache server, any interest?

[Sweere, Kevin E CTR USAF AFRL/RYT]

Greetings,
 
I work for the US Air Force.  We have a prototype that dramatically,
fundamentally increases a web server's security.  
 
We run an Apache server within a minimized, user-level-only, Linux variant
only within RAM and from only a DVD (no harddrive).  With no shells, hackers
have nowhere to go.  With no persistent memory, malware has no place to
reside.  A simple reboot restores the website to a pristine state within
minutes.  
 
Because a LiveDVD holds the OS, apps and content, its best for static,
non-interactive, low-volume, high-value, highly-targeted websites.  Any
change means burning a new DVD, but this also makes testing easier and less
noisy.  Logs are tricky to extract. 
 
While it has worked well, some of us believe its usability drawbacks (e.g.
limited ability to receive input from users, every change needs a new DVD)
outweigh its great security benefits making it unmarketable (in govt or
industry) and thus just another prototype to leave on the shelf.
 
I'm curious what your group thinks.  Thanks in advance -- I don't quite know
with whom to discuss this idea.
 
Kevin Sweere
 
 
 
 


smime.p7s
Description: S/MIME cryptographic signature

Re: balancer-manager and server-status feature request.

[Jim Jagielski]

On Nov 16, 2009, at 5:52 AM, Mladen Turk wrote:
> Regarding xml data, it is my long standing wish to create
> log output filter sub module system where the log lines would
> go trough a VFS filter capable of writing to xml, database, etc
> (depending on the VFS implementation).
> 


*grin*

I'd been mulling over the same idea... a data-set filter that
can take data and output it as XML, HTML, etc... 

Re: A fundamentally secure Apache server, any interest?

[Junyong Jiang]

I support you!

2009/11/16 Sweere, Kevin E CTR USAF AFRL/RYT 

> Greetings,
>
> I work for the US Air Force.  We have a prototype that dramatically,
> fundamentally increases a web server's security.
>
> We run an Apache server within a minimized, user-level-only, Linux variant
> only within RAM and from only a DVD (no harddrive).  With no shells,
> hackers
> have nowhere to go.  With no persistent memory, malware has no place to
> reside.  A simple reboot restores the website to a pristine state within
> minutes.
>
> Because a LiveDVD holds the OS, apps and content, its best for static,
> non-interactive, low-volume, high-value, highly-targeted websites.  Any
> change means burning a new DVD, but this also makes testing easier and less
> noisy.  Logs are tricky to extract.
>
> While it has worked well, some of us believe its usability drawbacks (e.g.
> limited ability to receive input from users, every change needs a new DVD)
> outweigh its great security benefits making it unmarketable (in govt or
> industry) and thus just another prototype to leave on the shelf.
>
> I'm curious what your group thinks.  Thanks in advance -- I don't quite
> know
> with whom to discuss this idea.
>
> Kevin Sweere
>
>
>
>
>

Re: A fundamentally secure Apache server, any interest?

[Mark Watts]

On Mon, 2009-11-16 at 08:42 -0500, Sweere, Kevin E CTR USAF AFRL/RYT
wrote:
> Greetings,
>  
> I work for the US Air Force.  We have a prototype that dramatically,
> fundamentally increases a web server's security.  
>  
> We run an Apache server within a minimized, user-level-only, Linux variant
> only within RAM and from only a DVD (no harddrive).  With no shells, hackers
> have nowhere to go.  With no persistent memory, malware has no place to
> reside.  A simple reboot restores the website to a pristine state within
> minutes.  
>  
> Because a LiveDVD holds the OS, apps and content, its best for static,
> non-interactive, low-volume, high-value, highly-targeted websites.  Any
> change means burning a new DVD, but this also makes testing easier and less
> noisy.  Logs are tricky to extract. 
>  
> While it has worked well, some of us believe its usability drawbacks (e.g.
> limited ability to receive input from users, every change needs a new DVD)
> outweigh its great security benefits making it unmarketable (in govt or
> industry) and thus just another prototype to leave on the shelf.
>  
> I'm curious what your group thinks.  Thanks in advance -- I don't quite know
> with whom to discuss this idea.
>  
> Kevin Sweere

Hi Kevin,

The idea of a CD/DVD-ROM based webserver isn't new, I know we did some
internal research into it many years ago and came to the same
conclusions you have - the level of security offered seriously impedes
your ability to use/manage the server.

You also run into problems if your servers don't actually have an
optical drive (eg: Blades).

If I was looking for that level of assurance that my data hasn't been
tampered with, I'd be looking at using a mechanism of snapshoting your
webserver in some way such that a rollback is trivial. Linux LVM,
Solaris ZFS or even VMWare all offer this kind of snapshot and rollback.
I'd also be using TripWire or something similar to verify my content
directories.

Apache configured with minimum modules to simply serve static ASCII and
image files is about as secure at it gets for that type of content.
SELinux stops a rogue CGI from reading /etc/shadow, and mod_security
helps to block a lot of crud from ever generating a response from the
server.


Read-Only web servers are certainly secure but by their nature, very
time-consuming to manage.


Mark.

-- 
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, Managed Services Manpower
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg


signature.asc
Description: This is a digitally signed message part

Re: one remaining mpms-shared quirk

[Jeff Trawick]

On Sun, Nov 15, 2009 at 5:09 PM, Jeff Trawick  wrote:
> On Sat, Nov 14, 2009 at 8:10 PM, William A. Rowe Jr.
>  wrote:
>> ./configure with both --with-mpm=worker --enable-mpms-shared provides a 
>> really
>> odd result;
>>
>> checking which MPM to use by default... worker
>> ../httpd-2.x/configure: line 25999: server/mpm/event/modules.mk: No such 
>> file or directory
>> checking for pthread_kill... yes
>> ../httpd-2.x/configure: line 26148: server/mpm/prefork/modules.mk: No such 
>> file or directory
>> ../httpd-2.x/configure: line 26197: server/mpm/simple/modules.mk: No such 
>> file or directory
>> ../httpd-2.x/configure: line 26293: server/mpm/worker/modules.mk: No such 
>> file or directory
>
> lack of vpath support is the problem; I need to mimic or share the
> builddir creation logic that exists for regular modules in order to
> create server/mpm/FOO before trying to create modules.mk there
>

r880775 :)

Re: mod_fcgid: different instances of the same program

[Felipe Alcacibar]

> On Mon, Nov 9, 2009 at 5:16 PM, Danny Sadinoff
>  wrote:
> 2) Virtual hosts
> The above item holds true even across virtual hosts.   So while
> it's possible to adjust the FcgidInitialEnv items on a per-vhost
> basis, this is a recipe for disaster if two vhosts point at the same
> fcgi executable, because the resulting processes with potentially
> different Environments will be inserted into the same pool.  Once that
> occurs, we may expect that a server spawned with config defined in
> vhost A will be parcelled out to vhost B.


The first time that i use mod_fcgid, i probe SetEnv instead
FcgidInitialEnv because i not readed complete the documentation, but
SetEnv and FcgidInitialEnv causes the same effect, the difference is
where you put the SetEnv. The FastCGI especification only needs the
environment variables that uses (FCGI_SOCKET, FCGI_USER, ALLOWED_ENV,
etc.) you could use SetEnv if need more flexibility than
FcgidInitialEnv.

-- 
Felipe Alcacibar Buccioni

Re: Server Gated Certs (Was: TLS renegotiation attack, mod_ssl and OpenSSL)

[Joe Orton]

On Fri, Nov 06, 2009 at 02:00:47AM +, Dirk-Willem van Gulik wrote:
> What we really need is 1) a pub/priv key pair of such a cert* (or use  
> attached CSR) of some random domain (ideally expired and with a totally  
> bogus CN valye so we can post the private key publicly) and 2) obviously  
> a browser which support this (but that we can handle).

Rick got me an SGC-enabled test cert (thanks a lot!) - I've installed it 
on box which can be accessed e.g. here:

  https://dougal.manyfish.co.uk/cgi-bin/printenv

with SSLCipherSuite tweaked to enable EXPORT ciphers; it now reads:

  SSLCipherSuite ALL:!ADH:EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

The box is running the RHEL 2.2.3 with the CVE-2009-3555 patch applied, 
so should reject any client-initiated renegotiations.  Note that the 
cert has expired already (intentionally), but is otherwise valid.

I've been trying to find a real browser to do SGC against this but have 
failed - help welcome here!  I've tried old releases of Netscape 4.0x 
but they predate the Verisign root from which the cert was issued, so, 
prerequisite "enable SGC" trust bit in the root CA bundle isn't there.

It seems like the best bet to get a working SGC-enabled browser might be 
Windows 2K or similar vintage with an old "export" (non-US) version of 
MSIE (4/5?).  Can anybody dig out such a best and try loading the above 
page?  

You'd need to verify it was an export version by loading some other SSL 
site and checking the cipher used, and/or verifying that SGC works 
against one of the sites mentioned ealier:

>https://www.chase.com
>https://www.wellsfargo.com

Regards, Joe

Re: A fundamentally secure Apache server, any interest?

[Sander Temme]

Hi Kevin, 

Definitely not the right list: this is where we discuss development of the 
Apache HTTP Server code.  us...@httpd.apache.org may be a better forum within 
apache.org.  Outside Apache, several initiatives exist to look into hardening 
web servers.  The Center for Internet Security  is 
one of them.  

On Nov 16, 2009, at 8:42 AM, Sweere, Kevin E CTR USAF AFRL/RYT wrote:

> I work for the US Air Force.  We have a prototype that dramatically,
> fundamentally increases a web server's security.  
>  
> We run an Apache server within a minimized, user-level-only, Linux variant
> only within RAM and from only a DVD (no harddrive).  With no shells, hackers
> have nowhere to go.  With no persistent memory, malware has no place to
> reside.  A simple reboot restores the website to a pristine state within
> minutes.  

I agree.  Putting the entire OS and content on a read-only device (whether DVD 
or otherwise) significantly reduces your exposure to attacks for all these 
reasons.  The OS will need *some* writable space (like /tmp and /var/run), but 
I assume you made like Knoppix and Ubuntu Live and their ilk, and use RAM disks 
for that.  

> Because a LiveDVD holds the OS, apps and content, its best for static,
> non-interactive, low-volume, high-value, highly-targeted websites.  Any
> change means burning a new DVD, but this also makes testing easier and less
> noisy.  Logs are tricky to extract. 

You could write logs to a RAM disk, with obvious implications on retention.  Or 
you could spool them to another server either through a network mount or 
mod_log_spread.  The httpd configuration language allows you to put log files 
in any place you like, and there are several approaches to rotating log files 
if space is an issue.  Or you can use a third party module to write logs like 
the aforementioned mod_log_spread, which is not part of httpd itself.  

> While it has worked well, some of us believe its usability drawbacks (e.g.
> limited ability to receive input from users, every change needs a new DVD)
> outweigh its great security benefits making it unmarketable (in govt or
> industry) and thus just another prototype to leave on the shelf.

You are in for a perpetual war between Operations (whose pager goes off when 
things break) and dev (whose time-to-market is implicated by the fixed 
environment).  You could mitigate that problem by reading site content from a 
remote machine, either continuously over a network mount or by copying it into 
a RAM disk on boot.  The former might be slower, but would allow for more 
frequent site updates.  It's a trade-off, as usual.  

Keeping the remote mount read-only (even for root) will allow you achieve your 
goal of a read-only environment.  

More comprehensive upgrades that would involve adding modules or changing 
configuration parameters should trigger a change management process that would 
lead to an update of the boot image.  

> I'm curious what your group thinks.  Thanks in advance -- I don't quite know
> with whom to discuss this idea.

As Mark points out, this would be very secure but very hard to manage, and my 
impression is that time-to-market pressure and available expertise frequently 
cause ideas like this to fall by the wayside.  

Fundamentally, booting web heads from a read-only medium like an optical drive 
or PXE is a sound idea.  Any initiative, installation method or distribution 
that makes this easier to manage might increase adoption.  

S. I'd base it on BSD though

-- 
Sander Temme
scte...@apache.org
PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF





smime.p7s
Description: S/MIME cryptographic signature

Re: A fundamentally secure Apache server, any interest?

[Jorge Schrauwen]

On Mon, Nov 16, 2009 at 5:11 PM, Sander Temme  wrote:
> Hi Kevin,
>
> Definitely not the right list: this is where we discuss development of the 
> Apache HTTP Server code.  us...@httpd.apache.org may be a better forum within 
> apache.org.  Outside Apache, several initiatives exist to look into hardening 
> web servers.  The Center for Internet Security  
> is one of them.
>
> On Nov 16, 2009, at 8:42 AM, Sweere, Kevin E CTR USAF AFRL/RYT wrote:
>
>> I work for the US Air Force.  We have a prototype that dramatically,
>> fundamentally increases a web server's security.
>>
>> We run an Apache server within a minimized, user-level-only, Linux variant
>> only within RAM and from only a DVD (no harddrive).  With no shells, hackers
>> have nowhere to go.  With no persistent memory, malware has no place to
>> reside.  A simple reboot restores the website to a pristine state within
>> minutes.
>
> I agree.  Putting the entire OS and content on a read-only device (whether 
> DVD or otherwise) significantly reduces your exposure to attacks for all 
> these reasons.  The OS will need *some* writable space (like /tmp and 
> /var/run), but I assume you made like Knoppix and Ubuntu Live and their ilk, 
> and use RAM disks for that.
>
>> Because a LiveDVD holds the OS, apps and content, its best for static,
>> non-interactive, low-volume, high-value, highly-targeted websites.  Any
>> change means burning a new DVD, but this also makes testing easier and less
>> noisy.  Logs are tricky to extract.
>
> You could write logs to a RAM disk, with obvious implications on retention.  
> Or you could spool them to another server either through a network mount or 
> mod_log_spread.  The httpd configuration language allows you to put log files 
> in any place you like, and there are several approaches to rotating log files 
> if space is an issue.  Or you can use a third party module to write logs like 
> the aforementioned mod_log_spread, which is not part of httpd itself.
>
>> While it has worked well, some of us believe its usability drawbacks (e.g.
>> limited ability to receive input from users, every change needs a new DVD)
>> outweigh its great security benefits making it unmarketable (in govt or
>> industry) and thus just another prototype to leave on the shelf.
>
> You are in for a perpetual war between Operations (whose pager goes off when 
> things break) and dev (whose time-to-market is implicated by the fixed 
> environment).  You could mitigate that problem by reading site content from a 
> remote machine, either continuously over a network mount or by copying it 
> into a RAM disk on boot.  The former might be slower, but would allow for 
> more frequent site updates.  It's a trade-off, as usual.
>
> Keeping the remote mount read-only (even for root) will allow you achieve 
> your goal of a read-only environment.
>
> More comprehensive upgrades that would involve adding modules or changing 
> configuration parameters should trigger a change management process that 
> would lead to an update of the boot image.
>

You could just mount everything ro from a remote host then, even the
config for example, could be mounted from a remote host.

I guess a rw usb key with OS + config would work too if you can
somehow force it ro only when booting from it.

>> I'm curious what your group thinks.  Thanks in advance -- I don't quite know
>> with whom to discuss this idea.
>
> As Mark points out, this would be very secure but very hard to manage, and my 
> impression is that time-to-market pressure and available expertise frequently 
> cause ideas like this to fall by the wayside.
>
> Fundamentally, booting web heads from a read-only medium like an optical 
> drive or PXE is a sound idea.  Any initiative, installation method or 
> distribution that makes this easier to manage might increase adoption.
>
> S. I'd base it on BSD though
>
> --
> Sander Temme
> scte...@apache.org
> PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF
>
>
>
>

Thats all that comes to mind atm.

Jorge

I'm stuck with an OS X module problem and -mmacosx-version-min=10.5

[Patrick McManus]

Hi All,

Here's my problem - I have an apache module that has been happily
running on Mac Leopard for quite a while. I installed snow leopard recently.
If I build it on the SL host, it works fine there too.

Of course, the SL binary will not run on 10.5 - I get linker errors when
apache loads it. That's expected (ok, I forgot it, but its normal) - and
then I rebuilt the module with -mmacosx-version-min=10.5.

The linker errors resolved themselves. Hurrah.

However now the module is broken for me. The handler will run and
the right content-body will flow out, but content-type is consistently
text/plain even though ap_set_content_type(r, "text/html") has been
called.

If I remove the version-min-flag the correct content-type comes out (but
of course the binary is not portable anymore).

I tracked it down further to some interactions with filters and the
brigade code.

I have a simple example below which sets up a brigade directly out of
the handler, but it doesn't even do anything with it. (the real code
inserts it as a filter and deals with data streamed in - this change is
made to create a minimal example - the symptoms are identical). If this
brigade could is called and version-min is set - then content-type is
not honored. If EITHER the brigade calls or the version-min is omitted,
then content-type works fine. You'll see nothing is even done with the
brigade, and the HTTP request is a GET anyhow.


#include 
#include 

#include "httpd.h"
#include "http_config.h"
#include "http_core.h"
#include "http_protocol.h"
#include "http_request.h"
#include "http_connection.h"
#include "http_log.h"

#include "apr_strings.h"
#include "apr_lib.h"/* for apr_isspace */
#include "apr_base64.h" /* for apr_base64_decode et al */
#include "apr_file_io.h"


#define MGI_TYPE   "application/mgi"
#define MGI_SCRIPT "mgi-script"


static int mgi_handler(request_rec *r)
{
  if (r->handler == NULL)
return DECLINED;
  
  if (strcmp(r->handler, MGI_TYPE) && strcmp(r->handler, MGI_SCRIPT))
return DECLINED;

  {
  apr_status_t rv;
  apr_bucket_brigade *bb;
  
  bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
  rv = ap_get_brigade(r->input_filters, bb, AP_MODE_READBYTES,
  APR_BLOCK_READ, HUGE_STRING_LEN);
  }
  
ap_set_content_type (r,"text/fake");
ap_rwrite ("foobit", 6, r);

return OK;
}

static void mgi_register_hooks(apr_pool_t *p)
{
ap_hook_handler (mgi_handler, NULL, NULL, APR_HOOK_MIDDLE);
return;
}

module AP_MODULE_DECLARE_DATA mgi_module =
{
STANDARD20_MODULE_STUFF,
NULL,/* per-directory config creator */
NULL, /* dir config merger */
NULL,
NULL,  /* server config merger */
NULL, // todo mgi_cmds, /* command table */
mgi_register_hooks,   /* set up other request processing hooks
*/
};

Compiled like so (on 10.6  xcode 3.2.1 (and 3.2.0 - same result) using
the apache 2.2.13 distributed on SL)

gcc -Wall -Wno-multichar -fPIC -g -arch x86_64 -arch i386 -arch ppc 
-mmacosx-version-min=10.5 -I /usr/include/apr-1.0/ -I /usr/include/apr-1/ -I 
`/usr/sbin/apxs -q INCLUDEDIR` -DAPACHE_MODULE -c -o mod_mgi.o mod_mgi.c
gcc -Wall -Wno-multichar -fPIC -g -arch x86_64 -arch i386 -arch ppc 
-mmacosx-version-min=10.5 -I /usr/include/apr-1.0/ -I /usr/include/apr-1/ -I 
`/usr/sbin/apxs -q INCLUDEDIR` -DAPACHE_MODULE -fPIC -shared -flat_namespace 
-undefined warning -o mod_mgi.so mod_mgi.o


Like I said, either removing that brigade code that doesn't do anything or
removing the compiler flag gets the content-type text/fake that you
would expect.. but this example has been boiled down, the real module of
course needs a useful input filter and its really strange to expect
different output results based on that flag anyhow.

Is there potentially a problem because core apache isn't compiled with
that flag? And if that's the case, how could a 3rd party construct a
module that can run on either revision of the OS? Do I need multiple
binaries that are sorted out by the installer? Yech, hope that isn't the
case.

Thanks again for any help you can muster. I appreciate it.

-Patrick

Re: TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Jean-Marc Desperrier]

Jean-Marc Desperrier wrote:

Everyone who uses client certificate authentication knows that they are
many apache configurations around that will force the user to repeatedly
reauthenticate himself for apparently no good reason.

It's hard to believe the explanation is only that all of the concerned
sites forgot to activate the "session resume" option.
SSLVerifyClient and SSLSessionCacheTimeout forcing unnecessary
renegotiation did seem like a very plausible alternative explanation.
[...]

I'll try to find out more about this, with so many users reporting that
problem, there should be a way to get some more detailed info about what
causes it, if it's related with erroneous renegotiation or not.


Ok, so in fact I have one apache instance available locally with a 
problem of this kind. It's configured to not require client 
authentication by defaut, but to require it on the /authentication url


So what happens truly is that after firefox does a request to 
/authentication, mod_ssl does renegociation 6 times in a row for 
apparently no reason.


An interesting point is that firefox is *not* reusing the ssl session in 
that case, for some reason it sends a SessionID of 0 after the "Hello 
Request" from the server. I'll forward that to the NSS team, because if 
that behavior continues, it will be no use to implement the IETF draft 
for secure renegociation, it will fail all the same.


In that specific case, there's not much useful to learn, we get several 
re-authentication where only one was needed, but after that it's OK, no 
more needless renegociation requests. But I hear that in some 
configuration there's a reauthentication everytime the user requires a 
resource. I'll check if I can find more, if it's not just a case of 
SSLSessionCache not being enabled.


Here's the wireshark captured exchange between the client and server, 
note that "Hello Request" always *immediatly* follows the end of the 
renegotiation. This is with Apache 2.2.11/Openssl 0.9.8i (not a 
production server) :

217 19:30:50.745606 client_ip   server_ip   HTTPGET 
/authentication/ HTTP/1.1
218 19:30:50.747473 server_ip   client_ip   TLSv1   Hello Request
219 19:30:50.747896 client_ip   server_ip   TLSv1   Client Hello
220 19:30:50.749114 server_ip   client_ip   TLSv1   Server Hello, 
Certificate, Certificate Request, Server Hello Done
257 19:30:59.267340 client_ip   server_ip   TLSv1   Certificate, 
Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
259 19:30:59.288262 server_ip   client_ip   TLSv1   Change Cipher 
Spec, Finished
260 19:30:59.289066 server_ip   client_ip   TLSv1   Hello Request
262 19:30:59.289511 client_ip   server_ip   TLSv1   Client Hello
263 19:30:59.290741 server_ip   client_ip   TLSv1   Server Hello, 
Certificate, Certificate Request, Server Hello Done
282 19:31:07.747697 client_ip   server_ip   TLSv1   Certificate, 
Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
284 19:31:07.768765 server_ip   client_ip   TLSv1   Change Cipher 
Spec, Finished
285 19:31:07.769681 server_ip   client_ip   TLSv1   Hello Request
287 19:31:07.770128 client_ip   server_ip   TLSv1   Client Hello
288 19:31:07.771261 server_ip   client_ip   TLSv1   Server Hello, 
Certificate, Certificate Request, Server Hello Done
309 19:31:12.237699 client_ip   server_ip   TLSv1   Certificate, 
Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
311 19:31:12.258634 server_ip   client_ip   TLSv1   Change Cipher 
Spec, Finished
312 19:31:12.259583 server_ip   client_ip   TLSv1   Hello Request
314 19:31:12.260044 client_ip   server_ip   TLSv1   Client Hello
315 19:31:12.261187 server_ip   client_ip   TLSv1   Server Hello, 
Certificate, Certificate Request, Server Hello Done
441 19:31:17.613690 client_ip   server_ip   TLSv1   Certificate, 
Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
443 19:31:17.634661 server_ip   client_ip   TLSv1   Change Cipher 
Spec, Finished
444 19:31:17.635574 server_ip   client_ip   TLSv1   Hello Request
446 19:31:17.636005 client_ip   server_ip   TLSv1   Client Hello
447 19:31:17.638787 server_ip   client_ip   TLSv1   Server Hello, 
Certificate, Certificate Request, Server Hello Done
467 19:31:21.725473 client_ip   server_ip   TLSv1   Certificate, 
Client Key Exchange, Certificate Verify, Change Cipher Spec, Finished
469 19:31:21.746555 server_ip   client_ip   TLSv1   Change Cipher 
Spec, Finished
470 19:31:21.747127 server_ip   client_ip   TLSv1   Hello Request
472 19:31:21.747552 client_ip   server_ip   TLSv1   Client Hello
473 19:31:21.748911 server_ip   client_ip   TLSv1   Server Hello, 
Certifica

intend to roll next alpha on 24th

[Paul Querna]

I'll try to do another 2.3.x on next Tuesday, the 24th.

We can vote on it over thanksgiving :-)

Thanks,

Paul

Re: TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Torsten Foertsch]

On Mon 16 Nov 2009, Jean-Marc Desperrier wrote:
> Here's the wireshark captured exchange between the client and server,
> note that "Hello Request" always *immediatly* follows the end of the
> renegotiation. This is with Apache 2.2.11/Openssl 0.9.8i (not a
>
> production server) :
> > 217   19:30:50.745606 client_ip   server_ip   HTTPGET
> > /authentication/ HTTP/1.1
> > 218   19:30:50.747473 server_ip   client_ip   TLSv1   Hello
> > Request
> > 219   19:30:50.747896 client_ip   server_ip   TLSv1   Clien
> >t Hello
> > 220   19:30:50.749114 server_ip   client_ip   TLSv1   Serve
> >r Hello, Certificate, Certificate Request, Server Hello Done
> > 257   19:30:59.267340 client_ip   server_ip   TLSv1   Certi
> >ficate, Client Key Exchange, Certificate Verify, Change Cipher Spec,
> > Finished
> > 259   19:30:59.288262 server_ip   client_ip   TLSv1   Chang
> >e Cipher Spec, Finished
> > 260   19:30:59.289066 server_ip   client_ip   TLSv1   Hello
> > Request
> > 262   19:30:59.289511 client_ip   server_ip   TLSv1   Clien
> >t Hello
...
> > 510   19:31:37.260057 server_ip   client_ip   HTTPHTTP/
> >1.1 200 OK  (text/html)

I have noticed something similar. Don't know if it applies to you. If 
your /authentication/ is a resource that generates a directory listing 
via mod_autoindex then apache issues a subrequest for each directory 
entry. Now, if only /authentication/ requires a client certificate but 
the VHost or base server does not then each entry leads to a 
renegotiation. Correct me if I am wrong but that is how I have 
explained the behavior for me.

Torsten

-- 
Need professional mod_perl support?
Just hire me: torsten.foert...@gmx.net

Re: TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Joe Orton]

On Mon, Nov 16, 2009 at 08:21:20PM +0100, Jean-Marc Desperrier wrote:
> Ok, so in fact I have one apache instance available locally with a  
> problem of this kind. It's configured to not require client  
> authentication by defaut, but to require it on the /authentication url
>
> So what happens truly is that after firefox does a request to  
> /authentication, mod_ssl does renegociation 6 times in a row for  
> apparently no reason.

Please file a bug and attach all of:

a) error_log output at "LogLevel debug" for that case 
b) the config snipping that you're using for /authentication
c) the mod_ssl configuration

Regards, Joe

handling request splicing in case of server initiated renegotiation CVE-2009-3555

[Hartmut Keil]

Hi everybody


for clarification of https://issues.apache.org/bugzilla/show_bug.cgi?id=48204
a more detailed explanation of the described attack scenario is given here.

With the patch CVE-2009-3555-2.2.patch client initiated renegotiation has been 
disabled,
as a consequence of CVE-2009-3555. But a MITM attacker can also use the server 
initiated
renegotiation for exceting an arbitrary request with the Cookie of a victim.
(see http://extendedsubset.com/Renegotiating_TLS.pdf )

The following is the output of ssldump between a MITM attacker and the server. 
Remarks from my side
are starting with '###', longish message are replace by ''.



New TCP connection #1: adnws121.zh.adnovum.ch(44889) <-> 
adnpool01.zh.adnovum.ch(44300)
1 1  0.4423 (0.4423)  C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  
1 2  0.4447 (0.0024)  S>C  Handshake
  ServerHello
Version 3.1
  
1 3  0.4448 (0.)  S>C  Handshake
  Certificate
1 4  0.4448 (0.)  S>C  Handshake
  ServerHelloDone
1 5  0.4546 (0.0098)  C>S  Handshake
  ClientKeyExchange
1 6  0.4595 (0.0049)  C>S  ChangeCipherSpec
1 7  0.5504 (0.0908)  C>S  Handshake
  Finished
1 8  0.5513 (0.0009)  S>C  ChangeCipherSpec
1 9  0.5513 (0.)  S>C  Handshake
  Finished

### up to here the MITM has established a normale handshake with the server

1 10 0.5521 (0.0008)  C>S  application_data
---
GET /cert/hacked.html HTTP/1.1
Host: adnpool01.zh.adnovum.ch

GET /cert/injected.html HTTP/1.1
Host: adnpool01.zh.adnovum.ch
X-Ignore: ---

### the MITM is sending two request in one chunk application_data: First a 
complete
request, that is causes the server to initiate a renegotiation. And second 
request, that
is incomplete, since the last header 'X-Ignore:' is not terminated by CRLF.
At this point the server has consumend and decrypted both requests, because 
the request line
will be readed with buffer size of 2*8190 (the default of the configuration 
paramter LimitRequestLine)

 rv = ap_rgetline(&(r->the_request), (apr_size_t)(r->server->limit_req_line 
+ 2),
 &len, r, 0, bb);

 And in ssl_io_input_getline(..) all decrypted data will be written to a 
buffer:

 static apr_status_t ssl_io_input_getline(bio_filter_in_ctx_t *inctx,
 char *buf,
 apr_size_t *len)
   {


if (pos) {
char *value;
int length;
apr_size_t bytes = pos - buf;

bytes += 1;
value = buf + bytes;
length = *len - bytes;

char_buffer_write(&inctx->cbuf, value, length);

*len = bytes;
}

return APR_SUCCESS;
   }

   And any subsequent call of ssl_io_input_read(..) will first consume the data 
in the buffer!

1 11 0.5584 (0.0062)  S>C  Handshake
  HelloRequest

### the server is initiating a renegation, due to 'SSLVerifyClient require' 
configuration
for the location /cert/*
From now on the MITM is just downstreaming the decrypted data to the 
victim. And upstreaming
the encrypted data to the server, until he observes the change_cipher_spec 
messages.

1 12 0.6504 (0.0919)  C>S  Handshake
  ClientHello
Version 3.1
cipher suites

1 13 0.6530 (0.0025)  S>C  Handshake
  ServerHello
Version 3.1
   
1 14 0.6530 (0.)  S>C  Handshake
  Certificate
1 15 0.6530 (0.)  S>C  Handshake
  CertificateRequest
  
  ServerHelloDone
1 16 4.5204 (3.8674)  C>S  Handshake
  Certificate
1 17 4.5204 (0.)  C>S  Handshake
  ClientKeyExchange
1 18 4.5204 (0.)  C>S  Handshake
  CertificateVerify
  
1 19 4.5204 (0.)  C>S  ChangeCipherSpec
1 20 4.5204 (0.)  C>S  Handshake
  Finished
1 21 4.5589 (0.0384)  S>C  ChangeCipherSpec
1 22 4.5589 (0.)  S>C  Handshake
  Finished
1 23 4.5619 (0.0030)  S>C  application_data
---
HTTP/1.1 200 OK
Date: Mon, 16 Nov 2009 19:53:25 GMT
Server: Apache
...

---
1 24 4.5619 (0.)  S>C  application_data
---

---
1 25 4.6604 (0.0984)  C>S  application_data
---
GET /cert/index.html HTTP/1.1
Cookie: JSESSIONID=bhe8r67be3wy943hf73


### After recieving the request of the victim the server can complete the 
incomplete second request of MITM.
The server is reading the following request header:

GET /cert/injected.html HTTP/1.1
X-Ignore: GET /cert/index.html HTTP/1.1
Cookie: JSESSIONID=

Re: mod_fcgid: different instances of the same program

[Rainer Jung]

On 16.11.2009 13:14, Jeff Trawick wrote:
> On Mon, Nov 16, 2009 at 5:03 AM, Danny Sadinoff  wrote:
>> On Tue, Nov 10, 2009 at 1:47 AM, Danny Sadinoff  wrote:
>>>
>>> On Tue, Nov 10, 2009 at 12:53 AM, Jeff Trawick  wrote:

 On Mon, Nov 9, 2009 at 5:16 PM, Danny Sadinoff
  wrote:
> 2) Virtual hosts
> The above item holds true even across virtual hosts.   So while
> it's possible to adjust the FcgidInitialEnv items on a per-vhost
> basis, this is a recipe for disaster if two vhosts point at the same
> fcgi executable, because the resulting processes with potentially
> different Environments will be inserted into the same pool.  Once that
> occurs, we may expect that a server spawned with config defined in
> vhost A will be parcelled out to vhost B.

 Where does this occur?  Entries in the process table are distinguished
 by virtual host.  (I think the implementation of this check is broken,
 in that it requires that ServerName is set in the virtual hosts.  Are
 you using a simple test config that doesn't have ServerName set?)
>>>
>>> My case is not yet simple.  I'll get back to you.
>>
>> It turns out that the problem was that I was using mod_fcgid-2.2   Upgrading
>> to mod_fcigd-2.3.4 fixed the problem. I apologize for the noise.
> 
> great/np
> 
>> Should this item (vhost independence) be added to the upgrade notes section?
> 
> Unfortunately, I'm still in the dark about vhost independence and how
> exactly the share_grp_id construct in the source code can change the
> process association without more code written to tie it to the
> configuration.  ISTR that Rainer made the same observation --
> share_grp_id isn't completely implemented.  (Maybe it once worked and
> some code was lost?)

Yes, the original mail was:

http://www.mail-archive.com/dev@httpd.apache.org/msg45336.html

Regards,

Rainer

Re: TLS renegotiation disabling : mod_ssl and OpenSSL 0.9.8l

[Rainer Jung]

On 16.11.2009 20:21, Jean-Marc Desperrier wrote:
> Jean-Marc Desperrier wrote:
> An interesting point is that firefox is *not* reusing the ssl session in
> that case, for some reason it sends a SessionID of 0 after the "Hello
> Request" from the server. I'll forward that to the NSS team, because if
> that behavior continues, it will be no use to implement the IETF draft
> for secure renegociation, it will fail all the same.

Only concerning empty session id: during testing the renegotiation fix I
was also looking at session IDs and got aware, that the TLS session
ticket extension will move the session status to the client. In this
case the requests belonging to the initial connection to the server will
not carry a session id. Later connections from the same client will
provide some client generated id to the server that the server replaces
it by a new one. The session ticket extension allows the server to not
maintain the session data in its own cache. In the presence of the
session ticket extension, session IDs observed on the server are no
longer a good measurement for session reuse.

In Firefox's about:config you can filter fot "tls" and switch session
tickets off.

Regards,

Rainer

Re: svn commit: r880981 - /httpd/httpd/trunk/build/instdso.sh

[Jeff Trawick]

On Mon, Nov 16, 2009 at 4:51 PM,   wrote:
> Author: trawick
> Date: Mon Nov 16 21:51:01 2009
> New Revision: 880981
>
> URL: http://svn.apache.org/viewvc?rev=880981&view=rev
> Log:
> tweak r823613/PR 47951 change to avoid /usr/sbin/install on
> Solaris
>
> (not compatible with BSD install)
>
> Modified:
>    httpd/httpd/trunk/build/instdso.sh
>
> Modified: httpd/httpd/trunk/build/instdso.sh
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/build/instdso.sh?rev=880981&r1=880980&r2=880981&view=diff
> ==
> --- httpd/httpd/trunk/build/instdso.sh (original)
> +++ httpd/httpd/trunk/build/instdso.sh Mon Nov 16 21:51:01 2009
> @@ -49,7 +49,15 @@
>     $CMD || exit $?
>  fi
>
> -type install >/dev/null 2>&1 && INSTALL_CMD=install || INSTALL_CMD=cp
> +case $SYS in
> +    SunOS)
> +        INSTALL_CMD=cp
> +        ;;
> +    *)
> +        type install >/dev/null 2>&1 && INSTALL_CMD=install || INSTALL_CMD=cp

Perhaps this is just a temporary change, but from the
/usr/sbin/install permission denied messages I saw I'd hate for anyone
to run "make install" or apxs with superuser privileges.

The original complaint in the PR was that using cp to install DSOs
over a running server caused the server to segfault.  If indeed
replacing shared libraries underneath a running server is something we
want to acknowledge, we could rm it in instdso.sh before running
libtool ... cp foo.la libexecdir.  We already do that on AIX for other
reasons.

Alternatively, we could get the results of AC_PROG_INSTALL into instdso.sh.

Re: handling request splicing in case of server initiated renegotiation CVE-2009-3555

[Joe Orton]

On Mon, Nov 16, 2009 at 09:59:12PM +0100, Hartmut Keil wrote:
> With the change described in 
> https://issues.apache.org/bugzilla/show_bug.cgi?id=48204
> the buffer used in ssl_io_input_read(..) will be reset, and so the second 
> request of
> the MITM will be dropped.
> The first request will be executed, but since there is no Cookie from the 
> victim is the
> less dangerous one.

Thanks for posting.  So you are proposing that mod_ssl discards any 
decoded app-data message (i.e. HTTP traffic) *subsequent* to the data 
making up the request which triggered the renegotiation, but before the 
renegotiation occurs.

This change would prevent a variant of the renegotiation prefix attack 
against a site which uses both client cert auth in a per-dir/location 
context, *and* HTTP-level authentication.  It makes no difference to a 
vulnerable site relying only on client cert auth in per-dir/loc context; 
this would still be vulnerable.

This would break HTTP pipelining over SSL (for affected configurations), 
and it might not fail gracefully - the server would appear to simply 
never receive the pipelined requests.  I'm relucant to do that.

A similar solution which detected pending buffered bytes after 
completing the handshake (SSL_pending(ssl) does this), and giving a hard 
failure (TLS level or HTTP level) might be better, but I haven't thought 
that through in detail.

Regards, Joe