Re: Dispatching MPM

[Hideki Noma]

Did you check Metux MPM?
It works by passing socket discriptor to worker process.

The project is on beta status but works fine.
Please see the following pages.
http://www.metux.de/mpm/
http://www.sannes.org/metuxmpm/

If you are interested in extending this MPM, join in the 
project. I'm sure some of the maintainers are still watching the
move.

Hideki Noma

> Hi,
> 
> I'm going to make a new MPM (I called it Dispatching MPM, or D-MPM) that
> would do what perchild is supposed to do.  Here is a rough proposal of 
> the architecture.
> 
> My general idea is shown on the following diagram of example state of
> D-MPM:
> 
>  +---+   +---++---+   +---+
>  |worker1|   |worker2||worker3|   |workerN|
>  |---|   |---||---|   |-- |
>  |user1  |   |user2  ||user1  |  ...  |userX  |
>  |group1 |   |group2 ||group2 |   |groupY |
>  |/home/a|   |/home/b||/home/c|   |/home/Z|
>  +---+   +---++---+   +---+
>  /\  /\  /\   /\  /\
>  ||  ||  ||   || ...  ||
>  ||  \/  \/   \/  \/
>  || ++ ++
>  || |   dispatcher   | | master |
>  || || ||
>  || |   nobody   | |root|
>  || |   nogroup  |  <===>  |root|
>  || |   /var/empty   | |/   |
>  || ++ ++
>  ||   /\
>  ||   ||
>  ||   ||
>  \/   \/
>   +---+   +---+
>   |client1|   |client2|
>   +---+   +---+
> 
> Arrows indicate communication, eg. using sockets. First line of each
> item contains its name, next lines contain user, group, and root
> directory it runs with, respectively.
> 
> System consists of 3 parts:
> 
> Workers--they are basically the same as in other MPMs, the only
> difference is that each has some set of permissions (uid, gid, root
> directory, etc.) of particular group of virtual hosts.
> 
> Dispatcher, accepting incoming connections, reading requests and
> redirecting them to one of appropriate workers (i.e. those having set
> of permissions needed to handle that virtual host).
> 
> Apache master process, listening to requests from dispatcher, and
> creating new workers with specified permissions, according to the
> requests.
> 
> In presented example, client1 has specified Host: header and has been
> redirected to worker1, while client2 hasn't specified it yet, so it is
> still connected to the dispatcher.  The diagram is created with
> assumption that socket passing will be used, not proxying.
> 
> Rationale for the Architecture
> 
>   Most existing solutions (that separate different virtual hosts) have
>   the architecture where many threads/processes (each running with
>   privilege of some virtual host) accept connections, and redirect them
>   if client wants another virtual host. This may create potential
>   security problem--if there were an exploitable bug in code before
>   redirection, an attacker would have a possibility to gain privilege
>   of user of any virtual host. In D-MPM, an attacker would gain only
>   dispatcher's permissions, which can be very restricted.
> 
>   In existing MPMs number of threads for each virtual host is static,
>   set in configuration file. D-MPM approach fixes that problem--new
>   workers are dynamically created by master process on dispatcher's
>   demand.
> 
>   Of course there might be a security hole in master's code. But
>   communication protocol between dispatcher and creator is going to be
>   rather simple: dispatcher may only request that new worker with
>   specified permissions should be created, creator returns socket
>   descriptor connected to a newly created child or error code.
> 
> Extensibility
> 
>   Nowadays permissions of process depend not only on process user and
>   group ID, but on many other settings that vary from system to system.
>   The most popular is process' root directory (changable by chroot(2)),
>   but there are many others (eg. POSIX capabilities, jails, other
>   security settings in RSBAC and SELinux).
> 
>   To allow to utilize any of security extensions particular system has,
>   D-MPM will have a loadable module support. Making D-MPM change some
>   non-standard permissions would involve only writing a small shared
>   library, without need for changing D-MPM's source or even recompiling
>   it.
> 
>   Someone creating a module needs to provide only two functions: first,
>   that reads part of virtual host's configuration, and second, that
>   changes process permissions according to that configuration.
> 
> Implementation issues
> 
>

Re: Initial mod_smtpd code.

[Rian Hunter]

Nifty! I had some compilation problems involving regex, so in the  
attached patch I use ap_regex.h and change some defines. Hope this  
doesn't break anything.


that was a good idea, ap_regex.h was implictly getting included for me.



The other bug I partially fixed was, strstr in smtp_protocol.c only  
does exact matches so uppercase commands like MAIL FROM would fail.  
I added support for the upper case, but this needs to be improved  
still because mixed case doesn't work. Is there an APR function  
like stristr?


agh i forgot about an upper case FROM: or TO:.  
process_smtp_connection_internal() already lowercases the actual  
command, but not the rest of the params. Either MAIL from: or RCPT  
to: would work, but not FROM or TO. Anyway, stristr can be easily  
hacked by calling strstr with two lowercased strings.


The overall structure and the approach you took is very nice, easy  
to understand. I would recommend adding a hook immediately upon the  
client connection, because an external module (maybe for DNSBLs, or  
some rate limiting control) might not even want us to return a  
greeting at all -- i.e. close with "554 Service unavailable" right  
away.


Hmm. That sounds like a good idea, maybe there already is a hook  
defined that could deal with this, I'll look into it.


But I like what you have, would be happy to keep working around  
this design.


Thanks! Do you now have svn access? After I apply your patches I'll  
see about checking this in so more of us can deal with this code. It  
sucks that most are busy with ApacheCon right now.

-rian

Apache in a loop during startup

[Dirk-Willem van Gulik]

On Freebsd 4.10, Apache/2.0.52 - during starttup when there is a zero byte
__db.ssl_cache sitting in /var/run:

drwxr-xr-x   4 root  wheel 1536 Jul 19 07:47 .
drwxr-xr-x  20 root  wheel  512 Sep 17  2004 ..
-rw-r--r--   1 root  wheel0 Jul 19 07:41 __db.ssl_scache
-rw---   1 root  wheel0 Dec 27  2004 accept.lock.51219
...

with config:

SSLSessionCache dbm:/var/run/ssl_scache
SSLSessionCacheTimeout  300
SSLMutexfile:/var/run/ssl_mutex

then apache sits in an endless loop during startup doing:

stat("/var/run/ssl_scache",0xbfbff570)   ERR#2 'No such file or 
directory'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
stat("/var/run/ssl_scache",0xbfbff570)   ERR#2 'No such file or 
directory'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
 repeat...

Before I dive into this - does this ring a bell with anyone ?

Dw

Re: Hackathon Attendees

[Roy T. Fielding]

Yeah, no kidding.  We haven't made any decisions yet.  Decisions
are what happens when people vote on releases or changes.  Did one
of those go by and I didn't notice?  Nope.


Yup - branching 2.1.x - 2.2.x - trunk.


No, branching is part of the task of an RM.  It is not a decision
of the group, nor is it subject to vote, for the same reason we
do not allow people to veto releases.  As long as the RM is doing
it for a good reason (i.e., preparing a tarball, not just dicking
around), then we want them to use the version management tools the
way that they are intended to be used.

Nobody is forced to support the new branch.  Nobody is forced to
vote to release it.  It doesn't exist for the public until we vote
to release it, so just keep working on what you are doing and ignore
the dude behind the curtains.  At least until he asks us to make
a decision.

Roy

Re: NTLM HTTP Authentication is insecure by design - a new writeup by Amit Klein

[Roy T. Fielding]

On Jul 18, 2005, at 12:30 PM, William A. Rowe, Jr. wrote:

NTLM HTTP Authentication
 (and possibly other connection-oriented
 HTTP authentication and authorization protocols)
 is insecure by design


Yep, no shit -- that's what the Microsoft fools were told when
they introduced it.


*) The web server (IIS/6.0) must receive a Via-less request. The
Microsoft implementation assumes that the Via header is always sent
by a proxy server, and this is indeed mandated by the HTTP/1.1 RFC
2616 (http://www.ietf.org/rfc/rfc2616.txt), section 14.45:
 The Via general-header field MUST be used by gateways and proxies
 to indicate the intermediate protocols and recipients between the
 user agent and the server on requests [...]


Yep.


However, it seems that not all servers adhere to this standard. For
example, Apache 2.0.54 mod_proxy does not generate a Via header by
default (see the ProxyVia directive -
http://httpd.apache.org/docs-2.0/mod/mod_proxy.html#proxyvia, yet
the default httpd.conf file contains a commented-out "ProxyVia On"
directive, so it's possible that many Apache proxy deployments do


WTF?  That's a bug.


*) Proxy vendors ­ do not to share TCP connections to the server
among several clients. Yes, it improves performance, but it's also
insecure and enables/aids 3 different attacks (the one described
here, HTTP Request Smuggling and HTTP Response Splitting).
Also, comply to the RFC and send the HTTP Via request header by
default (Apache Group - please take note).


As reverse proxy is never enabled without intent, the impact of
Apache on this vector is very low (note that Amit deliberately
introduced this into his reproduction case) - but anyone who has
intentionally used Apache as a reverse proxy to protect sensitive
IIS servers behind their DMZ using NTLM auth is vulnerable (as are
users of various NTLM Apache auth modules sitting behind Apache
reverse proxies.)


We don't really need to get in between their gun and their foot.


My thinking is that rather than 'revealing' the reverse proxy
origin server, we should be dumping the auth headers if they are
NTLM.


I don't think that will work from the browser's perspective.
I would just mark the connection as being in use by a single
client and remove it when the client closes.

Roy

Re: Apache in a loop during startup

[Joe Orton]

On Tue, Jul 19, 2005 at 01:02:56AM -0700, Dirk-Willem van Gulik wrote:
> 
> with config:
> 
>   SSLSessionCache dbm:/var/run/ssl_scache
>   SSLSessionCacheTimeout  300
>   SSLMutexfile:/var/run/ssl_mutex

Using s/shmcb/dbm/ and "SSLMutex default" is IMO the best default (on 
modern platforms).

> then apache sits in an endless loop during startup doing:
> 
>   stat("/var/run/ssl_scache",0xbfbff570)   ERR#2 'No such file or 
> directory'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>   stat("/var/run/ssl_scache",0xbfbff570)   ERR#2 'No such file or 
> directory'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>   open("/var/run/__db.ssl_scache",0xa02,0644)  ERR#17 'File exists'
>    repeat...
> 
> Before I dive into this - does this ring a bell with anyone ?

No, what's the backtrace when it's doing this?  I don't think mod_ssl 
would loop, nor APR.  What dbm library?

joe

Re: [vote] Revoke R-T-C [was: svn commit: r219520]

[Justin Erenkrantz]

--On July 18, 2005 1:19:54 PM -0500 "William A. Rowe, Jr." 
<[EMAIL PROTECTED]> wrote:



Some of us, trawick, orton and myself come to mind, are still up
for supporting our current users.  As it is, backports aren't
reviewed, or committed once they are (I even split STATUS just to
call out approved-yet-not-backported patches :-).


Well, why haven't you backported them then?  Why are you looking for someone 
else to do the work?



So, I call a vote that we drop R-T-C altogether.  It's pretty clear
to me that those interested in current / near-future / far-future
users are almost three distinct groups.  It will be up to those
small groups to call out and vote on changes within our individual
domains.


-1.  R-T-C is the crux of our stability pact with our users.  -- justin

Re: Dispatching MPM

[Nick Kew]

On Mon, 18 Jul 2005, Michal Kosek wrote:

[ replying as I read - some points here which you address later ]

It would be great to see feedback from the authors of the original
perchild here - I'd guess you'll have insights into the difficulties
Michal is likely to face.

> System consists of 3 parts:
>
> Workers--they are basically the same as in other MPMs, the only
> difference is that each has some set of permissions (uid, gid, root
> directory, etc.) of particular group of virtual hosts.
>
> Dispatcher, accepting incoming connections, reading requests and
> redirecting them to one of appropriate workers (i.e. those having set
> of permissions needed to handle that virtual host).
>
> Apache master process, listening to requests from dispatcher, and
> creating new workers with specified permissions, according to the
> requests.

Nice start.  From this point, you probably want to do the minimum
possible work, and re-use existing code wherever possible.


> In presented example, client1 has specified Host: header and has been
> redirected to worker1, while client2 hasn't specified it yet, so it is
> still connected to the dispatcher.  The diagram is created with
> assumption that socket passing will be used, not proxying.

That is how 1.x and its successors work, isn't it?

> Rationale for the Architecture
>
>   Most existing solutions (that separate different virtual hosts) have
>   the architecture where many threads/processes (each running with
>   privilege of some virtual host) accept connections, and redirect them
>   if client wants another virtual host. This may create potential
>   security problem--if there were an exploitable bug in code before
>   redirection, an attacker would have a possibility to gain privilege
>   of user of any virtual host.

I'm not quite sure what you're saying.  What existing solution do
you have in mind that is open to a transfer-of-privileges attack?
Surely not the standard MPMs and suexec?

>In D-MPM, an attacker would gain only
>   dispatcher's permissions, which can be very restricted.

Agreed.  A minimal dispatcher sounds great.  You should probably
aim to have it require chrooting, as well as running as a user with
no shell or login (perhaps drop those constraints under #ifdef DEBUG).

>   In existing MPMs number of threads for each virtual host is static,
>   set in configuration file. D-MPM approach fixes that problem--new
>   workers are dynamically created by master process on dispatcher's
>   demand.

Are you planning to use reslist to manage your pool of children
per-vhost?

> Extensibility
>
>   Nowadays permissions of process depend not only on process user and
>   group ID, but on many other settings that vary from system to system.
>   The most popular is process' root directory (changable by chroot(2)),
>   but there are many others (eg. POSIX capabilities, jails, other
>   security settings in RSBAC and SELinux).
>
>   To allow to utilize any of security extensions particular system has,
>   D-MPM will have a loadable module support. Making D-MPM change some
>   non-standard permissions would involve only writing a small shared
>   library, without need for changing D-MPM's source or even recompiling
>   it.

Do you mean it will implement a new optional hook specifically for
chroot-like operations?  That sound very nice: maybe just make my
"insist on chroot" suggestion the default if no module does
something else.

>   Someone creating a module needs to provide only two functions: first,
>   that reads part of virtual host's configuration, and second, that
>   changes process permissions according to that configuration.

Right:-)

> Implementation issues
>
>   Dispatcher
>
> It would be best to make dispatcher single-threaded, I suppose that
> creating separate thread for each connection would take much more
> resources.  OTOH it may cause some problems.  One I am currently
> aware of is mod_ssl, which AFAIK doesn't currently have non-blocking
> API.  Would it be difficult to create such API?

That would certainly be fair at this point.  And for a first-pass
implementation you might also want to ignore mod_ssl.

> I want dispatcher to be as simple as possible, and make workers do
> all the work (filters, handlers, etc.).  Of course, as dispatcher
> must read Host header, it must use connection filters.
>
> This headers is needed, because dispatcher has to decide to which
> worker a connection should be redirected.  I think that it would be
> best if it didn't do full header parsing, but just look for virtual
> host name.
>
> There are two ways of dispatching connection:
> - passing a socket descriptor (along with data that were already
>   read) to the appropriate worker,
> - acting as a proxy, redirecting data to the appropriate worker.

The tricky bit seems to me to be the conn_rec.  Of course a connection
_should_ only ever be between a single client and a single vhost, but
if 

Re: [vote] Revoke R-T-C [was: svn commit: r219520]

[Sander Striker]

William A. Rowe, Jr. wrote:

At 12:51 PM 7/18/2005, Roy T. Fielding wrote:



Or you could simply keep working on trunk like everyone else
and let releases be made when a tarball gets three +1s.
Version numbers are cheap.  Telling the entire group to stop while
you work on the next big patch is extremely expensive.



Ok, so we are now three levels deep?

  /trunk

C-T-R

  /branches/2.2.x  [misnomer, we don't have a beta yet]

R-T-C ?  If not now, when?

  /branches/2.0.x

R-T-C


There should be natural migration to 2.2.x.  2.0.x should be
considered closed for new features, that's what the development
line is for.

Triple commits to fix one, one-line segfault?  Well, this isn't 
workable.  I think lack of progress shows it's not workable.


The lack of progress is not due to having to commit to multiple
branches.


Some of us, trawick, orton and myself come to mind, are still up
for supporting our current users.


You make it sound like everybody else is dissing our current users.
This broad characterization is not exactly productive.

As it is, backports aren't 
reviewed, or committed once they are (I even split STATUS just to
call out approved-yet-not-backported patches :-).  


The person who proposed the backport is expected to perform the
backport when it has enough votes.  The person casting the 3rd
vote sometimes does the backport.  And there are cases when a
friendly RM clears the slate when it comes to approved proposed
backports.


Some were working on the stable release of 2.2.x, pquerna comes
firstmost to mind.


It isn't just Paul who wants to see 2.2.x finally materialize.
We've been trying to get 2.2.x out for quite a while.  We've come
very close a couple of times, and because we like consensus we've
not pushed too hard.  I for one don't want to sit here again next
year and still discuss what needs to be fixed/refactored/whatever
before 2.2.x can be released.  Let's just move on.  2.2.x is already
a lot better than 2.0.x; our users deserve a release.


Great progress is afoot, I see that release
going beta very soon, the number of issues has dropped quite
significantly.  [Although we have 29 PatchAvailable issues, not
sure how many correspond to 2.2 commits not backported.]



And others are yet working on 2.4.x.


2.3.x, leading up to 2.4.x.  As of the branch you are one of the
people working on that, what's the issue with that?


So, I call a vote that we drop R-T-C altogether.  It's pretty clear
to me that those interested in current / near-future / far-future
users are almost three distinct groups.  It will be up to those
small groups to call out and vote on changes within our individual
domains.


Define current, near-future, far-future.

current == 1.3?
near-future == 2.0?
far-future == 2.x?

As it stands, only trunk should be C-T-R IMO.  Stability on the
_stable_ branches needs to be ensured.


The question is; would we rather be saturated by commits we feel
need review, or exhausted waiting for commits to be approved?


The latter, but again, it's a broad characterization.


This means code might be committed to 2.0.x and never fixed in head,
it might mean code is fixed in head and never considered for backport
to our current users, but all in all, it beats the situation today.


No it doesn't.  trunk needs to be the branch that has the latest
code, and is most complete.


I'm not suggesting that the 2.0.x users would entertain a break to
ABI compatibility.  But I'm suggesting that parallel development
doesn't work when most folks are focused on tangential development.



Sander

Re: Initial mod_smtpd code.

[Nick Kew]

> Hi Rian,

Useful start: you seem to have dealt with the core SMTP stuff:-)

> I like how the code is done. I am not sure a hook for each smtp command
> is the good solution. Adding a new command here is very simple and quick.

That's my feeling too.  But I'm happy with the way Rian has done it -
and Jem was IIRC in favour of the same approach.

> the problem i found when i did my poc is when there is in the command,
> different destination email. It's difficult here to keep the virtualHost
> scheme.
> It would be nice to keep a conf file like
>
> 
> ServerName mail.bla.com
> SmtpUserMap mail.bla.com-user.map
> or SmtpRelay host
> 
> 

Yes.  I think there is a logical difficulty here.  smtpd_create_request
is run before process_smtp_connection_internal, and the design doesn't
allow for multiple recipients to be processed differently.

My own feeling was that multiple recipients require a request each.
Maybe we could get that in your design by creating a subrequest
for each RCPT TO?

We should also deal with spooling.  Maybe a protocol-level input
filter can do that.  I still like the idea of a spool bucket
(being a long-life file bucket with crash/recovery capabilities
and reference counting), in which case the input filter would
simply convert everything to that.  That would then run before
creating any subrequests.

-- 
Nick Kew

Re: Initial mod_smtpd code.

[Rian Hunter]

On Jul 19, 2005, at 6:51 AM, Nick Kew wrote:
the problem i found when i did my poc is when there is in the  
command,
different destination email. It's difficult here to keep the  
virtualHost

scheme.
It would be nice to keep a conf file like


ServerName mail.bla.com
SmtpUserMap mail.bla.com-user.map
or SmtpRelay host





Yes.  I think there is a logical difficulty here.   
smtpd_create_request

is run before process_smtp_connection_internal, and the design doesn't
allow for multiple recipients to be processed differently.

My own feeling was that multiple recipients require a request each.
Maybe we could get that in your design by creating a subrequest
for each RCPT TO?


This is possible but I'm not sure what the advantage is. Would you  
mind setting up a hypothetical situation where this is necessary? I  
figure that ultimately the handler will be responsible for dealing  
with each rcpt to differently.


Early on I wanted configuration possibility similar to:
#
Listen 21

# mod_smtpd conf
  SmtpProtocol On
SetHandler unix-module

# mod_smtpd_unix
AcceptDomains thelaststop.net www.thelaststop.net
Relay On

# mod_smtpd_easyfilter

# matches against email in MAIL TO: smtp command
RegexMailTo "/thelaststop.net$/"
SetHandler maildir-module

# mod_smtpd_maildir conf
MaildirBase "/usr/local/virtual"
MailboxLimit 50M


# Simple spam filter using mod_smtpd_easyfilter
# Default handler does nothing with mail message

RegexHeader "Subject" "/cialis/"
SetHandler none


##

In the case of this httpd.conf embedded filtering mechanism, I  
figured the handlers could be changed based on the certain RegExs  
right before ap_run_handler() was called. Maybe ap_run_fixups() could  
be called, and my hypothetical mod_smtpd_easyfilter would have a  
fixups hook where it accomplished something similar to this  
situation. Although after thinking about it I realize now that  
mod_smtpd_easyfilter couldn't set different handlers for different  
rcpt tos. Is this what you meant?


I think this requires some more thought considering different smtp  
connections and server requirements. The main drawback to sub- 
requesting each rcpt to is that we have two different handlers trying  
to read data from the socket. Is this problem solved by spooling the  
data, and letting the two separate requests read from the spool bucket?

-rian

Re: [vote] Revoke R-T-C [was: svn commit: r219520]

[David Reid]

-1

Re: Dispatching MPM

[Michal Kosek]

On Tue, Jul 19, 2005 at 11:14:51AM +0100, Nick Kew wrote:
> > In presented example, client1 has specified Host: header and has been
> > redirected to worker1, while client2 hasn't specified it yet, so it is
> > still connected to the dispatcher.  The diagram is created with
> > assumption that socket passing will be used, not proxying.
> 
> That is how 1.x and its successors work, isn't it?

No, diagram shows how D-MPM should work (other MPMs don't have anything
like dispatcher).  But if we decide to make dispatcher act as proxy (and
we probably must, if we want connection level filters work), clients
won't ever be connected to worker directly, but through dispatcher.

> >   Most existing solutions (that separate different virtual hosts) have
> >   the architecture where many threads/processes (each running with
> >   privilege of some virtual host) accept connections, and redirect them
> >   if client wants another virtual host. This may create potential
> >   security problem--if there were an exploitable bug in code before
> >   redirection, an attacker would have a possibility to gain privilege
> >   of user of any virtual host.
> 
> I'm not quite sure what you're saying.  What existing solution do
> you have in mind that is open to a transfer-of-privileges attack?
> Surely not the standard MPMs and suexec?

I compare design of D-MPM to design of perchild, Metux and peruser, ie.
other MPMs that aim for separating privileges of different virtual
hosts.  With D-MPM, some code will be executed with dispatcher's
permissions, and rest with permissions of appropriate worker.
Only spawning new children must be done by master apache process (with
root privileges), but it would be a small piece of code...

> >  In D-MPM, an attacker would gain only
> >   dispatcher's permissions, which can be very restricted.
> 
> Agreed.  A minimal dispatcher sounds great.  You should probably
> aim to have it require chrooting, as well as running as a user with
> no shell or login (perhaps drop those constraints under #ifdef DEBUG).

Of course, that's why I put /var/empty as dispatcher's root directory in
the example.

> >   To allow to utilize any of security extensions particular system has,
> >   D-MPM will have a loadable module support. Making D-MPM change some
> >   non-standard permissions would involve only writing a small shared
> >   library, without need for changing D-MPM's source or even recompiling
> >   it.
> 
> Do you mean it will implement a new optional hook specifically for
> chroot-like operations?

Yes, exactly.

> That sound very nice: maybe just make my
> "insist on chroot" suggestion the default if no module does
> something else.

[...]

> > Workers won't change much, so code from existing MPMs should be
> > somehow reused.  Even running different workers for different
> > virtual hosts should be possible.  So I'm thinking of possibility of
> > some generic code reuse, that would allow me to use workers from
> > almost any MPM (prefork, worker, event...).
> 
> Agreed.  For the time being, maybe just take the preforked processes.
> That'll mean you get to satisfy the PHP crowd.  Of course if you can
> harness Worker/Event/Other for no additional work, that would be the
> icing on the cake:-)

Prefork looks like a good place to start, as its the simplest of
existing stable MPMs...

> > Logging.  Currently every worker has descriptors of every log file.
> > To provide security, after creating a worker and changing its
> > permissions all the unnecessary log file descriptors will be closed.
> 
> Since a worker process only ever serves one vhost in this architecture,
> it'll only ever need a single server_rec.  For security, it shouldn't
> even see the server_recs for other vhosts.  A single server_rec has
> a single logfile descriptor open.

I actually want to have worker for each set of permissions (there may
be more vhosts with the same permissions).  It would have descriptors of
log files of all vhosts it serves, but as all of them have the same
owner, group, etc. it won't cause security problems.

Michał

Re: Dispatching MPM

[Michal Kosek]

On Tue, Jul 19, 2005 at 04:05:06PM +0900, Hideki Noma wrote:
> Did you check Metux MPM?
> It works by passing socket discriptor to worker process.

Yes, I looked at Metux MPM, but it also passes descriptors in the same
way as perchild, from process handling one vhost to another, and I don't
like it.  I think that the architecture with low-privileged dispatcher
is potentially more secure.  Moreover, with this architecture it would
be easier to handle eg. SSL.

Michał

Re: Initial mod_smtpd code.

[Joe Schaefer]

Rian Hunter <[EMAIL PROTECTED]> writes:

> I think this requires some more thought considering different
> smtp connections and server requirements. The main drawback to
> sub- requesting each rcpt to is that we have two different
> handlers trying to read data from the socket. Is this problem
> solved by spooling the data, and letting the two separate
> requests read from the spool bucket?

Hmm, what would the smtp return status for DATA be,
if only some of the RCPT_TO addresses are handled
successfully?

I've been assuming the http analog of "RCPT_TO: <[EMAIL PROTECTED]>"
was "POST: /foo\nHost: bar"  but I now think that's wrong
from a resource identifier standpoint.

-- 
Joe Schaefer

Re: AT_skip

[Randy Kobes]

On Tue, 19 Jul 2005, Joe Schaefer wrote:

> Joe Schaefer <[EMAIL PROTECTED]> writes:
>
> > Randy Kobes <[EMAIL PROTECTED]> writes:
> >
> >> Should AT_skip() print "ok ...", rather than "not ok ..."?
> >> If I change this to do so, then the tests are reported as
> >> all pass; however, the message that these tests have been
> >> skipped aren't seen.
> >
> > +1 to dropping the "not".
>
>
> Please apply that patch soonish, so I can roll rc4
> and subject it to a vote.

OK - thanks, that's been applied.

-- 
best regards,
randy

Re: Initial mod_smtpd code.

[Joe Schaefer]

Joe Schaefer <[EMAIL PROTECTED]> writes:

> Rian Hunter <[EMAIL PROTECTED]> writes:
>
>> I think this requires some more thought considering different
>> smtp connections and server requirements. The main drawback to
>> sub- requesting each rcpt to is that we have two different
>> handlers trying to read data from the socket. Is this problem
>> solved by spooling the data, and letting the two separate
>> requests read from the spool bucket?
>
> Hmm, what would the smtp return status for DATA be,
> if only some of the RCPT_TO addresses are handled
> successfully?
>
> I've been assuming the http analog of "RCPT_TO: <[EMAIL PROTECTED]>"
> was "POST: /foo\nHost: bar"  but I now think that's wrong
> from a resource identifier standpoint.

OTOH, maybe we should just return success in this case,
and only retry/bounce the failed subrequests later on.

-- 
Joe Schaefer

Re: svn commit: r218978 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c

[Joe Orton]

On Thu, Jul 14, 2005 at 07:43:35AM -0400, Jeff Trawick wrote:
> I'm so confused while trying to draw the line between
> 
> alternate RFC-compliant philosophy
> fixes for actual RFC violations
> fixes for security issues
> 
> I think CHANGES should be crystal clear on what change has a security
> implication.

I am also confused and still trying to catch up and understand these 
changes...

Bill, can you please describe *exactly* what security issues you see in 
the 2.0 proxy after the two already-committed patches (r219061).

CAN-2005-2088 MUST NOT be used to refer to anything other than specific 
issue described in the WatchFire report.  When you start bandying this 
CVE reference around with each new patch it defeats the purpose of 
having a CVE reference in the first place.  If there are wider issues in 
the proxy then they will need new CVE names assigned.

Regards,

joe

Re: Initial mod_smtpd code.

[Jem Berkes]

> Hmm. That sounds like a good idea, maybe there already is a hook
> defined that could deal with this, I'll look into it.

I could also start work on a mod_smtpd_dnsbl if the mentors feel that is 
worthwhile? This would look up a connecting IP address against a blacklist 
and return a descriptive string to mod_smtpd if the client should be 
rejected with an error: "550 5.7.1 Email rejected because 127.0.0.2 is 
listed by sbl-xbl.spamhaus.org"

I'd also like to include support for RHSBL, a newer type of listing by 
domain names from the envelope sender address. That's used by a growing 
number of projects.

Re: Initial mod_smtpd code.

[Nick Kew]

On Tue, 19 Jul 2005, Jem Berkes wrote:

> > Hmm. That sounds like a good idea, maybe there already is a hook
> > defined that could deal with this, I'll look into it.
>
> I could also start work on a mod_smtpd_dnsbl if the mentors feel that is
> worthwhile? This would look up a connecting IP address against a blacklist
> and return a descriptive string to mod_smtpd if the client should be
> rejected with an error: "550 5.7.1 Email rejected because 127.0.0.2 is
> listed by sbl-xbl.spamhaus.org"
>
> I'd also like to include support for RHSBL, a newer type of listing by
> domain names from the envelope sender address. That's used by a growing
> number of projects.

Happy to see you do either/both of those: it's all part of what
we want to see.

But is anyone dealing with outgoing SMTP via a proxy_smtp in the
mod_proxy framework?  I think you were discussing that a short while
ago, weren't you?  I think that might be higher priority.

-- 
Nick Kew

Re: [VOTE] libapreq2-2.06-dev-rc4

[Fred Moyer]

Please test & vote on rc4 for tomorrow's release of:

 http://people.apache.org/~joes/libapreq2-2.06-dev-rc4.tar.gz


From userland mod_perl 2.01, perl 5.8.6, httpd 2.0.54, linux 2.6.12, all 

tests pass, +1.

Re: Apache in a loop during startup

[Dirk-Willem van Gulik]

On Tue, 19 Jul 2005, Dirk-Willem van Gulik wrote:

> Before I dive into this - does this ring a bell with anyone ?

With help from Paul - seems to be a bug in BerklyDB where it scans through
possible backup index files before opening the real file - and then gets
confused; and gets its nickers in a twist.

Dw

Re: svn commit: r218978 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c

[William A. Rowe, Jr.]

At 10:07 AM 7/19/2005, Joe Orton wrote:
>On Thu, Jul 14, 2005 at 07:43:35AM -0400, Jeff Trawick wrote:
>> I'm so confused while trying to draw the line between
>> 
>> alternate RFC-compliant philosophy

Roy spelled it out, it's not in the RFC but if there is -any- way
we can use C-L, let's do it.  The current 2.0.x code doesn't actually 
do that because it doesn't check to see if we can read all of the
body within our hardcoded limit, before electing a C-L or T-E method.

Everything else represents violations.

>> fixes for actual RFC violations

The currently applied backport patch 171205 by Jeff...

  still believes cl_zero, even with T-E.
  still drops the body altogether for empty bodies in the
stream_chunked and spool_cl cases 
(an empty body is still a body!)
  still looks for a C-L value before looking for T-E 
  ignored the fact that *any* T-E overrides C-L
  introduced proxy-sendunchangedcl, something that an administrator
(as opposed to a developer) is unlikely to be certain of, and 
then never tested that the C-L values don't mismatch.
  provided no protection against proto_input_filters changing the 
body length

In other words, send_request_body is altogether bogus.  The problem
is exacerbated by the fact that send_request_body is out-of-line with
the rest of the header handling, making this harder code to review
than a single inline function to evaluate request headers+bodies.

>> fixes for security issues

Each RFC violation between proxy and origin server (or next hop)
becomes a cache poisioning/splitting/spoofing opportunity, I think
anyone who's read Watchfire groks that by now.

>> I think CHANGES should be crystal clear on what change has a security
>> implication.
>
>I am also confused and still trying to catch up and understand these 
>changes...
>
>Bill, can you please describe *exactly* what security issues you see in 
>the 2.0 proxy after the two already-committed patches (r219061).

E.g. the code remains bogus, as it once was, only with more
flavors of mishandling the body.

>CAN-2005-2088 MUST NOT be used to refer to anything other than specific 
>issue described in the WatchFire report.  When you start bandying this 
>CVE reference around with each new patch it defeats the purpose of 
>having a CVE reference in the first place.  If there are wider issues in 
>the proxy then they will need new CVE names assigned.

This patch addresses the fact that 2.0.x today HANGS when passed 
a C-L and T-E.  The 'protocol.c' patch previously committed does
mitigate the problem.  But Watchfire explicitly called out the
mishandling of C-L + T-E headers and the fact that the resulting
request body in some cases would have no request headers whatsoever.

The code in 2.0.x branch is ---still--- invalid, because the C-L
and T-E selection is invalid.  The symptoms are trivial; the proxy
never processes the body correctly because the wrong request
headers are ---still--- passed to the origin server.  And that is
exactly what Watchfire discussed.

I suggested Jeff's approach is *GOOD*, backport the correct body
handling code in its entirety.  Unfortunately, at that point in 
time, request body handling was still broken.

Since the intent was to backport, yet the elections and edge cases 
were not thought through, I'm vetoing and that specific backport.
It leaves me with a question;

is it better to revert [and then, re-credit Jeff in that CHANGES 
entry if/when we have an approved backport to reapply?] or better
to patch over to the current proxy_http body handling code?  I don't
know which will be more legible, later.

We enforce rules up front now with the protocol.c patch, masking
how problematic that mod_proxy_http.c still is;  always revert
  http://people.apache.org/~wrowe/httpd-2.0-cl-te-protocol.patch
before you test the newest mod_proxy_http patches.  Every other 
module which diddles the headers_in can re-trigger these edge cases, 
and anyone using the proxy_http.c as an example of anything will be 
led astray.  Yet, even with this wondrous sipmnle patch, the existing 
proxy_http code in 2.0.x trunk backported from 2.1.x still has issues.

In order to test, I'd applied;
  http://people.apache.org/~wrowe/httpd-2.0-trace.patch 
which provides trivial echo of the body and its headers, and then
used http://people.apache.org/~wrowe/httpd.proxy.conf to set up
a pretty wide range of tests from 1.3, 2.0, 2.1 against 1.3, 2.0
and 2.1 back-end servers.  You don't have to do this, of course,
but TraceEnable extended is simpler than sniffing sometimes.

And you can see from using netcat to pipe;
  http://people.apache.org/~wrowe/chunked20.req
through httpd-2.0 how badly this is all, still, mishandled, both
according to Roy's points r.e. C-L election, and sans the protocol
patch (which doesn't apply to module-mauled C-L and T-E headers).

Some examples;

** Using 2.0.54 proxy

HEAD /20/ HTTP/1.1
Host: localhost
Content-Length: 75
Transfer-Encoding: chunked

c
Test This!


0


** HANGS,

Pondering strings in Apache 3.x

[William A. Rowe, Jr.]

Greg and a few others voiced interest in moving from null-term
strings to counted strings for a future version of Apache.  
This was too broad a scope change to make it into 2.0, of course,
and was dropped on the floor for the time being.

I'm wondering today; what metadata interests us in an ap_string_t
prefix header?  I have a hunch that a short, 65536, is enough
to map most data we want to handle in one chunk; brigades are
better for handling large sets of data.  Of course we could push
that to an int, or size_t, but there would be a small memory
penalty.  It might be overcome by cpu-specific optimized int
or size_t handling behavior, since the assembly code wouldn't
need to truncate short values.

Perhaps, both bytes allocated/used, in order to play optimized
games with string allocation.  Perhaps, a refcount?  (This
doesn't play well with pool allocations, obviously.)

But the byte count clearly isn't enough.  I'm thinking of;

  encoding;  is this data URI escaped or un-escaped?

  tainted;   is it raw?  or has it been untainted with 
 context-specific validity checks?

  charset;   is this native?  (e.g. EBCDIC).  utf-8?
 opaque or otherwise a specific set?

What else interests us within an 'ap_string_t' header, that
would help eliminate bugs within httpd?  A random trailing
short following the string, in a 'string debug' mode, to 
detect buffer overflows?  Something similar to detect 
underflows?

Open to all ideas.

Bill

Re: [patch 1.3] The http_protocol.c C-L + T-E patch

[William A. Rowe, Jr.]

Paul?  Joe?  Jeff?  Someone?

This is the only showstopper to a 1.3.34 candidate today, 
since 1.3.x/src/modules/proxy/mod_proxy.c rejects T-E 
for proxy request bodies.

Bill

At 03:26 PM 7/15/2005, William A. Rowe, Jr. wrote:
>folks, the same patch Paul/Joe worked out for 2.1, then 2.0,
>should still probably fall on 1.3 even though proxy is not
>affected.  Other modules surely could be hit.
>
>Votes/Comments?  I think this is it for getting 1.3.34 out.
>
>Index: src/main/http_protocol.c
>===
>--- src/main/http_protocol.c(revision 209835)
>+++ src/main/http_protocol.c(working copy)
>@@ -1214,6 +1214,14 @@
> ap_log_transaction(r);
> return r;
> }
>+if (ap_table_get(r->headers_in, "Transfer-Encoding")
>+&& ap_table_get(r->headers_in, "Content-Length")) {
>+/* 2616 section 4.4, point 3: "if both Transfer-Encoding
>+ * and Content-Length are received, the latter MUST be
>+ * ignored"; so unset it here to prevent any confusion
>+ * later. */
>+ap_table_unset(r->headers_in, "Content-Length");
>+}
> }
> else {
> ap_kill_timeout(r);

Re: Initial mod_smtpd code.

[Matthieu Estrade]

Rian Hunter wrote:



On Jul 19, 2005, at 6:51 AM, Nick Kew wrote:


the problem i found when i did my poc is when there is in the  command,
different destination email. It's difficult here to keep the  
virtualHost

scheme.
It would be nice to keep a conf file like


ServerName mail.bla.com
SmtpUserMap mail.bla.com-user.map
or SmtpRelay host





Yes.  I think there is a logical difficulty here.   smtpd_create_request
is run before process_smtp_connection_internal, and the design doesn't
allow for multiple recipients to be processed differently.

My own feeling was that multiple recipients require a request each.
Maybe we could get that in your design by creating a subrequest
for each RCPT TO?



This is possible but I'm not sure what the advantage is. Would you  
mind setting up a hypothetical situation where this is necessary? I  
figure that ultimately the handler will be responsible for dealing  
with each rcpt to differently.


Early on I wanted configuration possibility similar to:
#
Listen 21

# mod_smtpd conf
  SmtpProtocol On
SetHandler unix-module

# mod_smtpd_unix
AcceptDomains thelaststop.net www.thelaststop.net
Relay On

# mod_smtpd_easyfilter

# matches against email in MAIL TO: smtp command
RegexMailTo "/thelaststop.net$/"
SetHandler maildir-module

# mod_smtpd_maildir conf
MaildirBase "/usr/local/virtual"
MailboxLimit 50M


# Simple spam filter using mod_smtpd_easyfilter
# Default handler does nothing with mail message

RegexHeader "Subject" "/cialis/"
SetHandler none


##


imho, your conf is too complex.
first, it would be good to create some  or 


and in each tag, put some process and filter setup.


ServerAlias prc.apache.org apr.apache.org

SmtpUser pam (or ldap or mysql etc like auth in httpd)
SmtpSpool /var/spool/mail (directive from mod smtp core)
or
SmtpRelay 1.2.3.4
SmtpRelay 1.3.4.5 (With something

(then here directive for other smtp modules



if we consider the SmtpUser as a function hooked in something created 
specialy, like ap_hook_smtp_user, we run this function to find the user 
inside the local/ldap/pam or whatever choosen. maybe a user profile 
provider.
We could also have a hook for register new command, like we could 
register new method in http.


it could be:

command 1 -> processed by function for this command
command 2 -> processed by function for this command
rcpt to: [EMAIL PROTECTED] -> get_domain then get_user(domain) then register 
user and action to do (maybe in a array) (SmtpSpool or SmtpRelay 
function, maybe a ap_hook_smtp_action)


command y -> processed by function for this command
end command

DATA

Here we execute actions (registered with users and domain before, as 
filter or as hook)

user 1 -> Relay -> execute relay
user 2 -> Local Delivery -> put mail inside Spool
user 3 -> Relay etc...

imho, delivering, relaying etc could be done sequentialy

Maybe we could have inside the user array, something ordered per vdomain 
coz they have the same action.

if [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
We could directly relay user1 and user3 in the same action and local 
deliver for user inside domain-local.


Cheers,

Matthieu

In the case of this httpd.conf embedded filtering mechanism, I  
figured the handlers could be changed based on the certain RegExs  
right before ap_run_handler() was called. Maybe ap_run_fixups() could  
be called, and my hypothetical mod_smtpd_easyfilter would have a  
fixups hook where it accomplished something similar to this  
situation. Although after thinking about it I realize now that  
mod_smtpd_easyfilter couldn't set different handlers for different  
rcpt tos. Is this what you meant?


I think this requires some more thought considering different smtp  
connections and server requirements. The main drawback to sub- 
requesting each rcpt to is that we have two different handlers trying  
to read data from the socket. Is this problem solved by spooling the  
data, and letting the two separate requests read from the spool bucket?

-rian

Re: [patch 1.3] The http_protocol.c C-L + T-E patch

[Joe Orton]

On Tue, Jul 19, 2005 at 02:59:14PM -0500, William Rowe wrote:
> Paul?  Joe?  Jeff?  Someone?
> 
> This is the only showstopper to a 1.3.34 candidate today, 
> since 1.3.x/src/modules/proxy/mod_proxy.c rejects T-E 
> for proxy request bodies.

Since the 1.3 proxy already rejects such requests what does this patch 
actually fix?

> 
> Bill
> 
> At 03:26 PM 7/15/2005, William A. Rowe, Jr. wrote:
> >folks, the same patch Paul/Joe worked out for 2.1, then 2.0,
> >should still probably fall on 1.3 even though proxy is not
> >affected.  Other modules surely could be hit.
> >
> >Votes/Comments?  I think this is it for getting 1.3.34 out.
> >
> >Index: src/main/http_protocol.c
> >===
> >--- src/main/http_protocol.c(revision 209835)
> >+++ src/main/http_protocol.c(working copy)
> >@@ -1214,6 +1214,14 @@
> > ap_log_transaction(r);
> > return r;
> > }
> >+if (ap_table_get(r->headers_in, "Transfer-Encoding")
> >+&& ap_table_get(r->headers_in, "Content-Length")) {
> >+/* 2616 section 4.4, point 3: "if both Transfer-Encoding
> >+ * and Content-Length are received, the latter MUST be
> >+ * ignored"; so unset it here to prevent any confusion
> >+ * later. */
> >+ap_table_unset(r->headers_in, "Content-Length");
> >+}
> > }
> > else {
> > ap_kill_timeout(r);
> 

Re: [patch 1.3] The http_protocol.c C-L + T-E patch

[William A. Rowe, Jr.]

At 04:11 PM 7/19/2005, Joe Orton wrote:
>On Tue, Jul 19, 2005 at 02:59:14PM -0500, William Rowe wrote:
>> Paul?  Joe?  Jeff?  Someone?
>> 
>> This is the only showstopper to a 1.3.34 candidate today, 
>> since 1.3.x/src/modules/proxy/mod_proxy.c rejects T-E 
>> for proxy request bodies.
>
>Since the 1.3 proxy already rejects such requests what does this patch 
>actually fix?

Hmmm...

  mod_isapi?
  mod_php?
  mod_cgi?
  mod_jk?

shall I keep digging?

Bill

Re: Initial mod_smtpd code.

[Jem Berkes]

> But is anyone dealing with outgoing SMTP via a proxy_smtp in the
> mod_proxy framework?  I think you were discussing that a short while
> ago, weren't you?  I think that might be higher priority.

I hesitated on that because I did not understand at all how mod_proxy fits 
into this. i.e. I don't see how the proxy mechanism helps in relaying out 
mail to other SMTP servers.

Here's an idea on how we can start on the outbound SMTP side of things:

I can start work on a "mod_smtp_relay" which takes an email and attempts to 
relay it via the appropriate MX relay. This involves some DNS queries for 
MX records, and making new TCP connections to another SMTP server. I 
recommend that mod_smtp_relay does not itself do any spooling or queueing, 
to isolate these tasks. i.e. some other delivery/scheduler will handle 
spooling and retries etc, and occasionally pass an email to mod_smtp_relay

So given an input message, mod_smtp_relay would make an immediate relay 
attempt and then return success (sent) or an error describing where along 
the lines things went wrong -- it could be DNS, TCP connect failure, or 
SMTP error dictating permanent failure or temporary failure, needing retry.

Re: Initial mod_smtpd code.

[Jorge Schrauwen]

Wouldn't something like


ServerName www.apache.org:81
ServerAlias httpd.apache.org internal_dev
ServerAdmin [EMAIL PROTECTED]
ErrorLog logs/error.log
TransferLog logs/access.log
SMTP On # like with mod_dav (dav on)


so reusing existing command, then this main an user of www.apache.org

lets call him

[EMAIL PROTECTED]
that the following addresse point to the same.
[EMAIL PROTECTED]
[EMAIL PROTECTED]

this seems to be more logical to me.

btw: i'm not a member, just someone who follow this list closely

Jorge

Re: mod-cache-requestor plan

[Ian Holsman]

Parin Shah wrote:

you should be using a mix of

# requests
last access time
cost of reproducing the request.




Just to double check, we would insert entry into the 'refresh queue'
only if the page is requested and the page is soon-to-be-expired. once
it is in the queue we would use above parameters to calculate the
priority. Is this correct? or let me know If I have mistaken it.


yep.
thats the idea.
refresh the most-popular pages first.




see memcache_gdsf_algorithm() in mod_mem_cache.c for an implementation
of this, which assumes 'length' of request is related to the cost of
reproducing the request.

the priority queue implementation is sitting in mod_mem_cache, and could
be used to implement the 'refresh' queue I would think.



I feel comfortable with mod-cache and mod-mem-cache code now. but we
also need to start new thread/process for mod-cache-requester when
server starts. I am not too sure how we could implement it. any
pointers to the similar piece of code would be really helpful to me.

I don't have any code which does this to share with you (others might 
know of some).




Thanks,
Parin.


--Ian

mod_mbox and generate_index

[russell johnson]

I trying to build the standalone helper program generate_index that
the catchup-archive scriptneeds.  This program does not seem to be
made by the makefile and the build-dso errors out looking for lucene4c
headers. Any ideas?

build-dso?

[steve johnson]

What (or who) generates the build-dso scripts for
modules?  What are they used for?  If it is used why
not use make?  I am fighting with mod_mbox in
particular but I'm also interested in a more
generalized answer if someone has one. 

build-dso?

[steve johnson]

What (or who) generates the build-dso scripts for
modules?  What are they used for?  If it is used why
not use make?  I am fighting with mod_mbox in
particular but I'm also interested in a more
generalized answer if someone has one. 

Re: Initial mod_smtpd code.

[Jorge Schrauwen]

Wouldn't something like


ServerName www.apache.org:81
ServerAlias httpd.apache.org internal_dev
ServerAdmin [EMAIL PROTECTED]
ErrorLog logs/error.log
TransferLog logs/access.log
SMTP On # like with mod_dav (dav on)


so reusing existing command, then this main an user of www.apache.org

lets call him

[EMAIL PROTECTED]
that the following addresse point to the same.
[EMAIL PROTECTED]
[EMAIL PROTECTED]

this seems to be more logical to me.

btw: i'm not a member, just someone who follow this list closely

Jorge

Re: Pondering strings in Apache 3.x

[Brian Pane]

On Jul 19, 2005, at 12:55 PM, William A. Rowe, Jr. wrote:


Greg and a few others voiced interest in moving from null-term
strings to counted strings for a future version of Apache.
This was too broad a scope change to make it into 2.0, of course,
and was dropped on the floor for the time being.

I'm wondering today; what metadata interests us in an ap_string_t
prefix header?  I have a hunch that a short, 65536, is enough
to map most data we want to handle in one chunk; brigades are
better for handling large sets of data.  Of course we could push
that to an int, or size_t, but there would be a small memory
penalty.  It might be overcome by cpu-specific optimized int
or size_t handling behavior, since the assembly code wouldn't
need to truncate short values.

Perhaps, both bytes allocated/used, in order to play optimized
games with string allocation.  Perhaps, a refcount?  (This
doesn't play well with pool allocations, obviously.)

But the byte count clearly isn't enough.  I'm thinking of;

  encoding;  is this data URI escaped or un-escaped?

  tainted;   is it raw?  or has it been untainted with
 context-specific validity checks?

  charset;   is this native?  (e.g. EBCDIC).  utf-8?
 opaque or otherwise a specific set?

What else interests us within an 'ap_string_t' header, that
would help eliminate bugs within httpd?  A random trailing
short following the string, in a 'string debug' mode, to
detect buffer overflows?  Something similar to detect
underflows?

Open to all ideas.


This may be a bit more radical than you were hoping for, but...

I like the idea of using a reference-counted, non-null-terminated
string type for 3.x.

More generally, it would be great to have overflow detection
on all arrays.

And although I like the performance benefits of the pool memory
allocators, I remember how tricky it was to debug some of the
pool and bucket lifetime problems that we encountered during
the development of 2.0 (especially in filters).  All things considered,
I don't think I'd mind the overhead of a garbage collection thread.

Thus I can't help but wonder: Would 3.0 be a good time to consider
trying a Java-based httpd?

Brian

Re: mod-cache-requestor plan

[Parin Shah]

Hi All,

We are now almost at consesus about this new mod-cache-requester
module's mechanism. and now I believe its good time to start
implementing the module.

But before I could do that, I need some help from you guys.

- I am now comfortable with mod-cache, mod-mem-cache, cache_storage.c,
cache_util.c etc.

- But still not too sure how to implement couple of things.

1. How to start the new thread/process for mod-cache-requester when
server start. any similar piece of code would help me a lot.

2. how mod-cache-requester can generate the sub request just to reload
the content in the cache.

3. In current scheme, whenever mod-cache-requester pulls first entry
from pqueue ('refresh' queue) it re-requests it to reload. now by the
time this re-request is done, page might actually have been expired
and removed from cache. in such case should mod-cache reload it or
should wait for next legitimate request.

Your thoughts on any/all on these issues would be really helpful.

Thanks
Parin.

On 7/19/05, Ian Holsman <[EMAIL PROTECTED]> wrote:
> Parin Shah wrote:
> >>you should be using a mix of
> >>
> >># requests
> >>last access time
> >>cost of reproducing the request.
> >>
> >
> >
> > Just to double check, we would insert entry into the 'refresh queue'
> > only if the page is requested and the page is soon-to-be-expired. once
> > it is in the queue we would use above parameters to calculate the
> > priority. Is this correct? or let me know If I have mistaken it.
> >
> yep.
> thats the idea.
> refresh the most-popular pages first.
> 
> >
> >>see memcache_gdsf_algorithm() in mod_mem_cache.c for an implementation
> >>of this, which assumes 'length' of request is related to the cost of
> >>reproducing the request.
> >>
> >>the priority queue implementation is sitting in mod_mem_cache, and could
> >>be used to implement the 'refresh' queue I would think.
> >>
> >
> > I feel comfortable with mod-cache and mod-mem-cache code now. but we
> > also need to start new thread/process for mod-cache-requester when
> > server starts. I am not too sure how we could implement it. any
> > pointers to the similar piece of code would be really helpful to me.
> >
> I don't have any code which does this to share with you (others might
> know of some).
> 
> 
> > Thanks,
> > Parin.
> >
> --Ian
> 
>

Re: NTLM HTTP Authentication is insecure by design - a new writeup by Amit Klein

[William A. Rowe, Jr.]

At 03:57 AM 7/19/2005, Roy T. Fielding wrote:
>On Jul 18, 2005, at 12:30 PM, William A. Rowe, Jr. wrote:
>>>NTLM HTTP Authentication
>>> (and possibly other connection-oriented
>>> HTTP authentication and authorization protocols)
>>> is insecure by design
>
>Yep, no shit -- that's what the Microsoft fools were told when
>they introduced it.

That doesn't surprise me :)

>>>*) The web server (IIS/6.0) must receive a Via-less request. The
>>>Microsoft implementation assumes that the Via header is always sent
>>>by a proxy server, and this is indeed mandated by the HTTP/1.1 RFC
>>>2616 (http://www.ietf.org/rfc/rfc2616.txt), section 14.45:
>>> The Via general-header field MUST be used by gateways and proxies
>>> to indicate the intermediate protocols and recipients between the
>>> user agent and the server on requests [...]
>
>Yep.
>
>>>However, it seems that not all servers adhere to this standard. For
>>>example, Apache 2.0.54 mod_proxy does not generate a Via header by
>>>default (see the ProxyVia directive -
>>>http://httpd.apache.org/docs-2.0/mod/mod_proxy.html#proxyvia, yet
>>>the default httpd.conf file contains a commented-out "ProxyVia On"
>>>directive, so it's possible that many Apache proxy deployments do
>
>WTF?  That's a bug.

I'd say.  The question is; what's the fix?

Today's syntax is ProxyVia on|off|full|block

The default is off, which is clearly invalid.

A default of full might be overkill, default of on 
should be sufficient

Are ProxyVia off and ProxyVia block invalid?

Technically, it's pretty clear that they are.

Realistically, if someone wants an invisible proxy, I guess
that's up to them, but then again it's their own damned fault
if they trip over something like this.  But we don't give them
much choice today, I don't think...

The 14.45 Via section defines a pretty clear acceptable flavor,


   [...] The
   received-by field is normally the host and optional port number of a
   recipient server or client that subsequently forwarded the message.
   However, if the real host is considered to be sensitive information,
   it MAY be replaced by a pseudonym. If the port is not given, it MAY
   be assumed to be the default port of the received-protocol.


Sooo... a new alternative, "ProxyVia alias shortname" would seem sane,
and a much better choice for most reverse proxies.


   For organizations that have strong privacy requirements for hiding
   internal structures, a proxy MAY combine an ordered subsequence of
   Via header field entries with identical received-protocol values into
   a single such entry. For example,

   Via: 1.0 ricky, 1.1 ethel, 1.1 fred, 1.0 lucy

could be collapsed to

   Via: 1.0 ricky, 1.1 mertz, 1.0 lucy

   Applications SHOULD NOT combine multiple entries unless they are all
   under the same organizational control and the hosts have already been
   replaced by pseudonyms. Applications MUST NOT combine entries which
   have different received-protocol values.

Sooo... "ProxyVia collapse shortname" would be an even terser flavor,
and I can't see a case where we would keep the truename of this via
server while collapsing the rest.

Thoughts?

Bill

Re: svn commit: r219520 - /httpd/httpd/branches/2.2.x/

[Joe Orton]

On Mon, Jul 18, 2005 at 11:58:21AM -0500, William Rowe wrote:
> Thanks Paul, you just collided with the refactoring of 2.1.x proxy.

Is this refactoring complete?  Apart from the compiler warnings, a bunch 
of the t/ssl/proxy.t tests have started failing with the trunk code.  
With worker, the server is dumping core:

(this seems to be a regression since 2005-07-15, when all the tests were 
passing with both prefork and worker)

Core was generated by `/tmp/regressk13532/trunk-worker-root/bin/httpd -d 
/tmp/regressk13532/pf-trunk-w'.
Program terminated with signal 11, Segmentation fault.

[...omitting other threads...]

#0  0x0032e452e989 in kill () from /lib64/tls/libc.so.6
No symbol table info available.
#1  
No symbol table info available.
#2  apr_pool_clear (pool=0x0) at memory/unix/apr_pools.c:332
freelist = (apr_memnode_t *) 0x0
max_index = 2
max_free_index = 0
next = (apr_memnode_t *) 0x56a3e8
index = 0
current_free_index = 4294967295
#3  0x002a97de8e31 in connection_cleanup (theconn=0x56a3e8) at 
proxy_util.c:1456
p = (apr_pool_t *) 0x80db08
conn = (proxy_conn_rec *) 0x809cd8
worker = (proxy_worker *) 0x5a0328
#4  0x002a97de95a6 in ap_proxy_release_connection (proxy_function=0x56a3e8 
"\001", conn=0x809cd8, s=0x0) at 
proxy_util.c:1714
No locals.
#5  0x002a980fd6c8 in ap_proxy_http_cleanup (scheme=0x0, r=0x89e4d8, 
backend=0x809cd8) at mod_proxy_http.c:1525
No locals.
#6  0x002a980fe910 in proxy_http_handler (r=0x89e4d8, 
worker=0x5a0328, 
conf=0x878b88, url=0x891168 "/require-ssl-cgi/env.pl", proxyname=0x0, 
proxyport=0) at mod_proxy_http.c:1162
here = (struct apr_bucket *) 0x0
status = 0
server_portstr = 
":8541\000\000\000Húj\225*\000\000\000\000\000\000\000\210\213\207\000\000\000\000"
scheme = 0x0
proxy_function = 0x2a980fee00 "HTTPS"
u = 0x0
backend = (proxy_conn_rec *) 0x809cd8
is_ssl = 1
p = (apr_pool_t *) 0x8903f8
uri = (apr_uri_t *) 0x8910a8
#7  0x002a97de3786 in proxy_run_scheme_handler (r=0x89e4d8, 
worker=0x5a0328, conf=0x878b88, 
url=0x8a0066 
"https://localhost.localdomain:8532/require-ssl-cgi/env.pl";, 
proxyhost=0x0, proxyport=0) at mod_proxy.c:1890
pHook = (proxy_LINK_scheme_handler_t *) 0x589448
n = 0
rv = 0

Re: Pondering strings in Apache 3.x

[André Malo]

* Brian Pane wrote:

> And although I like the performance benefits of the pool memory
> allocators, I remember how tricky it was to debug some of the
> pool and bucket lifetime problems that we encountered during
> the development of 2.0 (especially in filters).  All things considered,
> I don't think I'd mind the overhead of a garbage collection thread.

The pool problems should be solved now... (mostly)

> Thus I can't help but wonder: Would 3.0 be a good time to consider
> trying a Java-based httpd?

If you ask me: Nope. Try Tomcat instead ;)

What we need for 3.0 is just a clean design and definitions of what is core 
(not much, imo) and what is not core. This was started for 2.0 but never 
finished. Further a standardized exception handling would be nice (like 
svn's). 
The "core" could provide several convenience data types like ap_string_t.

I would, btw, just store the length of the string in such a type. Other 
properties (url-encoding state, ...) imo belong to a different layer. Like 
a bucket or just a wrapper type.

nd
-- 
"Das Verhalten von Gates hatte mir bewiesen, dass ich auf ihn und seine
beiden Gefährten nicht zu zählen brauchte" -- Karl May, "Winnetou III"

Im Westen was neues: