Re: preserving request body across redirects

[André Warnier]

Mark Hedges wrote:



[...]
Hi.

Not to discourage you, but to point out some additional aspects..

There is a Belgian proverb that says essentially one should not try to 
be more catholic than the pope.
What I mean by that is that there are many twists to web authentication, 
and it is really hard to try and cover all of them at once.  Your 
intentions are laudable, but you may end up having to chew more than you 
intend to.  There are some good reasons why the myriad mod_perl 
authentication modules only each cover part of the AAA scene.


The underlying truth is that HTTP was designed as a fire and forget 
protocol, where each request/response cycle is unique and essentially 
disconnected from any previous or subsequent request/response.
This means that, whatever you do, you first find yourself fighting 
against the fundamental limitations of the protocol.  In a way, it is as 
if everything you do would be a patch applied over an uncooperative 
and slippery basic surface.  Worse yet, you have to apply your own AAA 
patch over the existing patches (such as Keep-Alive, chunked encoding, 
NTLM authentication, Cookies, byte ranges, DAV, cgi-bin, proxying..), 
and make sure it works with all of them.


For example, if you want to be really agnostic, then you have to start 
by saving the whole request each time, no matter if the user is 
authenticated/authorised or not, because you do not know this yet.  It 
is only when the request has started to be parsed, that you know which 
of your Location or Directory sections apply, and thus which 
authentication/authorization applies.
There may be clever ways to do this, but I suspect that it might in any 
case be very inefficient, and not really workable for a production 
environment (the ScriptLog directive of Apache is an example).


You might want to wonder for instance if you cannot instead structure 
your application so that, prior to accessing a protected URL, it is not 
mandatory to do a first authentication GET to some document.


Another avenue may be to apply an old African proverb, which states that 
to eat an elephant, one should do it bit by bit.
In other words, specialising your authentication subs by HTTP method, 
and apply different schemes to HEAD, GET, POST, PUT, (MKCOL, OPTIONS..).


As an example of twist, imagine the following scenario : a user POSTs 
(or PUTs) a very large file to the server.  To my knowledge, Apache will 
read the POST/PUT entirely and store it somewhere, prior to even calling 
your authentication module.  Then you read the POST/PUT, save it 
somewhere again, and send them a login page to start your authentication 
cycle.
Now imagine the user just does not know a valid login, and closes his 
browser in despair.  How do you clean up ?


As another example, consider the following :
The DAV module/protocol allows one to create web folders in one's 
Windows Explorer, corresponding on the server side to directories, in 
which one can drag-and-drop files directly from one's desktop.

Of course you might want to authenticate/authorise the user doing this.
The Windows Explorer web folder implementation however does not 
support cookies set by the webserver, so it is hard to do anything based 
on cookies, be it only a cookie holding a session-id.  It also only 
supports Basic (and maybe Digest) authentication (no custom login pages 
thus), and it does not support OpenID.
The only way I found was to do Basic authentication, and store the 
authentication (or session) data in the Connection structure (like 
$r-connection-notes). In that case it fits, because Windows-style 
authentication is connection-oriented anyway.


And as long as we are talking Windows (and since at least 90% of 
corporations still use IE and Windows authentication internally), what 
if one of your customers insists that to authenticate external users, 
OpenID is OK, but for internal users they would really like you to use 
the integrated Windows authentication (NTLM), since the user is already 
authenticated in the Domain anyway.
NTLM authentication would play havoc with your scheme, because it 
already itself relies on a multi-step exchange between browser and server.

Re: preserving request body across redirects

[Adam Prime]

Mark Hedges wrote:


The question is, since the handler doing the preservation
and installing the input filter also instantiates an
Apache2::Request object, when it gets to the response phase
controller, will the response handler's Apache2::Request
instance read the replaced data or the cached data?



From what I understand, as soon as you try to use the body of a request 
using Apache2::Request, it reads it all in.  Which might play havoc with 
the idea of using a Filter to modify a request / inject a body to replay 
an old post request.


However, without that complication it may be possible.  Have you looked 
at the filter documentation on perl.apache.org? Particularly the 
examples in this section:


http://perl.apache.org/docs/2.0/user/handlers/filters.html#Input_Filters

My suggestion would be to try it and see what happens.  Filters are one 
aspect of mp2 that it seems like a lot of people haven't really used.  I 
personally have used OuputFilters for a bunch of things, but i haven't 
used InputFilters at all.  I'm also not familiar enough with how OpenID 
works to really be able to comment on what you're trying to do.


Adam

Re: preserving request body across redirects

[Mark Hedges]

On Mon, 29 Dec 2008, David Ihnen wrote:
  Say the application's session times out, but the user
  posts something by clicking submit.
 
  They are redirected to the OpenID server but it says
  they are still logged in and returns a positive
  response.

 But you can't know that they are authenticated without
 redirecting them, because you dropped the session data
 that held that information.

 I think the obvious answer here is to keep the session
 data around as long as the authentication is valid for.
 Or keep this piece of state in a cookie so that it doesn't
 vanish when your server decides to drop session state.

I'm not sure I understand what you mean here David.  They
don't get redirected for every request to the server.  The
session data keeps a timestamp.  They stay logged in (and do
not need to redirect thru the OpenID server) as long as the
timestamp is current.  The whole point is to keep the
session current with an abstract tracking mechanism (the
default mechanism in the framework is a session ID cookie.)

But if they time out, and click submit, but the independent
OpenID server says they're still logged in, or they re-auth
automatically with the server (e.g. VeriSign Seatbelt
Firefox plugin), then it should act as if they never timed
out, i.e. when they come back from the redirect cycle, it
should continue to submit whatever they clicked.

On Mon, 29 Dec 2008, Adam Prime wrote:
 From what I understand, as soon as you try to use the body
 of a request using Apache2::Request, it reads it all in.
 Which might play havoc with the idea of using a Filter to
 modify a request / inject a body to replay an old post
 request.

Ah - yes I may have to go to using only the methods of the
Apache2::RequestRec family to get at the 'openid_url' param.
Then an input filter could rewrite the buckets before the
Response phase controller instantiates the Apache2::Request.
I'll try it and see.

On Mon, 29 Dec 2008, André Warnier wrote:
 Not to discourage you, but to point out some additional
 aspects..

Thanks André these are very good points.  I had considered a
size limit option where you could say that the request body
would not be preserved if it was over 2k or something.  But
even that leaves the system open to DoS attacks unless the
application developer runs some very active cleanup script.

So I think the best plan is to preserve the request body
only if they already have an authenticated session and the
session timed out.

Good to know about other methods and not being able to use
cookies with Windows etc.  But the session tracking
mechanism in Apache2::Controller is abstracted, so a
non-cookie mechanism could be used instead, like an output
filter that rewrites links with a session id query argument,
or something.  Probably still would not work with DAV, but
it's the general idea.

 And as long as we are talking Windows (and since at least
 90% of corporations still use IE and Windows
 authentication internally), what if one of your customers
 insists that to authenticate external users, OpenID is OK,
 but for internal users they would really like you to use
 the integrated Windows authentication (NTLM), since the
 user is already authenticated in the Domain anyway. NTLM
 authentication would play havoc with your scheme, because
 it already itself relies on a multi-step exchange between
 browser and server.

In this case I might create an internal OpenID server that
interfaced to our Windows NTLM.  :-)  Or, I'd stack an NTLM
auth handler above the OpenID handler, so if NTLM succeeds,
then the session timestamp is updated, and the OpenID
handler says OK without checking anything.

The point of Apache2::Controller is to abstract different
facets into stackable, subclassable object-oriented handlers
and then dispatch into a nice MVC-style controller
subroutine, but without all the overhead of something like
Catalyst.  I dislike Catalyst's plugin architecture.  Why do
I need a plugin layer above everything else on CPAN?  I just
want to use something and do it and get it done.  Why do I
need a model abstraction layer?  I just want to use a
DBIx::Class schema or maybe sometimes I want to feed
straight out of DBI - the point is I don't want the
framework to get in your way.

Mark

Re: preserving request body across redirects

[David Ihnen]

Mark Hedges wrote:

On Mon, 29 Dec 2008, David Ihnen wrote:
  

Say the application's session times out, but the user
posts something by clicking submit.

They are redirected to the OpenID server but it says
they are still logged in and returns a positive
response.
  

But you can't know that they are authenticated without
redirecting them, because you dropped the session data
that held that information.

I think the obvious answer here is to keep the session
data around as long as the authentication is valid for.
Or keep this piece of state in a cookie so that it doesn't
vanish when your server decides to drop session state.



But if they time out, and click submit, but the independent
OpenID server says they're still logged in, or they re-auth
automatically with the server (e.g. VeriSign Seatbelt
Firefox plugin), then it should act as if they never timed
out, i.e. when they come back from the redirect cycle, it
should continue to submit whatever they clicked.
  
Maybe you could use something a bit more client side like an iframe 
target, which CAN be resubmitted because the form still exists in state 
on the page despite whats going on in the iframe.


- submit form target iframe
- iframe redirects for auth
- iframe gets auth
- success page activates parent re-submit
- submit form target iframe
- iframe success activates main page success state

Though that is, of course, specific to the application being programmed, 
utilizing client-side javascript active stuff rather than particular web 
server programming to transparently handle it on the server side using 
basic html2.0 type structure.


I have to agree with others that a whole proxying layer to allow it 
seems... excessive.


timtowtdi I guess.

David

Apache::Reload/ModPerl::Util/base.pm incompatibility prospective patch

[David Ihnen]

Philip M. Gollucci wrote:

David Ihnen wrote:

1. Problem Description:

While developing with CGI::Application and utilizing Apache::Reload, 
we encountered an issue where our modules were not being succesfully 
reinitialized on reload.  It was traced down to @ISA not containing 
the proper values after a 'use base' directive, but only on automatic 
reload, and freshly on perl 5.10, not our previous version of perl 
5.8.8!
Are you able to try 5.8.9, its at least a smaller change set to look 
at if 5.8.9 breaks too.


No, not easily when I considered the mod_perl ramifications 
particularly.  I did find a way around this particular problem on the 
web server though.


I noticed line 79. of base.pm

79 next if grep $_-isa($base), ($inheritor, @bases);

Experimentally, if I commented it out, the web server could reload 
properly!  Score!


Hm.   I know -isa is a UNIVERSAL function.  UNIVERSAL.pm is... not 
implimented in perl.  Must be in the C code.


So I found UNIVERSAL.c

whose isa function implimentation said after some error checks essentially

ST(0) = boolSV(sv_derived_from(sv, name));


which function had the documentation

=head1 SV Manipulation Functions

=for apidoc sv_derived_from

Returns a boolean indicating whether the SV is derived from the specified class
Iat the C level.  To check derivation at the Perl level, call Cisa() as a
normal Perl method.

=cut


Ooh. at the **C** level.  I bet the unload utility doesn't even *affect* 
the C level!


Those who understand perl C code might be better able to tell if 
unloading in the C levels is possible, a good idea, or just bad mojo.  
Having taken some medicine to deal with the C-inspired headache that 
inspection brought on, I returned to base.pm.


ISA is a package local variable list that lists the parents.  In order 
to see all-the-parents, or replicate -isa, I have to get the list of 
all the packages in ISA, all THEIR ISAs, and so on, as long as there are 
contents therein.  And my scan will pay attention only to the 
interpreter state, not the C state...


I should be able to do that in perl easily enough.  PATCH

--- 510base.pm2008-12-29 12:48:27.0 -0700
+++ /usr/share/perl/5.10.0/base.pm2008-12-29 12:48:48.0 -0700
@@ -76,7 +76,20 @@
warn Class '$inheritor' tried to inherit from itself\n;
}

-next if grep $_-isa($base), ($inheritor, @bases);
+my @classlist = ($inheritor, @bases);
+foreach ($inheritor, @bases) {
+my @parents = @$_::ISA;
+do {
+my $class = shift @parents;
+foreach my $parent (@$class::ISA) {
+unless (grep { $_ eq $parent } @classlist) {
+push @classlist, $parent;
+push @parents, @$parent::ISA;
+}
+}
+} while (scalar @parents);
+}
+next if grep {$_ eq $base} @classlist;

if (has_version($base)) {
${$base.'::VERSION'} = '-1, set by base.pm'

I know thats kind of ugly, and verbose, and maybe not very performant.  
But it works for the web server.  Its not in any of the ISAs at 
re-require time so it does not skip loading the module even if the C HAS 
seen it before, and the application continues happily.


So at any rate, its a patch that works for my case (much as I hate 
patching base.pm) - and to my somewhat weak understanding of the C code 
implimentation, does the same thing in perl that the isa-iteration did 
in c.  What is the proper path here?


David Ihnen
Perl Programmer
Norchem Laboratories, Inc

Re: [mp2] undefined symbol in make test with threaded Apache 2.2.11 RESOLVED

[craig]

Dropped back in configuration and component programs to a working  
combination, then advanced in smaller steps than I had done before.   
You were correct, Mr G, in your prediction that the lib tree was  
involved.


The 'Undefined symbol PL_markstack_ptr' in the mod_perl2 'make  
test' step:
1. appeared as soon as an option was added that changed the names of  
the architecturally dependent lib directories, and
2. went away when I removed the three old architecturally-dependent  
directories that I had tried to preserve, from the colon separated  
list of additional lib directories.


As a general statement of this, I would nominate: when changing Perl  
build options that change the names of architecturally-dependent lib  
directories, don't try to keep old arch-dependent directories in  
@INC.  With the note that such dependent directories contain  
hyphenated elements like 'i386-freebsd'.


Thanks for your help,
cmac


On Dec 27, 2008, at 11:44 PM, Philip M. Gollucci wrote:


Philip M. Gollucci wrote:

Craig MacKenna wrote:

On Dec 24 00:57, Philip M. Gollucci pgollu...@p6m7g8.com wrote:
Subject: Re: [mp2] undefined symbol in make test with threaded  
Apache 2.2.11
If you post your ./Configure for perl, ./configure for httpd, and  
perl Makefile.PL for mod_perl, I'll run it locally on my freebsd  
boxes.

http://people.apache.org/~pgollucci/mp2bug.txt

FWIW, I do this nearly weekly, so I'm not really expecting it to  
fail, but hey first time for everything.


Works like a charm though a few tests fail b/c its the event mpm as  
expected.

I stand by what I said before

  1) clean up your perl lib tree
  2) verify /usr/bin/perl is correctly symlinked to /usr/local/bin/ 
perl

Info about mp2 and threaded MPMs

[craig]

I'm about to try a threaded MPM for my largely mod_perl2-based web site.

Maybe my search skills are going downhill, but I haven't found much
material about how to modify my scripts for the threaded environment.

The first step is done, namely to minimize use of global variables.  I'm
pretty sure that the few that remain need to be marked as shared and
in some cases need to have mutual exclusion zones around their use.

It's confusing that Apache2 and perl5.8 have separate mutual-exclusion
mechanisms.  For no good reason my instinct is to use the Apache2
mutexes.

Comments on the mutual exclusion alternatives will be welcome, as will
suggestions for printed or online books, tutorials, and other words  
about

threading and mod_perl2.

Thanks and HNY,
cmac
www.animalhead.com

Re: Info about mp2 and threaded MPMs

[André Warnier]

cr...@animalhead.com wrote:
[...]
My own not very reliable 2 cent :

The mod_perl 2 User's Guide (Pub: Onyx Neon, authors: Stas Beckman and 
Jim Brandt) has apparently only part of 2 pages (365-366) on the subject.
Also, the Suse Enterprise Linux 10.x system that I recently installed 
for a customer has a worker (threaded) Apache2 + mod_perl2 installed by 
default.
None of the above constitute firm recommendations, but in the absence of 
forecasts of doom, I would tend to see these as encouraging signs.


Apart of the usual warnings about the underlying thread-safety of 
underlying C libraries, there are apparently a couple of potentially 
nasty side-effects, such as if you would be using chdir() in some of 
your modules (because a chdir in one thread affects all the others).

There is a reference in the book to this, which I have not checked myself :
http://www.perl.com/lpt/2002/06/11/threads.html

Re: Info about mp2 and threaded MPMs

[mackenna]

One of the fun aspects of this is that the dark-blue-going-
on-purple book (which I have) describes how global variables
are sometimes shared between threads if they're not declared
shared, while your linked page says only data that is
explicitly requested to be shared will be shared between
threads.

I think both are correct within their frame of reference,
the latter page is not talking about mod_perl2 nor threads
created by Apache2.

I used to like new subjects to learn about...
cmac


On Dec 29, 2008, at 2:37 PM, André Warnier wrote:


cr...@animalhead.com wrote:
[...]
My own not very reliable 2 cent :

The mod_perl 2 User's Guide (Pub: Onyx Neon, authors: Stas Beckman  
and Jim Brandt) has apparently only part of 2 pages (365-366) on  
the subject.
Also, the Suse Enterprise Linux 10.x system that I recently  
installed for a customer has a worker (threaded) Apache2 +  
mod_perl2 installed by default.
None of the above constitute firm recommendations, but in the  
absence of forecasts of doom, I would tend to see these as  
encouraging signs.


Apart of the usual warnings about the underlying thread-safety of  
underlying C libraries, there are apparently a couple of  
potentially nasty side-effects, such as if you would be using chdir 
() in some of your modules (because a chdir in one thread affects  
all the others).
There is a reference in the book to this, which I have not checked  
myself :

http://www.perl.com/lpt/2002/06/11/threads.html

Re: preserving request body across redirects

[Mark Hedges]

On Mon, 29 Dec 2008, David Ihnen wrote:

 Though that is, of course, specific to the application
 being programmed, utilizing client-side javascript active
 stuff rather than particular web server programming to
 transparently handle it on the server side using basic
 html2.0 type structure.

 I have to agree with others that a whole proxying layer to
 allow it seems... excessive.

I don't know who's recommending a proxying layer or who's
agreeing with you... are you aware of how OpenID works or
what it is?

Did you understand that the Apache2::Controller framework is
intended to be a general application framework, and should
not require you to implement pages in a certain way?

Mark

Re: preserving request body across redirects

[David Ihnen]

Mark Hedges wrote:

On Mon, 29 Dec 2008, David Ihnen wrote:
  

Though that is, of course, specific to the application
being programmed, utilizing client-side javascript active
stuff rather than particular web server programming to
transparently handle it on the server side using basic
html2.0 type structure.

I have to agree with others that a whole proxying layer to
allow it seems... excessive.



I don't know who's recommending a proxying layer or who's
agreeing with you... are you aware of how OpenID works or
what it is?
  
I think that any system that stores a post for later retrieval on a 
separate request is, indeed, proxying that request for later use in a 
different situation.  Thus, to program your system to hold onto a form 
put/post amounts to inserting a proxy-delay-for-authentication for that 
action - even if semantically you want to say its not a proxy because 
its not a separate application/codebase.  I admit to introducing the 
term proxy here without explaining why I consider the concept to be proxy.


Yes, I am aware of how OpenID works.  And it works in-band unless the 
application explicitly sidelines it - there is no inherent side-band 
communication that the client and server will use - otherwise, you 
wouldn't EVER do a main state redirect.  The moment you have to redirect 
to that openid server page, you have sidelined the entire stream of 
browser-server communication - and as you have found in the problem 
you're trying to solve - the state inherent therein, including the 
content of the original request.  Is the utilization of the stored form 
data going to be through a different connection/request entirely after 
authentication verification?  Would require some tests to see if the 
client behaves that way or not.  I suspect its not defined to be one way 
or the other, but I may be wrong.

Did you understand that the Apache2::Controller framework is
intended to be a general application framework, and should
not require you to implement pages in a certain way?
  
Not explicitly, but implicitly I would expect that, yes. 

Is the controller framework going to require me to depend my web system 
operations on some sort of semi-persistent pan-server correlated session 
state system?  Would that not be requiring me to implement my web 
application in a particular way?  Okay, that may indeed be the role of a 
framework though I'd no doubt chafe at the limitations myself.  If I 
have to write my web application a certain way, is it so unusual to have 
my pages need to interact with that application a certain way?  They're 
almost inevitably closely coupled.


This is a fairly sticky issue - if you have run out of local 
authentication token, its impolite to drop data they were submitting.  
But on the other hand, there's no particularly good way of *not* 
dropping it - you can't really handle information if they're not 
authenticated for it.  And out of pure defensive network traffic 
handling, we do the absolute minimum for people who aren't authenticated 
- so they can't consume our system resources, be that 
posts-to-sessions-that-don't-exist or what.  I can see programming the 
client side of the web application to handle this kind of 
token-loss-recovery automatically - the client has the form state and 
being able to recover the session state is valuable, and entirely 
independent from the framework utilized.  But I'm not convinced that the 
web server should be jumping through hoops/proxies to make it happen.  
(not that you have to convince me, I'm trying to present a perspective 
that may be novel and generally be helpful in improving your software, 
and we may just disagree on the role of the software involved)


David