Re: generic question on input filters
On Thu 09 Apr 2009, André Warnier wrote: Suspicion : I cannot read the content, and print nothing when I am called. Instead of accumulating the content and process it all at once at the end, should I process it chunk by chunk ? yes, as I said in my previous mail, do it chunkwise. Is it OK then when the Content-Length header is wrong (since I am changing the size of the data) ? As long as your handler does not rely on it you can ignore the Content-Length header. In fact, a request can also be sumbitted in Transfer-Encoding: chunked which means there is no CL header. The end of request body is determined by the HTTP_IN filter (ap_http_filter() in httpd/modules/http/http_filters.c) which is a protocol filter. That means it comes before your filter. Torsten -- Need professional mod_perl support? Just hire me: torsten.foert...@gmx.net
Re: generic question on input filters
Torsten Foertsch wrote: On Thu 09 Apr 2009, André Warnier wrote: Suspicion : I cannot read the content, and print nothing when I am called. Instead of accumulating the content and process it all at once at the end, should I process it chunk by chunk ? yes, as I said in my previous mail, do it chunkwise. Ok, I will change my filter to do it chunk by chunk. It is quite a pain though, as I have to look for '' boundaries between query parameters, to make sure I don't cut in the middle of a %-escaped sequence. I have the impression that I am getting this Apache error : [Wed Apr 08 22:46:22 2009] [crit] [Wed Apr 08 22:46:22 2009] file http_filters.c, line 1324, assertion !bb)-list))-next == (struct apr_bucket *)((char *)(((bb)-list)) - ((long) (((char *) struct apr_bucket*)((void *)0))-link))) - ((char *) ((void *)0)) failed [Wed Apr 08 22:46:22 2009] [notice] child pid 5518 exit signal Aborted (6) when I exceed some size in what I save with $f-ctx($data). (say with $data around 1100 bytes ?). Is that so ?
Re: generic question on input filters
On Thu 09 Apr 2009, André Warnier wrote: I have the impression that I am getting this Apache error : [Wed Apr 08 22:46:22 2009] [crit] [Wed Apr 08 22:46:22 2009] file http_filters.c, line 1324, assertion !bb)-list))-next == (struct apr_bucket *)((char *)(((bb)-list)) - ((long) (((char *) struct apr_bucket*)((void *)0))-link))) - ((char *) ((void *)0)) failed [Wed Apr 08 22:46:22 2009] [notice] child pid 5518 exit signal Aborted (6) Perhaps it's easier to implement a brigade based filter since this comes from here: rv = ap_get_brigade(r-input_filters, bb, AP_MODE_READBYTES, APR_BLOCK_READ, bufsiz); ... /* If this fails, it means that a filter is written incorrectly and that * it needs to learn how to properly handle APR_BLOCK_READ requests by * returning data when requested. */ AP_DEBUG_ASSERT(!APR_BRIGADE_EMPTY(bb)); Meaning your filter hasn't written anything. Torsten -- Need professional mod_perl support? Just hire me: torsten.foert...@gmx.net
Re: generic question on input filters
Torsten Foertsch wrote:
On Thu 09 Apr 2009, André Warnier wrote:
I have the impression that I am getting this Apache error :
[Wed Apr 08 22:46:22 2009] [crit] [Wed Apr 08 22:46:22 2009] file
http_filters.c, line 1324, assertion !bb)-list))-next ==
(struct apr_bucket *)((char *)(((bb)-list)) - ((long) (((char *)
struct apr_bucket*)((void *)0))-link))) - ((char *) ((void
*)0)) failed [Wed Apr 08 22:46:22 2009] [notice] child pid 5518
exit signal Aborted (6)
Perhaps it's easier to implement a brigade based filter since this comes
from here:
rv = ap_get_brigade(r-input_filters, bb, AP_MODE_READBYTES,
APR_BLOCK_READ, bufsiz);
...
/* If this fails, it means that a filter is written incorrectly and that
* it needs to learn how to properly handle APR_BLOCK_READ requests by
* returning data when requested.
*/
AP_DEBUG_ASSERT(!APR_BRIGADE_EMPTY(bb));
Meaning your filter hasn't written anything.
Hmmm.
So I cannot return nothing at any filter invocation.
Is there some hack allowing me to return something that will satisfy
Apache, without introducing input data I don't want ?
(sending like a null string brigade, but brigade anyway ?).
Can I for instance do a $f-print('') ?
On the input side, I get a URI-escaped query string, like
key1=FOO%D3D+I+SALUT+ARMATNI%3B%3B%3A%3Akey2=BAR%D3D+I+SALUT+ARMATNI%3B%3B%3A%3A
(but much longer).
I want to unescape this data, replace some of the characters, then
re-escape the data and forward it to the application.
Presumably, the input could be split over 2 bucket brigades, the first
one ending at some % or %3, leaving one trailing escape sequence
incomplete.
I was trying to avoid that, by only processing complete query
parameters key=value (which is also nicer for logging), but that of
course can lead to the case where I have a brigade like
key1=FOO%D3D+I+SALUT+ARMATNI%3
(with a yet-incomplete parameter, but nothing before it that I can process).
In that case, I was saving the string for later in $f-ctx, and
forwarding nothing this time around.
Looks like I'll have to drop the idea of processing complete query
parameters, and switch to processing complete escape sequences instead.
Or else, do a print of some dummy parameter which I know the application
will ignore. But that is a really application-specific hack.
Re: generic question on input filters
On Wed 08 Apr 2009, André Warnier wrote:
I want to write a mod_perl input filter, which essentially would
replace some characters from form-posted input on the fly, before
letting the request go to the real application.
The requests are POST, with content encoded in one long string,
URL-encoded. I am thinking of writing this as a FilterRequestHandler
filter, grabbing the content, splitting it on , URL-decoding the
so-splitted query parameters individually, then in the value
performing the character substitutions, then re-URL-encoding the
query parameters, re-joining them with , and essentially et voilà.
Basically, the type shown here :
http://perl.apache.org/docs/2.0/user/handlers/filters.html#toc_Stream
_oriented_Input_Filters
Is the above a realistic view, or am I missing something essential ?
Remember the request can consist of multiple brigades. Each filter
invocation gets one brigade. That means you have to be prepared to
store some context (store the /[^;]*$/ part in $f-ctx). Don't try to
read in the whole request. Operate on each part of it.
For example try the following PerlInputFilterHandler:
sub handler {
my ($f)=...@_;
$f-print(\n###\n);
my $buf;
while($f-read($buf, 1024)) {
$f-print(\n$buf\n);
undef $buf;
}
$f-print(\n:::\n);
return Apache2::Const::OK;
}
with this PerlResponseHandler:
sub handler {
my ($r)=...@_;
$r-content_type('text/plain');
my $buf;
while($r-read($buf, 15000)) {
$r-print($buf);
undef $buf;
}
return Apache2::Const::OK;
}
and feed it with more than 8000 bytes:
perl -e 'print xx1' | curl -qs -d @- http://localhost/iftest
You'll see multiple blocks wrapped in ... from the filter loop.
But you'll also see 2 blocks of # ... :: that mark the
filter invocations.
BTW, the buffer size in the response handler is that large because is
must be able to store a whole brigade's content due to a modperl bug,
see XXX: comment in modperl_request_read(). So, don't $f-print much
in one filter invocation.
Torsten
--
Need professional mod_perl support?
Just hire me: torsten.foert...@gmx.net
Re: generic question on input filters
Torsten Foertsch wrote: [...] Thanks for the previous answer and tips. I have written and tried the filter, and now I have a problem (at the end). Anyone knows what that means ? Suspicion : I cannot read the content, and print nothing when I am called. Instead of accumulating the content and process it all at once at the end, should I process it chunk by chunk ? Is it OK then when the Content-Length header is wrong (since I am changing the size of the data) ? main server error log: [Wed Apr 08 22:42:37 2009] [notice] Apache/2.2.3 (Linux/SUSE) mod_jk/1.2.28-dev-727724 mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations VirtualHost error log (logging by the filter, a few runs that seem to work fine, then suddenly the error. Sorry for the clutter, that's the html form sending stuff): [Wed Apr 08 22:43:22 2009] [warn] [client 10.13.16.93] FSWB: 1st call, referer: http://host.customer.com/starweb/wideoteka/servlet.starweb [Wed Apr 08 22:43:22 2009] [warn] [client 10.13.16.93] FSWB: buffer: [__sessionNumber=67__pageid=daneIndeksu__a1=__a2=__a3=__hiddenstyle=A__dirtyFlag=undefined__action=216Near=SelectedIndexTerms=__05.12.1971%3B%3B%3A%3ASelectedIndexTerms=__05.1970%3B%3B%3A%3ASelectedIndexTerms=__06+SIERPNIA+2008+R.%3B%3B%3A%3ASelectedIndexTerms=__06+SIERPNIA+2008+R.+-+SKR%D3T%3B%3B%3A%3ASelectedIndexTerms=__06.02.1972%3B%3B%3A%3ASelectedIndexTerms=__06.04.1969%3B%3B%3A%3ASelectedIndexTerms=__06.05.1968%3B%3B%3A%3ASelectedIndexTerms=__06.06.1971%3B%3B%3A%3ASelectedIndexTerms=__06.10.1965%3B%3B%3A%3ASelectedIndexTerms=__06.11.1970%3B%3B%3A%3ASelectedIndexTerms=__06.12.1968+ROK%3B%3B%3A%3ASelectedIndexTerms=__07+SIERPIE%D1+2008%3B%3B%3A%3ASelectedIndexTerms=__07+SIERPNIA+2008+R.%3B%3B%3A%3ASelectedIndexTerms=__07+ZG%A3O%3F+SI%CA%3B%3B%3A%3ASelectedIndexTerms=__07+ZG%A3O%3F+SI%CA%3B%3B%3A%3A__ReportId=histIndeksu%5E0], referer: http://host.customer.com/starweb/wideoteka/servlet.starweb [Wed Apr 08 22:43:22 2009] [warn] [client 10.13.16.93] FSWB: splitted params: __sessionNumber=67 || __pageid=daneIndeksu || __a1= || __a2= || __a3= || __hiddenstyle=A || __dirtyFlag=undefined || __action=216 || Near= || SelectedIndexTerms=__05.12.1971%3B%3B%3A%3A || SelectedIndexTerms=__05.1970%3B%3B%3A%3A || SelectedIndexTerms=__06+SIERPNIA+2008+R.%3B%3B%3A%3A || SelectedIndexTerms=__06+SIERPNIA+2008+R.+-+SKR%D3T%3B%3B%3A%3A || SelectedIndexTerms=__06.02.1972%3B%3B%3A%3A || SelectedIndexTerms=__06.04.1969%3B%3B%3A%3A || SelectedIndexTerms=__06.05.1968%3B%3B%3A%3A || SelectedIndexTerms=__06.06.1971%3B%3B%3A%3A || SelectedIndexTerms=__06.10.1965%3B%3B%3A%3A || SelectedIndexTerms=__06.11.1970%3B%3B%3A%3A || SelectedIndexTerms=__06.12.1968+ROK%3B%3B%3A%3A || SelectedIndexTerms=__07+SIERPIE%D1+2008%3B%3B%3A%3A || SelectedIndexTerms=__07+SIERPNIA+2008+R.%3B%3B%3A%3A || SelectedIndexTerms=__07+ZG%A3O%3F+SI%CA%3B%3B%3A%3A || SelectedIndexTerms=__07+ZG%A3O%3F+SI%CA%3B%3B%3A%3A || __ReportId=histIndeksu%5E0, referer: http://host.customer.com/starweb/wideoteka/servlet.starweb [Wed Apr 08 22:43:22 2009] [warn] [client 10.13.16.93] FSWB: new content: [__sessionNumber=67__pageid=daneIndeksu__a1=__a2=__a3=__hiddenstyle=A__dirtyFlag=undefined__action=216Near=SelectedIndexTerms%3D__05.12.1971%3B%3B%3A%3ASelectedIndexTerms%3D__05.1970%3B%3B%3A%3ASelectedIndexTerms%3D__06%2BSIERPNIA%2B2008%2BR.%3B%3B%3A%3ASelectedIndexTerms%3D__06%2BSIERPNIA%2B2008%2BR.%2B-%2BSKROT%3B%3B%3A%3ASelectedIndexTerms%3D__06.02.1972%3B%3B%3A%3ASelectedIndexTerms%3D__06.04.1969%3B%3B%3A%3ASelectedIndexTerms%3D__06.05.1968%3B%3B%3A%3ASelectedIndexTerms%3D__06.06.1971%3B%3B%3A%3ASelectedIndexTerms%3D__06.10.1965%3B%3B%3A%3ASelectedIndexTerms%3D__06.11.1970%3B%3B%3A%3ASelectedIndexTerms%3D__06.12.1968%2BROK%3B%3B%3A%3ASelectedIndexTerms%3D__07%2BSIERPIEN%2B2008%3B%3B%3A%3ASelectedIndexTerms%3D__07%2BSIERPNIA%2B2008%2BR.%3B%3B%3A%3ASelectedIndexTerms%3D__07%2BZGLO%3F%2BSIE%3B%3B%3A%3ASelectedIndexTerms%3D__07%2BZGLO%3F%2BSIE%3B%3B%3A%3A__ReportId%3DhistIndeksu%5E0], referer: http://host.customer.com/starweb/wideoteka/servlet.starweb [Wed Apr 08 22:45:35 2009] [warn] [client 10.13.16.93] FSWB: 1st call, referer: http://host.customer.com/starweb/wideoteka/servlet.starweb [Wed Apr 08 22:45:35 2009] [warn] [client 10.13.16.93] FSWB: buffer:
