Re: mod_proxy_fcgi and flush

[Nick Kew]

On Thu, 2017-07-06 at 14:08 -0400, Helmut K. C. Tessarek wrote:
> One of the comments on the documentation page of mod_proxy_fcgi
> (http://httpd.apache.org/docs/2.4/mod/mod_proxy_fcgi.html) mentions an
> issue with flush:
> 
> There is just no flush support it seems. I attempt to use PHP flush()
> and it won't work until you fill up a buffer first, rendering Server
> Sent Events impossible with proxy_fcgi.

Do we have any idea what (if anything) proxy_fcgi receives from a
PHP flush()?  Your fix could be as simple as propagating an event.

> > In a perfect world, I'm right there with you, but we've seen (as long as
> > technology has existed) that people twist the way how technology is
> > applied and used. It's hard to convince those people otherwise,
> > especially when most of the time it has been possible to get it to work

That takes me back ...
https://www.theregister.co.uk/2007/08/24/everything_over_http/

> > > It probably makes sense to work on a nonblocking architecture for
> > > proxied responses in general.

Is that really the issue in the first place?  We have a concept of
flushing, and can implement more finely-tuned throughput than merely
blocking vs non-blocking.  The point at issue is for PHP to communicate
a flush with proxy_fcgi, and for proxy_fcgi then to honour it.
It seems one or both of those things isn't happening.

> > I'm not familiar with that particular code, but would be interested in
> > looking into it. Does anybody volunteer as a mentor?

Best mentor is this list.  Or #httpd-dev on Freenode (IRC),
if someone's paying attention there.

> lookup https://sks-keyservers.net/i for KeyID 0xC11F128D

Please update that: it's too easy to spoof.
See https://evil32.com/ , or my blog article at
https://bahumbug.wordpress.com/2017/04/27/pretty-good-phishing/

-- 
Nick Kew

Re: [VOTE] Release httpd-2.2.34

[Eric Covener]

On Thu, Jul 6, 2017 at 3:33 PM, William A Rowe Jr  wrote:
> +/-1
> [ ] Release 2.2.34 as legacy GA
>
+1 aix/xlc/ppc64 100% pass

> +/-1
> [ ] Retire the 2.2.x branch from any further maintenance.

+1


-- 
Eric Covener
cove...@gmail.com

Re: mod_proxy_fcgi and flush

[Helmut K. C. Tessarek]

Hi Luca,

Thanks for looking into this.

On 2017-07-08 03:51, Luca Toscano wrote:
> I checked mod_fcgi as Helmut suggested and it seems to me that the
> -flush feature is a simple "flush every data that you receive", so I
> tested the following patch with Jacob's php example code and it seems
> doing what Helmut asked (please correct me if I am wrong).

No, you are correct. But I was merely referring to the comment on the
documentation page.
But yes, this seems to mirror the "-flush" option that is present for
FastCgiExternalServer.

> Caveat: I had to set output_buffering = Off in my php-fpm's php.ini
> config file.

This should be ok, since people who are using output buffering are
usually aware that it can influence the app's behaviour. I think that
was even stated in the PHP manual somewhere.
Although I also remember that there was a section that states that both
ob_flush() and flush() have to be called to make flush work with output
buffering on.

> So if what I wrote vaguely makes sense, we might think about adding a
> new directive that enables explicit flushing to mod_proxy_fcgi, so
> admins can twist its behavior if needed?

As far as I can tell it makes sense, but I haven't gone through the
entire proxy code yet.

Are you thinking of a directive like ProxyFCGIFlush or an option to the
proxy worker:




> Completely agree with Jacob, this use case might not be the best one for
> HTTP :)

As mentioned before, in a perfect world I'd agree. But this option was
available with FastCgiExternalServer, thus it should be available with
proxy_fcgi as well, especially when proxy_fcgi is the preferred way for
Apache >= 2.4.

Cheers,
 K. C.

-- 
regards Helmut K. C. Tessarek
lookup https://sks-keyservers.net/i for KeyID 0xC11F128D

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/

Re: [VOTE] Release Apache httpd 2.4.27 as GA

[Gregg Smith]

On Jul 6, 2017, at 1:45 PM, Jim Jagielski  wrote:

The pre-release test tarballs for Apache httpd
version 2.4.27 can be found at the usual place:

http://httpd.apache.org/dev/dist/

I'm calling a VOTE on releasing these as Apache httpd 2.4.27 GA.


+1 Windows 7/8.1/2012R2, x86 & 64 VC9, 11, 14, VC15 x64
APR 1.6.2 APU 1.6.0
OpenSSL 1.0.2l VC9, 11 & 14
OpenSSL 1.1.0f (VC14 & 15)
brotli 0.6.0, Expat 2.2.1

Re: [VOTE] Release Apache httpd 2.4.27 as GA

[Jacob Champion]

[x] +1: Good to go


Ubuntu 16.04 x64, test suites for httpd and mod_websocket 0.1.1 with

- APR 1.5.2, APR-Util 1.5.4
- OpenSSL 1.0.2g (Debian)
- event, prefork, worker
- no brotli, no mod_php

--Jacob

Re: [VOTE] Release httpd-2.2.34

[Jacob Champion]

On 07/06/2017 12:33 PM, William A Rowe Jr wrote:

[ ] Release 2.2.34 as legacy GA


+1 Ubuntu 16.04, worker/prefork and mod_websocket 0.1.1.

[ ] Retire the 2.2.x branch from any further maintenance.


+1

--Jacob

Re: [VOTE] Release Apache httpd 2.4.27 as GA

[Luca Toscano]

2017-07-06 19:45 GMT+02:00 Jim Jagielski :

> The pre-release test tarballs for Apache httpd
> version 2.4.27 can be found at the usual place:
>
> http://httpd.apache.org/dev/dist/
>
> I'm calling a VOTE on releasing these as Apache httpd 2.4.27 GA.
>

+1 on Debian Stretch

All green from the test suite with --enable-modules=all/most and
pre-installed APR (1.5.2-5).

(Will try to make more tests over the weekend)

Luca

Re: httpd 2.4.26 with apr 1.6 ExtFilterDefine

[Steffen]

Received more details:

Our client/server system needs encrypting to client. Almost files are 
statically encrypt, but some xml need to

modify by SSI then encrypt. So I made encrypting file
filter by exe (not Apache module). Its name is encect.exe
(x86) and it is using msvcrt.dll. Because encect.exe works
all Apache HTTP Server (x86/x64) which is enabled built-in
mod_ext_filter module.

(Details)

Some directory uses SYMLINKDed directory. Apache HTTP
Server too. Enabed symlink by .htaccess file in parent of
symlink.

  Options +FollowSymLinks

The modification & encrypt SSI template xml file are in
the above symlinked directory.

Apache HTTP Server's httpd.conf defines like follow.

ExtFilterDefine ENCECT mode=output cmd=modules/encect.exe
Alias /WebRoot/ "/Path/To/WebRoot/"

Options Indexes MultiViews IncludesNOEXEC
AddOutputFilter INCLUDES m3s inc ini
AddOutputFilter INCLUDES;ENCECT xmlpp
AllowOverride All
Require all granted


The xmlpp file size is 3,295 bytes. written in Shift_JIS.
Used SSI tag is three  like
follow.

  /...

Client access to xmlpp file, the encect.exe process is
remains in service. I'm memory dump encect.exe and load to
WinDbg. Call stack is ReadFile(), it means encect.exe
waiting stdin, but Apache does not send any. Kill the
encect.exe, client receives body from first SSIed parsed
text. (Loose until first SSI tag.)

I tested INCLUDE;ENCECT to..
(A) INCLUDES only, client receives plain complete SSI
parsed xmlpp file. OK.
(B) ENCECT only, client receives responce header without
body. NG. This must encrypt xmlpp file.

Next, I tested mod_ext_filter.so to 2.4.25, but still NG.

Next, I read Apache HTTP Server Change log, I found
mod_filter changes. I tested mod_filter.so and
mod_ext_filter.so to 2.4.25, NG.

Next, I tested httpd.exe to 2.4.25, NG.

Next, I read Apache Lounge's used modules, I found APR is
upgraded to 1.6.2 from 1.5.x. So I tested libapr-1.dll,
libaprutil-1.dll to 2.4.25, OK! Works like 2.4.25 and
earlier version of Apache HTTP Server.




On Friday 07/07/2017 at 20:13, William A Rowe Jr  wrote:

On Fri, Jul 7, 2017 at 7:13 AM, Jim Jagielski  wrote:


2.4.26/27 doesn't *require* APR/APU 1.6.x, but there are
some features that depend on it. If it's a bug in apr 1.6.x,
then it's not a httpd bug specifically... imo at least.

any further detail on how the below is actually borken??
What happens?



On Jul 7, 2017, at 7:41 AM, Steffen  wrote:


I got the following report. Is this known ?


Because apache http server all of 2.4.26 (vc11,vc14,vc15)
was not work ExtFilterDefine & OutputFilter. Its bug
exists in apr 1.6. I thought it need to inform.

Apache 2.4.26 changes apr 1.6, but it broke
ExtFilterDefine & OutputFilter.
(test) copy apache 2.4.25's libapr-1 & libaprutil to
apache 2.4.26, they worked correctly like before apaches.


Yes, we need the actual error messages.

I'm about ready to T apr 1.6.1 / apu 1.6.3 for the various fixes 
already
present, but if we can fix something specific that we are unaware 
of...

Re: mod_proxy_fcgi and flush

[Luca Toscano]

Hi Jacob, Helmut!

2017-07-06 20:54 GMT+02:00 Jacob Champion :

> On 07/06/2017 11:13 AM, Jim Jagielski wrote:
>
>> works 4 me...
>>
>
> Doesn't for me. E.g. with a script like
>
>print("hi!\n")
>   flush();
>   sleep(1);
>   print("hi!\n");
> ?>
>
> it takes 1 second to receive a single chunk with both lines in it.
>
> From a quick skim I assume this is because we don't use nonblocking
> sockets in the proxy implementation. (There's even a note in mod_proxy_fcgi
> that says, "Yes it sucks to [get the actual data] in a second recv call,
> this will eventually change when we move to real nonblocking recv calls.")


Quick check from my side too, so not sure if it makes sense. IIUC the
comment is about getting all the headers from the FCGI backend
(get_data_full(..., AP_FCGI_HEADER_LEN)), then get the response body in
another one (the [actual data]).

I checked mod_fcgi as Helmut suggested and it seems to me that the -flush
feature is a simple "flush every data that you receive", so I tested the
following patch with Jacob's php example code and it seems doing what
Helmut asked (please correct me if I am wrong).

Caveat: I had to set output_buffering = Off in my php-fpm's php.ini config
file.

http://home.apache.org/~elukey/httpd-2.4.x-mod_proxy_fcgi-force_flush.patch

I am still not sure if the get_data function (responsible to read the body
after the headers in dispatch()) works in this way only my local test or
not, but it seems simply calling apr_socket_recv with a buffer len and
return whatever data it comes from the backend. IIUC apr_socket_recv should
return data even if less than the buffer specified..

So if what I wrote vaguely makes sense, we might think about adding a new
directive that enables explicit flushing to mod_proxy_fcgi, so admins can
twist its behavior if needed?



> On 07/06/2017 11:08 AM, Helmut K. C. Tessarek wrote:
> > What is your take on this?
>
> If I'm honest, my brutally blunt take on it is "stop using HTTP to try to
> emulate push notifications within a single response; pretty much everything
> in the ecosystem is actively working against you at this point; responses
> are designed to be cacheable and deliverable as a unit, not as multiple
> pieces; and we've had *real* solutions like WebSocket for five years now
> rabble rabble rabble." But I don't actually think that's going to be the
> accepted answer.
>
> It probably makes sense to work on a nonblocking architecture for proxied
> responses in general.
>
> -Jacob
>

Completely agree with Jacob, this use case might not be the best one for
HTTP :)

Luca