Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl

[Jacob Champion]

On 11/11/2016 11:20 AM, Jacob Champion wrote:

I suspect that these failures have nothing to do with this patch, but I
just want to confirm that before it goes into trunk.


Yeah, the failures show up with or without the patch, so I added a 
couple comments and committed.


The SSL failures in the trunk test suite are going to make this 
difficult to test fully; has anyone seen them before or already have a 
fix tucked away privately?


--Jacob

Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl

[Jacob Champion]

On 11/11/2016 10:15 AM, William A Rowe Jr wrote:

The patch looks reasonable to me. As you point out, fixing this in APR
might be worthwhile, but doesn't protect mod_ssl from other OpenSSL
consuming logic such as OpenLDAP auth, database connections or
other plug-ins that may leave a lingering SSL_error state lying about.


Looks good to me too -- thanks for the bump, Paul.

I haven't committed this yet because of some SSL-related test failures 
in our suite when running against httpd-trunk:


- mod_session_crypto appears to be segfaulting the server for some reason
- mod_proxy tests hang the suite when run in -ssl test mode, and httpd 
complains about malformed request lines


I suspect that these failures have nothing to do with this patch, but I 
just want to confirm that before it goes into trunk.


--Jacob

Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl

[William A Rowe Jr]

On Fri, Nov 11, 2016 at 9:01 AM, Paul Spangler  wrote:

> On 10/17/2016 2:04 PM, Paul Spangler wrote:
>
>> Hello,
>>
>> Due to the way OpenSSL stores errors in a per-thread queue, functions
>> such as SSL_read followed by SSL_get_error may not produce the desired
>> result if the error queue is not empty prior to calling SSL_read[1]. For
>> example, a non-blocking read reports that no data is available by
>> setting up SSL_get_error to return SSL_ERROR_WANT_READ. But if an error
>> is already present in the queue, SSL_get_error sees that error instead
>> and returns SSL_ERROR_SSL.
>>
>> I found at least one case where the error queue may be non-empty prior
>> to a non-blocking read[2] that involves combining mod_session_crypto
>> (which leaves the error queue non-empty) and the third-party
>> mod_websocket (which uses non-blocking reads), resulting in the
>> connection being closed. I included a potential patch to mod_ssl for
>> consideration on the bug report that simply clears the error queue prior
>> to any of the three SSL_* calls that mod_ssl makes. An ideal fix might
>> be to keep the error queue empty at all times (i.e. patch the APR crypto
>> library), but I propose that this patch is more robust in a modular
>> environment.
>>
>> Thanks for your consideration.
>>
>> [1]
>> https://www.openssl.org/docs/manmaster/ssl/SSL_get_error.html#DESCRIPTION
>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=60223
>>
>> Anyone have any thoughts on this small patch? It addresses an issue with
> OpenSSL's per-thread state causing connections to fail.
>
>
> Regards,
> Paul Spangler
> LabVIEW R
> National Instruments
>


The patch looks reasonable to me. As you point out, fixing this in APR
might be worthwhile, but doesn't protect mod_ssl from other OpenSSL
consuming logic such as OpenLDAP auth, database connections or
other plug-ins that may leave a lingering SSL_error state lying about.

Re: mod_ftp segaults on rheloids

[William A Rowe Jr]

On Fri, Nov 11, 2016 at 2:24 AM, Benjamin Lefoul 
wrote:

> Is this list still active?
> Maybe it was not the right place to ask about mod_ftp?
>

It is the place to ask, and was on my list to respond to your question...
sorry I'm overwhelmed with other code this week.

The last 'release' is quite dated, and several things changed within
httpd 2.4 that prove incompatible with the last official mod_ftp 0.9.6
release. I did try to push out another release, but it never got enough
votes to be 'released' by the community, and since has undergone
a number of bug fixes to work with the current httpd 2.4.23 (although
it should still be compatible with an older httpd release like 2.4.6.)

http://svn.apache.org/repos/asf/httpd/mod_ftp/trunk/
is the repository of the current source code. The primary defect I'm
aware of in an incompatibility with mod_ftp and the httpd core that
prevents us from successfully serving FTP Explicit SSL connections.
An implicit SSL connection, e.g. ftps on port 990 (using TLS from the
time of connecting) is functional.

Once other httpd projects are wrapped up, I'm coming back to httpd
2.4 and mod_ftp to figure out how to get Explicit SSL functioning again.
If you would grab the latest development code I'd love to read about
your success or any defects you encounter.

Cheers,

Bill

Re: [PATCH 60223] non-empty OpenSSL error queue preventing non-blocking reads through mod_ssl

[Paul Spangler]

On 10/17/2016 2:04 PM, Paul Spangler wrote:

Hello,

Due to the way OpenSSL stores errors in a per-thread queue, functions
such as SSL_read followed by SSL_get_error may not produce the desired
result if the error queue is not empty prior to calling SSL_read[1]. For
example, a non-blocking read reports that no data is available by
setting up SSL_get_error to return SSL_ERROR_WANT_READ. But if an error
is already present in the queue, SSL_get_error sees that error instead
and returns SSL_ERROR_SSL.

I found at least one case where the error queue may be non-empty prior
to a non-blocking read[2] that involves combining mod_session_crypto
(which leaves the error queue non-empty) and the third-party
mod_websocket (which uses non-blocking reads), resulting in the
connection being closed. I included a potential patch to mod_ssl for
consideration on the bug report that simply clears the error queue prior
to any of the three SSL_* calls that mod_ssl makes. An ideal fix might
be to keep the error queue empty at all times (i.e. patch the APR crypto
library), but I propose that this patch is more robust in a modular
environment.

Thanks for your consideration.

[1]
https://www.openssl.org/docs/manmaster/ssl/SSL_get_error.html#DESCRIPTION
[2] https://bz.apache.org/bugzilla/show_bug.cgi?id=60223

Anyone have any thoughts on this small patch? It addresses an issue with 
OpenSSL's per-thread state causing connections to fail.


Regards,
Paul Spangler
LabVIEW R
National Instruments

Re: mod_ftp segaults on rheloids

[Benjamin Lefoul]

On Fri, 2016-11-11 at 13:22 +0100, Rainer Canavan wrote:
> On Fri, Nov
11, 2016 at 10:27 AM, Benjamin Lefoul
> 
wrote:
> > 
> > On Fri, 2016-11-11 at 08:53 +, Nick Kew wrote:
> > > 
>
> > Did you build mod_ftp yourself on a distro-provided httpd?
> > Yes.
I did build myself from the latest svn.
> > I installed:
> > httpd-devel-
2.4.6-40.sl7.4.x86_64 from Scientific Linux 7
> > httpd-devel-2.4.6-
40.el7.centos.4.x86_64 on centOS7
> > (so, the same, essentially), as
well as the "Development Tools"
> > group.
> 
> If you have a core dump,
try analyzing it with gdb, otherwise, since
> you can apparently
trivially reproduce the segfault, just attach gdb
> to
> any httpd
process (except maybe the one that has pid 1 as its parent)
> with
> 
>
gdb $(which httpd) 
> 
> it will tell you that lots of debug symbols
are absent, and how to
> run
> debuginfo-install
> to install them. Do
that as instructed, re-run gdb, verify that no
> "symbols missing"
messages
> are printed on startup. Execute "cont" at the gdb prompt and
start
> causing segfaults  again
> until gdb returns to the prompt. Do a
"bt full". Not sure if httpd-
> dev
> is the right place, it
> might be a
bug in CentOS and/or the ancient httpd they're using.
> 

Thanks for your help and patience. I wasn't able to get the prompt back
so after a while I just C-c'd. Any comment?
This is what I get:

(gdb) cont
Continuing.
Detaching after fork from child process 13499.
Detaching after fork from child process 13500.
Detaching after fork from child process 13501.
Detaching after fork from child process 13502.
Detaching after fork from child process 13503.
^C
Program received signal SIGINT, Interrupt.
0x7f598fc773f3 in __select_nocancel () at ../sysdeps/unix/syscall-
template.S:81
81  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt full
#0  0x7f598fc773f3 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x7f599038f585 in apr_sleep (t=t@entry=100) at
time/unix/time.c:246
tv = {tv_sec = 0, tv_usec = 713807}
#2  0x7f59916b1dd1 in ap_wait_or_timeout (status=status@entry=0x7ff
dd22c85f8, exitcode=exitcode@entry=0x7ffdd22c85fc, ret=ret@entry=0x7ffd
d22c8600, p=, s=) at mpm_common.c:195
rv = 
#3  0x7f5986e8a13e in prefork_run (_pconf=,
plog=, s=) at prefork.c:1016
status = 11
pid = {pid = -1, in = 0x7f599308ef58, out = 0x34a934a9, err
= 0x7f599300c340}
child_slot = 
exitwhy = APR_PROC_SIGNAL
processed_status = 
index = 
remaining_children_to_start = 0
rv = 
#4  0x7f59916b15be in ap_run_mpm (pconf=pconf@entry=0x7f5992fe3158,
plog=0x7f5993010378, s=0x7f599300c340) at mpm_common.c:96
pHook = 
n = 0
rv = -1
#5  0x7f59916aab46 in main (argc=2, argv=0x7ffdd22c88f8) at
main.c:777
c = 68 'D'
showcompile = 0
showdirectives = 0
confname = 0x7f59916e68af "conf/httpd.conf"
def_server_root = 0x7f59916e68a4 "/etc/httpd"
temp_error_log = 0x0
error = 
process = 0x7f5992fe1238
pconf = 0x7f5992fe3158
plog = 0x7f5993010378
ptemp = 0x7f599300e368
pcommands = 0x7f5993005268
opt = 0x7f5993005358
rv = 
mod = 0x7f5991904098 
opt_arg = 0x7ffdd22c9f7d "FOREGROUND"
signal_server = 




And the error log:

[Fri Nov 11 12:50:45.402984 2016] [mpm_prefork:notice] [pid 13481]
AH00163: Apache/2.4.6 (CentOS) mod_ftp/1.0.1-dev OpenSSL/1.0.1e-fips
mod_fcgid/2.3.9 configured -- resuming normal operations
[Fri Nov 11 12:50:45.403003 2016] [core:notice] [pid 13481] AH00094:
Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Nov 11 12:50:45.427847 2016] [ftp:notice] [pid 13482] FTP low
numbered port daemon waiting for port requests
[Fri Nov 11 12:51:26.868223 2016] [core:notice] [pid 13481] AH00052:
child pid 13484 exit signal Segmentation fault (11)
[Fri Nov 11 12:51:56.902013 2016] [core:notice] [pid 13481] AH00052:
child pid 13485 exit signal Segmentation fault (11)
[Fri Nov 11 12:52:15.923595 2016] [core:notice] [pid 13481] AH00052:
child pid 13486 exit signal Segmentation fault (11)
[Fri Nov 11 12:52:22.932293 2016] [core:notice] [pid 13481] AH00052:
child pid 13487 exit signal Segmentation fault (11)
[Fri Nov 11 12:52:25.935810 2016] [core:notice] [pid 13481] AH00052:
child pid 13488 exit signal Segmentation fault (11)

Re: mod_ftp segaults on rheloids

[Benjamin Lefoul]

On Fri, 2016-11-11 at 08:53 +, Nick Kew wrote:
> On Fri, 2016-11-11 at 08:24 +, Benjamin Lefoul wrote:
> > 
> > Is this list still active?
> > Maybe it was not the right place to ask about mod_ftp?
> 
> I guess noone picked up the baton.  Too many blanks to fill.
> 
> Did you build mod_ftp yourself on a distro-provided httpd?
Yes. I did build myself from the latest svn.
I installed:
httpd-devel-2.4.6-40.sl7.4.x86_64 from Scientific Linux 7
httpd-devel-2.4.6-40.el7.centos.4.x86_64 on centOS7
(so, the same, essentially), as well as the "Development Tools" group.

> What toolchain did you use for that?
> What happens if you build everything yourself,
I'd rather focus on trying to get it to work on the distro provided
apache for this week (I already have some services neatly set there).
(I'll append here if I can clear some time to try later this month.)

> or alternatively use everything from the distro?
> Where's the traceback from your segfault?
> 
Can you help me with this? I guess I should run gdb and attach the
correct process, but how do I set the debug flags?

Thank you very much,

Benjamin

Re: mod_ftp segaults on rheloids

[Nick Kew]

On Fri, 2016-11-11 at 08:24 +, Benjamin Lefoul wrote:
> Is this list still active?
> Maybe it was not the right place to ask about mod_ftp?

I guess noone picked up the baton.  Too many blanks to fill.

Did you build mod_ftp yourself on a distro-provided httpd?
What toolchain did you use for that?
What happens if you build everything yourself,
or alternatively use everything from the distro?
Where's the traceback from your segfault?

-- 
Nick Kew

Re: mod_ftp segaults on rheloids

[Benjamin Lefoul]

Is this list still active?
Maybe it was not the right place to ask about mod_ftp?

Benjamin


On Tue, 2016-11-08 at 09:36 +0100, Benjamin Lefoul wrote:
> Hi,
> 
> I have built mod_ftp on fresh RHELoids (tried on both CentOS7 and
> Scientific Linux 7).
> Whether I try the default conf or my own, the module always
> immediately segfaults:
> 
> [Tue Nov 08 08:15:52.798201 2016] [suexec:notice] [pid 1225] AH01232:
> suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
> [Tue Nov 08 08:15:52.890162 2016] [auth_digest:notice] [pid 1225]
> AH01757: generating secret for digest authentication ...
> [Tue Nov 08 08:15:52.891402 2016] [lbmethod_heartbeat:notice] [pid
> 1225] AH02282: No slotmem from mod_heartmonitor
> [Tue Nov 08 08:15:52.898709 2016] [ftp:notice] [pid 1333] FTP low
> numbered port daemon waiting for port requests
> [Tue Nov 08 08:15:52.914261 2016] [mpm_prefork:notice] [pid 1225]
> AH00163: Apache/2.4.6 (CentOS) mod_ftp/1.0.1-dev OpenSSL/1.0.1e-fips
> mod_fcgid/2.3.9 configured -- resuming normal operations
> [Tue Nov 08 08:15:52.914279 2016] [core:notice] [pid 1225] AH00094:
> Command line: '/usr/sbin/httpd -D FOREGROUND'
> [Tue Nov 08 08:25:02.543360 2016] [core:notice] [pid 1225] AH00052:
> child pid 1337 exit signal Segmentation fault (11)
> [Tue Nov 08 08:25:32.577127 2016] [core:notice] [pid 1225] AH00052:
> child pid 1338 exit signal Segmentation fault (11)
> [Tue Nov 08 08:25:40.586853 2016] [core:notice] [pid 1225] AH00052:
> child pid 1339 exit signal Segmentation fault (11)
> 
> Each of these segfaults is me typing "ls" in lftp.
> I juste want to get the ftp feature running. I can provide more info
> if you tell me how :)
> 
> Thanks!
> 
> Benjamin Lefoul
> nWISE AB