Re: [VOTE] Release Apache httpd 2.4.28 as GA
An update concerning the sporadic proxy test failures for prefork on Solaris that I observed: These are a bit difficult to nail down, because running only the proxy tests does not fail. One has to run a relatively big part of the test suite to have a chance of triggering the problem during the proxy test. I can reproduce in about 50% of the test runs. Currently I have the impression, that the problem only happens with apr 1.6.x (in about 50% of the test runs), but never with apr 1.5.x. Whether I use apu 1.5.x or apu 1.6.x does not seem to matter. So I need to dig deeper, what behavioral changes are expected between apr 1.5.x and 1.6.x especially on Solaris (and which only matter for prefork). Still IMHO not a show-stopper but of course worth fixing. Regards, Rainer
Re: [VOTE] Release Apache httpd 2.4.28 as GA
On Mon, Sep 25, 2017 at 12:11 PM, Steffenwrote: > On Windows it does not build out of the box. > > Missing modules/core include for mod_watchdog.h in > mod_proxy_balancer.dsp/mak and libhttp.dsp/mak . Did not checked cmake. Seems baffling, but it is pretty straightforward in hindsight. libhttpd is built first before modules/* libhttpd compiles in mod_so (alongside core, win32, mpm_winnt, and http.) libhttpd starts by copying mod_so.h to include/, originally the only module, just as the unix build had long done so when mod_so was enabled. There was no need to /I modules/core with that header in include/ modules/core/mod_watchdog was added - but built as a loadable module. The .h was neither copied to include/ nor was /I include/core added. mod_heartbeat, mod_heartmonitor, mod_proxy_balancer and mod_proxy_hcheck all include mod_watchdog; it makes more sense to copy this header as well, rather than modify four projects. I'm a little confused how libhttpd would need it itself, but that's where include copying should occur for now. Fixed in r1810012
Re: [VOTE] Release Apache httpd 2.4.28 as GA
On 2017-09-28 09:51, Jim Jagielski wrote: > Personally, I don't think the age or heritage of the build-logic is the issue, > but rather a lack of people *really* testing 2.4.x until a release is tagged. > It's for that reason that I tend to pre-announce a T well in advance of > actually > DOING the T so that people can test HEAD in hopes that the tag will already > have had some good testing beforehand. I have a question. Why are you tagging a release and do testing? Most of the time a problem is found and a new release is tagged and it starts over (I think the max was a 3 or 4 patch level jump). Why not tagging an RC? People test the RC. When all is ok, the RC is released. If not a new RC is tagged. Cheers, K. C. -- regards Helmut K. C. Tessarek KeyID 0xF7832007C11F128D Key fingerprint = 28A3 1666 4FE8 D72C CFD5 8B23 F783 2007 C11F 128D /* Thou shalt not follow the NULL pointer for chaos and madness await thee at its end. */
Re: [VOTE] Release Apache httpd 2.4.28 as GA
If the issue is "real" and a patch is forthcoming, I agree that we mark 2.4.28 as DOA and proceed w/ 2.4.29 next week. I am comfortable keeping the VOTE open longer that the normal/required 72hours to be sure one way or another. Personally, I don't think the age or heritage of the build-logic is the issue, but rather a lack of people *really* testing 2.4.x until a release is tagged. It's for that reason that I tend to pre-announce a T well in advance of actually DOING the T so that people can test HEAD in hopes that the tag will already have had some good testing beforehand. > On Sep 27, 2017, at 11:21 PM, William A Rowe Jrwrote: > > The assert() has me concerned, and Steffen's report is problematic. He has a > vote but hasn't cast it. At this moment I'm -0 and would spin a 2.4.29 next > week to address these issues, unless you decide to respin before this > release, yourself. > > Nothing I've changed today altered the httpd tarball significantly. Studying > Steffen's report next, along with some apparently missing glue for brotli. > > My solution is going to be radical, shove every last d*mned modules/foo into > the /I path includes list so this can't happen again during 2.4, and > hopefully not until 20 year old build logic is discarded. One less thing to > worry about or pre-review when RM's loudly announce an upcoming tag. > > > > > On Sep 25, 2017 07:13, "Jim Jagielski" wrote: > The pre-release test tarballs for Apache httpd > version 2.4.28 can be found at the usual place: > > http://httpd.apache.org/dev/dist/ > > I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA. > > [ ] +1: Good to go > [ ] +0: meh > [ ] -1: Danger Will Robinson. And why. > > Vote will last the normal 72 hrs. > > NOTE: The *-deps are only there for convenience. > > Thx!
Re: [VOTE] Release Apache httpd 2.4.28 as GA
Am 25.09.2017 um 14:13 schrieb Jim Jagielski: The pre-release test tarballs for Apache httpd version 2.4.28 can be found at the usual place: http://httpd.apache.org/dev/dist/ I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA. [X] +1: Good to go [ ] +0: meh [ ] -1: Danger Will Robinson. And why. Vote will last the normal 72 hrs. NOTE: The *-deps are only there for convenience. +1 to release and thanks a bunch for RMing and to all who contributed to the release. In short: No regressions found. Detailed report: - Sigs and hashes OK - contents of tarballs identical - contents of tag and tarballs identical except for expected deltas Built on - Solaris 10 Sparc as 32 Bit Binaries - SLES 11+12 (64 Bits) - RHEL 6+7 (64 Bits) For all platforms built - with default (shared), shared and static modules - with module sets none, few, most, all, reallyall and default (always mod_privileges disabled) - using --enable-load-all-modules - against "included" APR/APU from deps tarball, plus external APR/APU 1.6.2/1.6.0, 1.5.2/1.5.4, 1.6.x branches head and trunk head. - using external libraries - expat 2.2.4 - pcre 8.41 - openssl 1.0.2k plus patches - lua 5.3.4 (compiled with LUA_COMPAT_MODULE) - distcache 1.5.1 - libxml2 2.9.5 - libnghttp2 1.26.0 - brotli 1.0.1 - Tool chain: - platform gcc except on Solaris (gcc 4.1.2 for Solaris 8 and 7.1.0 Solaris 10 (only older APR/APU 1.5.x compiled with older gcc 4.9.2) - CFLAGS: -O2 -g -Wall -fno-strict-aliasing - on Solaris additionally -mpcu=v9, -D_XOPEN_SOURCE, -D_XOPEN_SOURCE_EXTENDED=1 and -D__EXTENSIONS__ plus -D_XPG6 when using gcc 7.1.0 350 build succeeded, 2 builds both on SLES 11 crashed due to segmentation faults in ksh during make, but could be completed by restarting make. Not a regression. - compiler warnings: - modules/core/mod_watchdog.c:436: warning: 'rv' may be used uninitialized in this function -> warning is correct but nt critical (debug log), will fix this soon; not a regression and only on Solaris (gcc 7.1.0) - server/util.c:2667:10: warning: 'escaped_char[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] - srclib/apr-util/xlate/xlate.c:120:38: warning: passing argument 2 of 'iconv' from incompatible pointer type [-Wincompatible-pointer-types] - srclib/apr-util/xlate/xlate.c:343:42: warning: passing argument 2 of 'iconv' from incompatible pointer type [-Wincompatible-pointer-types] Tested for - Solaris 10, SLES 11+12, RHEL 6+7 - MPMs prefork, worker, event - default, shared and static modules - log level trace8 - module set reallyall (125 modules plus MPMs) The following test failures were seen: a Lots of tests in t/module/session.t fail always for static builds The whole setup for the /sesiontest uri is missing in the generated t/conf/httpd.conf. This is due to it missing from the also generated filet/conf/apache_test_config.pm. I do not know yet, why it is missing there, but this seems to be a test framework problem. Not a regression. b Test 5 in t/modules/dav.t: Happens for 7 out of about 225 runs Creation, modified and now times not in the correct order. This seems to be a system issue, all tests done on NFS, many tested on virtualized guests. Not a regression. c Test 59 of t/modules/include.t only and always on Solaris. This is due to a bug in the test, which uses strftime() with a "%s" pattern that is not supported on Solaris. Until recently the server and the test client both returned verbatim "%s" and the test succeeded. After updating some Perl modules for the http2 tests, the perl client even on Solaris now supports "%s" in strftime and the test starts to fail. It seems we have to fix the test. Not a regression d Various tests in t/apache/expr_string.t: (3, 6, 11, 14, 17, 20 ,23) Happens for 8 out of about 225 runs (this time all on RHEL6). The failure is always on line 68, where the error_log contents are checked. Not a regression. e Only on Solaris and only with prefork proxy tests sometimes seem to hang until timeout. Some test runs complete without the issue. Once the issue hits, all remaining proxy tests during that run face that issue. Not observed for static builds. Of the 8 non-static builds using APR/APU 1.6 or trunk, 5 had the problem. The two APR/APU builds did not have it. First observed when testing 2.4.26. I'll try to investigate further, but IMHO this is not a show-stopper. All in all I don't see a critical problem. Regards, Rainer
Re: [VOTE] Release Apache httpd 2.4.28 as GA
Update: disregard the man behind the curtain - for now. I have the strangest effects on my main machine under native macOS 10.12.6, which do not happen on my parallels ubuntu image and my laptop with macOS 10.13.0. I tested with hyperthreading disabled to exclude the Intel *lake bugs, but it made no difference. I will upgrade the machine over night to macOS 10.13.0 and check again tomorrow if that had any effect. Regardless of all this, I managed to find an edge case assertion fail and made a fix on trunk. If we go for 2.4.29, I'll propose that for inclusion. It is not a regression to 2.4.27, so no veto from me on 2.4.28. Cheers, Stefan > Am 28.09.2017 um 09:54 schrieb Stefan Eissing: > > Analyzing... > > my checkout of branch/2.4.x does not show this. > my buildt of 2.4.28 with -deps as provided, does after while of letting: > > while true; do h2load -t 8 -c 100 -n 1 -m 5 -d gen/data-10k > http://test.example.org:12345/; done > > run (data-10k is a 10k text file). The whole server setup is according to the > mod_http2 test cases > in https://svn.apache.org/repos/asf/httpd/test/mod_h2/trunk > > In 2.4.28 I have apr 1.6.2 and in my branch/2.4.x is a 1.5.x (which shows > 1.5.3 in its header). > > Will check if that is connected or not. Will also check if this happens for > me on my Ubuntu. > > Sporadically (less frequent), I can also see (not both 2.4.x and 2.4.28): > Thread 3 Crashed: > 0 libsystem_kernel.dylib0x7fffd3c34d42 __pthread_kill + 10 > 1 libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90 > 2 libsystem_c.dylib 0x7fffd3b9a4bb __abort + 140 > 3 libsystem_c.dylib 0x7fffd3b9a42f abort + 144 > 4 libsystem_pthread.dylib 0x7fffd3d23bc7 __pthread_abort + 49 > 5 libsystem_pthread.dylib 0x7fffd3d23c7b > __pthread_abort_reason + 180 > 6 libsystem_pthread.dylib 0x7fffd3d1fd93 > _pthread_mutex_unlock_drop + 167 > 7 mod_http2.so 0x00010cb42b2b h2_beam_send + 1819 > (h2_bucket_beam.c:965) > 8 mod_http2.so 0x00010cb59d31 send_out + 257 > (h2_task.c:100) > 9 mod_http2.so 0x00010cb59146 > h2_filter_slave_output + 294 (h2_task.c:176) > 10 httpd 0x00010c598829 > ap_process_request_after_handler + 89 (http_request.c:366) > 11 httpd 0x00010c598ab6 ap_process_request + > 22 (http_request.c:473) > 12 mod_http2.so 0x00010cb5874e h2_task_process_conn > + 398 (h2_task.c:682) > 13 httpd 0x00010c5705f7 > ap_run_process_connection + 55 (connection.c:42) > 14 mod_http2.so 0x00010cb599db h2_task_do + 539 > (h2_task.c:640) > 15 mod_http2.so 0x00010cb5df24 slot_run + 260 > (h2_workers.c:233) > 16 libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 180 > 17 libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + 286 > 18 libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13 > > which is a call to apr_thread_mutex_unlock(). Without having checkinto > pthread version, the closest > similar report I could find on the net is > https://sourceware.org/bugzilla/show_bug.cgi?id=17514 > > Will report when I know more. > > -Stefan > >> Am 27.09.2017 um 18:04 schrieb Luca Toscano : >> >> Hi Stefan, >> >> 2017-09-27 17:32 GMT+02:00 Stefan Eissing : >> On my h2 load tests, the server sometimes crashes in an assertion failure >> (build in maintainer mode): >> >> How did you make the tests? It would be good for people to attempt to >> reproduce.. >> >> >> macOS crash reporter: >> Thread 54 Crashed: >> 0 libsystem_kernel.dylib 0x7fffd3c34d42 __pthread_kill + >> 10 >> 1 libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90 >> 2 libsystem_c.dylib 0x7fffd3b9a4bb __abort + 140 >> 3 libsystem_c.dylib 0x7fffd3b9a42f abort + 144 >> 4 httpd 0x00010be99282 ap_log_assert + >> 130 (log.c:1696) >> 5 mod_mpm_event.so0x00010c50b51c ap_queue_push + >> 188 >> 6 mod_mpm_event.so0x00010c509be2 listener_thread + >> 2226 (event.c:1749) >> 7 libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + >> 180 >> 8 libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + >> 286 >> 9 libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13 >> >> error_log: >> [Wed Sep 27 17:27:09.941040 2017] [mpm_event:notice] [pid 23404:tid >> 140736895005632] AH00489: Apache/2.4.28 (Unix) OpenSSL/1.1.0e configured -- >> resuming normal operations >> [Wed Sep 27 17:27:09.941055 2017]
Re: [VOTE] Release Apache httpd 2.4.28 as GA
Analyzing... my checkout of branch/2.4.x does not show this. my buildt of 2.4.28 with -deps as provided, does after while of letting: while true; do h2load -t 8 -c 100 -n 1 -m 5 -d gen/data-10k http://test.example.org:12345/; done run (data-10k is a 10k text file). The whole server setup is according to the mod_http2 test cases in https://svn.apache.org/repos/asf/httpd/test/mod_h2/trunk In 2.4.28 I have apr 1.6.2 and in my branch/2.4.x is a 1.5.x (which shows 1.5.3 in its header). Will check if that is connected or not. Will also check if this happens for me on my Ubuntu. Sporadically (less frequent), I can also see (not both 2.4.x and 2.4.28): Thread 3 Crashed: 0 libsystem_kernel.dylib 0x7fffd3c34d42 __pthread_kill + 10 1 libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90 2 libsystem_c.dylib 0x7fffd3b9a4bb __abort + 140 3 libsystem_c.dylib 0x7fffd3b9a42f abort + 144 4 libsystem_pthread.dylib 0x7fffd3d23bc7 __pthread_abort + 49 5 libsystem_pthread.dylib 0x7fffd3d23c7b __pthread_abort_reason + 180 6 libsystem_pthread.dylib 0x7fffd3d1fd93 _pthread_mutex_unlock_drop + 167 7 mod_http2.so0x00010cb42b2b h2_beam_send + 1819 (h2_bucket_beam.c:965) 8 mod_http2.so0x00010cb59d31 send_out + 257 (h2_task.c:100) 9 mod_http2.so0x00010cb59146 h2_filter_slave_output + 294 (h2_task.c:176) 10 httpd 0x00010c598829 ap_process_request_after_handler + 89 (http_request.c:366) 11 httpd 0x00010c598ab6 ap_process_request + 22 (http_request.c:473) 12 mod_http2.so0x00010cb5874e h2_task_process_conn + 398 (h2_task.c:682) 13 httpd 0x00010c5705f7 ap_run_process_connection + 55 (connection.c:42) 14 mod_http2.so0x00010cb599db h2_task_do + 539 (h2_task.c:640) 15 mod_http2.so0x00010cb5df24 slot_run + 260 (h2_workers.c:233) 16 libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 180 17 libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + 286 18 libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13 which is a call to apr_thread_mutex_unlock(). Without having checkinto pthread version, the closest similar report I could find on the net is https://sourceware.org/bugzilla/show_bug.cgi?id=17514 Will report when I know more. -Stefan > Am 27.09.2017 um 18:04 schrieb Luca Toscano: > > Hi Stefan, > > 2017-09-27 17:32 GMT+02:00 Stefan Eissing : > On my h2 load tests, the server sometimes crashes in an assertion failure > (build in maintainer mode): > > How did you make the tests? It would be good for people to attempt to > reproduce.. > > > macOS crash reporter: > Thread 54 Crashed: > 0 libsystem_kernel.dylib 0x7fffd3c34d42 __pthread_kill + 10 > 1 libsystem_pthread.dylib 0x7fffd3d22457 pthread_kill + 90 > 2 libsystem_c.dylib 0x7fffd3b9a4bb __abort + 140 > 3 libsystem_c.dylib 0x7fffd3b9a42f abort + 144 > 4 httpd 0x00010be99282 ap_log_assert + > 130 (log.c:1696) > 5 mod_mpm_event.so0x00010c50b51c ap_queue_push + 188 > 6 mod_mpm_event.so0x00010c509be2 listener_thread + > 2226 (event.c:1749) > 7 libsystem_pthread.dylib 0x7fffd3d1f93b _pthread_body + 180 > 8 libsystem_pthread.dylib 0x7fffd3d1f887 _pthread_start + > 286 > 9 libsystem_pthread.dylib 0x7fffd3d1f08d thread_start + 13 > > error_log: > [Wed Sep 27 17:27:09.941040 2017] [mpm_event:notice] [pid 23404:tid > 140736895005632] AH00489: Apache/2.4.28 (Unix) OpenSSL/1.1.0e configured -- > resuming normal operations > [Wed Sep 27 17:27:09.941055 2017] [core:notice] [pid 23404:tid > 140736895005632] AH00094: Command line: '/opt/apache-2.4.x/bin/httpd -d > /Users/sei/projects/httpd/test/mod_h2/2.4.x/gen/apache' > [Wed Sep 27 17:29:43.207923 2017] [mpm_event:alert] [pid 30381:tid > 123145361334272] (35)Resource temporarily unavailable: AH03104: > apr_thread_create: unable to create worker thread > [Wed Sep 27 17:29:43.208904 2017] [mpm_event:crit] [pid 30381:tid > 123145369382912] (9)Bad file descriptor: apr_pollset_poll failed. Attempting > to shutdown process gracefully > [Wed Sep 27 17:29:44.000147 2017] [core:notice] [pid 23404:tid > 140736895005632] AH00052: child pid 30381 exit signal Segmentation fault (11) > [Wed Sep 27 17:29:44.208035 2017] [mpm_event:alert] [pid 30382:tid > 123145438388224] (35)Resource temporarily unavailable: AH03104: > apr_thread_create: unable to create worker thread >
Re: [VOTE] Release Apache httpd 2.4.28 as GA
+1 to doing a 2.4.29 next week with these issues fixed. On 9/27/2017 9:21 PM, William A Rowe Jr wrote: The assert() has me concerned, and Steffen's report is problematic. He has a vote but hasn't cast it. At this moment I'm -0 and would spin a 2.4.29 next week to address these issues, unless you decide to respin before this release, yourself. Nothing I've changed today altered the httpd tarball significantly. Studying Steffen's report next, along with some apparently missing glue for brotli. My solution is going to be radical, shove every last d*mned modules/foo into the /I path includes list so this can't happen again during 2.4, and hopefully not until 20 year old build logic is discarded. One less thing to worry about or pre-review when RM's loudly announce an upcoming tag. On Sep 25, 2017 07:13, "Jim Jagielski"wrote: The pre-release test tarballs for Apache httpd version 2.4.28 can be found at the usual place: http://httpd.apache.org/dev/dist/ I'm calling a VOTE on releasing these as Apache httpd 2.4.28 GA. [ ] +1: Good to go [ ] +0: meh [ ] -1: Danger Will Robinson. And why. Vote will last the normal 72 hrs. NOTE: The *-deps are only there for convenience. Thx!