Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Mon, Aug 04, 2014 at 12:25:48PM +0300, Niko Tyni wrote: Not sure how useful it is to continue including the error log with build failures, but I'll leave it in place for now. FWIW I changed my mind and have removed it in 2.0.9~1604013-1 after too many times of losing the actual test results with manual builds. I don't think we ever got anything useful out of it. -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
severity 711167 important thanks On Wed, Jun 19, 2013 at 09:57:37PM +0300, Niko Tyni wrote: I guess I'll upload something that shows the error log when the tests fail then, and start hoping for the worst... :) Done in 2.0.8+httpd24-r1449661-5. Of course, the worst didn't happen and alkman managed a successful ia64 build. I'm downgrading this to 'important'; there's nothing we can do until it happens again and we see the error log. -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
Quoting Niko Tyni nt...@debian.org: Thanks for trying. Do you have a chance to test the theory that this somehow depends on the kernel? A squeeze system with a sid chroot should work for that. I tested building of the libapache2-mod-perl2 package on a sid chroot on a Debian squeeze (ia64, kernel 2.6.32). Unfortunately it was successfully. The build log had the known messages about starting of the Apache server and the list of test steps; each with an 'ok'. Stephan -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Wed, Jun 19, 2013 at 08:47:17PM +0200, Stephan Schreiber wrote: Quoting Niko Tyni nt...@debian.org: Thanks for trying. Do you have a chance to test the theory that this somehow depends on the kernel? A squeeze system with a sid chroot should work for that. I tested building of the libapache2-mod-perl2 package on a sid chroot on a Debian squeeze (ia64, kernel 2.6.32). Unfortunately it was successfully. The build log had the known messages about starting of the Apache server and the list of test steps; each with an 'ok'. Unfortunate indeed, but thanks again for trying! I guess I'll upload something that shows the error log when the tests fail then, and start hoping for the worst... :) -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Mon, Jun 17, 2013 at 05:24:22AM +0200, Stephan Schreiber wrote: Quoting Niko Tyni nt...@debian.org: Cc'ing the debian-ia64 list. Can anybody reproduce the sid build failure of libapache2-mod-perl2 2.0.8+httpd24-r1449661-4 ? What's in t/logs/error_log? I'm using Debian unstable on ia64. I could not reproduce the problem; I was able to build the libapache2-mod-perl2 (2.0.8+httpd24-r1449661-4) package with success. Thanks for trying. Do you have a chance to test the theory that this somehow depends on the kernel? A squeeze system with a sid chroot should work for that. -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
[context: libapache2-mod-perl2 2.0.8+httpd24-r1449661-4 fails to build on ia64 only.] On Wed, Jun 05, 2013 at 06:16:38PM +0200, gregor herrmann wrote: On Wed, 05 Jun 2013 10:52:37 +0300, Niko Tyni wrote: From the ia64 build log: still waiting for server to warm up: the server is down, giving up after 301 secs [ error] failed to start server! (please examine t/logs/error_log) -2 built successfully (on a different buildd). I just tried on merulo, and the package builds without any changes. OK, the SIGSEGVs were fixed with #711213 but this ia64-specific problem is still present. It looks like it happens on caballero (which tried to build 2.0.8+httpd24-r1449661-3 and -4) but not alkman (which built -2) or merulo (where I can't reproduce it either.) It might be related to the kernel version: the build logs indicate caballero is running 2.6.32-bpo.5-mckinley while alkman and merulo are on 3.2.0-4-mckinley. Cc'ing the debian-ia64 list. Can anybody reproduce the sid build failure of libapache2-mod-perl2 2.0.8+httpd24-r1449661-4 ? What's in t/logs/error_log? If that doesn't help, I suppose the last resort is to upload something like 'make test || (cat t/logs/error.log; exit 1)' to sid and hope caballero picks it up too (or try it first in experimental and see how mundy fares.) -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
Quoting Niko Tyni nt...@debian.org: Cc'ing the debian-ia64 list. Can anybody reproduce the sid build failure of libapache2-mod-perl2 2.0.8+httpd24-r1449661-4 ? What's in t/logs/error_log? I'm using Debian unstable on ia64. I could not reproduce the problem; I was able to build the libapache2-mod-perl2 (2.0.8+httpd24-r1449661-4) package with success. Stephan -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Fri, Jun 07, 2013 at 02:23:43PM +0300, Niko Tyni wrote: I can reproduce the SIGSEGV at the end of the main test suite (#711213) on amd64. The armel problem might well be related, as the log ends at the same point. My easiest recipe is APACHE_TEST_EXTRA_ARGS=-httpd_conf /home/niko/apache2.conf make run_tests TEST_FILES=t/user/rewrite.t where apache2.conf is like the one attached. It doesn't matter which tests get run; I managed to trigger it with no tests at all by just running apache2 manually and then killing it with SIGTERM. That means it looks like an apache2 bug. I haven't had the time to bisect the generated real apache2 config file further to get a minimal test case for the apache2 maintainers. I'm somewhat further now: what happens is that register_auth_provider() in modperl_util.c calls apr_pool_pre_cleanup_register(pool, NULL, cleanup_perl_global_providers); once in the parent process, then another time in a child. For some reason that I do not understand yet, the cleanup_perl_global_providers() function resides at a different memory location (with a 0x2c000 offset or so) on the second time. The first location has at that point become an invalid memory address, resulting in a SIGSEGV when libapr calls the registered cleanup functions and jumps into the old location. The perl module seems to get unloaded and loaded several times, but the location of cleanup_perl_global_providers() seems to stay the same in the parent process but differ in the child process. Perhaps a (re)load occurring right after a fork() can make the memory layout differ? I see this with both mpm_worker and mpm_prefork. It's something of a heisenbug; I had to give up bisecting the config file when very unrelated small changes would make it go away. Running apache2 under gdb or limiting the stack size in any way also hides it here, as does recompiling apache2 (but not apr or mod_perl itself.) This makes proper instrumentation on apache2 itself rather hard... Will keep on glaring as time permits. -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Thu, Jun 06, 2013 at 10:54:03PM +0200, Hector Oron wrote: On Wed, Jun 05, 2013 at 10:52:37AM +0300, Niko Tyni wrote: Package: libapache2-mod-perl2 Version: 2.0.8+httpd24-r1449661-3 Severity: serious The package versioned as 2.0.8+httpd24-r1449661-3 also fails on armel architecture, but hangs the build daemon. It took out of bussiness ancina armel buildd, then it was given back and it hanged alwyn. As the package has not failed, the logs is not visible online, I have taken out the build log from ancina and place it at: http://people.debian.org/~zumbi/tmp/libapache2-mod-perl2_2.0.8+httpd24-r1449661-3_armel-20130604-2030 Clarification: does it hang the kernel altogether? That's really unfortunate. I wonder if it's reproducible on the porter boxes. I assume a bug should be filed against the kernel. I probably can't work on this on the weekend, so I'll share what I have now: I can reproduce the SIGSEGV at the end of the main test suite (#711213) on amd64. The armel problem might well be related, as the log ends at the same point. My easiest recipe is APACHE_TEST_EXTRA_ARGS=-httpd_conf /home/niko/apache2.conf make run_tests TEST_FILES=t/user/rewrite.t where apache2.conf is like the one attached. It doesn't matter which tests get run; I managed to trigger it with no tests at all by just running apache2 manually and then killing it with SIGTERM. That means it looks like an apache2 bug. I haven't had the time to bisect the generated real apache2 config file further to get a minimal test case for the apache2 maintainers. If others have trouble reproducing it: running 'ulimit -s unlimited' first might help. Earlier I had to set that to trigger it, which was really weird. It's not needed for me anymore, so that part still remains a bit of a mystery. Finally, for people testing this: each crash with mod_digest loaded leaves dangling IPC objects which will eventually cause problems. I'm running this for cleanup: #!/bin/sh ipcs -m | grep ^0x | while read s rest; do ipcrm -M $s; done ipcs -s | grep ^0x | while read k s rest; do ipcrm -s $s; done -- Niko Tyni nt...@debian.org ## # these modules are needed for the test suite LoadModule mpm_worker_module /usr/lib/apache2/modules/mod_mpm_worker.so LoadModule authn_core_module /usr/lib/apache2/modules/mod_authn_core.so LoadModule auth_basic_module /usr/lib/apache2/modules/mod_auth_basic.so # LoadModule auth_digest_module /usr/lib/apache2/modules/mod_auth_digest.so LoadModule authz_core_module /usr/lib/apache2/modules/mod_authz_core.so LoadModule access_compat_module /usr/lib/apache2/modules/mod_access_compat.so LoadModule authz_user_module /usr/lib/apache2/modules/mod_authz_user.so LoadModule authn_file_module /usr/lib/apache2/modules/mod_authn_file.so LoadModule authz_groupfile_module /usr/lib/apache2/modules/mod_authz_groupfile.so LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so LoadModule authz_host_module /usr/lib/apache2/modules/mod_authz_host.so LoadModule alias_module /usr/lib/apache2/modules/mod_alias.so LoadModule deflate_module /usr/lib/apache2/modules/mod_deflate.so # LoadModule include_module /usr/lib/apache2/modules/mod_include.so ## LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so ## LoadModule env_module /usr/lib/apache2/modules/mod_env.so ## LoadModule mime_module /usr/lib/apache2/modules/mod_mime.so ## LoadModule dir_module /usr/lib/apache2/modules/mod_dir.so ## LoadModule cgi_module /usr/lib/apache2/modules/mod_cgi.so ## # LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so ## # commented out because: ## # .AH00526: Syntax error on line 233 of /home/niko/libapache2-mod-perl2/ModPerl-Registry/t/conf/extra.conf: ## # Invalid command 'RewriteLogLevel', perhaps misspelled or defined by a module not included in the server configuration
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
Hello, On Wed, Jun 05, 2013 at 10:52:37AM +0300, Niko Tyni wrote: Package: libapache2-mod-perl2 Version: 2.0.8+httpd24-r1449661-3 Severity: serious The package versioned as 2.0.8+httpd24-r1449661-3 also fails on armel architecture, but hangs the build daemon. It took out of bussiness ancina armel buildd, then it was given back and it hanged alwyn. As the package has not failed, the logs is not visible online, I have taken out the build log from ancina and place it at: http://people.debian.org/~zumbi/tmp/libapache2-mod-perl2_2.0.8+httpd24-r1449661-3_armel-20130604-2030 Regards, -- Hector Oron signature.asc Description: Digital signature
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Thu, 06 Jun 2013 22:54:03 +0200, Hector Oron wrote: Hello, ¡Hola! On Wed, Jun 05, 2013 at 10:52:37AM +0300, Niko Tyni wrote: Package: libapache2-mod-perl2 Version: 2.0.8+httpd24-r1449661-3 Severity: serious The package versioned as 2.0.8+httpd24-r1449661-3 also fails on armel architecture, but hangs the build daemon. It took out of bussiness ancina armel buildd, then it was given back and it hanged alwyn. As the package has not failed, the logs is not visible online, I have taken out the build log from ancina and place it at: http://people.debian.org/~zumbi/tmp/libapache2-mod-perl2_2.0.8+httpd24-r1449661-3_armel-20130604-2030 Thanks, that's interesting and helpful! (The missing log was indeed the missing piece :) So far we've only seen the ia64 failure which was not reproducible.) A short view shows that - the log doesn't actually show anything :/ (i.e. no problem) - it ends after the first test suite, when the test server is brought down - which might indeed be similar to the segfaults we've seen on other architectures at this stage (cf. #711213) - so this might be the core issue here? Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT SPI, fellow of the Free Software Foundation Europe `- NP: Funny van Dannen: Hauptstadt der Gefühle signature.asc Description: Digital signature
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
Package: libapache2-mod-perl2 Version: 2.0.8+httpd24-r1449661-3 Severity: serious From the ia64 build log: still waiting for server to warm up: the server is down, giving up after 301 secs [ error] failed to start server! (please examine t/logs/error_log) [ error] oh nuts, server dumped core [ error] for stacktrace, run: gdb /usr/sbin/apache2 -core /build/buildd-libapache2-mod-perl2_2.0.8+httpd24-r1449661-3-ia64-Qo0mXO/libapache2-mod-perl2-2.0.8+httpd24-r1449661/t/core ++ | Please file a bug report: http://perl.apache.org/bugs/ | ++ [ error] oh crap, server dumped core [ error] for stacktrace, run: gdb /usr/sbin/apache2 -core /build/buildd-libapache2-mod-perl2_2.0.8+httpd24-r1449661-3-ia64-Qo0mXO/libapache2-mod-perl2-2.0.8+httpd24-r1449661/t/core make[2]: *** [run_tests] Error 1 -2 built successfully (on a different buildd). I'm away from my keys; could someone please check if this is reproducible on the ia64 porter box and see if it's the addition of mod_auth_digest in debian/apache2.conf that triggers it? -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Wed, 05 Jun 2013 10:52:37 +0300, Niko Tyni wrote: From the ia64 build log: still waiting for server to warm up: the server is down, giving up after 301 secs [ error] failed to start server! (please examine t/logs/error_log) [ error] oh nuts, server dumped core [ error] for stacktrace, run: gdb /usr/sbin/apache2 -core /build/buildd-libapache2-mod-perl2_2.0.8+httpd24-r1449661-3-ia64-Qo0mXO/libapache2-mod-perl2-2.0.8+httpd24-r1449661/t/core ++ | Please file a bug report: http://perl.apache.org/bugs/ | ++ [ error] oh crap, server dumped core [ error] for stacktrace, run: gdb /usr/sbin/apache2 -core /build/buildd-libapache2-mod-perl2_2.0.8+httpd24-r1449661-3-ia64-Qo0mXO/libapache2-mod-perl2-2.0.8+httpd24-r1449661/t/core make[2]: *** [run_tests] Error 1 -2 built successfully (on a different buildd). I'm away from my keys; could someone please check if this is reproducible on the ia64 porter box and see if it's the addition of mod_auth_digest in debian/apache2.conf that triggers it? I just tried on merulo, and the package builds without any changes. (apache2 dumps core _after_ the successful tests). If I remove the mod_auth_digest line, t/hooks/authen_digest.t fails unsurprisingly (and we get a core dump _after_ finishing the tests again). After marking the tests as 'todo', all tests pass again (plus the segfault afterwards). Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT SPI, fellow of the Free Software Foundation Europe `- signature.asc Description: Digital signature
Bug#711167: libapache2-mod-perl2: FTBFS on ia64: test suite failed to start server
On Wed, Jun 05, 2013 at 06:16:38PM +0200, gregor herrmann wrote: On Wed, 05 Jun 2013 10:52:37 +0300, Niko Tyni wrote: [ error] oh nuts, server dumped core [ error] for stacktrace, run: gdb /usr/sbin/apache2 -core /build/buildd-libapache2-mod-perl2_2.0.8+httpd24-r1449661-3-ia64-Qo0mXO/libapache2-mod-perl2-2.0.8+httpd24-r1449661/t/core I just tried on merulo, and the package builds without any changes. (apache2 dumps core _after_ the successful tests). If I remove the mod_auth_digest line, t/hooks/authen_digest.t fails unsurprisingly (and we get a core dump _after_ finishing the tests again). After marking the tests as 'todo', all tests pass again (plus the segfault afterwards). OK, thanks. I suppose retrying it on the buildds might succeed then. Running the build in a loop or something to get t/logs/error_log and a backtrace would be interesting. (My guess is that the other core dump at the end is the same as #711213 and unrelated.) -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
