On Mon, Mar 12, 2012, at 10:10 PM, Bron Gondwana wrote: > On Mon, Mar 12, 2012 at 05:03:06PM -0400, Jenkins wrote: > > See <http://ci.cyrusimap.org/job/cyrus-imapd-master/449/changes> > > No idea what actually went wrong there - I don't know how to read > these stack dumps. It's uglier than Java I tell you! >
Cassandane is run in -v (verbose) mode, which generates enormous amounts of output, which is shoved aside into the file cass.errs, which you can view from the web interface thusly: http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/cass.errs/*view*/ The last few lines are ... =====> Instance[688] start main instance for test test_cve_2011_3208_list_active: basedir /var/tmp/cass/21021782 =====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f" "/var/tmp/cass/21021782/conf/sasldb2" "-c" "-p" "admin" =====> Instance[526] _start_master: logging to /var/tmp/cass/21021782/conf/master.log =====> Instance[1063] Running: "/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/master" "-C" "/var/tmp/cass/21021782/conf/imapd.conf" "-l" "255" "-p" "/var/tmp/cass/21021782/run/cyrus.pid" "-d" "-M" "/var/tmp/cass/21021782/conf/cyrus.conf" "-L" "/var/tmp/cass/21021782/conf/master.log" =====> Instance[535] _start_master: waiting for PID file =====> Instance[538] _start_master: PID file present and correct =====> Instance[544] _start_master: PID waiting for services =====> Service[269] is_listening: service nntp is listening on 127.0.0.1:9101 =====> Instance[555] _start_master: all services listening =====> Instance[652] create user cassandane =====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f" "/var/tmp/cass/21021782/conf/sasldb2" "-c" "-p" "cassandane" News::NNTPClient::SOCK1 connecting to localhost.localdomain:9101 News::NNTPClient unexpected EOF on News::NNTPClient::SOCK1 at Cassandane/Cyrus/Nntp.pm line 66 News::NNTPClient::SOCK1 command: AUTHINFO USER cassandane That looks like the NNTPClient Perl code failed because the nntpd it was talking to dropped the connection, probably a result of terminating prematurely. Also, every other test seems to follow a pattern like =====> Instance[688] start main instance for test test_set_system_flag_deliver: basedir /var/tmp/cass/2102173 =====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f" "/var/tmp/cass/2102173/conf/sasldb2" "-c" "-p" "admin" =====> Instance[526] _start_master: logging to /var/tmp/cass/2102173/conf/master.log =====> Instance[1063] Running: "/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/master" "-C" "/var/tmp/cass/2102173/conf/imapd.conf" "-l" "255" "-p" "/var/tmp/cass/2102173/run/cyrus.pid" "-d" "-M" "/var/tmp/cass/2102173/conf/cyrus.conf" "-L" "/var/tmp/cass/2102173/conf/master.log" =====> Instance[535] _start_master: waiting for PID file =====> Instance[538] _start_master: PID file present and correct =====> Instance[544] _start_master: PID waiting for services WARNING: Port 9102 never freed. Allocated at Cassandane/Service.pm line 100 Cassandane::Service::port('Cassandane::IMAPService=HASH(0xa7b84c0)') called at Cassandane/Service.pm line 142 [...Perl stack trace...] This is missing the bulk of the test xlog() calls, and seems to have failed early in instance startup. Let's have a look to see if there's any cores. Normally, tests will be failed if we get to the end of them and Cassandane notices there are cores in the instance directories, but in this case because of the "Port 9102 never freed" warning I suspect instances are not being shut down properly. root@hudson-01 2004 cd /var/tmp/cass root@hudson-01 2004 find * -name core\* -type f | while read f ; do echo -n "$f: " ; gdb -q --batch -c $f | sed -ne 's|Core was generated by ||p' ; done 2102171/conf/cores/core.20475: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021710/conf/cores/core.20536: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021711/conf/cores/core.20543: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021712/conf/cores/core.20550: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021713/conf/cores/core.20557: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021714/conf/cores/core.20564: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021715/conf/cores/core.20570: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021716/conf/cores/core.20577: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021717/conf/cores/core.20583: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021718/conf/cores/core.20589: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021719/conf/cores/core.20594: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102172/conf/cores/core.20483: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021720/conf/cores/core.20601: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021721/conf/cores/core.20606: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021722/conf/cores/core.20612: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021723/conf/cores/core.20618: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021724/conf/cores/core.20624: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021725/conf/cores/core.20630: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021726/conf/cores/core.20636: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021727/conf/cores/core.20642: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021728/conf/cores/core.20648: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021729/conf/cores/core.20654: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102173/conf/cores/core.20489: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102174/conf/cores/core.20497: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021746/conf/cores/core.21098: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021747/conf/cores/core.21104: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021748/conf/cores/core.21111: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021749/conf/cores/core.21117: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102175/conf/cores/core.20502: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021750/conf/cores/core.21122: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021751/conf/cores/core.21128: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021752/conf/cores/core.21134: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021753/conf/cores/core.21140: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021754/conf/cores/core.21146: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021755/conf/cores/core.21152: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021756/conf/cores/core.21158: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021757/conf/cores/core.21164: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021758/conf/cores/core.21170: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021759/conf/cores/core.21176: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102176/conf/cores/core.20508: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021760/conf/cores/core.21182: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021761/conf/cores/core.21188: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021762/conf/cores/core.21194: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021763/conf/cores/core.21200: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021764/conf/cores/core.21206: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021765/conf/cores/core.21212: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021766/conf/cores/core.21218: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021767/conf/cores/core.21224: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021768/conf/cores/core.21230: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021769/conf/cores/core.21236: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102177/conf/cores/core.20516: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021770/conf/cores/core.21242: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021771/conf/cores/core.21248: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021772/conf/cores/core.21254: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021773/conf/cores/core.21260: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021774/conf/cores/core.21266: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021775/conf/cores/core.21272: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021776/conf/cores/core.21278: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021777/conf/cores/core.21284: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021778/conf/cores/core.21290: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021779/conf/cores/core.21296: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 2102178/conf/cores/core.20522: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021780/conf/cores/core.21302: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021781/conf/cores/core.21310: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. 21021782/conf/cores/core.21318: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. 21021782/conf/cores/core.21320: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. 21021782/conf/cores/core.21322: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. 21021782/conf/cores/core.21319: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. 21021782/conf/cores/core.21321: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. 2102179/conf/cores/core.20529: `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'. My, that's a lot of cores, most from ctl_cyrus but some from nntpd. The `ctl_cyrus' cores are actually from ctl_cyrusdb and are doing Program terminated with signal 6, Aborted. #0 0x0000003665830265 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003665830265 in raise () from /lib64/libc.so.6 #1 0x0000003665831d10 in abort () from /lib64/libc.so.6 #2 0x00000036658296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00000000004a4b1b in config_getenum (opt=IMAPOPT_VIRTDOMAINS) at libconfig.c:138 #4 0x00000000004a5c85 in config_read ( alt_config=0x7fff871b47b5 "/var/tmp/cass/21021775/conf/imapd.conf") at libconfig.c:386 #5 0x000000000041df80 in cyrus_init ( alt_config=0x7fff871b47b5 "/var/tmp/cass/21021775/conf/imapd.conf", ident=0x4acbf2 "ctl_cyrusdb", flags=0) at global.c:152 #6 0x0000000000406ada in main (argc=4, argv=0x7fff871b4268) at ctl_cyrusdb.c:272 The nntpd cores are doing this root@hudson-01 2005 gdb -q /var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd 21021782/conf/cores/core.21321 .Core was generated by `/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd -C'. Program terminated with signal 6, Aborted. #0 0x0000003665830265 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003665830265 in raise () from /lib64/libc.so.6 #1 0x0000003665831d10 in abort () from /lib64/libc.so.6 #2 0x00000036658296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00000000004dd7ff in config_getenum (opt=IMAPOPT_VIRTDOMAINS) at libconfig.c:138 #4 0x00000000004de969 in config_read ( alt_config=0x7fff726407a0 "/var/tmp/cass/21021782/conf/imapd.conf") at libconfig.c:386 #5 0x0000000000471170 in cyrus_init ( alt_config=0x7fff726407a0 "/var/tmp/cass/21021782/conf/imapd.conf", ident=0x129f2530 "nntp", flags=0) at global.c:152 #6 0x0000000000438cf6 in main (argc=3, argv=0x7fff7263eed8, envp=0x7fff7263eef8) at service.c:337 .. (gdb) up 3 #3 0x00000000004dd7ff in config_getenum (opt=IMAPOPT_VIRTDOMAINS) at libconfig.c:138 138 assert(imapopts[opt].t == OPT_ENUM); (gdb) p opt $1 = IMAPOPT_VIRTDOMAINS (gdb) p imapopts[opt] $2 = {opt = IMAPOPT_VIRTDOMAINS, optname = 0x5efbbf "unixhierarchysep", seen = 0, t = OPT_SWITCH, val = {s = 0x0, i = 0, b = 0, e = IMAP_ENUM_ZERO, x = 0}, def = {s = 0x0, i = 0, b = 0, e = IMAP_ENUM_ZERO, x = 0}, enum_options = {{name = 0x0, val = IMAP_ENUM_ZERO} <repeats 22 times>}} Woops. It looks like there's at least one commit > Commit a95d47894f08539f794e020f334d8327015f13ef by brong > Bug #3642 - add 'specialusealways' switch to imapd.conf that added a new imapd.conf option, but the changes didn't flow through to the object code properly. Clearly there's a Makefile bug here. Aha: -rw-r--r-- 1 tomcat tomcat 21063 Mar 12 05:01 libconfig.c -rw-rw-r-- 1 tomcat tomcat 63616 Mar 12 05:01 libconfig.o -rw-r--r-- 1 tomcat tomcat 62904 Mar 12 17:01 imapoptions -rw-rw-r-- 1 tomcat tomcat 132440 Mar 12 17:01 imapopts.o -rw-rw-r-- 1 tomcat tomcat 10129 Mar 12 17:01 imapopts.h -rw-rw-r-- 1 tomcat tomcat 55749 Mar 12 17:01 imapopts.c So on this last run, make rebuilt: imapoptions -> imapopts.c, imapopts.h imapopts.c -> imapopts.o but it did NOT build libconfig,c -> libconfig.o presumably because it lack the dependency information to know that it should, as libconfig.a #includes imapopts.h. Stupid dependency crap...we need automake!! Anyway, the easiest workaround is probably to do a "make clean" in the autobuild.sh script to remove all the .o files. -- Greg.