Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 4 Mar 2015, at 15:25, Shyam wrote: > On 03/03/2015 11:27 PM, Justin Clift wrote: >> 2 x Coredumps >> * >> >> * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ >> >> IP - 104.130.74.142 >> >> This coredump run also failed on: >> >> * tests/basic/fops-sanity.t >> (Wstat: 0 Tests: 11 Failed: 1) >> Failed test: 10 >> >> * tests/bugs/glusterfs-server/bug-861542.t >> (Wstat: 0 Tests: 13 Failed: 1) >> Failed test: 10 >> >> * tests/performance/open-behind.t >> (Wstat: 0 Tests: 17 Failed: 1) >> Failed test: 17 > > FWIW, this is the same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 > >> >> * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ >> >> IP - 104.130.74.143 >> >> This coredump run also failed on: >> >> * tests/basic/afr/entry-self-heal.t >> (Wstat: 0 Tests: 180 Failed: 2) >> Failed tests: 127-128 >> >> * tests/bugs/glusterfs-server/bug-861542.t >> (Wstat: 0 Tests: 13 Failed: 1) >> Failed test: 10 > > So is this one. i.e same as > https://bugzilla.redhat.com/show_bug.cgi?id=1195415 Thanks Shyam. Somehow missed your email earlier, but all good now. :) + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/04/2015 09:57 AM, Justin Clift wrote: Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 Nice catch by regression. Fix: http://review.gluster.org/9817 Pranith * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/04/2015 10:25 AM, Nithya Balachandran wrote: Fix for the spurious bug-1117851.t failure at http://review.gluster.org/#/c/9798/ Thanks, Shyam and Nithya for a quick turnaround on this spurious failure! -Vijay ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Fix for the spurious bug-1117851.t failure at http://review.gluster.org/#/c/9798/ Regards, Nithya - Original Message - From: "Justin Clift" To: "Nithya Balachandran" Cc: "Gluster Devel" Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran wrote: > I'll take a look at tests/bugs/distribute/bug-1117851.t > > Regards, > Nithya > > - Original Message - > From: "Justin Clift" > To: "Gluster Devel" > Sent: Wednesday, 4 March, 2015 9:57:00 AM > Subject: [Gluster-devel] Spurious failure report for master branch - > 2015-03-03 > > Ran 20 x regression tests on our GlusterFS master branch code > as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. > > 5 of them were successful (25%), 15 of them failed in various ways > (75%). > > We need to get this down to about 5% or less (preferably 0%), as it's > killing our development iteration speed. We're wasting huge amounts > of time working around this. :( > > > Spurious failures > * > > * 5 x tests/bugs/distribute/bug-1117851.t >(Wstat: 0 Tests: 24 Failed: 1) >Failed test: 15 > >This one is causing a 25% failure rate all by itself. :( > >This needs fixing soon. :) > > > * 3 x tests/bugs/geo-replication/bug-877293.t >(Wstat: 0 Tests: 15 Failed: 1) >Failed test: 11 > > * 2 x tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * 1 x tests/basic/ec/ec-12-4.t >(Wstat: 0 Tests: 541 Failed: 2) >Failed tests: 409, 441 > > * 1 x tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * 1 x tests/basic/uss.t >(Wstat: 0 Tests: 160 Failed: 1) >Failed test: 26 > > * 1 x tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * 1 x tests/bugs/distribute/bug-884455.t >(Wstat: 0 Tests: 22 Failed: 1) >Failed test: 11 > > * 1 x tests/bugs/fuse/bug-1126048.t >(Wstat: 0 Tests: 12 Failed: 1) >Failed test: 10 > > * 1 x tests/bugs/quota/bug-1038598.t >(Wstat: 0 Tests: 28 Failed: 1) >Failed test: 28 > > > 2 x Coredumps > * > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ > >IP - 104.130.74.142 > >This coredump run also failed on: > > * tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > * tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ > >IP - 104.130.74.143 > >This coredump run also failed on: > > * tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > Both VMs are also online, in case they're useful to log into > for investigation (root / the jenkins slave pw). > > If they're not, please let me know so I can blow them away. :) > > > 1 x hung host > * > > Hung on tests/bugs/posix/bug-1113960.t > > root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh > root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh > root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl > /usr/bin/prove -rf --timer ./tests > root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
On 03/03/2015 11:27 PM, Justin Clift wrote: 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 FWIW, this is the same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 So is this one. i.e same as https://bugzilla.redhat.com/show_bug.cgi?id=1195415 Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Hi, I will also look at the hang - can I have the setup for a couple of days? Regards, Nithya - Original Message - From: "Justin Clift" To: "Gluster Devel" Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Hi, I had a look at tests/bugs/distribute/bug-1117851.t The test fails at : EXPECT_WITHIN 75 "done" cat $M0/status_0 The test uses a status file to check if the file rename operation (where a 1000 files are renamed) which runs in the background is over. The status file $M0/status_0 is created before the rename begins and the string "running" is written to it. Once the rename is done, the string "done" is written to the file. So it turns out the renames are actually finishing well in time - roughly 40 seconds. But the status_0 file is not present so cat fails on the file. The logs for two regression runs that failed confirm this (http://build.gluster.org/job/rackspace-regression-2GB/951/console and http://build.gluster.org/job/rackspace-regression-2GB/983/console). cat: /mnt/glusterfs/0/status_0: No such file or directory [14:53:50] ./tests/bugs/distribute/bug-1117851.t . not ok 15 Got "" instead of "done" Failed 1/24 subtests The test runs successfully on my local setup and has failed only twice on the VM Justin provided(out of about 50 runs), so I am still looking into why it cannot find the file. Regards, Nithya - Original Message - From: "Justin Clift" To: "Nithya Balachandran" Cc: "Gluster Devel" Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran wrote: > I'll take a look at tests/bugs/distribute/bug-1117851.t > > Regards, > Nithya > > - Original Message - > From: "Justin Clift" > To: "Gluster Devel" > Sent: Wednesday, 4 March, 2015 9:57:00 AM > Subject: [Gluster-devel] Spurious failure report for master branch - > 2015-03-03 > > Ran 20 x regression tests on our GlusterFS master branch code > as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. > > 5 of them were successful (25%), 15 of them failed in various ways > (75%). > > We need to get this down to about 5% or less (preferably 0%), as it's > killing our development iteration speed. We're wasting huge amounts > of time working around this. :( > > > Spurious failures > * > > * 5 x tests/bugs/distribute/bug-1117851.t >(Wstat: 0 Tests: 24 Failed: 1) >Failed test: 15 > >This one is causing a 25% failure rate all by itself. :( > >This needs fixing soon. :) > > > * 3 x tests/bugs/geo-replication/bug-877293.t >(Wstat: 0 Tests: 15 Failed: 1) >Failed test: 11 > > * 2 x tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * 1 x tests/basic/ec/ec-12-4.t >(Wstat: 0 Tests: 541 Failed: 2) >Failed tests: 409, 441 > > * 1 x tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * 1 x tests/basic/uss.t >(Wstat: 0 Tests: 160 Failed: 1) >Failed test: 26 > > * 1 x tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * 1 x tests/bugs/distribute/bug-884455.t >(Wstat: 0 Tests: 22 Failed: 1) >Failed test: 11 > > * 1 x tests/bugs/fuse/bug-1126048.t >(Wstat: 0 Tests: 12 Failed: 1) >Failed test: 10 > > * 1 x tests/bugs/quota/bug-1038598.t >(Wstat: 0 Tests: 28 Failed: 1) >Failed test: 28 > > > 2 x Coredumps > * > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ > >IP - 104.130.74.142 > >This coredump run also failed on: > > * tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > * tests/performance/open-behind.t >(Wstat: 0 Tests
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Looks like I will need a VM - the test passes consistently on my local setup. Can you please send me the details? Regards, Nithya - Original Message - From: "Justin Clift" To: "Nithya Balachandran" Cc: "Gluster Devel" Sent: Wednesday, 4 March, 2015 10:12:17 AM Subject: Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran wrote: > I'll take a look at tests/bugs/distribute/bug-1117851.t > > Regards, > Nithya > > - Original Message - > From: "Justin Clift" > To: "Gluster Devel" > Sent: Wednesday, 4 March, 2015 9:57:00 AM > Subject: [Gluster-devel] Spurious failure report for master branch - > 2015-03-03 > > Ran 20 x regression tests on our GlusterFS master branch code > as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. > > 5 of them were successful (25%), 15 of them failed in various ways > (75%). > > We need to get this down to about 5% or less (preferably 0%), as it's > killing our development iteration speed. We're wasting huge amounts > of time working around this. :( > > > Spurious failures > * > > * 5 x tests/bugs/distribute/bug-1117851.t >(Wstat: 0 Tests: 24 Failed: 1) >Failed test: 15 > >This one is causing a 25% failure rate all by itself. :( > >This needs fixing soon. :) > > > * 3 x tests/bugs/geo-replication/bug-877293.t >(Wstat: 0 Tests: 15 Failed: 1) >Failed test: 11 > > * 2 x tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * 1 x tests/basic/ec/ec-12-4.t >(Wstat: 0 Tests: 541 Failed: 2) >Failed tests: 409, 441 > > * 1 x tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * 1 x tests/basic/uss.t >(Wstat: 0 Tests: 160 Failed: 1) >Failed test: 26 > > * 1 x tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * 1 x tests/bugs/distribute/bug-884455.t >(Wstat: 0 Tests: 22 Failed: 1) >Failed test: 11 > > * 1 x tests/bugs/fuse/bug-1126048.t >(Wstat: 0 Tests: 12 Failed: 1) >Failed test: 10 > > * 1 x tests/bugs/quota/bug-1038598.t >(Wstat: 0 Tests: 28 Failed: 1) >Failed test: 28 > > > 2 x Coredumps > * > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ > >IP - 104.130.74.142 > >This coredump run also failed on: > > * tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > * tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ > >IP - 104.130.74.143 > >This coredump run also failed on: > > * tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > Both VMs are also online, in case they're useful to log into > for investigation (root / the jenkins slave pw). > > If they're not, please let me know so I can blow them away. :) > > > 1 x hung host > * > > Hung on tests/bugs/posix/bug-1113960.t > > root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh > root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh > root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl > /usr/bin/prove -rf --timer ./tests > root 22018 12519 0 00:17 ? S
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Few more test cases causing spurious failures: ./tests/basic/ec/ec-5-1.t Failed test: 69 ./tests/basic/ec/ec-5-2.t Failed test: 69 ./tests/bugs/disperse/bug-1187474.t Failed tests: 11-12 ./tests/basic/ec/nfs.t Failed test: 9 The above failures were seen for the patches which were ineffective, i.e. the code that was modified was never executed as it had no callers. Regards, Poornima - Original Message - > From: "Justin Clift" > To: "Gluster Devel" > Sent: Wednesday, March 4, 2015 9:57:00 AM > Subject: [Gluster-devel] Spurious failure report for master branch - > 2015-03-03 > > Ran 20 x regression tests on our GlusterFS master branch code > as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. > > 5 of them were successful (25%), 15 of them failed in various ways > (75%). > > We need to get this down to about 5% or less (preferably 0%), as it's > killing our development iteration speed. We're wasting huge amounts > of time working around this. :( > > > Spurious failures > * > > * 5 x tests/bugs/distribute/bug-1117851.t > (Wstat: 0 Tests: 24 Failed: > 1) > Failed test: 15 > > This one is causing a 25% failure rate all by itself. :( > > This needs fixing soon. :) > > > * 3 x tests/bugs/geo-replication/bug-877293.t > (Wstat: 0 Tests: 15 Failed: 1) > Failed test: 11 > > * 2 x tests/basic/afr/entry-self-heal.t > (Wstat: 0 Tests: 180 > Failed: 2) > Failed tests: 127-128 > > * 1 x tests/basic/ec/ec-12-4.t > (Wstat: 0 Tests: > 541 Failed: 2) > Failed tests: 409, 441 > > * 1 x tests/basic/fops-sanity.t > (Wstat: 0 Tests: > 11 Failed: 1) > Failed test: 10 > > * 1 x tests/basic/uss.t > (Wstat: 0 > Tests: 160 Failed: 1) > Failed test: 26 > > * 1 x tests/performance/open-behind.t > (Wstat: 0 Tests: 17 > Failed: 1) > Failed test: 17 > > * 1 x tests/bugs/distribute/bug-884455.t > (Wstat: 0 Tests: 22 Failed: > 1) > Failed test: 11 > > * 1 x tests/bugs/fuse/bug-1126048.t > (Wstat: 0 Tests: 12 > Failed: 1) > Failed test: 10 > > * 1 x tests/bugs/quota/bug-1038598.t > (Wstat: 0 Tests: 28 > Failed: 1) > Failed test: 28 > > > 2 x Coredumps > * > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ > > IP - 104.130.74.142 > > This coredump run also failed on: > > * tests/basic/fops-sanity.t > (Wstat: 0 > Tests: 11 Failed: 1) > Failed test: 10 > > * tests/bugs/glusterfs-server/bug-861542.t > (Wstat: 0 Tests: 13 Failed: > 1) > Failed test: 10 > > * tests/performance/open-behind.t > (Wstat: 0 Tests: 17 > Failed: 1) > Failed test: 17 > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ > > IP - 104.130.74.143 > > This coredump run also failed on: > > * tests/basic/afr/entry-self-heal.t > (Wstat: 0 Tests: 180 > Failed: 2) > Failed tests: 127-128 > > * tests/bugs/glusterfs-server/bug-861542.t > (Wstat: 0 Tests: 13 Failed: > 1) > Failed test: 10 > > Both VMs are also online, in case they're useful to log into > for investigation (root / the jenkins slave pw). > > If they're not, please let me know so I can blow them away. :) > > > 1 x hung host > * > > Hung on tests/bugs/posix/bug-1113960.t > > root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh > root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh > root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl > /usr/bin/prove -rf --timer ./tests > root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash > ./tests/bugs/posix/bug-1113960.t > root 30002 22018 0 01:57 ? S 0:00 \_ mv > /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ > > This VM (23.253.53.111) is still online + untouched (still hung), > if someone wants to log in to investigate. (root / the jenkins > slave pw) > > Hope that's helpful. :) > > Regards and best wishes, > > Justin Clift > > -- > GlusterFS - http://www.gluster.org > > An open source, distributed file system scaling to several > petabytes, and handling thousands of clients. > > My personal twitter: twitter.com/realjustinclift > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
Thanks. :) If you need a VM setup in Rackspace for you to investigate on, it's easy to do. Let me know if so. :) + Justin On 4 Mar 2015, at 04:37, Nithya Balachandran wrote: > I'll take a look at tests/bugs/distribute/bug-1117851.t > > Regards, > Nithya > > - Original Message - > From: "Justin Clift" > To: "Gluster Devel" > Sent: Wednesday, 4 March, 2015 9:57:00 AM > Subject: [Gluster-devel] Spurious failure report for master branch - > 2015-03-03 > > Ran 20 x regression tests on our GlusterFS master branch code > as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. > > 5 of them were successful (25%), 15 of them failed in various ways > (75%). > > We need to get this down to about 5% or less (preferably 0%), as it's > killing our development iteration speed. We're wasting huge amounts > of time working around this. :( > > > Spurious failures > * > > * 5 x tests/bugs/distribute/bug-1117851.t >(Wstat: 0 Tests: 24 Failed: 1) >Failed test: 15 > >This one is causing a 25% failure rate all by itself. :( > >This needs fixing soon. :) > > > * 3 x tests/bugs/geo-replication/bug-877293.t >(Wstat: 0 Tests: 15 Failed: 1) >Failed test: 11 > > * 2 x tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * 1 x tests/basic/ec/ec-12-4.t >(Wstat: 0 Tests: 541 Failed: 2) >Failed tests: 409, 441 > > * 1 x tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * 1 x tests/basic/uss.t >(Wstat: 0 Tests: 160 Failed: 1) >Failed test: 26 > > * 1 x tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * 1 x tests/bugs/distribute/bug-884455.t >(Wstat: 0 Tests: 22 Failed: 1) >Failed test: 11 > > * 1 x tests/bugs/fuse/bug-1126048.t >(Wstat: 0 Tests: 12 Failed: 1) >Failed test: 10 > > * 1 x tests/bugs/quota/bug-1038598.t >(Wstat: 0 Tests: 28 Failed: 1) >Failed test: 28 > > > 2 x Coredumps > * > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ > >IP - 104.130.74.142 > >This coredump run also failed on: > > * tests/basic/fops-sanity.t >(Wstat: 0 Tests: 11 Failed: 1) >Failed test: 10 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > * tests/performance/open-behind.t >(Wstat: 0 Tests: 17 Failed: 1) >Failed test: 17 > > * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ > >IP - 104.130.74.143 > >This coredump run also failed on: > > * tests/basic/afr/entry-self-heal.t >(Wstat: 0 Tests: 180 Failed: 2) >Failed tests: 127-128 > > * tests/bugs/glusterfs-server/bug-861542.t >(Wstat: 0 Tests: 13 Failed: 1) >Failed test: 10 > > Both VMs are also online, in case they're useful to log into > for investigation (root / the jenkins slave pw). > > If they're not, please let me know so I can blow them away. :) > > > 1 x hung host > * > > Hung on tests/bugs/posix/bug-1113960.t > > root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh > root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh > root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl > /usr/bin/prove -rf --timer ./tests > root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash > ./tests/bugs/posix/bug-1113960.t > root 30002 22018 0 01:57 ? S 0:00 \_ mv > /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ > > This VM (23.253.53.111) is still online + untouched (still hung), > if someone wants to log in to investigate. (root / the jenkins > slave pw) > > Hope that's helpful. :) > > Regards and best wishes, > > Justin Clift > > -- > GlusterFS - http://www.gluster.org > > An open source, distributed file system scaling to several > petabytes, and handling thousands of clients. > > My personal twitter: twitter.com/realjustinclift > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel -- GlusterF
Re: [Gluster-devel] Spurious failure report for master branch - 2015-03-03
I'll take a look at tests/bugs/distribute/bug-1117851.t Regards, Nithya - Original Message - From: "Justin Clift" To: "Gluster Devel" Sent: Wednesday, 4 March, 2015 9:57:00 AM Subject: [Gluster-devel] Spurious failure report for master branch - 2015-03-03 Ran 20 x regression tests on our GlusterFS master branch code as of a few hours ago, commit 95d5e60afb29aedc29909340e7564d54a6a247c2. 5 of them were successful (25%), 15 of them failed in various ways (75%). We need to get this down to about 5% or less (preferably 0%), as it's killing our development iteration speed. We're wasting huge amounts of time working around this. :( Spurious failures * * 5 x tests/bugs/distribute/bug-1117851.t (Wstat: 0 Tests: 24 Failed: 1) Failed test: 15 This one is causing a 25% failure rate all by itself. :( This needs fixing soon. :) * 3 x tests/bugs/geo-replication/bug-877293.t (Wstat: 0 Tests: 15 Failed: 1) Failed test: 11 * 2 x tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * 1 x tests/basic/ec/ec-12-4.t (Wstat: 0 Tests: 541 Failed: 2) Failed tests: 409, 441 * 1 x tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * 1 x tests/basic/uss.t (Wstat: 0 Tests: 160 Failed: 1) Failed test: 26 * 1 x tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * 1 x tests/bugs/distribute/bug-884455.t (Wstat: 0 Tests: 22 Failed: 1) Failed test: 11 * 1 x tests/bugs/fuse/bug-1126048.t (Wstat: 0 Tests: 12 Failed: 1) Failed test: 10 * 1 x tests/bugs/quota/bug-1038598.t (Wstat: 0 Tests: 28 Failed: 1) Failed test: 28 2 x Coredumps * * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk5/ IP - 104.130.74.142 This coredump run also failed on: * tests/basic/fops-sanity.t (Wstat: 0 Tests: 11 Failed: 1) Failed test: 10 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 * tests/performance/open-behind.t (Wstat: 0 Tests: 17 Failed: 1) Failed test: 17 * http://mirror.salasaga.org/gluster/master/2015-03-03/bulk8/ IP - 104.130.74.143 This coredump run also failed on: * tests/basic/afr/entry-self-heal.t (Wstat: 0 Tests: 180 Failed: 2) Failed tests: 127-128 * tests/bugs/glusterfs-server/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 Both VMs are also online, in case they're useful to log into for investigation (root / the jenkins slave pw). If they're not, please let me know so I can blow them away. :) 1 x hung host * Hung on tests/bugs/posix/bug-1113960.t root 12497 1290 0 Mar03 ? S 0:00 \_ /bin/bash /opt/qa/regression.sh root 12504 12497 0 Mar03 ? S 0:00 \_ /bin/bash ./run-tests.sh root 12519 12504 0 Mar03 ? S 0:03 \_ /usr/bin/perl /usr/bin/prove -rf --timer ./tests root 22018 12519 0 00:17 ? S 0:00 \_ /bin/bash ./tests/bugs/posix/bug-1113960.t root 30002 22018 0 01:57 ? S 0:00 \_ mv /mnt/glusterfs/0/longernamedir1/longernamedir2/longernamedir3/ This VM (23.253.53.111) is still online + untouched (still hung), if someone wants to log in to investigate. (root / the jenkins slave pw) Hope that's helpful. :) Regards and best wishes, Justin Clift -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
