Re: [Gluster-devel] regression burn-in summary over the last 7 days
On Wed, Aug 3, 2016 at 8:00 PM, Vijay Bellur wrote: > [Thanks to Jeff Darcy, Poornima G and Raghavendra Talur for > extras/failed-tests.py which generates this summary] > > Found cores: > > Component: [u'glusterfs-1879.core'] > Regression Link: > https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull > > ./tests/performance/open-behind.t ; Failed 3 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1443/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1426/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1425/consoleFull > > ./tests/bugs/glusterd/bug-963541.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1431/consoleFull > > ./tests/basic/ec/ec-new-entry.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull > > ./tests/bugs/bug-1110262.t ; Failed 2 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1424/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1418/consoleFull > > ./tests/bugs/glusterd/bug-1089668.t ; Failed 2 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1447/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1435/consoleFull > > This has been RCAed and an email on this [1] was sent. Can someone from DHT team send the fix? [1] https://www.gluster.org/pipermail/gluster-devel/2016-July/050129.html > ./tests/basic/volume-snapshot.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1429/consoleFull > > ./tests/basic/gfapi/bug1291259.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1450/consoleFull > > ./tests/bugs/quick-read/bug-846240.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1416/consoleFull > > ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull > > maintainers & owners of above tests - can we please address these failures? > > Thanks, > Vijay > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > -- --Atin ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
I think we should compare n-1th week report with nth and any common tests out of that comparison having more than or equal to 3 instances of failures SHOULD immediately marked as bad and any component having more than 5 bad tests should be BLOCKED for further merges till bad tests are fixed. What do others think about it? On Wednesday 3 August 2016, Vijay Bellur wrote: > [Thanks to Jeff Darcy, Poornima G and Raghavendra Talur for > extras/failed-tests.py which generates this summary] > > Found cores: > > Component: [u'glusterfs-1879.core'] > Regression Link: > https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull > > ./tests/performance/open-behind.t ; Failed 3 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1443/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1426/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1425/consoleFull > > ./tests/bugs/glusterd/bug-963541.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1431/consoleFull > > ./tests/basic/ec/ec-new-entry.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull > > ./tests/bugs/bug-1110262.t ; Failed 2 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1424/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1418/consoleFull > > ./tests/bugs/glusterd/bug-1089668.t ; Failed 2 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1447/consoleFull > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1435/consoleFull > > ./tests/basic/volume-snapshot.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1429/consoleFull > > ./tests/basic/gfapi/bug1291259.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1450/consoleFull > > ./tests/bugs/quick-read/bug-846240.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1416/consoleFull > > ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull > > maintainers & owners of above tests - can we please address these failures? > > Thanks, > Vijay > -- --Atin ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On Wed, Aug 03, 2016 at 10:30:28AM -0400, Vijay Bellur wrote: > ./tests/basic/gfapi/bug1291259.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1450/consoleFull I'm modifying this test for a cleanup in gfapi+upcall and found a problem with it already. This will get addressed this week or early next. > ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times > Regression Links: > https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull I have not seen this fail yet... All gfapi tests are running in a loop on a test-system now, we'll see if it reproducible in a few days or so. Niels signature.asc Description: PGP signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On Wed, Aug 3, 2016 at 8:19 PM, Atin Mukherjee wrote: > > > On Wed, Aug 3, 2016 at 8:00 PM, Vijay Bellur wrote: > >> [Thanks to Jeff Darcy, Poornima G and Raghavendra Talur for >> extras/failed-tests.py which generates this summary] >> >> Found cores: >> >> Component: [u'glusterfs-1879.core'] >> Regression Link: >> https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull >> >> ./tests/performance/open-behind.t ; Failed 3 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1443/consoleFull >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1426/consoleFull >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1425/consoleFull >> >> ./tests/bugs/glusterd/bug-963541.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1431/consoleFull >> >> ./tests/basic/ec/ec-new-entry.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1451/consoleFull >> >> ./tests/bugs/bug-1110262.t ; Failed 2 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1424/consoleFull >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1418/consoleFull >> >> ./tests/bugs/glusterd/bug-1089668.t ; Failed 2 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1447/consoleFull >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1435/consoleFull >> >> > This has been RCAed and an email on this [1] was sent. Can someone from > DHT team send the fix? > > [1] https://www.gluster.org/pipermail/gluster-devel/2016-July/050129.html > I've sent a patch http://review.gluster.org/15085 for this. > > > >> ./tests/basic/volume-snapshot.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1429/consoleFull >> >> ./tests/basic/gfapi/bug1291259.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1450/consoleFull >> >> ./tests/bugs/quick-read/bug-846240.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1416/consoleFull >> >> ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times >> Regression Links: >> https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull >> >> maintainers & owners of above tests - can we please address these >> failures? >> >> Thanks, >> Vijay >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel >> > > > > -- > > --Atin > -- --Atin ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On Thu, Aug 04, 2016 at 12:00:53AM +0200, Niels de Vos wrote: > On Wed, Aug 03, 2016 at 10:30:28AM -0400, Vijay Bellur wrote: > ... > > ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times > > Regression Links: > > https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull > > I have not seen this fail yet... All gfapi tests are running in a loop > on a test-system now, we'll see if it reproducible in a few days or so. It seems that glfs_fini() returns -1 every now and then (once after 1027 iterations, once after 287). Some of the gfapi test cases actually succeed their intended test, but still return an error when glfs_fini() fails. I am tempted to just skip this error in most tests and have only tests/basic/gfapi/libgfapi-fini-hang error out on it. (Obviously also intend to fix the failure.) Thoughts? Niels signature.asc Description: PGP signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On 08/04/2016 08:07 AM, Niels de Vos wrote: > On Thu, Aug 04, 2016 at 12:00:53AM +0200, Niels de Vos wrote: >> On Wed, Aug 03, 2016 at 10:30:28AM -0400, Vijay Bellur wrote: >> > ... >>> ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times >>> Regression Links: >>> https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull >> >> I have not seen this fail yet... All gfapi tests are running in a loop >> on a test-system now, we'll see if it reproducible in a few days or so. > > It seems that glfs_fini() returns -1 every now and then (once after 1027 > iterations, once after 287). Some of the gfapi test cases actually > succeed their intended test, but still return an error when glfs_fini() > fails. I am tempted to just skip this error in most tests and have only > tests/basic/gfapi/libgfapi-fini-hang error out on it. (Obviously also > intend to fix the failure.) If you fix the bug in glfs_fini() then it should not be necessary to ignore the failure in the tests, right? Just fix the bug, don't hack the test. -- Kaleb signature.asc Description: OpenPGP digital signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On 08/04/2016 05:40 PM, Kaleb KEITHLEY wrote: On 08/04/2016 08:07 AM, Niels de Vos wrote: On Thu, Aug 04, 2016 at 12:00:53AM +0200, Niels de Vos wrote: On Wed, Aug 03, 2016 at 10:30:28AM -0400, Vijay Bellur wrote: ... ./tests/bugs/gfapi/bug-1093594.t ; Failed 1 times Regression Links: https://build.gluster.org/job/regression-test-burn-in/1423/consoleFull I have not seen this fail yet... All gfapi tests are running in a loop on a test-system now, we'll see if it reproducible in a few days or so. It seems that glfs_fini() returns -1 every now and then (once after 1027 iterations, once after 287). Some of the gfapi test cases actually succeed their intended test, but still return an error when glfs_fini() fails. I am tempted to just skip this error in most tests and have only tests/basic/gfapi/libgfapi-fini-hang error out on it. (Obviously also intend to fix the failure.) If you fix the bug in glfs_fini() then it should not be necessary to ignore the failure in the tests, right? Just fix the bug, don't hack the test. -- Kaleb I've faced similar issues with glfs_fini() while working on the bareos integration. When using a libgfapi built with --enable-debug, an assert causes the process to dump core. https://bugzilla.redhat.com/show_bug.cgi?id=1233136 may be worth addressing. Milind ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] regression burn-in summary over the last 7 days
On Wed, Aug 3, 2016 at 1:37 PM, Atin Mukherjee wrote: > I think we should compare n-1th week report with nth and any common tests > out of that comparison having more than or equal to 3 instances of failures > SHOULD immediately marked as bad and any component having more than 5 bad > tests should be BLOCKED for further merges till bad tests are fixed. What do > others think about it? > If we can automate this workflow, I am in favor of the proposal. -Vijay ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
