As our regression tests have grown bigger and bigger - i have been observing a pattern of repeated builds which are requested every patch.
Here are the list of scenarios that i have observed - Testcases pass on local laptop, while fail on Jenkins - Testcases fail on local laptop, while pass on jenkins - Testcases fail on both laptop and jenkins but on a second run they pass - Testcases fail randomly for a totally unrelated patch example Example: http://review.gluster.com/#/c/7317/ Gluster Build System Mar 28 9:20 AM --------------------------------------------------------- Patch Set 6: Verified-1 http://build.gluster.org/job/regression/3885/consoleFull : FAILED --------------------------------------------------------- Gluster Build System Mar 28 8:55 PM --------------------------------------------------------- Patch Set 6: Verified+1 http://build.gluster.org/job/regression/3893/consoleFull : SUCCESS --------------------------------------------------------- There has been no change the patch which was submitted. Another example - http://review.gluster.com/#/c/7278/ --------------------------------------------------------- Gluster Build System Mar 31 10:22 AM Patch Set 4: Verified-1 http://build.gluster.org/job/regression/3900/consoleFull : FAILED Kaleb KEITHLEY Mar 31 11:02 AM Patch Set 4: build.gluster.org:/d/cores_from_mar_31/core.27188 is from one of these tests: ... [10:51:52] ./tests/bugs/bug-1034085.t ........................ ok 11 s No volumes present setfattr: /mnt/glusterfs/0/: Operation not permitted setfattr: /mnt/glusterfs/0/: Operation not permitted [10:52:05] ./tests/bugs/bug-1034716.t ........................ ok 13 s [10:52:24] ./tests/bugs/bug-1035576.t ........................ ok 19 s No volumes present [10:52:54] ./tests/bugs/bug-1037501.t ........................ ok 30 s No volumes present 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0248987 s, 168 MB/s 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0393089 s, 160 MB/s [10:53:16] ./tests/bugs/bug-1038598.t ........................ ok 22 s Gluster Build System Mar 31 11:54 AM Patch Set 4: Verified+1 http://build.gluster.org/job/regression/3901/consoleFull : SUCCESS --------------------------------------------------------- My general growing concern is we might be taking in patches which are not failing where they should have while a growing number of valid patches might just fail and vice-versa. This will grow larger and larger as the individual test cases are not consistent in the way they behave in different outcomes - we are looking a major issue here. Since we can't be certain of the "regression" validity that a bug is fixed or the same bug is re-introduced by a new patch. Cheers -- Religious confuse piety with mere ritual, the virtuous confuse regulation with outcomes _______________________________________________ Gluster-devel mailing list Gluster-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/gluster-devel
