[Dev] Random test failures in Jenkins
I have noticed random test failures in Jenkins is getting increase specially in Java7 builds, as an example among 5 product-greg[1] builds 4 were unstable due to random test failures ( each time due to different test cases ). In our local builds we are not getting any of these random failures and no SCM changes as well, it looks like Jenkins issue to me. G-Reg team looked into this but still no luck, can Jenkins volunteers team look into these issues ? FYI product-greg job info during 5 consecutive runs. Build #439 - No SCM changes org.wso2.carbon.registry.app.test.CommentTestCase.AddComment Build #440 - No SCM changes --- org.wso2.carbon.registry.reporting.test.ReportClassTestCases.testgetAttributeNames org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs org.wso2.carbon.registry.tck.cmis.CMISTck.runTckTest Build #441 - No SCM changes -- org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs Build #442 - No SCM changes - org.wso2.carbon.registry.app.test.FileSystemImportExportTestCase.FileImportTest Build #443 - No SCM changes -- Success [1] - https://wso2.org/jenkins/view/product-builds/job/product-greg/ Thanks ! -- Sagara Gunathunga Senior Technical Lead; WSO2, Inc.; http://wso2.com V.P Apache Web Services;http://ws.apache.org/ Linkedin; http://www.linkedin.com/in/ssagara Blog ; http://ssagara.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Random test failures in Jenkins
Hi Sagara, While analyzing the configuration history of product-greg [1] (those who have permission can view this) found that jdk version is changed from Oracle-JDK-1.6.0_33 to Oracle-JDK-1.7.0_51. I think this issue was introduced by this JDK change. When the unit test are run in Java7 it may run in different order, Order of test cases are not guaranteed unless it is specifically specified. So you have to add "@FixMethodOrder(MethodSorters.NAME_ASCENDING)" to all your test classes to guarantee the order. Please try with the above fix. [1] https://wso2.org/jenkins/job/product-greg/jobConfigHistory/showDiffFiles?timestamp1=2015-03-07_21-35-42×tamp2=2015-03-11_12-14-37 Thanks, Jayanga. *Jayanga Dissanayake* Senior Software Engineer WSO2 Inc. - http://wso2.com/ lean . enterprise . middleware email: jaya...@wso2.com mobile: +94772207259 On Sun, Mar 15, 2015 at 8:22 AM, Sagara Gunathunga wrote: > > I have noticed random test failures in Jenkins is getting increase > specially in Java7 builds, as an example among 5 product-greg[1] builds 4 > were unstable due to random test failures ( each time due to different test > cases ). In our local builds we are not getting any of these random > failures and no SCM changes as well, it looks like Jenkins issue to me. > > G-Reg team looked into this but still no luck, can Jenkins volunteers team > look into these issues ? > > > FYI product-greg job info during 5 consecutive runs. > > Build #439 - No SCM changes > > org.wso2.carbon.registry.app.test.CommentTestCase.AddComment > > > Build #440 - No SCM changes > --- > > org.wso2.carbon.registry.reporting.test.ReportClassTestCases.testgetAttributeNames > > org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs > org.wso2.carbon.registry.tck.cmis.CMISTck.runTckTest > > > Build #441 - No SCM changes > -- > > org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs > > Build #442 - No SCM changes > - > > org.wso2.carbon.registry.app.test.FileSystemImportExportTestCase.FileImportTest > > > Build #443 - No SCM changes > -- > Success > > > [1] - https://wso2.org/jenkins/view/product-builds/job/product-greg/ > > > Thanks ! > -- > Sagara Gunathunga > > Senior Technical Lead; WSO2, Inc.; http://wso2.com > V.P Apache Web Services;http://ws.apache.org/ > Linkedin; http://www.linkedin.com/in/ssagara > Blog ; http://ssagara.blogspot.com > > ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Random test failures in Jenkins
On Sun, Mar 15, 2015 at 10:32 AM, Jayanga Dissanayake wrote: > Hi Sagara, > > While analyzing the configuration history of product-greg [1] (those who > have permission can view this) found that jdk version is changed from > Oracle-JDK-1.6.0_33 > to Oracle-JDK-1.7.0_51. I think this issue was introduced by this JDK > change. > Yes this is intentional change, now all G-Reg own projects should compile on Java7. We also noticed this issue started after moving to Java7 but only in Jenkins, as C 4.4 based products will be based on Java7/8 we need to find why these Java7 tests are randomly fail only in Jenkins. > > When the unit test are run in Java7 it may run in different order, Order > of test cases are not guaranteed unless it is specifically specified. > So you have to add "@FixMethodOrder(MethodSorters.NAME_ASCENDING)" to all > your test classes to guarantee the order. > > Please try with the above fix. > AFAIK we got this ordering issue earlier and already fixed, BTW Chandana/Prasanna can you guys double check whether we have missed anything ? Thanks ! > > [1] > https://wso2.org/jenkins/job/product-greg/jobConfigHistory/showDiffFiles?timestamp1=2015-03-07_21-35-42×tamp2=2015-03-11_12-14-37 > > Thanks, > Jayanga. > > *Jayanga Dissanayake* > Senior Software Engineer > WSO2 Inc. - http://wso2.com/ > lean . enterprise . middleware > email: jaya...@wso2.com > mobile: +94772207259 > > On Sun, Mar 15, 2015 at 8:22 AM, Sagara Gunathunga > wrote: > >> >> I have noticed random test failures in Jenkins is getting increase >> specially in Java7 builds, as an example among 5 product-greg[1] builds 4 >> were unstable due to random test failures ( each time due to different test >> cases ). In our local builds we are not getting any of these random >> failures and no SCM changes as well, it looks like Jenkins issue to me. >> >> G-Reg team looked into this but still no luck, can Jenkins volunteers >> team look into these issues ? >> >> >> FYI product-greg job info during 5 consecutive runs. >> >> Build #439 - No SCM changes >> >> org.wso2.carbon.registry.app.test.CommentTestCase.AddComment >> >> >> Build #440 - No SCM changes >> --- >> >> org.wso2.carbon.registry.reporting.test.ReportClassTestCases.testgetAttributeNames >> >> org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs >> org.wso2.carbon.registry.tck.cmis.CMISTck.runTckTest >> >> >> Build #441 - No SCM changes >> -- >> >> org.wso2.carbon.registry.server.mgt.test.ServerStartupTestCase.testVerifyLogs >> >> Build #442 - No SCM changes >> - >> >> org.wso2.carbon.registry.app.test.FileSystemImportExportTestCase.FileImportTest >> >> >> Build #443 - No SCM changes >> -- >> Success >> >> >> [1] - https://wso2.org/jenkins/view/product-builds/job/product-greg/ >> >> >> Thanks ! >> -- >> Sagara Gunathunga >> >> Senior Technical Lead; WSO2, Inc.; http://wso2.com >> V.P Apache Web Services;http://ws.apache.org/ >> Linkedin; http://www.linkedin.com/in/ssagara >> Blog ; http://ssagara.blogspot.com >> >> > -- Sagara Gunathunga Senior Technical Lead; WSO2, Inc.; http://wso2.com V.P Apache Web Services;http://ws.apache.org/ Linkedin; http://www.linkedin.com/in/ssagara Blog ; http://ssagara.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
