Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
We had a debug session to find the root cause. The issue with incorrect usage with depends on annotation in SchemaAddTestCase. Only one test case depends on another case and rest of the test case execution happened in random order. After putting depends on annotation and getting the tests to run in order solve the problem. We might need to document this behavior with TestNG in our wiki. I will check whether this is TestNG bug or not. Thanks, Krishantha. On Fri, Jul 11, 2014 at 2:18 PM, Sameera Kannangara samee...@wso2.com wrote: HI Krishantha, Please find TA JIRA created at [1] https://wso2.org/jira/browse/TA-857. We will exclude test failing due to interleaving, when taking test summaries. Thank you, SameeraK [1] https://wso2.org/jira/browse/TA-857 On Fri, Jul 11, 2014 at 12:57 PM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi Sameera, It will take bit of time to detect root cause of this problem. So can you disable the interleaving test cases and create TA JIRA to track it. Thanks, Krishantha. On Fri, Jul 11, 2014 at 12:12 AM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi all, You can find reasons for this kind of intermittent issue at TAF wiki - http://docs.wso2.com/display/TA430/FAQ#FAQ-Whydoesmytestmodulecontainintermittenttestfailures? I suspect this issue is something to do with test resource cleanup. Need to check run all tests and do extensive debug session. Please share active GIT repo URL. Thanks, Krishantha. On Thu, Jul 10, 2014 at 4:46 PM, Subash Chaturanga sub...@wso2.com wrote: Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Sameera Kannangara, Software Engineer, WSO2 Inc. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
Noted. Thanks, Tania On Mon, Jul 14, 2014 at 6:44 PM, Krishantha Samaraweera krishan...@wso2.com wrote: We had a debug session to find the root cause. The issue with incorrect usage with depends on annotation in SchemaAddTestCase. Only one test case depends on another case and rest of the test case execution happened in random order. After putting depends on annotation and getting the tests to run in order solve the problem. We might need to document this behavior with TestNG in our wiki. I will check whether this is TestNG bug or not. Thanks, Krishantha. On Fri, Jul 11, 2014 at 2:18 PM, Sameera Kannangara samee...@wso2.com wrote: HI Krishantha, Please find TA JIRA created at [1] https://wso2.org/jira/browse/TA-857. We will exclude test failing due to interleaving, when taking test summaries. Thank you, SameeraK [1] https://wso2.org/jira/browse/TA-857 On Fri, Jul 11, 2014 at 12:57 PM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi Sameera, It will take bit of time to detect root cause of this problem. So can you disable the interleaving test cases and create TA JIRA to track it. Thanks, Krishantha. On Fri, Jul 11, 2014 at 12:12 AM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi all, You can find reasons for this kind of intermittent issue at TAF wiki - http://docs.wso2.com/display/TA430/FAQ#FAQ-Whydoesmytestmodulecontainintermittenttestfailures? I suspect this issue is something to do with test resource cleanup. Need to check run all tests and do extensive debug session. Please share active GIT repo URL. Thanks, Krishantha. On Thu, Jul 10, 2014 at 4:46 PM, Subash Chaturanga sub...@wso2.com wrote: Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Sameera Kannangara, Software Engineer, WSO2 Inc. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Tania Mahanama Senior Technical Writer Contact: Mob: +94 077 5129270 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
Hi Sameera, It will take bit of time to detect root cause of this problem. So can you disable the interleaving test cases and create TA JIRA to track it. Thanks, Krishantha. On Fri, Jul 11, 2014 at 12:12 AM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi all, You can find reasons for this kind of intermittent issue at TAF wiki - http://docs.wso2.com/display/TA430/FAQ#FAQ-Whydoesmytestmodulecontainintermittenttestfailures? I suspect this issue is something to do with test resource cleanup. Need to check run all tests and do extensive debug session. Please share active GIT repo URL. Thanks, Krishantha. On Thu, Jul 10, 2014 at 4:46 PM, Subash Chaturanga sub...@wso2.com wrote: Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
HI Krishantha, Please find TA JIRA created at [1] https://wso2.org/jira/browse/TA-857. We will exclude test failing due to interleaving, when taking test summaries. Thank you, SameeraK [1] https://wso2.org/jira/browse/TA-857 On Fri, Jul 11, 2014 at 12:57 PM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi Sameera, It will take bit of time to detect root cause of this problem. So can you disable the interleaving test cases and create TA JIRA to track it. Thanks, Krishantha. On Fri, Jul 11, 2014 at 12:12 AM, Krishantha Samaraweera krishan...@wso2.com wrote: Hi all, You can find reasons for this kind of intermittent issue at TAF wiki - http://docs.wso2.com/display/TA430/FAQ#FAQ-Whydoesmytestmodulecontainintermittenttestfailures? I suspect this issue is something to do with test resource cleanup. Need to check run all tests and do extensive debug session. Please share active GIT repo URL. Thanks, Krishantha. On Thu, Jul 10, 2014 at 4:46 PM, Subash Chaturanga sub...@wso2.com wrote: Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [G-Reg][integration-testing] Interleaving test classes
Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [G-Reg][integration-testing] Interleaving test classes
Hi all, You can find reasons for this kind of intermittent issue at TAF wiki - http://docs.wso2.com/display/TA430/FAQ#FAQ-Whydoesmytestmodulecontainintermittenttestfailures? I suspect this issue is something to do with test resource cleanup. Need to check run all tests and do extensive debug session. Please share active GIT repo URL. Thanks, Krishantha. On Thu, Jul 10, 2014 at 4:46 PM, Subash Chaturanga sub...@wso2.com wrote: Adding Krishantha On Thu, Jul 10, 2014 at 3:57 PM, Sameera Kannangara samee...@wso2.com wrote: Hi all, When running G-Reg tests I've encountered several test cases which passes when the containing class executed individually and fails when tests are run as packages. Following are few examples. deleteResources(org.wso2.carbon.registry.metadata.test.schema.SchemaAddTestCase) addAssociationToRoot(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) cleanup(org.wso2.carbon.registry.ws.client.test.general.old.TestAssociationWSTestCase) When remote debugging these test classes it is observed that tests from different classes run interleaved, and that some times lead to changing and removing shared resources, results in test failures. How can we solve this problem? Thank you, SameeraK -- Sameera Kannangara, Software Engineer, WSO2 Inc. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Thanks /subash *Subash Chaturanga* Senior Software Engineer Lead WSO2 Governance Registry Platform TG; WSO2 Inc. http://wso2.com Contact: email: sub...@wso2.com blog: http://subashsdm.blogspot.com/ twitter: @subash89 phone: +9477 2225922 Lean . Enterprise . Middleware -- Krishantha Samaraweera Senior Technical Lead - Test Automation Mobile: +94 77 7759918 WSO2, Inc.; http://wso2.com/ lean . enterprise . middlewear. ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
