[Dev] Automation framework's inconsistent behavior
Hi All,
Apparently there's some serious ordering/execution issue when the bunch of
tests run with the framework.
For example if you run a single class LCStateIdNameTestCase all test passes
but fails when running only with the package
(org.wso2.carbon.registry.lifecycle.test in this case). Failure below with
the reason for the same. This fails as the logs indicate a different LC is
attached to the service while executing the method.
So how this is possible? There can be multiple reasons but one can be the
use of same service/metadata in multiple test cases and another test case
attaches a different LC to the same service referred by the current test
case.
One way this can be avoided (at least for the most part) is re-factor the
test cases to avoid these multiple test methods with @Test (to add
service/LC etc...) in that the main test case in focus do the necessary
method calls directly (to setup required services etc...) and do the
asserts (This can be done some restructuring of the current tests) itself.
Is there any other way this interference can be avoided within the
framework?
[2013-08-07 16:33:49,811] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - On test success..
[2013-08-07 16:33:49,812] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - Running the test
method ---
org.wso2.carbon.registry.lifecycle.test.LCStateIdNameTestCase.testAddLcToService
+++ LC PR:
+/trunk/services/com/abb/IntergalacticService, *
StateIdNameLC*
++ LC PROP: +++registry.lifecycle.*
TransitionApprovalLC*.state
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item
[2013-08-07 16:33:50,082] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} - On Test failure..
[2013-08-07 16:33:50,083] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} -
*java.lang.AssertionError:
LifeCycle not added to service expected [true] but found [false]*
Regards,
Vijitha.
--
Vijitha Kumara
Senior Software Engineer; WSO2, Inc.; http://wso2.com/
email: viji...@wso2.com
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Automation framework's inconsistent behavior
This happens when the resources used for a test is not cleared
properly after the test. Those clearance should be called under AfterClass
or After Test annotated methods. Then Those methods will call even if the
test is failed or passed. Then there are no effects of other test classes.
Some test methods have all the steps inside it. Once it gets failed, It may
affect other test too. If we do that properly we can use the same
resources in many test cases. if not we have to use different names for
artifact we use.
Thanks,
Nuwanw
On Wed, Aug 7, 2013 at 5:44 PM, Vijitha Kumara viji...@wso2.com wrote:
Hi All,
Apparently there's some serious ordering/execution issue when the bunch of
tests run with the framework.
For example if you run a single class LCStateIdNameTestCase all test
passes but fails when running only with the package
(org.wso2.carbon.registry.lifecycle.test in this case). Failure below with
the reason for the same. This fails as the logs indicate a different LC is
attached to the service while executing the method.
So how this is possible? There can be multiple reasons but one can be the
use of same service/metadata in multiple test cases and another test case
attaches a different LC to the same service referred by the current test
case.
One way this can be avoided (at least for the most part) is re-factor the
test cases to avoid these multiple test methods with @Test (to add
service/LC etc...) in that the main test case in focus do the necessary
method calls directly (to setup required services etc...) and do the
asserts (This can be done some restructuring of the current tests) itself.
Is there any other way this interference can be avoided within the
framework?
[2013-08-07 16:33:49,811] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - On test success..
[2013-08-07 16:33:49,812] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - Running the test
method ---
org.wso2.carbon.registry.lifecycle.test.LCStateIdNameTestCase.testAddLcToService
+++ LC PR:
+/trunk/services/com/abb/IntergalacticService, *
StateIdNameLC*
++ LC PROP: +++registry.lifecycle.*
TransitionApprovalLC*.state
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item
[2013-08-07 16:33:50,082] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} - On Test failure..
[2013-08-07 16:33:50,083] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} -
*java.lang.AssertionError:
LifeCycle not added to service expected [true] but found [false]*
Regards,
Vijitha.
--
Vijitha Kumara
Senior Software Engineer; WSO2, Inc.; http://wso2.com/
email: viji...@wso2.com
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
--
Nuwan Wimalasekara
Senior Software Engineer - Test Automation
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
phone: +94 71 668 4620
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Automation framework's inconsistent behavior
Hi Nuwan,
I see most of the test cases have cleanup code to do the necessary stuff
but we need to review all of them as well if there's any doubt.
But this is not about resource cleanup rather the issue with the inferences
and the related consequences I explained. The test in focus is not done but
while invoking one of its methods it has met with an unexpected state (that
can be a result of another test case which might also not finished).
If the latter is finished then it might have cleaned up the service (same
service used across both classes in this case) so it might fail the current
test case with no service found, which is another side effect of these. So
this might have to do with ordering/execution as well. I'm not sure about
the implementation but we can analyze that.
On Wed, Aug 7, 2013 at 6:02 PM, Nuwan Wimalasekara nuw...@wso2.com wrote:
This happens when the resources used for a test is not cleared
properly after the test. Those clearance should be called under AfterClass
or After Test annotated methods. Then Those methods will call even if the
test is failed or passed. Then there are no effects of other test classes.
Some test methods have all the steps inside it. Once it gets failed, It
may affect other test too. If we do that properly we can use the same
resources in many test cases. if not we have to use different names for
artifact we use.
That's also a possible solution for these kind which requires redoing all
test classes again which I'm not sure an option with the time frames of the
current release.
Regards,
Vijitha.
Thanks,
Nuwanw
On Wed, Aug 7, 2013 at 5:44 PM, Vijitha Kumara viji...@wso2.com wrote:
Hi All,
Apparently there's some serious ordering/execution issue when the bunch
of tests run with the framework.
For example if you run a single class LCStateIdNameTestCase all test
passes but fails when running only with the package
(org.wso2.carbon.registry.lifecycle.test in this case). Failure below with
the reason for the same. This fails as the logs indicate a different LC is
attached to the service while executing the method.
So how this is possible? There can be multiple reasons but one can be the
use of same service/metadata in multiple test cases and another test case
attaches a different LC to the same service referred by the current test
case.
One way this can be avoided (at least for the most part) is re-factor the
test cases to avoid these multiple test methods with @Test (to add
service/LC etc...) in that the main test case in focus do the necessary
method calls directly (to setup required services etc...) and do the
asserts (This can be done some restructuring of the current tests) itself.
Is there any other way this interference can be avoided within the
framework?
[2013-08-07 16:33:49,811] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - On test success..
[2013-08-07 16:33:49,812] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - Running the test
method ---
org.wso2.carbon.registry.lifecycle.test.LCStateIdNameTestCase.testAddLcToService
+++ LC PR:
+/trunk/services/com/abb/IntergalacticService, *
StateIdNameLC*
++ LC PROP: +++registry.lifecycle.*
TransitionApprovalLC*.state
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item
[2013-08-07 16:33:50,082] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} - On Test failure..
[2013-08-07 16:33:50,083] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} -
*java.lang.AssertionError:
LifeCycle not added to service expected [true] but found [false]*
Regards,
Vijitha.
--
Vijitha Kumara
Senior Software Engineer; WSO2, Inc.; http://wso2.com/
email: viji...@wso2.com
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
--
Nuwan Wimalasekara
Senior Software Engineer - Test Automation
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
phone: +94 71 668 4620
--
Vijitha Kumara
Senior Software Engineer; WSO2, Inc.; http://wso2.com/
email: viji...@wso2.com
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Automation framework's inconsistent behavior
First of all this is not framework issue. The issue has caused by the way
we have done test design. The same behavior was observed when using
priority and dependOn annotation together (could be caused by testNG issue)
I have explained the reason for the using of duplicate create
service/LC methods
in another mail thread.
I don't think we need to redo the test code here. All these tests were
successful with G-Reg 4.5.3. IMO we can use unique service names and fix
the problem.
Thanks,
Krishantha.
On Wed, Aug 7, 2013 at 7:32 PM, Vijitha Kumara viji...@wso2.com wrote:
Hi Nuwan,
I see most of the test cases have cleanup code to do the necessary stuff
but we need to review all of them as well if there's any doubt.
But this is not about resource cleanup rather the issue with the
inferences and the related consequences I explained. The test in focus is
not done but while invoking one of its methods it has met with an
unexpected state (that can be a result of another test case which might
also not finished).
If the latter is finished then it might have cleaned up the service (same
service used across both classes in this case) so it might fail the current
test case with no service found, which is another side effect of these. So
this might have to do with ordering/execution as well. I'm not sure about
the implementation but we can analyze that.
On Wed, Aug 7, 2013 at 6:02 PM, Nuwan Wimalasekara nuw...@wso2.comwrote:
This happens when the resources used for a test is not cleared
properly after the test. Those clearance should be called under AfterClass
or After Test annotated methods. Then Those methods will call even if the
test is failed or passed. Then there are no effects of other test classes.
Some test methods have all the steps inside it. Once it gets failed, It
may affect other test too. If we do that properly we can use the same
resources in many test cases. if not we have to use different names for
artifact we use.
That's also a possible solution for these kind which requires redoing all
test classes again which I'm not sure an option with the time frames of the
current release.
Regards,
Vijitha.
Thanks,
Nuwanw
On Wed, Aug 7, 2013 at 5:44 PM, Vijitha Kumara viji...@wso2.com wrote:
Hi All,
Apparently there's some serious ordering/execution issue when the bunch
of tests run with the framework.
For example if you run a single class LCStateIdNameTestCase all test
passes but fails when running only with the package
(org.wso2.carbon.registry.lifecycle.test in this case). Failure below with
the reason for the same. This fails as the logs indicate a different LC is
attached to the service while executing the method.
So how this is possible? There can be multiple reasons but one can be
the use of same service/metadata in multiple test cases and another test
case attaches a different LC to the same service referred by the current
test case.
One way this can be avoided (at least for the most part) is re-factor
the test cases to avoid these multiple test methods with @Test (to add
service/LC etc...) in that the main test case in focus do the necessary
method calls directly (to setup required services etc...) and do the
asserts (This can be done some restructuring of the current tests) itself.
Is there any other way this interference can be avoided within the
framework?
[2013-08-07 16:33:49,811] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - On test success..
[2013-08-07 16:33:49,812] INFO
{org.wso2.carbon.automation.core.PlatformTestManager} - Running the test
method ---
org.wso2.carbon.registry.lifecycle.test.LCStateIdNameTestCase.testAddLcToService
+++ LC PR:
+/trunk/services/com/abb/IntergalacticService, *
StateIdNameLC*
++ LC PROP: +++registry.lifecycle.*
TransitionApprovalLC*.state
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.1.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.2.item.permission
++ LC PROP:
+++registry.custom_lifecycle.checklist.option.0.item
[2013-08-07 16:33:50,082] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} - On Test failure..
[2013-08-07 16:33:50,083] ERROR
{org.wso2.carbon.automation.core.PlatformTestManager} -
*java.lang.AssertionError:
LifeCycle not added to service expected [true] but found [false]*
Regards,
Vijitha.
--
Vijitha Kumara
Senior Software Engineer; WSO2, Inc.; http://wso2.com/
email: viji...@wso2.com
Lean . Enterprise .
