First of all this is not framework issue. The issue has caused by the way
we have done test design. The same behavior was observed when using
priority and dependOn annotation together (could be caused by testNG issue)
I have explained the reason for the using of duplicate create
service/LC methods
in another mail thread.
I don't think we need to redo the test code here. All these tests were
successful with G-Reg 4.5.3. IMO we can use unique service names and fix
the problem.
Thanks,
Krishantha.
On Wed, Aug 7, 2013 at 7:32 PM, Vijitha Kumara <viji...@wso2.com> wrote:
> Hi Nuwan,
>
> I see most of the test cases have cleanup code to do the necessary stuff
> but we need to review all of them as well if there's any doubt.
>
> But this is not about resource cleanup rather the issue with the
> inferences and the related consequences I explained. The test in focus is
> not done but while invoking one of its methods it has met with an
> unexpected state (that can be a result of another test case which might
> also not finished).
>
> If the latter is finished then it might have cleaned up the service (same
> service used across both classes in this case) so it might fail the current
> test case with no service found, which is another side effect of these. So
> this might have to do with ordering/execution as well. I'm not sure about
> the implementation but we can analyze that.
>
>
> On Wed, Aug 7, 2013 at 6:02 PM, Nuwan Wimalasekara <nuw...@wso2.com>wrote:
>
>> This happens when the resources used for a test is not cleared
>> properly after the test. Those clearance should be called under AfterClass
>> or After Test annotated methods. Then Those methods will call even if the
>> test is failed or passed. Then there are no effects of other test classes.
>> Some test methods have all the steps inside it. Once it gets failed, It
>> may affect other test too. If we do that properly we can use the same
>> resources in many test cases. if not we have to use different names for
>> artifact we use.
>>
>
> That's also a possible solution for these kind which requires redoing all
> test classes again which I'm not sure an option with the time frames of the
> current release.
>
>
>
>
> Regards,
> Vijitha.
>
>
>
>>
>> Thanks,
>> Nuwanw
>>
>>
>> On Wed, Aug 7, 2013 at 5:44 PM, Vijitha Kumara <viji...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> Apparently there's some serious ordering/execution issue when the bunch
>>> of tests run with the framework.
>>>
>>> For example if you run a single class LCStateIdNameTestCase all test
>>> passes but fails when running only with the package
>>> (org.wso2.carbon.registry.lifecycle.test in this case). Failure below with
>>> the reason for the same. This fails as the logs indicate a different LC is
>>> attached to the service while executing the method.
>>>
>>> So how this is possible? There can be multiple reasons but one can be
>>> the use of same service/metadata in multiple test cases and another test
>>> case attaches a different LC to the same service referred by the current
>>> test case.
>>>
>>> One way this can be avoided (at least for the most part) is re-factor
>>> the test cases to avoid these multiple test methods with @Test (to add
>>> service/LC etc...) in that the main test case in focus do the necessary
>>> method calls directly (to setup required services etc...) and do the
>>> asserts (This can be done some restructuring of the current tests) itself.
>>>
>>> Is there any other way this interference can be avoided within the
>>> framework?
>>>
>>>
>>>
>>>
>>> [2013-08-07 16:33:49,811] INFO
>>> {org.wso2.carbon.automation.core.PlatformTestManager} - On test success..
>>> [2013-08-07 16:33:49,812] INFO
>>> {org.wso2.carbon.automation.core.PlatformTestManager} - Running the test
>>> method ---
>>> org.wso2.carbon.registry.lifecycle.test.LCStateIdNameTestCase.testAddLcToService
>>> ----
>>> +++++++++++++++++++ LC PR:
>>> +++++++++++++++++++++++++/trunk/services/com/abb/IntergalacticService, *
>>> StateIdNameLC*
>>> ++++++++++++++++++ LC PROP: +++++++++++++++registry.lifecycle.*
>>> TransitionApprovalLC*.state
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.1.item
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.1.item.permission
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.2.item
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.0.item.permission
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.2.item.permission
>>> ++++++++++++++++++ LC PROP:
>>> +++++++++++++++registry.custom_lifecycle.checklist.option.0.item
>>> [2013-08-07 16:33:50,082] ERROR
>>> {org.wso2.carbon.automation.core.PlatformTestManager} - On Test failure..
>>> [2013-08-07 16:33:50,083] ERROR
>>> {org.wso2.carbon.automation.core.PlatformTestManager} -
>>> *java.lang.AssertionError:
>>> LifeCycle not added to service expected [true] but found [false]*
>>>
>>>
>>>
>>>
>>> Regards,
>>> Vijitha.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Vijitha Kumara
>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com/
>>> email: viji...@wso2.com
>>>
>>>
>>> Lean . Enterprise . Middleware
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Nuwan Wimalasekara
>> Senior Software Engineer - Test Automation
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> phone: +94 71 668 4620
>>
>>
>>
>>
>
>
> --
> Vijitha Kumara
> Senior Software Engineer; WSO2, Inc.; http://wso2.com/
> email: viji...@wso2.com
>
>
> Lean . Enterprise . Middleware
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
