Re: Some questions about testing framework(Need Help!)

[York Shen]

I think we need to answer the following two question before we do the real 
coding.
What kinds of test do we need? Unit Test or End-to-end test(Integrate test)?
How do we trigger the test? Automation or manual?

For the first problems, I think the answer is clear based on this mailing list. 
We need end-to-end(E2E) test. It would be better if we can run E2E test without 
emulator, which will make the test real fast. However, we have to mock all the 
Android API on a normal JVM if there is not Android emulator. Although there 
are some framework for this purpose(Robolectric and Mockito), I am afraid it’s 
almost impossible to mock all Android API and load C++ shared library of which 
we don’t even have source code in a normal JVM. So we have to run the E2E test 
on an emulator. Based on the poor behavior of Android emulator on TravisCI, I 
think SauceLab is the places where we host the mobile emulator.

For for problems, I think running test before PR is merged would be great. 
There is even SauceLab plugin for TravisCI [1].

FYI: I have submitted the application for requesting an open source account in 
SauceLab in the name of ASF. I expect to receive response from SauceLab in 5 
business day.

[1] https://docs.travis-ci.com/user/sauce-connect/ 


> 在 2019年8月18日，16:30，王仁敏  写道：
> 
> I spent some time building an app for end2end testing, and I build a simple 
> iOS and Android Demo.the details as shown below:
> 
> 
> ## Description
> 1. I build a demo app whose home page is a TableView, and each Cell is a test 
> case.
> 2. Each Cell will be bound to a URL link. Clicking on Cell will use a URL 
> link to build the ViewController and pass it to the ViewController with an 
> identifier.
> 3. At the end of the ViewController's viewdidLoad() function, start an 
> asynchronous detection task with delay.
> 4. After the delay, the asynchronous detection task will be executed. Taking 
> the UI test as an example, the View property will be compared with the View 
> property when the test is correct to determine whether to pass the detection 
> task. Since this operation is on the Native side, It's very easy.
> 5. Pass the message that the test is correct or failed to the front-end test 
> framework.
> 
> 
> Because all of the Cells are ViewControllers built using URLs, we can share 
> all of the test cases with a ViewController, which can perform different 
> asynchronous detection tasks depending on the incoming identifier.
> 
> 
> ## How To Run
> The test will run as follows:
> 1. Appium framework controls JS click on Cell to enter test case
> 2. Once the Cell clicks, the ViewController will initialize and trigger an 
> asynchronous detection task.
> 3. The Appium framework test case waits for a while and checks if the 
> received message is "Success" or "Fail" to determine if the test is correct.
> 4. Take the next test.
> In the above scheme, the Appium framework is equivalent to controlling the 
> entry into the test case, and the specific detection task is placed on the 
> Native side.
> 
> 
> 
> 
> ## About Simulator
> And after investigation, SauceLab is in line with our requirements. First of 
> all, they can provide free real machines, although the real iPhone only 
> provides iPhone6, Android provide Samsung Galaxy S6(Android7.0),but in our 
> case, it's enough! By the way, Android provides ARM simulator.
> 
> 
> 
> 
> ## TravisCI And SauceLab:
> 1. Configure the environment variables required by SauceLab in TravisCI
> 2. In Travis CI, we first build the project and then upload the app (iOS) or 
> apk (Android) to the SauceLab server.
> 3. In Travis CI, start the Appium test and run the E2E test with the remote 
> SauceLab simulator and app/apk.
> 
> 
> If you have any suggestions, please let me know. your suggestions are very 
> helpful to me!
> 
> 
> 
> Best Wishes
> Renmin
> 
> 
> Links：
> 1. How To Upload 
> App/Apk:https://wiki.saucelabs.com/display/DOCS/Uploading+Your+App+to+Real+Device+Storage+with+the+REST+API
> 2. SauceLab Mobile Test Document: 
> https://wiki.saucelabs.com/display/DOCS/Mobile+Application+Testing
> 3. Config the Appium Test With SauceLab:  
> https://wiki.saucelabs.com/display/DOCS/Platform+Configurator#/
> 4. Appium SauceLab Python Example: 
> https://wiki.saucelabs.com/display/DOCS/Python+Example+Script+for+iOS+Mobile+Application+Tests
> 
> 5. Supported Simulators:https://app.saucelabs.com/live/web-testing
> 
> 
> | |
> 王仁敏
> |
> |
> hnht...@163.com
> |
> 签名由网易邮箱大师定制
> 
> 
> On 08/16/2019 17:04，York Shen wrote：
> Some thoughts about End to End test on Weex Android. I spent some time this 
> week coding for Android End to End test using the test framework Android 
> itself provided. After all, any fancy test frameworks for Android is an 
> encapsulation of Android Test framework. Here is what I got:
> 
> If you run a test case without Android Emulator, all the Android API in a 
> test case is actually mock APIs by a test framework, 

Re: Some questions about testing framework(Need Help!)

[王仁敏]

I spent some time building an app for end2end testing, and I build a simple iOS 
and Android Demo.the details as shown below:


## Description
1. I build a demo app whose home page is a TableView, and each Cell is a test 
case.
2. Each Cell will be bound to a URL link. Clicking on Cell will use a URL link 
to build the ViewController and pass it to the ViewController with an 
identifier.
3. At the end of the ViewController's viewdidLoad() function, start an 
asynchronous detection task with delay.
4. After the delay, the asynchronous detection task will be executed. Taking 
the UI test as an example, the View property will be compared with the View 
property when the test is correct to determine whether to pass the detection 
task. Since this operation is on the Native side, It's very easy.
5. Pass the message that the test is correct or failed to the front-end test 
framework.


Because all of the Cells are ViewControllers built using URLs, we can share all 
of the test cases with a ViewController, which can perform different 
asynchronous detection tasks depending on the incoming identifier.


## How To Run
The test will run as follows:
1. Appium framework controls JS click on Cell to enter test case
2. Once the Cell clicks, the ViewController will initialize and trigger an 
asynchronous detection task.
3. The Appium framework test case waits for a while and checks if the received 
message is "Success" or "Fail" to determine if the test is correct.
4. Take the next test.
In the above scheme, the Appium framework is equivalent to controlling the 
entry into the test case, and the specific detection task is placed on the 
Native side.




## About Simulator
And after investigation, SauceLab is in line with our requirements. First of 
all, they can provide free real machines, although the real iPhone only 
provides iPhone6, Android provide Samsung Galaxy S6(Android7.0),but in our 
case, it's enough! By the way, Android provides ARM simulator.




## TravisCI And SauceLab:
1. Configure the environment variables required by SauceLab in TravisCI
2. In Travis CI, we first build the project and then upload the app (iOS) or 
apk (Android) to the SauceLab server.
3. In Travis CI, start the Appium test and run the E2E test with the remote 
SauceLab simulator and app/apk.


If you have any suggestions, please let me know. your suggestions are very 
helpful to me!



Best Wishes
Renmin


Links：
1. How To Upload 
App/Apk:https://wiki.saucelabs.com/display/DOCS/Uploading+Your+App+to+Real+Device+Storage+with+the+REST+API
2. SauceLab Mobile Test Document: 
https://wiki.saucelabs.com/display/DOCS/Mobile+Application+Testing
3. Config the Appium Test With SauceLab:  
https://wiki.saucelabs.com/display/DOCS/Platform+Configurator#/
4. Appium SauceLab Python Example: 
https://wiki.saucelabs.com/display/DOCS/Python+Example+Script+for+iOS+Mobile+Application+Tests

5. Supported Simulators:https://app.saucelabs.com/live/web-testing


| |
王仁敏
|
|
hnht...@163.com
|
签名由网易邮箱大师定制


On 08/16/2019 17:04，York Shen wrote：
Some thoughts about End to End test on Weex Android. I spent some time this 
week coding for Android End to End test using the test framework Android itself 
provided. After all, any fancy test frameworks for Android is an encapsulation 
of Android Test framework. Here is what I got:

If you run a test case without Android Emulator, all the Android API in a test 
case is actually mock APIs by a test framework, not provided by Android system, 
which means the behavior of mocks and Android system itself is different in 
some aspect. In the situation of Weex, I am stocked by loading and running 
shared library running on Android in test framework (Robolectric). It is very 
unlikely that we could load and run all of the 8 shared libraries built for 
Mobile x86 platform in Weex originally on Linux x86 platform when running test 
case, especially some of them dynamically links to  shared libraries in Android 
system itself.
If we are running test case with Android Emulator on Travis CI, we can get 
everything we get want by expanding a test framework called espresso[1]. The 
only problem here is Android emulator on Travis CI is really slow due to 
nested-virtualization problems of Travis-CI [2]. As this is caused by the KVM 
virtualization technology, I don’t think we can get over the emulator 
performance problem in a easy way.

[1] https://developer.android.com/training/testing/espresso 

[2] https://travis-ci.community/t/add-kvm-support/1406/10 


在 2019年8月12日，11:08，王仁敏  写道：

I have recently designed a test framework with York, and we want to add
some end-to-end test cases to improve the stability and robustness of Weex.


Our expected testing framework want to verify basic properties of the View,
instead of plain text(Because plain text verification has too many
limitations)



For all test cases, the following 

Re: Some questions about testing framework(Need Help!)

[王仁敏]

Thank you for your detailed reply.

The test cases I want to create are far more complex than the AnimationTest
mentioned earlier.

With regard to AnimationTest, York built a test case to determine whether
the test case was successful by comparing the screenshots of the test case
with the screenshots in the correct case. If the two are the same, then the
test case is successful. but some test case that should be successful
failed because there are subtle differences between the pixels of the
image.I think video will encounter the same problem.

I've built several test cases locally for iOS based on the ideas mentioned
in my first email, they worked locally.

The biggest problem is the Android. As for Android emulator, there are some
problems with Android emulator based on TravisCI as you mentioned, so I
want to know more about Android emulator of Sauce Lab.

I will spend some time to investigate Sauce Lab and really hope that
SauceLab's android emulator meets our requirements.

Jan Piotrowski  于2019年8月13日周二 下午4:41写道：

> In the context of a Cordova platform, end to end indeed means testing
> if the whole CLI and script flow works - adding the platform to a
> project, building the project etc. There is no real testing if what is
> created works in a specific way on the UI - as a platform itself has
> no UI - that comes via the app of the user or specific plugins.
>
> If you want to create an app with weex and see if the animation takes
> a specific amount of time, SauceLabs via Appium is indeed the way to
> go. You get a video recording of what your app did and can look at it
> to confirm any assertions your tests passed or failed.
>
> I can not tell you any specific about the Android emulators on Sauce
> Labs, as Cordova is currently not using it because it used to only run
> tests on SauceLabs. For iOS it was simple enough to add the local
> simulators, but for Android we hit several problems, see this issue
> for the request to add this functionality:
> https://github.com/apache/cordova-paramedic/issues/134
>
> What paramedic does is described here:
> https://github.com/apache/cordova-paramedic#what-does-it-do I think
> you will probably want to build something like that for weex, where a
> project is created, some code added like a user would write it (via a
> template perhaps) and then build the app to be able to run it on
> different configurations (iOS, Android, versions) and run some tests
> on it.
>
> Paramedic is really old and flaky, so you probably do not want to base
> what you create on it - but you definitely can take the general idea.
> If you are interested in more details, I can write up the "flow" of
> things for one of the plugins that have actual Appium tests that
> confirm native behavior (unfortunately that is currently all kinds of
> broken because of upgrades, so the tests are actually not in
> production right now - so you can definitely do that better than
> Cordova ;) )
>
> J
>
> Am Di., 13. Aug. 2019 um 07:36 Uhr schrieb 王仁敏 :
> >
> > >
> > > Thanks for your detailed reply, and it help me a lot.  I'm sorry to
> reply
> > >> so later , I was very busy yesterday.
> > >
> > >
> > >> After read the document of  Cordova testing, I know that you have
> done a
> > >> really great work about test.
> > >
> > >
> > >> in Weex, we have also build  some native Unit Tests in both iOS and
> > >> Android to check if individual pieces of code produce the expected
> > >> results.But as you know,in Weex,our users build their app by writing
> > >> javascript code, so Unit Tests is not enough.We need e2e/Integration
> > >> testing.
> > >
> > >
> > >> for example, I want to test if the animation works as expected.the
> > >> animation is (x+300,y+300,during=0.3s),
> > >
> > >
> > >> if we use js test-framework,we can easy to simulate user input. but
> we
> > >> can't get the animation result.(because we should compare the view's
> > >> properties with it's expected)
> > >
> > > if we use Native,we should firstly Create UIViewController/Activity and
> > >> then Simulate user input.
> > >
> > >
> > >> Correct me if I'm wrong.I have read the e2e test of cordova-ios and
> > >> cordova-android, the e2e-test only test if  project is created
> successful
> > >> and  project is builded successful. I'm curious how you designed a
> test
> > >> case like this.
> > >
> > >
> > >> After I read Sauce Lab document, I think maybe Sauce Lab is better
> > >> solution than Travis CI to run android emulator-based test case. It
> also
> > >> provide free resource for open source project and the sauce lab can
> > >> integrated into Travis CI.
> > >
> > >
> > >> can you tell me something  about ARM android emulator in Sauce Lab,
> For
> > >> example, the stability and the speed at which test cases are
> executed. It's
> > >> very important to me.
> > >
> > >
> > >>
> > >
> > >> 1. Free to Apple For SauceLab: https://saucelabs.com/open-source
> > >
> > > 2. Travis CI with Sauce Lab:
> > >> 

Re: Some questions about testing framework(Need Help!)

[王仁敏]

>
> Thanks for your detailed reply, and it help me a lot.  I'm sorry to reply
>> so later , I was very busy yesterday.
>
>
>> After read the document of  Cordova testing, I know that you have done a
>> really great work about test.
>
>
>> in Weex, we have also build  some native Unit Tests in both iOS and
>> Android to check if individual pieces of code produce the expected
>> results.But as you know,in Weex,our users build their app by writing
>> javascript code, so Unit Tests is not enough.We need e2e/Integration
>> testing.
>
>
>> for example, I want to test if the animation works as expected.the
>> animation is (x+300,y+300,during=0.3s),
>
>
>> if we use js test-framework,we can easy to simulate user input. but  we
>> can't get the animation result.(because we should compare the view's
>> properties with it's expected)
>
> if we use Native,we should firstly Create UIViewController/Activity and
>> then Simulate user input.
>
>
>> Correct me if I'm wrong.I have read the e2e test of cordova-ios and
>> cordova-android, the e2e-test only test if  project is created successful
>> and  project is builded successful. I'm curious how you designed a test
>> case like this.
>
>
>> After I read Sauce Lab document, I think maybe Sauce Lab is better
>> solution than Travis CI to run android emulator-based test case. It also
>> provide free resource for open source project and the sauce lab can
>> integrated into Travis CI.
>
>
>> can you tell me something  about ARM android emulator in Sauce Lab, For
>> example, the stability and the speed at which test cases are executed. It's
>> very important to me.
>
>
>>
>
>> 1. Free to Apple For SauceLab: https://saucelabs.com/open-source
>
> 2. Travis CI with Sauce Lab:
>> https://docs.travis-ci.com/user/sauce-connect/
>
> 3. supported Android emulators:
>> https://wiki.saucelabs.com/display/DOCS/Supported+Android+Emulators
>
>

Some questions about testing framework(Need Help!)

[王仁敏]

I have recently designed a test framework with York, and we want to add
some end-to-end test cases to improve the stability and robustness of Weex.


Our expected testing framework want to verify basic properties of the View,
instead of plain text(Because plain text verification has too many
limitations)



For all test cases, the following three processes are required:

1. Create UIViewController/Activity

2. Simulate user input

3. Check if the result is right


We thought of several possible solutions:


## 1. Based on the pure Native testing framework.

Weex provides the ability to load pages from a URL file. If you don't rely
on the emulator, you need call some interface to convert the URL to a page
(York said this process can't build success)


## 2. Based on the Detox(end-to-end framework) and simulator.

   - Create page: build the page based on the URL (the entire Android
   project is already running, so we can call the interface)
   - Simulate user input: (1) simulate user input via JS (2) simulate user
   click event using native code
   - Test result: on the Native Side, compare View's properties with the
   expected. If it is consistent, notify the JS the test case is succeeded,
   otherwise, notify the JS the test case is failed.

But Android emulator has many limitations. According to our survey, the
Travis CI Android emulator is slower, and the X86 emulator Google has not
been updated for three years, and the ARM emulator is subject to the Travis
CI architecture.


@Jan, I learn about you have done some works about
Cordova-testing-framework recently. How do you solve this problem?


I will be very appreciated if anyone can help me, thanks very much!


Best wished.

Renmin