I think we need to answer the following two question before we do the real coding. What kinds of test do we need? Unit Test or End-to-end test(Integrate test)? How do we trigger the test? Automation or manual?
For the first problems, I think the answer is clear based on this mailing list. We need end-to-end(E2E) test. It would be better if we can run E2E test without emulator, which will make the test real fast. However, we have to mock all the Android API on a normal JVM if there is not Android emulator. Although there are some framework for this purpose(Robolectric and Mockito), I am afraid it’s almost impossible to mock all Android API and load C++ shared library of which we don’t even have source code in a normal JVM. So we have to run the E2E test on an emulator. Based on the poor behavior of Android emulator on TravisCI, I think SauceLab is the places where we host the mobile emulator. For for problems, I think running test before PR is merged would be great. There is even SauceLab plugin for TravisCI [1]. FYI: I have submitted the application for requesting an open source account in SauceLab in the name of ASF. I expect to receive response from SauceLab in 5 business day. [1] https://docs.travis-ci.com/user/sauce-connect/ <https://docs.travis-ci.com/user/sauce-connect/> > 在 2019年8月18日，16:30，王仁敏 <hnht...@163.com> 写道： > > I spent some time building an app for end2end testing, and I build a simple > iOS and Android Demo.the details as shown below: > > > ## Description > 1. I build a demo app whose home page is a TableView, and each Cell is a test > case. > 2. Each Cell will be bound to a URL link. Clicking on Cell will use a URL > link to build the ViewController and pass it to the ViewController with an > identifier. > 3. At the end of the ViewController's viewdidLoad() function, start an > asynchronous detection task with delay. > 4. After the delay, the asynchronous detection task will be executed. Taking > the UI test as an example, the View property will be compared with the View > property when the test is correct to determine whether to pass the detection > task. Since this operation is on the Native side, It's very easy. > 5. Pass the message that the test is correct or failed to the front-end test > framework. > > > Because all of the Cells are ViewControllers built using URLs, we can share > all of the test cases with a ViewController, which can perform different > asynchronous detection tasks depending on the incoming identifier. > > > ## How To Run > The test will run as follows: > 1. Appium framework controls JS click on Cell to enter test case > 2. Once the Cell clicks, the ViewController will initialize and trigger an > asynchronous detection task. > 3. The Appium framework test case waits for a while and checks if the > received message is "Success" or "Fail" to determine if the test is correct. > 4. Take the next test. > In the above scheme, the Appium framework is equivalent to controlling the > entry into the test case, and the specific detection task is placed on the > Native side. > > > > > ## About Simulator > And after investigation, SauceLab is in line with our requirements. First of > all, they can provide free real machines, although the real iPhone only > provides iPhone6, Android provide Samsung Galaxy S6(Android7.0),but in our > case, it's enough! By the way, Android provides ARM simulator. > > > > > ## TravisCI And SauceLab: > 1. Configure the environment variables required by SauceLab in TravisCI > 2. In Travis CI, we first build the project and then upload the app (iOS) or > apk (Android) to the SauceLab server. > 3. In Travis CI, start the Appium test and run the E2E test with the remote > SauceLab simulator and app/apk. > > > If you have any suggestions, please let me know. your suggestions are very > helpful to me! > > > > Best Wishes > Renmin > > > Links： > 1. How To Upload > App/Apk:https://wiki.saucelabs.com/display/DOCS/Uploading+Your+App+to+Real+Device+Storage+with+the+REST+API > 2. SauceLab Mobile Test Document: > https://wiki.saucelabs.com/display/DOCS/Mobile+Application+Testing > 3. Config the Appium Test With SauceLab: > https://wiki.saucelabs.com/display/DOCS/Platform+Configurator#/ > 4. Appium SauceLab Python Example: > https://wiki.saucelabs.com/display/DOCS/Python+Example+Script+for+iOS+Mobile+Application+Tests > > 5. Supported Simulators:https://app.saucelabs.com/live/web-testing > > > | | > 王仁敏 > | > | > hnht...@163.com > | > 签名由网易邮箱大师定制 > > > On 08/16/2019 17:04，York Shen<shenyua...@gmail.com> wrote： > Some thoughts about End to End test on Weex Android. I spent some time this > week coding for Android End to End test using the test framework Android > itself provided. After all, any fancy test frameworks for Android is an > encapsulation of Android Test framework. Here is what I got: > > If you run a test case without Android Emulator, all the Android API in a > test case is actually mock APIs by a test framework, not provided by Android > system, which means the behavior of mocks and Android system itself is > different in some aspect. In the situation of Weex, I am stocked by loading > and running shared library running on Android in test framework > (Robolectric). It is very unlikely that we could load and run all of the 8 > shared libraries built for Mobile x86 platform in Weex originally on Linux > x86 platform when running test case, especially some of them dynamically > links to shared libraries in Android system itself. > If we are running test case with Android Emulator on Travis CI, we can get > everything we get want by expanding a test framework called espresso[1]. The > only problem here is Android emulator on Travis CI is really slow due to > nested-virtualization problems of Travis-CI [2]. As this is caused by the KVM > virtualization technology, I don’t think we can get over the emulator > performance problem in a easy way. > > [1] https://developer.android.com/training/testing/espresso > <https://developer.android.com/training/testing/espresso> > [2] https://travis-ci.community/t/add-kvm-support/1406/10 > <https://travis-ci.community/t/add-kvm-support/1406/10> > > 在 2019年8月12日，11:08，王仁敏 <wrmwindm...@gmail.com> 写道： > > I have recently designed a test framework with York, and we want to add > some end-to-end test cases to improve the stability and robustness of Weex. > > > Our expected testing framework want to verify basic properties of the View, > instead of plain text(Because plain text verification has too many > limitations) > > > > For all test cases, the following three processes are required: > > 1. Create UIViewController/Activity > > 2. Simulate user input > > 3. Check if the result is right > > > We thought of several possible solutions: > > > ## 1. Based on the pure Native testing framework. > > Weex provides the ability to load pages from a URL file. If you don't rely > on the emulator, you need call some interface to convert the URL to a page > (York said this process can't build success) > > > ## 2. Based on the Detox(end-to-end framework) and simulator. > > - Create page: build the page based on the URL (the entire Android > project is already running, so we can call the interface) > - Simulate user input: (1) simulate user input via JS (2) simulate user > click event using native code > - Test result: on the Native Side, compare View's properties with the > expected. If it is consistent, notify the JS the test case is succeeded, > otherwise, notify the JS the test case is failed. > > But Android emulator has many limitations. According to our survey, the > Travis CI Android emulator is slower, and the X86 emulator Google has not > been updated for three years, and the ARM emulator is subject to the Travis > CI architecture. > > > @Jan, I learn about you have done some works about > Cordova-testing-framework recently. How do you solve this problem? > > > I will be very appreciated if anyone can help me, thanks very much! > > > Best wished. > > Renmin >
