Supre, when we first discussed this weeks ago, I didn't realize we will have the students lead this!! Glad to see this.
CLI verification tool seems to become a very important part of the e2e part, which is great. I want to ask, how the docker-compose lands in your mind? I think CLI can't control this part. Sheng Wu 吴晟 Twitter, wusheng1108 kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午5:27写道： > Hi the community, I want to raise a discussion to build the next > generation of E2E test framework for Apache SkyWalking, which may not be a > high priority but a necessity. > > As we already know, tests in SkyWalking play a very important role in the > contribution process, and the current E2E tests just work well to verify > every single pull request before merging, so why bother to build the next > generation of E2E test framework? > > 1. In the SkyWalking's ecosystem, there are many sub-projects that vary in > terms of language, almost all of them need E2E tests to help us verify the > pull requests, but we can see that they reimplement the E2E test framework > in their languages, (or even worse, some of them don’t have E2E tests at > all). Reimplementing the E2E test framework is unnecesarry and introduces > more duplicated work when adding a new test case. The ultimate goal is to > reuse all the test cases when needed. > > 2. The current E2E framework is built with Java, running Java-based > program is not an easy thing for non-Java developers (maintainers of other > sub-projects such as PHP, C++, LUA, etc.), they need to set up the > environment correctly and then run the tests, neither is it an easy skill > to debug the tests. > > 3. It’s a good opportunity to optimize the E2E tests because we added many > cases and many duplicated codes that need to be removed. > > Therefore we're planning to build an unified, easy-to-use, and fast E2E > test framework to benifit all the sub-projects. > > Here are some rough ideas about this framework: > > 0. (Unchanged) We will continue to use docker-compose.yaml to orchestrate > the services that are to be tested, it’s proved to be friendly and > debuggable because we can start it directly even if we are not writing E2E > tests. > 1. This framework will take full advantages of the CLI project to query > the traces/metrics/logs, we don’t want to maintain two versions of query > codes anymore, (now we have a version in the test codes and one in the CLI). > 2. We will build a CLI tool to compare the expected data and the actual > data, we now have a custom schema of the expected data yaml file, the > verification codes should be packaged into an executable command so that it > can be executed standalone without Java and maven knowledge. I hope we can > leverage existing standards to write the YAML files and do verifications. > 3. Build an orchestrating tool to start the `docker-compose.yaml`, health > check, query actual data, and verify the result. > 4. (Nice to have) wrap all the tools as a GitHub Actions to share between > sub-projects. > > If you have any other better idea or want to complement to this proposal, > please reply here. I will create an issue to track these tasks later. > > We have two students (who just finished the Summer Code 2020 Projects) to > lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a not > high priority, you have adequate time to get yourselves familiar with how > the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as well > as the ecosystem. Thanks for your interests and willingness to help. > > [1] https://github.com/Humbertzhang > [2] https://github.com/fgksgf/ > > ————————— > Zhenxu Ke (柯振旭) > GitHub @kezhenxu94 > >
