> So, we would provide a script set to drive this? Yes, a script or a GoLang-written CLI, it depends.
> My question is more about, > how to work with docker-compose It’s the same as what we’re doing now in the current version of E2E tests, now we have many `docker-compose.yaml` files, and all the `docker-compose.yaml` files are reusable in the new framework, we will keep this mechanism as it is proved to be convenient and easy to use (remember when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not yet merged though, it’s fast/easy to add a new case). ————————— Zhenxu Ke (柯振旭) GitHub @kezhenxu94 > On Nov 15, 2020, at 9:29 PM, Sheng Wu <[email protected]> wrote: > > kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午9:16写道： > >> >>> I want to ask, how the docker-compose lands in your mind? I think CLI >> can't >>> control this part. >> >> >> docker-compose just fits the requirements perfectly that we need, we have >> been using it in the current E2E and turns out it has advantages in >> starting many services correctly with health check and dependency >> declaration. (One may say Kubernetes can do the same thing, but it’s kind >> of heavy in E2E and GitHub Actions and complex). >> >> The SkyWalking-CLI won’t get involved with docker-compose related things, >> it just provides query ability (the same as what it is doing now, just need >> to check whether it covers all the needed query interfaces or not), we will >> have a dedicated control program to >> >> (1) start docker-compose (existing standard mechanism, no new knowledge); >> > > So, we would provide a script set to drive this? My question is more about, > how to work with docker-compose, rather than challanging the choice. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > >> (2) query actual data (by SkyWalking-CLI, existing repository and codes); >> (3) verify the actual data (by verification tool, a new tool, TODO); >> (4) and determine the test result (success or failed); >> >> >> ————————— >> Zhenxu Ke (柯振旭) >> GitHub @kezhenxu94 >> >> >> >>> Sheng Wu 吴晟 >>> Twitter, wusheng1108 >>> >>> >>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午5:27写道： >>> >>>> Hi the community, I want to raise a discussion to build the next >>>> generation of E2E test framework for Apache SkyWalking, which may not >> be a >>>> high priority but a necessity. >>>> >>>> As we already know, tests in SkyWalking play a very important role in >> the >>>> contribution process, and the current E2E tests just work well to verify >>>> every single pull request before merging, so why bother to build the >> next >>>> generation of E2E test framework? >>>> >>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary >> in >>>> terms of language, almost all of them need E2E tests to help us verify >> the >>>> pull requests, but we can see that they reimplement the E2E test >> framework >>>> in their languages, (or even worse, some of them don’t have E2E tests at >>>> all). Reimplementing the E2E test framework is unnecesarry and >> introduces >>>> more duplicated work when adding a new test case. The ultimate goal is >> to >>>> reuse all the test cases when needed. >>>> >>>> 2. The current E2E framework is built with Java, running Java-based >>>> program is not an easy thing for non-Java developers (maintainers of >> other >>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the >>>> environment correctly and then run the tests, neither is it an easy >> skill >>>> to debug the tests. >>>> >>>> 3. It’s a good opportunity to optimize the E2E tests because we added >> many >>>> cases and many duplicated codes that need to be removed. >>>> >>>> Therefore we're planning to build an unified, easy-to-use, and fast E2E >>>> test framework to benifit all the sub-projects. >>>> >>>> Here are some rough ideas about this framework: >>>> >>>> 0. (Unchanged) We will continue to use docker-compose.yaml to >> orchestrate >>>> the services that are to be tested, it’s proved to be friendly and >>>> debuggable because we can start it directly even if we are not writing >> E2E >>>> tests. >>>> 1. This framework will take full advantages of the CLI project to query >>>> the traces/metrics/logs, we don’t want to maintain two versions of query >>>> codes anymore, (now we have a version in the test codes and one in the >> CLI). >>>> 2. We will build a CLI tool to compare the expected data and the actual >>>> data, we now have a custom schema of the expected data yaml file, the >>>> verification codes should be packaged into an executable command so >> that it >>>> can be executed standalone without Java and maven knowledge. I hope we >> can >>>> leverage existing standards to write the YAML files and do >> verifications. >>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`, >> health >>>> check, query actual data, and verify the result. >>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share >> between >>>> sub-projects. >>>> >>>> If you have any other better idea or want to complement to this >> proposal, >>>> please reply here. I will create an issue to track these tasks later. >>>> >>>> We have two students (who just finished the Summer Code 2020 Projects) >> to >>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a >> not >>>> high priority, you have adequate time to get yourselves familiar with >> how >>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as >> well >>>> as the ecosystem. Thanks for your interests and willingness to help. >>>> >>>> [1] https://github.com/Humbertzhang >>>> [2] https://github.com/fgksgf/ >>>> >>>> ————————— >>>> Zhenxu Ke (柯振旭) >>>> GitHub @kezhenxu94 >>>> >>>> >> >> ————————— >> Zhenxu Ke (柯振旭) >> GitHub @kezhenxu94
