Inline kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午11:16写道：
> Hi Jiajing, > > Thanks for sharing your context and joining the discussion. It’s true that > we’re actually tackling the same problems in this domain. Some of the > issues you mentioned are solved in the current E2E framework (like `retry`, > `timeout`, etc.), while some others are still there without ideal solution. > Other discussions, please see my comments inline. > > > - the first issue is useful in health check steps, just like the ones > > defined in readiness probe and liveness probe in kubelet. > > Yes, the reason why we decided to pick docker-compose is that, it has the > same functionality with kubelet in terms of dependent services startup > order and healthiness checking, while it’s more lite-weight and easy for > both developers and GitHub Actions environment. Right? > > > - the second issue is critical for highly-customized assertions, we > > can leverage the `text/template` module to provide highly extensive > > assertion functions > > `text/template` is also the first thought that came to my mind when > considering customized assertors, customized assertors are for sure > critical in SkyWalking E2E tests because we have many kinds of assertions, > some of which are not foreseen until we actually need them, > `AtLeastOneGreaterThanZero` is an example. > > > - the last but not the least, the lifecycle hooks can be used for > > accurate control of test setup and teardown, such as `docker-compose > > build, up -d, down` > > The lifecycle hooks are what I didn’t take into consideration because I > thought it is quite natural for test framework to provide this features, > maybe Ke Zhang and Huaxi can do some research on this part. > > > Together with the aforementioned ideas, we will have a go-written, > > yaml-data driven e2e testing framework which can overcome the current > > problems that occur, > > I agree. > > > in which the most annoying issue is the fragmentation of different > > scripts in different submodules, IMO. > > That’s why I list the last item in the previous email, (Nice to have, wrap > them into a GitHub Action), with that GitHub Action, we can simply > configure a workflow/job in every submodule, and the tests are executed > with the shared scripts, no copy-pasting. > Note, GitHub Action will require an Apache release every time because we are distributing the binary. Sheng Wu 吴晟 Twitter, wusheng1108 > > > > > Regards, > > Jiajing > > > > On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <[email protected]> > wrote: > >> > >>> So, we would provide a script set to drive this? > >> > >> Yes, a script or a GoLang-written CLI, it depends. > >> > >>> My question is more about, > >>> how to work with docker-compose > >> > >> It’s the same as what we’re doing now in the current version of E2E > tests, now we have many `docker-compose.yaml` files, and all the > `docker-compose.yaml` files are reusable in the new framework, we will keep > this mechanism as it is proved to be convenient and easy to use (remember > when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not > yet merged though, it’s fast/easy to add a new case). > >> > >> ————————— > >> Zhenxu Ke (柯振旭) > >> GitHub @kezhenxu94 > >> > >>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <[email protected]> > wrote: > >>> > >>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午9:16写道： > >>> > >>>> > >>>>> I want to ask, how the docker-compose lands in your mind? I think CLI > >>>> can't > >>>>> control this part. > >>>> > >>>> > >>>> docker-compose just fits the requirements perfectly that we need, we > have > >>>> been using it in the current E2E and turns out it has advantages in > >>>> starting many services correctly with health check and dependency > >>>> declaration. (One may say Kubernetes can do the same thing, but it’s > kind > >>>> of heavy in E2E and GitHub Actions and complex). > >>>> > >>>> The SkyWalking-CLI won’t get involved with docker-compose related > things, > >>>> it just provides query ability (the same as what it is doing now, > just need > >>>> to check whether it covers all the needed query interfaces or not), > we will > >>>> have a dedicated control program to > >>>> > >>>> (1) start docker-compose (existing standard mechanism, no new > knowledge); > >>>> > >>> > >>> So, we would provide a script set to drive this? My question is more > about, > >>> how to work with docker-compose, rather than challanging the choice. > >>> > >>> Sheng Wu 吴晟 > >>> Twitter, wusheng1108 > >>> > >>> > >>>> (2) query actual data (by SkyWalking-CLI, existing repository and > codes); > >>>> (3) verify the actual data (by verification tool, a new tool, TODO); > >>>> (4) and determine the test result (success or failed); > >>>> > >>>> > >>>> ————————— > >>>> Zhenxu Ke (柯振旭) > >>>> GitHub @kezhenxu94 > >>>> > >>>> > >>>> > >>>>> Sheng Wu 吴晟 > >>>>> Twitter, wusheng1108 > >>>>> > >>>>> > >>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午5:27写道： > >>>>> > >>>>>> Hi the community, I want to raise a discussion to build the next > >>>>>> generation of E2E test framework for Apache SkyWalking, which may > not > >>>> be a > >>>>>> high priority but a necessity. > >>>>>> > >>>>>> As we already know, tests in SkyWalking play a very important role > in > >>>> the > >>>>>> contribution process, and the current E2E tests just work well to > verify > >>>>>> every single pull request before merging, so why bother to build the > >>>> next > >>>>>> generation of E2E test framework? > >>>>>> > >>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that > vary > >>>> in > >>>>>> terms of language, almost all of them need E2E tests to help us > verify > >>>> the > >>>>>> pull requests, but we can see that they reimplement the E2E test > >>>> framework > >>>>>> in their languages, (or even worse, some of them don’t have E2E > tests at > >>>>>> all). Reimplementing the E2E test framework is unnecesarry and > >>>> introduces > >>>>>> more duplicated work when adding a new test case. The ultimate goal > is > >>>> to > >>>>>> reuse all the test cases when needed. > >>>>>> > >>>>>> 2. The current E2E framework is built with Java, running Java-based > >>>>>> program is not an easy thing for non-Java developers (maintainers of > >>>> other > >>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the > >>>>>> environment correctly and then run the tests, neither is it an easy > >>>> skill > >>>>>> to debug the tests. > >>>>>> > >>>>>> 3. It’s a good opportunity to optimize the E2E tests because we > added > >>>> many > >>>>>> cases and many duplicated codes that need to be removed. > >>>>>> > >>>>>> Therefore we're planning to build an unified, easy-to-use, and fast > E2E > >>>>>> test framework to benifit all the sub-projects. > >>>>>> > >>>>>> Here are some rough ideas about this framework: > >>>>>> > >>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to > >>>> orchestrate > >>>>>> the services that are to be tested, it’s proved to be friendly and > >>>>>> debuggable because we can start it directly even if we are not > writing > >>>> E2E > >>>>>> tests. > >>>>>> 1. This framework will take full advantages of the CLI project to > query > >>>>>> the traces/metrics/logs, we don’t want to maintain two versions of > query > >>>>>> codes anymore, (now we have a version in the test codes and one in > the > >>>> CLI). > >>>>>> 2. We will build a CLI tool to compare the expected data and the > actual > >>>>>> data, we now have a custom schema of the expected data yaml file, > the > >>>>>> verification codes should be packaged into an executable command so > >>>> that it > >>>>>> can be executed standalone without Java and maven knowledge. I hope > we > >>>> can > >>>>>> leverage existing standards to write the YAML files and do > >>>> verifications. > >>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`, > >>>> health > >>>>>> check, query actual data, and verify the result. > >>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share > >>>> between > >>>>>> sub-projects. > >>>>>> > >>>>>> If you have any other better idea or want to complement to this > >>>> proposal, > >>>>>> please reply here. I will create an issue to track these tasks > later. > >>>>>> > >>>>>> We have two students (who just finished the Summer Code 2020 > Projects) > >>>> to > >>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is > a > >>>> not > >>>>>> high priority, you have adequate time to get yourselves familiar > with > >>>> how > >>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work > as > >>>> well > >>>>>> as the ecosystem. Thanks for your interests and willingness to help. > >>>>>> > >>>>>> [1] https://github.com/Humbertzhang > >>>>>> [2] https://github.com/fgksgf/ > >>>>>> > >>>>>> ————————— > >>>>>> Zhenxu Ke (柯振旭) > >>>>>> GitHub @kezhenxu94 > >>>>>> > >>>>>> > >>>> > >>>> ————————— > >>>> Zhenxu Ke (柯振旭) > >>>> GitHub @kezhenxu94 > >> > > > > ————————— > Zhenxu Ke (柯振旭) > GitHub @kezhenxu94 > >
