Thanks Hongtao’s suggestion, I’ve thought about adopting Kubernetes-based tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I have been thinking whether or not it is overkill for testing purposes to introduce Kubernetes things.
Since you mentioned this, we are open to discuss this much more deeply: It would be definitely a benefit to involve more “end”s in the so-call “end to end” tests, I’m +1 to leverage SWCK as well. If we are going to use KIND, I hope we can give a script or something similar to install all the needed components and create cluster for convenient testing locally, Ke Zhang and Huaxi. ————————— Zhenxu Ke (柯振旭) GitHub @kezhenxu94 > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> wrote: > > May SWCK help about provision and demo > > 1. SWCK is able to provide a stable and healthy OAP standalone instance or > cluster for e2e. We could compose indicated CR yamls for different > scenarios. > 2. Different demo for different cases is mandatory for e2e and swck, we > could build a group of demo projects for them. With Kubernetes's support, > we could > get free of verbose scripts that are written for installation, checking > the state of running applications, collecting logs. > > Ppl might have concerts about the minkube we currently have, which's hard > to debug and runs slowly. > > I prefer to kind[1] instead of minkube. From my experience, kind works fine > in an e2e scenario(limited cpu, memory resources), and it is run million > times > on my team's e2e environment, which proves the stability and > maintainability of it. > > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ > > regards, Hongtao > > Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： > >> Inline >> >> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午11:16写道： >> >>> Hi Jiajing, >>> >>> Thanks for sharing your context and joining the discussion. It’s true >> that >>> we’re actually tackling the same problems in this domain. Some of the >>> issues you mentioned are solved in the current E2E framework (like >> `retry`, >>> `timeout`, etc.), while some others are still there without ideal >> solution. >>> Other discussions, please see my comments inline. >>> >>>> - the first issue is useful in health check steps, just like the ones >>>> defined in readiness probe and liveness probe in kubelet. >>> >>> Yes, the reason why we decided to pick docker-compose is that, it has the >>> same functionality with kubelet in terms of dependent services startup >>> order and healthiness checking, while it’s more lite-weight and easy for >>> both developers and GitHub Actions environment. Right? >>> >>>> - the second issue is critical for highly-customized assertions, we >>>> can leverage the `text/template` module to provide highly extensive >>>> assertion functions >>> >>> `text/template` is also the first thought that came to my mind when >>> considering customized assertors, customized assertors are for sure >>> critical in SkyWalking E2E tests because we have many kinds of >> assertions, >>> some of which are not foreseen until we actually need them, >>> `AtLeastOneGreaterThanZero` is an example. >>> >>>> - the last but not the least, the lifecycle hooks can be used for >>>> accurate control of test setup and teardown, such as `docker-compose >>>> build, up -d, down` >>> >>> The lifecycle hooks are what I didn’t take into consideration because I >>> thought it is quite natural for test framework to provide this features, >>> maybe Ke Zhang and Huaxi can do some research on this part. >>> >>>> Together with the aforementioned ideas, we will have a go-written, >>>> yaml-data driven e2e testing framework which can overcome the current >>>> problems that occur, >>> >>> I agree. >>> >>>> in which the most annoying issue is the fragmentation of different >>>> scripts in different submodules, IMO. >>> >>> That’s why I list the last item in the previous email, (Nice to have, >> wrap >>> them into a GitHub Action), with that GitHub Action, we can simply >>> configure a workflow/job in every submodule, and the tests are executed >>> with the shared scripts, no copy-pasting. >>> >> >> Note, GitHub Action will require an Apache release every time because we >> are distributing the binary. >> >> >> Sheng Wu 吴晟 >> Twitter, wusheng1108 >> >> >>> >>>> >>>> Regards, >>>> Jiajing >>>> >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < >> [email protected]> >>> wrote: >>>>> >>>>>> So, we would provide a script set to drive this? >>>>> >>>>> Yes, a script or a GoLang-written CLI, it depends. >>>>> >>>>>> My question is more about, >>>>>> how to work with docker-compose >>>>> >>>>> It’s the same as what we’re doing now in the current version of E2E >>> tests, now we have many `docker-compose.yaml` files, and all the >>> `docker-compose.yaml` files are reusable in the new framework, we will >> keep >>> this mechanism as it is proved to be convenient and easy to use (remember >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, >> not >>> yet merged though, it’s fast/easy to add a new case). >>>>> >>>>> ————————— >>>>> Zhenxu Ke (柯振旭) >>>>> GitHub @kezhenxu94 >>>>> >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <[email protected]> >>> wrote: >>>>>> >>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午9:16写道： >>>>>> >>>>>>> >>>>>>>> I want to ask, how the docker-compose lands in your mind? I think >> CLI >>>>>>> can't >>>>>>>> control this part. >>>>>>> >>>>>>> >>>>>>> docker-compose just fits the requirements perfectly that we need, we >>> have >>>>>>> been using it in the current E2E and turns out it has advantages in >>>>>>> starting many services correctly with health check and dependency >>>>>>> declaration. (One may say Kubernetes can do the same thing, but it’s >>> kind >>>>>>> of heavy in E2E and GitHub Actions and complex). >>>>>>> >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose related >>> things, >>>>>>> it just provides query ability (the same as what it is doing now, >>> just need >>>>>>> to check whether it covers all the needed query interfaces or not), >>> we will >>>>>>> have a dedicated control program to >>>>>>> >>>>>>> (1) start docker-compose (existing standard mechanism, no new >>> knowledge); >>>>>>> >>>>>> >>>>>> So, we would provide a script set to drive this? My question is more >>> about, >>>>>> how to work with docker-compose, rather than challanging the choice. >>>>>> >>>>>> Sheng Wu 吴晟 >>>>>> Twitter, wusheng1108 >>>>>> >>>>>> >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository and >>> codes); >>>>>>> (3) verify the actual data (by verification tool, a new tool, TODO); >>>>>>> (4) and determine the test result (success or failed); >>>>>>> >>>>>>> >>>>>>> ————————— >>>>>>> Zhenxu Ke (柯振旭) >>>>>>> GitHub @kezhenxu94 >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Sheng Wu 吴晟 >>>>>>>> Twitter, wusheng1108 >>>>>>>> >>>>>>>> >>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午5:27写道： >>>>>>>> >>>>>>>>> Hi the community, I want to raise a discussion to build the next >>>>>>>>> generation of E2E test framework for Apache SkyWalking, which may >>> not >>>>>>> be a >>>>>>>>> high priority but a necessity. >>>>>>>>> >>>>>>>>> As we already know, tests in SkyWalking play a very important role >>> in >>>>>>> the >>>>>>>>> contribution process, and the current E2E tests just work well to >>> verify >>>>>>>>> every single pull request before merging, so why bother to build >> the >>>>>>> next >>>>>>>>> generation of E2E test framework? >>>>>>>>> >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that >>> vary >>>>>>> in >>>>>>>>> terms of language, almost all of them need E2E tests to help us >>> verify >>>>>>> the >>>>>>>>> pull requests, but we can see that they reimplement the E2E test >>>>>>> framework >>>>>>>>> in their languages, (or even worse, some of them don’t have E2E >>> tests at >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry and >>>>>>> introduces >>>>>>>>> more duplicated work when adding a new test case. The ultimate >> goal >>> is >>>>>>> to >>>>>>>>> reuse all the test cases when needed. >>>>>>>>> >>>>>>>>> 2. The current E2E framework is built with Java, running >> Java-based >>>>>>>>> program is not an easy thing for non-Java developers (maintainers >> of >>>>>>> other >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the >>>>>>>>> environment correctly and then run the tests, neither is it an >> easy >>>>>>> skill >>>>>>>>> to debug the tests. >>>>>>>>> >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we >>> added >>>>>>> many >>>>>>>>> cases and many duplicated codes that need to be removed. >>>>>>>>> >>>>>>>>> Therefore we're planning to build an unified, easy-to-use, and >> fast >>> E2E >>>>>>>>> test framework to benifit all the sub-projects. >>>>>>>>> >>>>>>>>> Here are some rough ideas about this framework: >>>>>>>>> >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to >>>>>>> orchestrate >>>>>>>>> the services that are to be tested, it’s proved to be friendly and >>>>>>>>> debuggable because we can start it directly even if we are not >>> writing >>>>>>> E2E >>>>>>>>> tests. >>>>>>>>> 1. This framework will take full advantages of the CLI project to >>> query >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two versions of >>> query >>>>>>>>> codes anymore, (now we have a version in the test codes and one in >>> the >>>>>>> CLI). >>>>>>>>> 2. We will build a CLI tool to compare the expected data and the >>> actual >>>>>>>>> data, we now have a custom schema of the expected data yaml file, >>> the >>>>>>>>> verification codes should be packaged into an executable command >> so >>>>>>> that it >>>>>>>>> can be executed standalone without Java and maven knowledge. I >> hope >>> we >>>>>>> can >>>>>>>>> leverage existing standards to write the YAML files and do >>>>>>> verifications. >>>>>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`, >>>>>>> health >>>>>>>>> check, query actual data, and verify the result. >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share >>>>>>> between >>>>>>>>> sub-projects. >>>>>>>>> >>>>>>>>> If you have any other better idea or want to complement to this >>>>>>> proposal, >>>>>>>>> please reply here. I will create an issue to track these tasks >>> later. >>>>>>>>> >>>>>>>>> We have two students (who just finished the Summer Code 2020 >>> Projects) >>>>>>> to >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this >> is >>> a >>>>>>> not >>>>>>>>> high priority, you have adequate time to get yourselves familiar >>> with >>>>>>> how >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking >> work >>> as >>>>>>> well >>>>>>>>> as the ecosystem. Thanks for your interests and willingness to >> help. >>>>>>>>> >>>>>>>>> [1] https://github.com/Humbertzhang >>>>>>>>> [2] https://github.com/fgksgf/ >>>>>>>>> >>>>>>>>> ————————— >>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>> GitHub @kezhenxu94 >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> ————————— >>>>>>> Zhenxu Ke (柯振旭) >>>>>>> GitHub @kezhenxu94 >>>>> >>> >>> >>> >>> ————————— >>> Zhenxu Ke (柯振旭) >>> GitHub @kezhenxu94 >>> >>> >>
