> 1. Docker-compose and kind are 2 options to set up the environments. > 2. Totally replace the Java and Maven based driving system. > 3. Enhance CLI to validate the GraphQL > 4. Keep the agent test tool unchanged for now as it is already a separate > system from the e2e.
This conclusion is good for me. What are others opinions? ————————— Zhenxu Ke (柯振旭) GitHub @kezhenxu94 > On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> wrote: > > Using k8s and docker-compose as 2 options in the test process is > reasonable, and should be supported. > > Let's keep the conclusion as simple as possible. In the next generation e2e > framework > 1. Docker-compose and kind are 2 options to set up the environments. > 2. Totally replace the Java and Maven based driving system. > 3. Enhance CLI to validate the GraphQL > 4. Keep the agent test tool unchanged for now as it is already a separate > system from the e2e. > > Are these good enough for everyone? > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > Hongtao Gao <[email protected]> 于2020年11月17日周二 下午12:41写道： > >>> >>> We can see from above, your discussion actually is only related to 3. >> >> >> That's the only I intend to discuss here if you notice my first >> response('May SWCK help about provision and demo") in this thread. For >> other parts, I don't have any tips to share, though. >> Let me explain it more clearly. I want kubernetes to be seen as an >> alternative to docker-compose to play a running-engine role in the test >> framework. If kubernetes can't replace >> the latter one Is it able to participate in it to solve the dedicated or >> special issues. >> >> Our e2e test and plugin test frameworks are already top-level complicity >>> thing in the worldwide open source field. Please don't over-design it, >> and >>> too aggressive, the world will stay in the hybrid env for a very long >> time, >>> same as the developer. >> >> >> Reusing current elaborate output might be reasonable. But in this thread, >> we're talking about the potential solution to build a >> next-generation framework, which might mean that the current framework >> is too complicated to maintain; we need a more tiny way to support more and >> more cases. >> >> And finally, SWCK is gonna and has to build a similar test framework. If >> the test framework opts for the kubernetes solution, we could share test >> cases and the underlying framework. It's a more efficient >> and consistent path for the entire system. >> >> regards, Hongtao. >> >> Sheng Wu <[email protected]> 于2020年11月17日周二 上午10:33写道： >> >>> Hi >>> >>> I don't think we have to make the decision on k8s or out-of-k8s. >>> e2e and all integration tests have their scenarios like SkyWalking have >>> multiple languages based and mesh-based, even Prometheus adoption. >>> The same thing happens on the test field, k8s, and out of k8s. >>> >>> I would like to agree with both of you, k8s is needed, no matter in mesh >>> solution(ALS, telemetry v2), but also not anyone needs that, such as >> agent >>> developer. We wouldn't deny the fact, the docker-compose is more >>> lightweight and easier to learn. >>> So, let me get this straight, for the test you need >>> 1. Build the source from different repos >>> 2. Build the images >>> 3. Use some platform(docker-compose or k8s) to run the images in one >> piece. >>> 4. Give some inputs to the env >>> 5. Use some tools/ways to check what should happen according to (4)'s >> input >>> 6. Checked or timeout failure. >>> >>> We can see from above, your discussion actually is only related to 3. >> Let's >>> not see this as a battle, we need both. Many developers/users use OAP and >>> SkyWalking out of k8s, that is a simple fact. >>> Our e2e test and plugin test frameworks are already top-level complicity >>> thing in the worldwide open source field. Please don't over-design it, >> and >>> too aggressive, the world will stay in the hybrid env for a very long >> time, >>> same as the developer. >>> >>> Also, let's keep in mind, k8s is popular becomes it doesn't require the >>> developers to understand as the VM did. We as an open-source community, >> are >>> focusing on the developer's capabilities. >>> >>> Sheng Wu 吴晟 >>> Twitter, wusheng1108 >>> >>> >>> Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： >>> >>>>> >>>>> but I have been thinking whether or not it is overkill for testing >>>>> purposes to introduce Kubernetes things. >>>> >>>> >>>> we all know the fact docker-compose is more portable than Kubernetes, >>>> friendly to local running. But there're also some benefits to replace >> it >>>> with kubernetes system: >>>> >>>> 1. SWCK has to test based on a real Kubernetes environment if the main >>>> repo test framework doesn't support it(following the docker-compose >>> stack), >>>> That means skywalking ecosystem MUST introduce kubernetes which is not >> an >>>> optional component. >>>> >>>> 2. Kubernetes support to scale applications at runtime, that help >>> improve >>>> the process of e2e. For instance, we could merge a single instance and >>>> cluster test, by scaling up replicas. This strategy will reduce the >> total >>>> time sharply. >>>> >>>> If we are going to use KIND, I hope we can give a script or something >>>>> similar to install all the needed components and create cluster for >>>>> convenient testing locally, Ke Zhang and Huaxi. >>>> >>>> >>>> That's SWCK's capacity to provide a standard and simple way to create a >>>> cluster in every environment, >>>> >>>> kezhenxu94@apache <[email protected]> 于2020年11月16日周一 下午7:00写道： >>>> >>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting >>> Kubernetes-based >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too, >> but I >>>>> have been thinking whether or not it is overkill for testing purposes >>> to >>>>> introduce Kubernetes things. >>>>> >>>>> Since you mentioned this, we are open to discuss this much more >> deeply: >>>>> >>>>> It would be definitely a benefit to involve more “end”s in the >> so-call >>>>> “end to end” tests, I’m +1 to leverage SWCK as well. >>>>> >>>>> If we are going to use KIND, I hope we can give a script or something >>>>> similar to install all the needed components and create cluster for >>>>> convenient testing locally, Ke Zhang and Huaxi. >>>>> >>>>> >>>>> ————————— >>>>> Zhenxu Ke (柯振旭) >>>>> GitHub @kezhenxu94 >>>>> >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> >>> wrote: >>>>>> >>>>>> May SWCK help about provision and demo >>>>>> >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone >>> instance >>>>> or >>>>>> cluster for e2e. We could compose indicated CR yamls for different >>>>>> scenarios. >>>>>> 2. Different demo for different cases is mandatory for e2e and >> swck, >>> we >>>>>> could build a group of demo projects for them. With Kubernetes's >>>> support, >>>>>> we could >>>>>> get free of verbose scripts that are written for installation, >>>>> checking >>>>>> the state of running applications, collecting logs. >>>>>> >>>>>> Ppl might have concerts about the minkube we currently have, >> which's >>>> hard >>>>>> to debug and runs slowly. >>>>>> >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind >>> works >>>>> fine >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run >>>> million >>>>>> times >>>>>> on my team's e2e environment, which proves the stability and >>>>>> maintainability of it. >>>>>> >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ >>>>>> >>>>>> regards, Hongtao >>>>>> >>>>>> Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： >>>>>> >>>>>>> Inline >>>>>>> >>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >> 下午11:16写道： >>>>>>> >>>>>>>> Hi Jiajing, >>>>>>>> >>>>>>>> Thanks for sharing your context and joining the discussion. It’s >>> true >>>>>>> that >>>>>>>> we’re actually tackling the same problems in this domain. Some of >>> the >>>>>>>> issues you mentioned are solved in the current E2E framework >> (like >>>>>>> `retry`, >>>>>>>> `timeout`, etc.), while some others are still there without ideal >>>>>>> solution. >>>>>>>> Other discussions, please see my comments inline. >>>>>>>> >>>>>>>>> - the first issue is useful in health check steps, just like the >>>> ones >>>>>>>>> defined in readiness probe and liveness probe in kubelet. >>>>>>>> >>>>>>>> Yes, the reason why we decided to pick docker-compose is that, it >>> has >>>>> the >>>>>>>> same functionality with kubelet in terms of dependent services >>>> startup >>>>>>>> order and healthiness checking, while it’s more lite-weight and >>> easy >>>>> for >>>>>>>> both developers and GitHub Actions environment. Right? >>>>>>>> >>>>>>>>> - the second issue is critical for highly-customized assertions, >>> we >>>>>>>>> can leverage the `text/template` module to provide highly >>> extensive >>>>>>>>> assertion functions >>>>>>>> >>>>>>>> `text/template` is also the first thought that came to my mind >> when >>>>>>>> considering customized assertors, customized assertors are for >> sure >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of >>>>>>> assertions, >>>>>>>> some of which are not foreseen until we actually need them, >>>>>>>> `AtLeastOneGreaterThanZero` is an example. >>>>>>>> >>>>>>>>> - the last but not the least, the lifecycle hooks can be used >> for >>>>>>>>> accurate control of test setup and teardown, such as >>> `docker-compose >>>>>>>>> build, up -d, down` >>>>>>>> >>>>>>>> The lifecycle hooks are what I didn’t take into consideration >>>> because I >>>>>>>> thought it is quite natural for test framework to provide this >>>>> features, >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. >>>>>>>> >>>>>>>>> Together with the aforementioned ideas, we will have a >> go-written, >>>>>>>>> yaml-data driven e2e testing framework which can overcome the >>>> current >>>>>>>>> problems that occur, >>>>>>>> >>>>>>>> I agree. >>>>>>>> >>>>>>>>> in which the most annoying issue is the fragmentation of >> different >>>>>>>>> scripts in different submodules, IMO. >>>>>>>> >>>>>>>> That’s why I list the last item in the previous email, (Nice to >>> have, >>>>>>> wrap >>>>>>>> them into a GitHub Action), with that GitHub Action, we can >> simply >>>>>>>> configure a workflow/job in every submodule, and the tests are >>>> executed >>>>>>>> with the shared scripts, no copy-pasting. >>>>>>>> >>>>>>> >>>>>>> Note, GitHub Action will require an Apache release every time >>> because >>>> we >>>>>>> are distributing the binary. >>>>>>> >>>>>>> >>>>>>> Sheng Wu 吴晟 >>>>>>> Twitter, wusheng1108 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Jiajing >>>>>>>>> >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < >>>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> So, we would provide a script set to drive this? >>>>>>>>>> >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. >>>>>>>>>> >>>>>>>>>>> My question is more about, >>>>>>>>>>> how to work with docker-compose >>>>>>>>>> >>>>>>>>>> It’s the same as what we’re doing now in the current version of >>> E2E >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the >>>>>>>> `docker-compose.yaml` files are reusable in the new framework, we >>>> will >>>>>>> keep >>>>>>>> this mechanism as it is proved to be convenient and easy to use >>>>> (remember >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E >>>> test, >>>>>>> not >>>>>>>> yet merged though, it’s fast/easy to add a new case). >>>>>>>>>> >>>>>>>>>> ————————— >>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>> >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < >>> [email protected]> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>> 下午9:16写道： >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I >>>> think >>>>>>> CLI >>>>>>>>>>>> can't >>>>>>>>>>>>> control this part. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> docker-compose just fits the requirements perfectly that we >>> need, >>>>> we >>>>>>>> have >>>>>>>>>>>> been using it in the current E2E and turns out it has >>> advantages >>>> in >>>>>>>>>>>> starting many services correctly with health check and >>> dependency >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing, >> but >>>>> it’s >>>>>>>> kind >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). >>>>>>>>>>>> >>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose >>> related >>>>>>>> things, >>>>>>>>>>>> it just provides query ability (the same as what it is doing >>> now, >>>>>>>> just need >>>>>>>>>>>> to check whether it covers all the needed query interfaces or >>>> not), >>>>>>>> we will >>>>>>>>>>>> have a dedicated control program to >>>>>>>>>>>> >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no new >>>>>>>> knowledge); >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So, we would provide a script set to drive this? My question >> is >>>> more >>>>>>>> about, >>>>>>>>>>> how to work with docker-compose, rather than challanging the >>>> choice. >>>>>>>>>>> >>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository >>> and >>>>>>>> codes); >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new tool, >>>>> TODO); >>>>>>>>>>>> (4) and determine the test result (success or failed); >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ————————— >>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>> 下午5:27写道： >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build the >>>> next >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, >> which >>>> may >>>>>>>> not >>>>>>>>>>>> be a >>>>>>>>>>>>>> high priority but a necessity. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very >> important >>>>> role >>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>>>> contribution process, and the current E2E tests just work >>> well >>>> to >>>>>>>> verify >>>>>>>>>>>>>> every single pull request before merging, so why bother to >>>> build >>>>>>> the >>>>>>>>>>>> next >>>>>>>>>>>>>> generation of E2E test framework? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many >> sub-projects >>>>> that >>>>>>>> vary >>>>>>>>>>>> in >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to >> help >>> us >>>>>>>> verify >>>>>>>>>>>> the >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the E2E >>>> test >>>>>>>>>>>> framework >>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t have >>> E2E >>>>>>>> tests at >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry >>> and >>>>>>>>>>>> introduces >>>>>>>>>>>>>> more duplicated work when adding a new test case. The >>> ultimate >>>>>>> goal >>>>>>>> is >>>>>>>>>>>> to >>>>>>>>>>>>>> reuse all the test cases when needed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running >>>>>>> Java-based >>>>>>>>>>>>>> program is not an easy thing for non-Java developers >>>> (maintainers >>>>>>> of >>>>>>>>>>>> other >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set >>> up >>>>> the >>>>>>>>>>>>>> environment correctly and then run the tests, neither is it >>> an >>>>>>> easy >>>>>>>>>>>> skill >>>>>>>>>>>>>> to debug the tests. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests >> because >>> we >>>>>>>> added >>>>>>>>>>>> many >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use, >>> and >>>>>>> fast >>>>>>>> E2E >>>>>>>>>>>>>> test framework to benifit all the sub-projects. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are some rough ideas about this framework: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml >> to >>>>>>>>>>>> orchestrate >>>>>>>>>>>>>> the services that are to be tested, it’s proved to be >>> friendly >>>>> and >>>>>>>>>>>>>> debuggable because we can start it directly even if we are >>> not >>>>>>>> writing >>>>>>>>>>>> E2E >>>>>>>>>>>>>> tests. >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI >>> project >>>> to >>>>>>>> query >>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two >>> versions >>>>> of >>>>>>>> query >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes and >>> one >>>>> in >>>>>>>> the >>>>>>>>>>>> CLI). >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data >> and >>>> the >>>>>>>> actual >>>>>>>>>>>>>> data, we now have a custom schema of the expected data yaml >>>> file, >>>>>>>> the >>>>>>>>>>>>>> verification codes should be packaged into an executable >>>> command >>>>>>> so >>>>>>>>>>>> that it >>>>>>>>>>>>>> can be executed standalone without Java and maven >> knowledge. >>> I >>>>>>> hope >>>>>>>> we >>>>>>>>>>>> can >>>>>>>>>>>>>> leverage existing standards to write the YAML files and do >>>>>>>>>>>> verifications. >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the >>>>> `docker-compose.yaml`, >>>>>>>>>>>> health >>>>>>>>>>>>>> check, query actual data, and verify the result. >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to >>>> share >>>>>>>>>>>> between >>>>>>>>>>>>>> sub-projects. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If you have any other better idea or want to complement to >>> this >>>>>>>>>>>> proposal, >>>>>>>>>>>>>> please reply here. I will create an issue to track these >>> tasks >>>>>>>> later. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have two students (who just finished the Summer Code >> 2020 >>>>>>>> Projects) >>>>>>>>>>>> to >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said >>>> this >>>>>>> is >>>>>>>> a >>>>>>>>>>>> not >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves >>>> familiar >>>>>>>> with >>>>>>>>>>>> how >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of >>> SkyWalking >>>>>>> work >>>>>>>> as >>>>>>>>>>>> well >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and willingness >>> to >>>>>>> help. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang >>>>>>>>>>>>>> [2] https://github.com/fgksgf/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ————————— >>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ————————— >>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>> GitHub @kezhenxu94 >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >>
