Hey, I’ve just drafted a design doc here [1], anyone interested in this please feel free to leave your comments, thanks!
[1] https://github.com/apache/skywalking-website/pull/172 > On Nov 26, 2020, at 14:41, Sheng Wu <[email protected]> wrote: > > You could learn from the Satellite project design, > 1. Discuss here, https://github.com/apache/skywalking/issues/5871 > 2. Final design posted as a blog, > http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/ > > People could learn from these things when you join the community in the > future. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > Hoshea Jiang <[email protected]> 于2020年11月26日周四 下午1:38写道： > >> I agree with the conclusion, and I hope to have some diagrams such as >> flowcharts or data flow diagrams, which help provide an overview of the E2E >> test before we start to do this. >> >> Ke Zhang <[email protected]> 于2020年11月26日周四 下午12:34写道： >> >>> I think supporting both docker-compose and KIND in the E2E test is >>> reasonable and necessary, and I also agree with other conclusions.😀 >>> >>> kezhenxu94@apache <[email protected]> 于2020年11月26日周四 上午11:58写道： >>> >>>> >>>>> 1. Docker-compose and kind are 2 options to set up the environments. >>>>> 2. Totally replace the Java and Maven based driving system. >>>>> 3. Enhance CLI to validate the GraphQL >>>>> 4. Keep the agent test tool unchanged for now as it is already a >>> separate >>>>> system from the e2e. >>>> >>>> This conclusion is good for me. >>>> >>>> What are others opinions? >>>> >>>> ————————— >>>> Zhenxu Ke (柯振旭) >>>> GitHub @kezhenxu94 >>>> >>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> >>> wrote: >>>>> >>>>> Using k8s and docker-compose as 2 options in the test process is >>>>> reasonable, and should be supported. >>>>> >>>>> Let's keep the conclusion as simple as possible. In the next >> generation >>>> e2e >>>>> framework >>>>> 1. Docker-compose and kind are 2 options to set up the environments. >>>>> 2. Totally replace the Java and Maven based driving system. >>>>> 3. Enhance CLI to validate the GraphQL >>>>> 4. Keep the agent test tool unchanged for now as it is already a >>> separate >>>>> system from the e2e. >>>>> >>>>> Are these good enough for everyone? >>>>> >>>>> Sheng Wu 吴晟 >>>>> Twitter, wusheng1108 >>>>> >>>>> >>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 下午12:41写道： >>>>> >>>>>>> >>>>>>> We can see from above, your discussion actually is only related to >> 3. >>>>>> >>>>>> >>>>>> That's the only I intend to discuss here if you notice my first >>>>>> response('May SWCK help about provision and demo") in this thread. >> For >>>>>> other parts, I don't have any tips to share, though. >>>>>> Let me explain it more clearly. I want kubernetes to be seen as an >>>>>> alternative to docker-compose to play a running-engine role in the >>> test >>>>>> framework. If kubernetes can't replace >>>>>> the latter one Is it able to participate in it to solve the >> dedicated >>> or >>>>>> special issues. >>>>>> >>>>>> Our e2e test and plugin test frameworks are already top-level >>> complicity >>>>>>> thing in the worldwide open source field. Please don't over-design >>> it, >>>>>> and >>>>>>> too aggressive, the world will stay in the hybrid env for a very >> long >>>>>> time, >>>>>>> same as the developer. >>>>>> >>>>>> >>>>>> Reusing current elaborate output might be reasonable. But in this >>>> thread, >>>>>> we're talking about the potential solution to build a >>>>>> next-generation framework, which might mean that the current >> framework >>>>>> is too complicated to maintain; we need a more tiny way to support >>> more >>>> and >>>>>> more cases. >>>>>> >>>>>> And finally, SWCK is gonna and has to build a similar test >> framework. >>> If >>>>>> the test framework opts for the kubernetes solution, we could share >>> test >>>>>> cases and the underlying framework. It's a more efficient >>>>>> and consistent path for the entire system. >>>>>> >>>>>> regards, Hongtao. >>>>>> >>>>>> Sheng Wu <[email protected]> 于2020年11月17日周二 上午10:33写道： >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I don't think we have to make the decision on k8s or out-of-k8s. >>>>>>> e2e and all integration tests have their scenarios like SkyWalking >>> have >>>>>>> multiple languages based and mesh-based, even Prometheus adoption. >>>>>>> The same thing happens on the test field, k8s, and out of k8s. >>>>>>> >>>>>>> I would like to agree with both of you, k8s is needed, no matter in >>>> mesh >>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such >> as >>>>>> agent >>>>>>> developer. We wouldn't deny the fact, the docker-compose is more >>>>>>> lightweight and easier to learn. >>>>>>> So, let me get this straight, for the test you need >>>>>>> 1. Build the source from different repos >>>>>>> 2. Build the images >>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in >> one >>>>>> piece. >>>>>>> 4. Give some inputs to the env >>>>>>> 5. Use some tools/ways to check what should happen according to >> (4)'s >>>>>> input >>>>>>> 6. Checked or timeout failure. >>>>>>> >>>>>>> We can see from above, your discussion actually is only related to >> 3. >>>>>> Let's >>>>>>> not see this as a battle, we need both. Many developers/users use >> OAP >>>> and >>>>>>> SkyWalking out of k8s, that is a simple fact. >>>>>>> Our e2e test and plugin test frameworks are already top-level >>>> complicity >>>>>>> thing in the worldwide open source field. Please don't over-design >>> it, >>>>>> and >>>>>>> too aggressive, the world will stay in the hybrid env for a very >> long >>>>>> time, >>>>>>> same as the developer. >>>>>>> >>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require >>> the >>>>>>> developers to understand as the VM did. We as an open-source >>> community, >>>>>> are >>>>>>> focusing on the developer's capabilities. >>>>>>> >>>>>>> Sheng Wu 吴晟 >>>>>>> Twitter, wusheng1108 >>>>>>> >>>>>>> >>>>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： >>>>>>> >>>>>>>>> >>>>>>>>> but I have been thinking whether or not it is overkill for >> testing >>>>>>>>> purposes to introduce Kubernetes things. >>>>>>>> >>>>>>>> >>>>>>>> we all know the fact docker-compose is more portable than >>> Kubernetes, >>>>>>>> friendly to local running. But there're also some benefits to >>> replace >>>>>> it >>>>>>>> with kubernetes system: >>>>>>>> >>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the >>> main >>>>>>>> repo test framework doesn't support it(following the >> docker-compose >>>>>>> stack), >>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is >>> not >>>>>> an >>>>>>>> optional component. >>>>>>>> >>>>>>>> 2. Kubernetes support to scale applications at runtime, that help >>>>>>> improve >>>>>>>> the process of e2e. For instance, we could merge a single instance >>> and >>>>>>>> cluster test, by scaling up replicas. This strategy will reduce >> the >>>>>> total >>>>>>>> time sharply. >>>>>>>> >>>>>>>> If we are going to use KIND, I hope we can give a script or >>> something >>>>>>>>> similar to install all the needed components and create cluster >> for >>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. >>>>>>>> >>>>>>>> >>>>>>>> That's SWCK's capacity to provide a standard and simple way to >>> create >>>> a >>>>>>>> cluster in every environment, >>>>>>>> >>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月16日周一 >> 下午7:00写道： >>>>>>>> >>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting >>>>>>> Kubernetes-based >>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too, >>>>>> but I >>>>>>>>> have been thinking whether or not it is overkill for testing >>> purposes >>>>>>> to >>>>>>>>> introduce Kubernetes things. >>>>>>>>> >>>>>>>>> Since you mentioned this, we are open to discuss this much more >>>>>> deeply: >>>>>>>>> >>>>>>>>> It would be definitely a benefit to involve more “end”s in the >>>>>> so-call >>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well. >>>>>>>>> >>>>>>>>> If we are going to use KIND, I hope we can give a script or >>> something >>>>>>>>> similar to install all the needed components and create cluster >> for >>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. >>>>>>>>> >>>>>>>>> >>>>>>>>> ————————— >>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>> GitHub @kezhenxu94 >>>>>>>>> >>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> May SWCK help about provision and demo >>>>>>>>>> >>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone >>>>>>> instance >>>>>>>>> or >>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for >> different >>>>>>>>>> scenarios. >>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and >>>>>> swck, >>>>>>> we >>>>>>>>>> could build a group of demo projects for them. With Kubernetes's >>>>>>>> support, >>>>>>>>>> we could >>>>>>>>>> get free of verbose scripts that are written for installation, >>>>>>>>> checking >>>>>>>>>> the state of running applications, collecting logs. >>>>>>>>>> >>>>>>>>>> Ppl might have concerts about the minkube we currently have, >>>>>> which's >>>>>>>> hard >>>>>>>>>> to debug and runs slowly. >>>>>>>>>> >>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind >>>>>>> works >>>>>>>>> fine >>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run >>>>>>>> million >>>>>>>>>> times >>>>>>>>>> on my team's e2e environment, which proves the stability and >>>>>>>>>> maintainability of it. >>>>>>>>>> >>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ >>>>>>>>>> >>>>>>>>>> regards, Hongtao >>>>>>>>>> >>>>>>>>>> Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： >>>>>>>>>> >>>>>>>>>>> Inline >>>>>>>>>>> >>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>> 下午11:16写道： >>>>>>>>>>> >>>>>>>>>>>> Hi Jiajing, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for sharing your context and joining the discussion. >> It’s >>>>>>> true >>>>>>>>>>> that >>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some >>> of >>>>>>> the >>>>>>>>>>>> issues you mentioned are solved in the current E2E framework >>>>>> (like >>>>>>>>>>> `retry`, >>>>>>>>>>>> `timeout`, etc.), while some others are still there without >>> ideal >>>>>>>>>>> solution. >>>>>>>>>>>> Other discussions, please see my comments inline. >>>>>>>>>>>> >>>>>>>>>>>>> - the first issue is useful in health check steps, just like >>> the >>>>>>>> ones >>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet. >>>>>>>>>>>> >>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that, >>> it >>>>>>> has >>>>>>>>> the >>>>>>>>>>>> same functionality with kubelet in terms of dependent services >>>>>>>> startup >>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight >> and >>>>>>> easy >>>>>>>>> for >>>>>>>>>>>> both developers and GitHub Actions environment. Right? >>>>>>>>>>>> >>>>>>>>>>>>> - the second issue is critical for highly-customized >>> assertions, >>>>>>> we >>>>>>>>>>>>> can leverage the `text/template` module to provide highly >>>>>>> extensive >>>>>>>>>>>>> assertion functions >>>>>>>>>>>> >>>>>>>>>>>> `text/template` is also the first thought that came to my mind >>>>>> when >>>>>>>>>>>> considering customized assertors, customized assertors are for >>>>>> sure >>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of >>>>>>>>>>> assertions, >>>>>>>>>>>> some of which are not foreseen until we actually need them, >>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example. >>>>>>>>>>>> >>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used >>>>>> for >>>>>>>>>>>>> accurate control of test setup and teardown, such as >>>>>>> `docker-compose >>>>>>>>>>>>> build, up -d, down` >>>>>>>>>>>> >>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration >>>>>>>> because I >>>>>>>>>>>> thought it is quite natural for test framework to provide this >>>>>>>>> features, >>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. >>>>>>>>>>>> >>>>>>>>>>>>> Together with the aforementioned ideas, we will have a >>>>>> go-written, >>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the >>>>>>>> current >>>>>>>>>>>>> problems that occur, >>>>>>>>>>>> >>>>>>>>>>>> I agree. >>>>>>>>>>>> >>>>>>>>>>>>> in which the most annoying issue is the fragmentation of >>>>>> different >>>>>>>>>>>>> scripts in different submodules, IMO. >>>>>>>>>>>> >>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice >> to >>>>>>> have, >>>>>>>>>>> wrap >>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can >>>>>> simply >>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are >>>>>>>> executed >>>>>>>>>>>> with the shared scripts, no copy-pasting. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Note, GitHub Action will require an Apache release every time >>>>>>> because >>>>>>>> we >>>>>>>>>>> are distributing the binary. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Jiajing >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < >>>>>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> So, we would provide a script set to drive this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> My question is more about, >>>>>>>>>>>>>>> how to work with docker-compose >>>>>>>>>>>>>> >>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version >>> of >>>>>>> E2E >>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all >> the >>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework, >>> we >>>>>>>> will >>>>>>>>>>> keep >>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to >> use >>>>>>>>> (remember >>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage >>> E2E >>>>>>>> test, >>>>>>>>>>> not >>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case). >>>>>>>>>>>>>> >>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < >>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>>>> 下午9:16写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your >> mind? I >>>>>>>> think >>>>>>>>>>> CLI >>>>>>>>>>>>>>>> can't >>>>>>>>>>>>>>>>> control this part. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that >> we >>>>>>> need, >>>>>>>>> we >>>>>>>>>>>> have >>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has >>>>>>> advantages >>>>>>>> in >>>>>>>>>>>>>>>> starting many services correctly with health check and >>>>>>> dependency >>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same >> thing, >>>>>> but >>>>>>>>> it’s >>>>>>>>>>>> kind >>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose >>>>>>> related >>>>>>>>>>>> things, >>>>>>>>>>>>>>>> it just provides query ability (the same as what it is >> doing >>>>>>> now, >>>>>>>>>>>> just need >>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces >>> or >>>>>>>> not), >>>>>>>>>>>> we will >>>>>>>>>>>>>>>> have a dedicated control program to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no >>> new >>>>>>>>>>>> knowledge); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So, we would provide a script set to drive this? My >> question >>>>>> is >>>>>>>> more >>>>>>>>>>>> about, >>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging >> the >>>>>>>> choice. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing >>> repository >>>>>>> and >>>>>>>>>>>> codes); >>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new >>> tool, >>>>>>>>> TODO); >>>>>>>>>>>>>>>> (4) and determine the test result (success or failed); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>>>>> 下午5:27写道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build >>> the >>>>>>>> next >>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, >>>>>> which >>>>>>>> may >>>>>>>>>>>> not >>>>>>>>>>>>>>>> be a >>>>>>>>>>>>>>>>>> high priority but a necessity. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very >>>>>> important >>>>>>>>> role >>>>>>>>>>>> in >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just >> work >>>>>>> well >>>>>>>> to >>>>>>>>>>>> verify >>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother >> to >>>>>>>> build >>>>>>>>>>> the >>>>>>>>>>>>>>>> next >>>>>>>>>>>>>>>>>> generation of E2E test framework? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many >>>>>> sub-projects >>>>>>>>> that >>>>>>>>>>>> vary >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to >>>>>> help >>>>>>> us >>>>>>>>>>>> verify >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the >>> E2E >>>>>>>> test >>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t >>> have >>>>>>> E2E >>>>>>>>>>>> tests at >>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is >> unnecesarry >>>>>>> and >>>>>>>>>>>>>>>> introduces >>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The >>>>>>> ultimate >>>>>>>>>>> goal >>>>>>>>>>>> is >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> reuse all the test cases when needed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running >>>>>>>>>>> Java-based >>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers >>>>>>>> (maintainers >>>>>>>>>>> of >>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to >>> set >>>>>>> up >>>>>>>>> the >>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is >>> it >>>>>>> an >>>>>>>>>>> easy >>>>>>>>>>>>>>>> skill >>>>>>>>>>>>>>>>>> to debug the tests. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests >>>>>> because >>>>>>> we >>>>>>>>>>>> added >>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified, >> easy-to-use, >>>>>>> and >>>>>>>>>>> fast >>>>>>>>>>>> E2E >>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use >> docker-compose.yaml >>>>>> to >>>>>>>>>>>>>>>> orchestrate >>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be >>>>>>> friendly >>>>>>>>> and >>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we >> are >>>>>>> not >>>>>>>>>>>> writing >>>>>>>>>>>>>>>> E2E >>>>>>>>>>>>>>>>>> tests. >>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI >>>>>>> project >>>>>>>> to >>>>>>>>>>>> query >>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two >>>>>>> versions >>>>>>>>> of >>>>>>>>>>>> query >>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes >>> and >>>>>>> one >>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>>>>>> CLI). >>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data >>>>>> and >>>>>>>> the >>>>>>>>>>>> actual >>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data >>> yaml >>>>>>>> file, >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable >>>>>>>> command >>>>>>>>>>> so >>>>>>>>>>>>>>>> that it >>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven >>>>>> knowledge. >>>>>>> I >>>>>>>>>>> hope >>>>>>>>>>>> we >>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and >> do >>>>>>>>>>>>>>>> verifications. >>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the >>>>>>>>> `docker-compose.yaml`, >>>>>>>>>>>>>>>> health >>>>>>>>>>>>>>>>>> check, query actual data, and verify the result. >>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions >>> to >>>>>>>> share >>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>> sub-projects. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement >> to >>>>>>> this >>>>>>>>>>>>>>>> proposal, >>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these >>>>>>> tasks >>>>>>>>>>>> later. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code >>>>>> 2020 >>>>>>>>>>>> Projects) >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I >>> said >>>>>>>> this >>>>>>>>>>> is >>>>>>>>>>>> a >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves >>>>>>>> familiar >>>>>>>>>>>> with >>>>>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of >>>>>>> SkyWalking >>>>>>>>>>> work >>>>>>>>>>>> as >>>>>>>>>>>>>>>> well >>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and >>> willingness >>>>>>> to >>>>>>>>>>> help. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang >>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ————————— >>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >> ————————— Zhenxu Ke (柯振旭) GitHub @kezhenxu94
