Because now we have skywalking-eyes repository to host the infra tools, we can put the new E2E framework there so that it doesn’t mess up the CLI repo. Here [1] I created a new directory to host the E2E codes.
[1] https://github.com/apache/skywalking-eyes/pull/11 ————————— Zhenxu Ke (柯振旭) GitHub @kezhenxu94 > On Dec 17, 2020, at 17:45, kezhenxu94@apache <[email protected]> wrote: > > Hey, I’ve just drafted a design doc here [1], anyone interested in this > please feel free to leave your comments, thanks! > > [1] https://github.com/apache/skywalking-website/pull/172 > >> On Nov 26, 2020, at 14:41, Sheng Wu <[email protected]> wrote: >> >> You could learn from the Satellite project design, >> 1. Discuss here, https://github.com/apache/skywalking/issues/5871 >> 2. Final design posted as a blog, >> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/ >> >> People could learn from these things when you join the community in the >> future. >> >> Sheng Wu 吴晟 >> Twitter, wusheng1108 >> >> >> Hoshea Jiang <[email protected]> 于2020年11月26日周四 下午1:38写道： >> >>> I agree with the conclusion, and I hope to have some diagrams such as >>> flowcharts or data flow diagrams, which help provide an overview of the E2E >>> test before we start to do this. >>> >>> Ke Zhang <[email protected]> 于2020年11月26日周四 下午12:34写道： >>> >>>> I think supporting both docker-compose and KIND in the E2E test is >>>> reasonable and necessary, and I also agree with other conclusions.😀 >>>> >>>> kezhenxu94@apache <[email protected]> 于2020年11月26日周四 上午11:58写道： >>>> >>>>> >>>>>> 1. Docker-compose and kind are 2 options to set up the environments. >>>>>> 2. Totally replace the Java and Maven based driving system. >>>>>> 3. Enhance CLI to validate the GraphQL >>>>>> 4. Keep the agent test tool unchanged for now as it is already a >>>> separate >>>>>> system from the e2e. >>>>> >>>>> This conclusion is good for me. >>>>> >>>>> What are others opinions? >>>>> >>>>> ————————— >>>>> Zhenxu Ke (柯振旭) >>>>> GitHub @kezhenxu94 >>>>> >>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> >>>> wrote: >>>>>> >>>>>> Using k8s and docker-compose as 2 options in the test process is >>>>>> reasonable, and should be supported. >>>>>> >>>>>> Let's keep the conclusion as simple as possible. In the next >>> generation >>>>> e2e >>>>>> framework >>>>>> 1. Docker-compose and kind are 2 options to set up the environments. >>>>>> 2. Totally replace the Java and Maven based driving system. >>>>>> 3. Enhance CLI to validate the GraphQL >>>>>> 4. Keep the agent test tool unchanged for now as it is already a >>>> separate >>>>>> system from the e2e. >>>>>> >>>>>> Are these good enough for everyone? >>>>>> >>>>>> Sheng Wu 吴晟 >>>>>> Twitter, wusheng1108 >>>>>> >>>>>> >>>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 下午12:41写道： >>>>>> >>>>>>>> >>>>>>>> We can see from above, your discussion actually is only related to >>> 3. >>>>>>> >>>>>>> >>>>>>> That's the only I intend to discuss here if you notice my first >>>>>>> response('May SWCK help about provision and demo") in this thread. >>> For >>>>>>> other parts, I don't have any tips to share, though. >>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an >>>>>>> alternative to docker-compose to play a running-engine role in the >>>> test >>>>>>> framework. If kubernetes can't replace >>>>>>> the latter one Is it able to participate in it to solve the >>> dedicated >>>> or >>>>>>> special issues. >>>>>>> >>>>>>> Our e2e test and plugin test frameworks are already top-level >>>> complicity >>>>>>>> thing in the worldwide open source field. Please don't over-design >>>> it, >>>>>>> and >>>>>>>> too aggressive, the world will stay in the hybrid env for a very >>> long >>>>>>> time, >>>>>>>> same as the developer. >>>>>>> >>>>>>> >>>>>>> Reusing current elaborate output might be reasonable. But in this >>>>> thread, >>>>>>> we're talking about the potential solution to build a >>>>>>> next-generation framework, which might mean that the current >>> framework >>>>>>> is too complicated to maintain; we need a more tiny way to support >>>> more >>>>> and >>>>>>> more cases. >>>>>>> >>>>>>> And finally, SWCK is gonna and has to build a similar test >>> framework. >>>> If >>>>>>> the test framework opts for the kubernetes solution, we could share >>>> test >>>>>>> cases and the underlying framework. It's a more efficient >>>>>>> and consistent path for the entire system. >>>>>>> >>>>>>> regards, Hongtao. >>>>>>> >>>>>>> Sheng Wu <[email protected]> 于2020年11月17日周二 上午10:33写道： >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s. >>>>>>>> e2e and all integration tests have their scenarios like SkyWalking >>>> have >>>>>>>> multiple languages based and mesh-based, even Prometheus adoption. >>>>>>>> The same thing happens on the test field, k8s, and out of k8s. >>>>>>>> >>>>>>>> I would like to agree with both of you, k8s is needed, no matter in >>>>> mesh >>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such >>> as >>>>>>> agent >>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more >>>>>>>> lightweight and easier to learn. >>>>>>>> So, let me get this straight, for the test you need >>>>>>>> 1. Build the source from different repos >>>>>>>> 2. Build the images >>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in >>> one >>>>>>> piece. >>>>>>>> 4. Give some inputs to the env >>>>>>>> 5. Use some tools/ways to check what should happen according to >>> (4)'s >>>>>>> input >>>>>>>> 6. Checked or timeout failure. >>>>>>>> >>>>>>>> We can see from above, your discussion actually is only related to >>> 3. >>>>>>> Let's >>>>>>>> not see this as a battle, we need both. Many developers/users use >>> OAP >>>>> and >>>>>>>> SkyWalking out of k8s, that is a simple fact. >>>>>>>> Our e2e test and plugin test frameworks are already top-level >>>>> complicity >>>>>>>> thing in the worldwide open source field. Please don't over-design >>>> it, >>>>>>> and >>>>>>>> too aggressive, the world will stay in the hybrid env for a very >>> long >>>>>>> time, >>>>>>>> same as the developer. >>>>>>>> >>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require >>>> the >>>>>>>> developers to understand as the VM did. We as an open-source >>>> community, >>>>>>> are >>>>>>>> focusing on the developer's capabilities. >>>>>>>> >>>>>>>> Sheng Wu 吴晟 >>>>>>>> Twitter, wusheng1108 >>>>>>>> >>>>>>>> >>>>>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： >>>>>>>> >>>>>>>>>> >>>>>>>>>> but I have been thinking whether or not it is overkill for >>> testing >>>>>>>>>> purposes to introduce Kubernetes things. >>>>>>>>> >>>>>>>>> >>>>>>>>> we all know the fact docker-compose is more portable than >>>> Kubernetes, >>>>>>>>> friendly to local running. But there're also some benefits to >>>> replace >>>>>>> it >>>>>>>>> with kubernetes system: >>>>>>>>> >>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the >>>> main >>>>>>>>> repo test framework doesn't support it(following the >>> docker-compose >>>>>>>> stack), >>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is >>>> not >>>>>>> an >>>>>>>>> optional component. >>>>>>>>> >>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help >>>>>>>> improve >>>>>>>>> the process of e2e. For instance, we could merge a single instance >>>> and >>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce >>> the >>>>>>> total >>>>>>>>> time sharply. >>>>>>>>> >>>>>>>>> If we are going to use KIND, I hope we can give a script or >>>> something >>>>>>>>>> similar to install all the needed components and create cluster >>> for >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. >>>>>>>>> >>>>>>>>> >>>>>>>>> That's SWCK's capacity to provide a standard and simple way to >>>> create >>>>> a >>>>>>>>> cluster in every environment, >>>>>>>>> >>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月16日周一 >>> 下午7:00写道： >>>>>>>>> >>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting >>>>>>>> Kubernetes-based >>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too, >>>>>>> but I >>>>>>>>>> have been thinking whether or not it is overkill for testing >>>> purposes >>>>>>>> to >>>>>>>>>> introduce Kubernetes things. >>>>>>>>>> >>>>>>>>>> Since you mentioned this, we are open to discuss this much more >>>>>>> deeply: >>>>>>>>>> >>>>>>>>>> It would be definitely a benefit to involve more “end”s in the >>>>>>> so-call >>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well. >>>>>>>>>> >>>>>>>>>> If we are going to use KIND, I hope we can give a script or >>>> something >>>>>>>>>> similar to install all the needed components and create cluster >>> for >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ————————— >>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>> >>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> May SWCK help about provision and demo >>>>>>>>>>> >>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone >>>>>>>> instance >>>>>>>>>> or >>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for >>> different >>>>>>>>>>> scenarios. >>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and >>>>>>> swck, >>>>>>>> we >>>>>>>>>>> could build a group of demo projects for them. With Kubernetes's >>>>>>>>> support, >>>>>>>>>>> we could >>>>>>>>>>> get free of verbose scripts that are written for installation, >>>>>>>>>> checking >>>>>>>>>>> the state of running applications, collecting logs. >>>>>>>>>>> >>>>>>>>>>> Ppl might have concerts about the minkube we currently have, >>>>>>> which's >>>>>>>>> hard >>>>>>>>>>> to debug and runs slowly. >>>>>>>>>>> >>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind >>>>>>>> works >>>>>>>>>> fine >>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run >>>>>>>>> million >>>>>>>>>>> times >>>>>>>>>>> on my team's e2e environment, which proves the stability and >>>>>>>>>>> maintainability of it. >>>>>>>>>>> >>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ >>>>>>>>>>> >>>>>>>>>>> regards, Hongtao >>>>>>>>>>> >>>>>>>>>>> Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： >>>>>>>>>>> >>>>>>>>>>>> Inline >>>>>>>>>>>> >>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>>> 下午11:16写道： >>>>>>>>>>>> >>>>>>>>>>>>> Hi Jiajing, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for sharing your context and joining the discussion. >>> It’s >>>>>>>> true >>>>>>>>>>>> that >>>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some >>>> of >>>>>>>> the >>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework >>>>>>> (like >>>>>>>>>>>> `retry`, >>>>>>>>>>>>> `timeout`, etc.), while some others are still there without >>>> ideal >>>>>>>>>>>> solution. >>>>>>>>>>>>> Other discussions, please see my comments inline. >>>>>>>>>>>>> >>>>>>>>>>>>>> - the first issue is useful in health check steps, just like >>>> the >>>>>>>>> ones >>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet. >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that, >>>> it >>>>>>>> has >>>>>>>>>> the >>>>>>>>>>>>> same functionality with kubelet in terms of dependent services >>>>>>>>> startup >>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight >>> and >>>>>>>> easy >>>>>>>>>> for >>>>>>>>>>>>> both developers and GitHub Actions environment. Right? >>>>>>>>>>>>> >>>>>>>>>>>>>> - the second issue is critical for highly-customized >>>> assertions, >>>>>>>> we >>>>>>>>>>>>>> can leverage the `text/template` module to provide highly >>>>>>>> extensive >>>>>>>>>>>>>> assertion functions >>>>>>>>>>>>> >>>>>>>>>>>>> `text/template` is also the first thought that came to my mind >>>>>>> when >>>>>>>>>>>>> considering customized assertors, customized assertors are for >>>>>>> sure >>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of >>>>>>>>>>>> assertions, >>>>>>>>>>>>> some of which are not foreseen until we actually need them, >>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example. >>>>>>>>>>>>> >>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used >>>>>>> for >>>>>>>>>>>>>> accurate control of test setup and teardown, such as >>>>>>>> `docker-compose >>>>>>>>>>>>>> build, up -d, down` >>>>>>>>>>>>> >>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration >>>>>>>>> because I >>>>>>>>>>>>> thought it is quite natural for test framework to provide this >>>>>>>>>> features, >>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. >>>>>>>>>>>>> >>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a >>>>>>> go-written, >>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the >>>>>>>>> current >>>>>>>>>>>>>> problems that occur, >>>>>>>>>>>>> >>>>>>>>>>>>> I agree. >>>>>>>>>>>>> >>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of >>>>>>> different >>>>>>>>>>>>>> scripts in different submodules, IMO. >>>>>>>>>>>>> >>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice >>> to >>>>>>>> have, >>>>>>>>>>>> wrap >>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can >>>>>>> simply >>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are >>>>>>>>> executed >>>>>>>>>>>>> with the shared scripts, no copy-pasting. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Note, GitHub Action will require an Apache release every time >>>>>>>> because >>>>>>>>> we >>>>>>>>>>>> are distributing the binary. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Jiajing >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My question is more about, >>>>>>>>>>>>>>>> how to work with docker-compose >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version >>>> of >>>>>>>> E2E >>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all >>> the >>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework, >>>> we >>>>>>>>> will >>>>>>>>>>>> keep >>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to >>> use >>>>>>>>>> (remember >>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage >>>> E2E >>>>>>>>> test, >>>>>>>>>>>> not >>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < >>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>>>>> 下午9:16写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your >>> mind? I >>>>>>>>> think >>>>>>>>>>>> CLI >>>>>>>>>>>>>>>>> can't >>>>>>>>>>>>>>>>>> control this part. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that >>> we >>>>>>>> need, >>>>>>>>>> we >>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has >>>>>>>> advantages >>>>>>>>> in >>>>>>>>>>>>>>>>> starting many services correctly with health check and >>>>>>>> dependency >>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same >>> thing, >>>>>>> but >>>>>>>>>> it’s >>>>>>>>>>>>> kind >>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose >>>>>>>> related >>>>>>>>>>>>> things, >>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is >>> doing >>>>>>>> now, >>>>>>>>>>>>> just need >>>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces >>>> or >>>>>>>>> not), >>>>>>>>>>>>> we will >>>>>>>>>>>>>>>>> have a dedicated control program to >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no >>>> new >>>>>>>>>>>>> knowledge); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My >>> question >>>>>>> is >>>>>>>>> more >>>>>>>>>>>>> about, >>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging >>> the >>>>>>>>> choice. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing >>>> repository >>>>>>>> and >>>>>>>>>>>>> codes); >>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new >>>> tool, >>>>>>>>>> TODO); >>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Sheng Wu 吴晟 >>>>>>>>>>>>>>>>>> Twitter, wusheng1108 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 >>>>>>>>>> 下午5:27写道： >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build >>>> the >>>>>>>>> next >>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, >>>>>>> which >>>>>>>>> may >>>>>>>>>>>>> not >>>>>>>>>>>>>>>>> be a >>>>>>>>>>>>>>>>>>> high priority but a necessity. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very >>>>>>> important >>>>>>>>>> role >>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just >>> work >>>>>>>> well >>>>>>>>> to >>>>>>>>>>>>> verify >>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother >>> to >>>>>>>>> build >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> next >>>>>>>>>>>>>>>>>>> generation of E2E test framework? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many >>>>>>> sub-projects >>>>>>>>>> that >>>>>>>>>>>>> vary >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to >>>>>>> help >>>>>>>> us >>>>>>>>>>>>> verify >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the >>>> E2E >>>>>>>>> test >>>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t >>>> have >>>>>>>> E2E >>>>>>>>>>>>> tests at >>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is >>> unnecesarry >>>>>>>> and >>>>>>>>>>>>>>>>> introduces >>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The >>>>>>>> ultimate >>>>>>>>>>>> goal >>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> reuse all the test cases when needed. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running >>>>>>>>>>>> Java-based >>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers >>>>>>>>> (maintainers >>>>>>>>>>>> of >>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to >>>> set >>>>>>>> up >>>>>>>>>> the >>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is >>>> it >>>>>>>> an >>>>>>>>>>>> easy >>>>>>>>>>>>>>>>> skill >>>>>>>>>>>>>>>>>>> to debug the tests. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests >>>>>>> because >>>>>>>> we >>>>>>>>>>>>> added >>>>>>>>>>>>>>>>> many >>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified, >>> easy-to-use, >>>>>>>> and >>>>>>>>>>>> fast >>>>>>>>>>>>> E2E >>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use >>> docker-compose.yaml >>>>>>> to >>>>>>>>>>>>>>>>> orchestrate >>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be >>>>>>>> friendly >>>>>>>>>> and >>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we >>> are >>>>>>>> not >>>>>>>>>>>>> writing >>>>>>>>>>>>>>>>> E2E >>>>>>>>>>>>>>>>>>> tests. >>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI >>>>>>>> project >>>>>>>>> to >>>>>>>>>>>>> query >>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two >>>>>>>> versions >>>>>>>>>> of >>>>>>>>>>>>> query >>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes >>>> and >>>>>>>> one >>>>>>>>>> in >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> CLI). >>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data >>>>>>> and >>>>>>>>> the >>>>>>>>>>>>> actual >>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data >>>> yaml >>>>>>>>> file, >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable >>>>>>>>> command >>>>>>>>>>>> so >>>>>>>>>>>>>>>>> that it >>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven >>>>>>> knowledge. >>>>>>>> I >>>>>>>>>>>> hope >>>>>>>>>>>>> we >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and >>> do >>>>>>>>>>>>>>>>> verifications. >>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the >>>>>>>>>> `docker-compose.yaml`, >>>>>>>>>>>>>>>>> health >>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result. >>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions >>>> to >>>>>>>>> share >>>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>>> sub-projects. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement >>> to >>>>>>>> this >>>>>>>>>>>>>>>>> proposal, >>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these >>>>>>>> tasks >>>>>>>>>>>>> later. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code >>>>>>> 2020 >>>>>>>>>>>>> Projects) >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I >>>> said >>>>>>>>> this >>>>>>>>>>>> is >>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves >>>>>>>>> familiar >>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of >>>>>>>> SkyWalking >>>>>>>>>>>> work >>>>>>>>>>>>> as >>>>>>>>>>>>>>>>> well >>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and >>>> willingness >>>>>>>> to >>>>>>>>>>>> help. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang >>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ————————— >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ————————— >>>>>>>>>>>>> Zhenxu Ke (柯振旭) >>>>>>>>>>>>> GitHub @kezhenxu94 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> > > > > ————————— > Zhenxu Ke (柯振旭) > GitHub @kezhenxu94
