kezhenxu94 commented on a change in pull request #172: URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545563202
########## File path: content/blog/e2e-design/index.md ########## @@ -0,0 +1,282 @@ +--- +title: "[Design] NGE2E - Next Generation End-to-End Testing Framework" +date: 2020-12-14 +author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer" +description: "The design of Next Generation End-to-End Testing Framework" +--- + +NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo. + +# Goal + +- Keep the feature parity with the existing E2E framework in SkyWalking main repo; +- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments; +- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes; + +# Non-Goal + +- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this; +- This project doesn't take the plugin tests into account, at least for now; +- This project doesn't mean to add/remove any new/existing test case to/from the main repo; +- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation; + +# Design + +Before diving into the design details, let's take a quick look at how the end user might use NGE2E. + +> All the following commands are mock, and are open to debate. + +To run a test case in a directory `/path/to/the/case/directory` + +```shell +swctl e2e run /path/to/the/case/directory + +# or + +cd /path/to/the/case/directory && swctl e2e run +``` + +This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case: + +**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line); + +### Set Up Services + +```shell +swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health +swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready Review comment: > Some questions: > > 1. How to sync images from the build process? Would we sync them after kind boot successfully? This can be done by the `--command` option, see line 145. > 2. Is one resource applied depends on its previous is ready? I don't think so, according to Kubernetes eventual consistency, is that necessary? > 3. We need a `label selector` to select the workloads we care about. Can you please elaborate more why we need this in a test case (Do the workloads matter)? > 4. Could we wait for deployment/statefulset/daemonset `available` status instead of the pod ready. `Available` means the readiness prod is successful, and pods are working continually for at least several seconds that are specificated by `MinReadySeconds` The `--wait=for=pod-ready` is only an example, there can be `--wait=for=service/ready`, `--wait=for=deployments/available`, `--wait=for=daemonset/available`, etc. > 5. Pls replace `resources` with `manifests` to follow k8s' convention. Will do. > 6. I failed to find any steps to setup skywalking components in this example. In that case, my preference is to depend on `swck` to provision them since they are all go based project. SkyWalking components should be set up via `--resource/manifests=` or `--command` IMO because how to set them up varies from case to case, you may want to set up via `swck`, but for sub-project like http://github.com/apache/skywalking-kubernetes/ , we may want to set them up via Helm (because the setup step itself is a test case!), or in `--env=docker-compose` case, they're set up via `docker-compose`. Does it make sense? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
