Hi Tao This is great, we are super interested. Let's track this effort with https://issues.apache.org/jira/browse/YUNIKORN-33. A couple of comments
1. There might be one thing missing is, how we can make this sustainable as "tests". which means we can 1) run this easily, 2) generate a result, and then 3) verify the result. Do we have #3 now? If not, we need to think about how to design a proper algorithm to verify if each scenario passes. 2. Another thing to add is the chaos monkey tests. This can be phase-2 but it might be good we can consider this now, to make sure our structure can support that. We can set up some meetings next week to discuss this in more detail. Thanks On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <ytaofight...@gmail.com> wrote: > Hello everyone > > We are planning to add integration testing framework and initial test cases > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts are > as follows. > > Basic testing framework includes: > 1. AppInfo struct: define basic information, requests and status of an > application > 2. AppManager interface and its implementations like DeploymentAppManager: > support useful operations like create/delete/refresh(status)/wait(for apps > to be satisfied or cleaned up) for testing. > 3. CommonConfig struct: keep several common configurations for testing like > schedulerName, outputRootPath, namespace, queue etc. > 4. Output tools like chart painter and file writer with specific format > (like csv). > > Initial test cases: > 1. Throughput : Request a certain number of pods via different schedulers > and then record the distributions of scheduled pods, draw results of > different schedulers on the same chart or write results into a file. > 2. Node Fairness: Request a certain number of pods via different schedulers > and then record the distributions of node usage, draw results on charts > separately for different schedulers and write results into a file. > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't support it > yet): Prepare queues with different capacities, request pods with different > number or resource for these queues, then record the usage of queues, draw > results on the same chart and write results into a file. > > Implementation: > 1. Package structure: add test/integration package in yunikorn-k8shim > project, the source files with structures or tools of testing framework > will be directly putted in this package, and test cases will be seperately > managed by case package in sub-directories: cases/<special-case> such as > cases/throughput, cases/node-fairness, cases/queue-fairness > 2. Configurations: cases keep their specific configurations themselves and > all configurations can be maintained in a single yaml file together with > common configurations, the config structure like this: > ``` > common: > schedulerName: yunikorn > maxWaitSeconds: 60 > queue: root.default > namespace: default > outputRootPath: /tmp > > cases: > throughput: > schedulerNames: > - yunikorn > - default-scheduelr > appName: throughput-test > numPods: 10000 > ... > node-fairness: > ... > queue-fairness: > ... > ``` > 3. Executions: all cases have main function themselves and need one > command-line argument: configPath, so that we can directly run specified > case and easily perform integration testing on different scenarios. > > Any comments and suggestions are welcome! > > Thanks, > Tao >
