The tests with wrk are comprised of 6 different tests. The first four utilize fewer concurrencies while the other two utilize 100 concurrencies. After changing the pause grace, it is more resilient with lower concurrency tests but still be failed with higher concurrency.
One strange thing is I opened a new PR copying the original one, and it passes the tests. https://github.com/apache/openwhisk/pull/5189 No matter how many I tried the original PR did not pass the tests, but with the new one with the same change, it passed the tests. It however does not mean the performance test is working as expected, there are still failed activations. It just does not go into an unresponsive state. I left a query to the Travis team asking if there are any differences between the two. New PRs might pass the tests and we can disable tests with higher concurrency only for now. But I feel eventually it's better to run them in another environment as tests are actually being failed. I confirmed the VM(openwhisk-vm1-he-de.apache.org) we have has 8 cores with 32GB memory. But it seems it is being used as a Jenkins slave. I am not quite sure if we can just use it or should we ask for another VM. -dom 2022년 1월 5일 (수) 오전 5:53, Davi Ribeiro <daviantune...@gmail.com>님이 작성: > hi, > > What you think about reduce the load in performance test in wrk to see what > happens? > > Davi Ribeiro > > > > Em seg., 3 de jan. de 2022 às 21:26, Dominic Kim <style9...@gmail.com> > escreveu: > > > Dear whiskers. > > > > I've found the performance tests have not been working so far. > > According to the following test, it is shown it passed the test but > > actually, there are some activations with other than 2xx,3xx response. > > https://app.travis-ci.com/github/apache/openwhisk/jobs/479539830 > > > > Even with some wrong results, there was no issue running the subsequent > > Gatling tests so far. > > But now, the system becomes unavailable after running tests with `wrk`. > > > > I got inside the VM and observed status of runtime containers kept > changing > > between running and paused. > > I suspect this is because the VM has too little resources to run all OW > > components and the wrk client so it could not generate/forward > activations > > to the invoker in time. > > And such Docker workloads made the Docker engine become abnormal. > > I have no idea what has changed on the Travis VM side yet, but at least, > I > > found the docker version is changed. > > > > I am not sure it is effective to run performance tests on a small VM > with 2 > > cores and 8GB memory. > > Now it is rather an obstacle to merging new PRs. > > > > I propose temporarily disabling the performance tests until we find a > > better way. > > Maybe we can utilize a bigger VM provided by ASF. > > > > Please share your opinion. > > > > Thanks > > Regards > > Dominic > > > > Reference: > > https://github.com/apache/openwhisk/pull/5158#issuecomment-1003900935 > > >
