Hi Markus, that’s great! Love the approach.
To answer your question: a while back I did some digging if there are ASF projects that run such tests (performance, scalability, etc) in the open, but could not find anything. Basically, I was looking for an ASF-like way to procure infrastructure for a limited time for running distributed tests. What i have seen is: * running the tests locally – which gives a hint on some degradations/improvements, but does not take into account effects of distributed setups * companies (like my employer Adobe) running the tests on internal setups and then publishing the results (the test scripts themselves might still be open). That works in a way, but is not ideal. It works because someone is footing the bill for the necessary infrastructure. But it is not ideal because usually a only a limited number of people get to investigate the setups. The latter could be fixed, but is cumbersome organisationally. Maybe our mentors can chime in. Has this been discussed in the ASF board or so? AFAIK Microsoft sponsors the ASF with some Azure credentials. Could these also be used/extended for such purposes? @Isabelle: how does Mahout do this? Or any other Hadoop-area project you are aware of? Cheers Michael From: Markus Thömmes <markusthoem...@me.com<mailto:markusthoem...@me.com>> Reply-To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" <dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>> Date: Tuesday 25 April 2017 15:55 To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" <dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>> Subject: Performance tests for OpenWhisk Hi all, Since many have asked I today started an effort to bring a comprehensive performance test suite for OpenWhisk out in the open. Without much talking, here's the repository: https://github.com/markusthoemmes/openwhisk-performance My initial idea was to have the tests run on Travis to get comparable numbers. Turns out, the Travis machines are way to weak (especially in terms of CPU cores) to host a whole OpenWhisk system and be able to perform extraordinary good. The tests I did are based on the npm loadtest module, which is pretty much Apache Benchmark, with a few more features and nicer output. Currently there is only one test which surfaces end-to-end user latency of blocking invocations and maximum throughput. README.md has some more information on how exactly the tests are set up. I do realize that those tests are the simplest you can get, but that's where I'd like to start. We can get more sophisticated tests any time. To proceed from here, I'd like to have those tests run on beefier machines, to flesh out the real performance. Any ideas on how to do this in an open-source fashion, e.g. "out in the wild"? Cheers Markus
