Hi all, Since many have asked I today started an effort to bring a comprehensive performance test suite for OpenWhisk out in the open.
Without much talking, here's the repository: https://github.com/markusthoemmes/openwhisk-performance My initial idea was to have the tests run on Travis to get comparable numbers. Turns out, the Travis machines are way to weak (especially in terms of CPU cores) to host a whole OpenWhisk system and be able to perform extraordinary good. The tests I did are based on the npm loadtest module, which is pretty much Apache Benchmark, with a few more features and nicer output. Currently there is only one test which surfaces end-to-end user latency of blocking invocations and maximum throughput. README.md has some more information on how exactly the tests are set up. I do realize that those tests are the simplest you can get, but that's where I'd like to start. We can get more sophisticated tests any time. To proceed from here, I'd like to have those tests run on beefier machines, to flesh out the real performance. Any ideas on how to do this in an open-source fashion, e.g. "out in the wild"? Cheers Markus
