This is an automated email from the ASF dual-hosted git repository. markusthoemmes pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git
The following commit(s) were added to refs/heads/master by this push: new 56891fb Add gatling as performance test suite. (#3526) 56891fb is described below commit 56891fbeb1ca306b4685396f13f07cb4b60a3022 Author: Christian Bickel <git...@cbickel.de> AuthorDate: Fri Apr 13 10:16:38 2018 +0200 Add gatling as performance test suite. (#3526) Adds an initial gatling based performance test, including a framework to talk to the OpenWhisk API in a declarative way. This first implementation only asserts /api/v1 performance, which can be helpful to measure general network/router performance. Co-authored-by: Markus Thömmes <markusthoem...@me.com> --- .travis.yml | 1 + performance/README.md | 44 +++++++++++--- performance/gatling_tests/build.gradle | 19 ++++++ .../src/gatling/resources/conf/logback.xml | 14 +++++ .../src/gatling/scala/ApiV1Simulation.scala | 51 ++++++++++++++++ .../extension/whisk/OpenWhiskActionBuilder.scala | 38 ++++++++++++ .../scala/extension/whisk/OpenWhiskDsl.scala | 26 +++++++++ .../extension/whisk/OpenWhiskProtocolBuilder.scala | 67 ++++++++++++++++++++++ .../src/gatling/scala/extension/whisk/Predef.scala | 20 +++++++ performance/preparation/deploy.sh | 1 + settings.gradle | 1 + 11 files changed, 275 insertions(+), 7 deletions(-) diff --git a/.travis.yml b/.travis.yml index 52d6109..7fc1587 100644 --- a/.travis.yml +++ b/.travis.yml @@ -64,5 +64,6 @@ jobs: - ./performance/preparation/deploy.sh - TERM=dumb ./performance/wrk_tests/latency.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" 2m - TERM=dumb ./performance/wrk_tests/throughput.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" 4 2 2m + - OPENWHISK_HOST="172.17.0.1" CONNECTIONS="100" REQUESTS_PER_SEC="1" ./gradlew gatlingRun-ApiV1Simulation env: - DESCRIPTION="Execute wrk-performance test suite." diff --git a/performance/README.md b/performance/README.md index c1d668f..f089021 100644 --- a/performance/README.md +++ b/performance/README.md @@ -2,7 +2,7 @@ A few simple but efficient test suites for determining the maximum throughput and end-user latency of the Apache OpenWhisk system. ## Workflow -- A standard OpenWhisk system is deployed. (_Note that the edge NGINX router and API Gateway are currently left out. As a consequence, the tests talk directly to the controller._) +- A standard OpenWhisk system is deployed. (_Note that the API Gateway is currently left out for the tests._) - All limits are set to 999999, which in our current use case means "No throttling at all". - The deployment is using the docker setup proposed by the OpenWhisk development team: `overlay` driver and HTTP API enabled via a UNIX port. @@ -13,8 +13,10 @@ The [machine provided by Travis](https://docs.travis-ci.com/user/ci-environment/ ## Suites -### Latency Test -Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system. +### wrk + +#### Latency Test +Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system. The requests are directly against the controller. - 1 HTTP request at a time (concurrency: 1) - You can specify how long this test will run. Default are 30s. @@ -22,12 +24,40 @@ Determines the end-to-end latency a user experience when doing a blocking invoca **Note:** The throughput number has a 100% correlation with the latency in this case. This test does not serve to determine the maximum throughput of the system. -### Throughput Test -Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation. +#### Throughput Test +Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation. The requests are directly against the controller. - 4 HTTP requests at a time (concurrency: 4) (using CPU cores * 2 to exploit some buffering) - 10.000 samples with a single user - no-op action -## Running tests against your own system is simple too! -All you have to do is use the corresponding script located in /*_tests folder, remembering that the parameters are defined inline. +#### Running tests against your own system is simple too! +All you have to do is use the corresponding script located in `/*_tests` folder, remembering that the parameters are defined inline. + +### gatling + +#### Simulations + +You can specify two thresholds for the simulations. +The reason is, that Gatling is able to handle each assertion as a JUnit test. +On using CI/CD pipelines (e.g. Jenkins) you will be able to set a threshold on an amount of failed testcases to mark the build as stable, unstable and failed. + +##### ApiV1Simulation + +This Simulation calls the `api/v1`. +You can specify the endpoint, the amount of connections against the backend and the duration of this burst. + +Available environment variables: + +``` +OPENWHISK_HOST (required) +CONNECTIONS (required) +SECONDS (default: 10) +REQUESTS_PER_SEC (required) +MIN_REQUESTS_PER_SEC (default: REQUESTS_PER_SEC) +``` + +You can run the simulation with (in OPENWHISK_HOME) +``` +OPENWHISK_HOST="openwhisk.mydomain.com" CONNECTIONS="10" REQUESTS_PER_SEC="50" ./gradlew gatlingRun-ApiV1Simulation +``` diff --git a/performance/gatling_tests/build.gradle b/performance/gatling_tests/build.gradle new file mode 100644 index 0000000..7e8f1ad --- /dev/null +++ b/performance/gatling_tests/build.gradle @@ -0,0 +1,19 @@ +plugins { + id "com.github.lkishalmi.gatling" version "0.7.1" +} + +apply plugin: 'eclipse' +apply plugin: 'scala' + +repositories { + mavenCentral() +} + +dependencies { + compile "org.scala-lang:scala-library:${gradle.scala.version}" + compile "io.gatling.highcharts:gatling-charts-highcharts:2.2.5" +} + +tasks.withType(ScalaCompile) { + scalaCompileOptions.additionalParameters = gradle.scala.compileFlags +} diff --git a/performance/gatling_tests/src/gatling/resources/conf/logback.xml b/performance/gatling_tests/src/gatling/resources/conf/logback.xml new file mode 100644 index 0000000..62bd2b2 --- /dev/null +++ b/performance/gatling_tests/src/gatling/resources/conf/logback.xml @@ -0,0 +1,14 @@ +<?xml version="1.0" encoding="UTF-8"?> +<configuration> + <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> + <encoder> + <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern> + <immediateFlush>false</immediateFlush> + </encoder> + </appender> + <!-- This will log the details of the failed requests --> + <logger name="io.gatling.http.ahc" level="DEBUG" /> + <root level="WARN"> + <appender-ref ref="CONSOLE"/> + </root> +</configuration> diff --git a/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala b/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala new file mode 100644 index 0000000..91c5302 --- /dev/null +++ b/performance/gatling_tests/src/gatling/scala/ApiV1Simulation.scala @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import extension.whisk.Predef._ +import io.gatling.core.Predef._ + +import scala.concurrent.duration._ + +class ApiV1Simulation extends Simulation { + + // Specify parameters for the run + val host = sys.env("OPENWHISK_HOST") + val connections = sys.env("CONNECTIONS").toInt + val seconds = sys.env.getOrElse("SECONDS", "10").toInt.seconds + + // Specify thresholds + val requestsPerSec = sys.env("REQUESTS_PER_SEC").toInt + val minimalRequestsPerSec = sys.env.getOrElse("MIN_REQUESTS_PER_SEC", requestsPerSec.toString).toInt + + // Generate the OpenWhiskProtocol + val openWhiskProtocol = openWhisk.apiHost(host) + + // Define scenario + val test = scenario("api/v1 endpoint") + .during(seconds) { + exec(openWhisk("Call api/v1 endpoint").info()) + } + + setUp(test.inject(atOnceUsers(connections))) + .protocols(openWhiskProtocol) + // One failure will make the build yellow + .assertions(details("Call api/v1 endpoint").requestsPerSec.gt(minimalRequestsPerSec)) + .assertions(details("Call api/v1 endpoint").requestsPerSec.gt(requestsPerSec)) + // Mark the build yellow, if there are failed requests. And red if both conditions fail. + .assertions(details("Call api/v1 endpoint").failedRequests.count.is(0)) + .assertions(details("Call api/v1 endpoint").failedRequests.percent.lte(0.1)) +} diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala new file mode 100644 index 0000000..ef6d2ef --- /dev/null +++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskActionBuilder.scala @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package extension.whisk + +import io.gatling.core.Predef._ +import io.gatling.core.action.Action +import io.gatling.core.action.builder.ActionBuilder +import io.gatling.core.session.Expression +import io.gatling.core.structure.ScenarioContext +import io.gatling.http.request.builder.{Http, HttpRequestBuilder} + +case class OpenWhiskActionBuilderBase(requestName: Expression[String]) { + + implicit private val http = new Http(requestName) + + def info() = OpenWhiskActionBuilder(http.get("/api/v1")) +} + +case class OpenWhiskActionBuilder(http: HttpRequestBuilder) extends ActionBuilder { + override def build(ctx: ScenarioContext, next: Action): Action = { + http.build(ctx, next) + } +} diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala new file mode 100644 index 0000000..ea51847 --- /dev/null +++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskDsl.scala @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package extension.whisk + +import io.gatling.core.session.Expression + +trait OpenWhiskDsl { + def openWhisk = OpenWhiskProtocolBuilderBase + + def openWhisk(requestName: Expression[String]) = new OpenWhiskActionBuilderBase(requestName) +} diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala new file mode 100644 index 0000000..fe499cf --- /dev/null +++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/OpenWhiskProtocolBuilder.scala @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package extension.whisk + +import com.softwaremill.quicklens._ +import io.gatling.core.Predef._ +import io.gatling.core.config.GatlingConfiguration +import io.gatling.http.Predef._ +import io.gatling.http.protocol.HttpProtocol + +import scala.language.implicitConversions + +/** + * This is the OpenWhiskProtocol. + * + * @param apiHost url or address to connect to + * @param protocol protocol to use. e.g. https + * @param port port to use. e.g. 443 + */ +case class OpenWhiskProtocol(apiHost: String, protocol: String = "https", port: Int = 443) + +case object OpenWhiskProtocolBuilderBase { + def apiHost(url: String): OpenWhiskProtocolBuilder = OpenWhiskProtocolBuilder(OpenWhiskProtocol(url)) +} + +object OpenWhiskProtocolBuilder { + + /** convert the OpenWhiskProtocolBuilder to an HttpProtocol. */ + implicit def toHttpProtocol(builder: OpenWhiskProtocolBuilder)( + implicit configuration: GatlingConfiguration): HttpProtocol = builder.build +} + +case class OpenWhiskProtocolBuilder(private val protocol: OpenWhiskProtocol) { + + /** set the api host */ + def apiHost(url: String): OpenWhiskProtocolBuilder = this.modify(_.protocol.apiHost).setTo(url) + + /** set the protocol */ + def protocol(protocol: String): OpenWhiskProtocolBuilder = this.modify(_.protocol.protocol).setTo(protocol) + + /** set the port */ + def port(port: Int): OpenWhiskProtocolBuilder = this.modify(_.protocol.port).setTo(port) + + /** build the http protocol with the parameters provided by the openwhisk-protocol. */ + def build(implicit configuration: GatlingConfiguration) = { + http + .baseURL(s"${protocol.protocol}://${protocol.apiHost}:${protocol.port}") + .contentTypeHeader("application/json") + .userAgentHeader("gatlingLoadTest") + .build + } +} diff --git a/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala b/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala new file mode 100644 index 0000000..5733f7d --- /dev/null +++ b/performance/gatling_tests/src/gatling/scala/extension/whisk/Predef.scala @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package extension.whisk + +object Predef extends OpenWhiskDsl diff --git a/performance/preparation/deploy.sh b/performance/preparation/deploy.sh index 0c9bfbe..8a11fd6 100755 --- a/performance/preparation/deploy.sh +++ b/performance/preparation/deploy.sh @@ -21,3 +21,4 @@ $ANSIBLE_CMD wipe.yml $ANSIBLE_CMD kafka.yml $ANSIBLE_CMD controller.yml $ANSIBLE_CMD invoker.yml +$ANSIBLE_CMD edge.yml diff --git a/settings.gradle b/settings.gradle index 4c76198..d3f56e2 100644 --- a/settings.gradle +++ b/settings.gradle @@ -17,6 +17,7 @@ include 'sdk:docker' include 'tests' include 'tests:dat:blackbox:badaction' include 'tests:dat:blackbox:badproxy' +include 'performance:gatling_tests' rootProject.name = 'openwhisk' -- To stop receiving notification emails like this one, please contact markusthoem...@apache.org.