Re: BDD Test Cases Contribution to complement existing test cases and coverage
Hi Rahul/all,
this is certainly an interesting approach to testing. Let me mention
some points.
The test structure is more flat - either all commands at the same level
or at most one/two calls to "features" in separate files. This gives the
user more insight into all details of the request/response. However, it
is questionable is the test itself is more readable than the Scala code.
The tests are a mix of Gherkin Feature file commands, embedded Java
snippets, and JavaScript as opposed to just Scala in the current test suite.
The tests in the Scala test suite are already written in the way:
"Component XY should do this and that" which is close to BDD.
I guess the most important point is whether OpenWhisk does BDD which is
really a style of development where behaviour is defined first as well
as tests with exact specification and coding follows. Does OpenWhisk
development work in this way or want to work this way? Then these tests
might be useful, especially for new features and test cases. I don't see
so much value in re-writing the existing tests because the community is
familiar with them and not sure how many people are familiar with the
Karate framework.
Perhaps other questions are:
* is there any code completion for the features files in some IDEs?
Especially for the embedded code snippets.
* do the tests run more quickly or slower than the current scala test suite?
Cheers,
Martin Gencur
QE, Red Hat
On 13.8.2018 12:00, Rahul Tripathi wrote:
Hi Markus,
I am yet to get the permission to add the proposal to the OW Wiki but though of
sharing it here:
What is Karate?
Karate is based on Cucumber which is a BDD framework.
What is BDD Testing?
BDD Testing is a testing approach based on Behavioural Driven Development. It
uses Given ,When ,Then statements to create test scenario's. Example:
TEST CASE-1
Feature: Get List of actions based on the NameSpace
Scenario: As a user I want to get the list of actions available for the given
namespace
* def path = '/api/v1/namespaces/'+nameSpace+'/actions?limit=30&skip=0'
Given url BaseUrl+path
And header Authorization = Auth
And header Content-Type = 'application/json'
When method get
Then status 200
And def json = response
The above example tests that the List Actions API's is returning a 200 ok . A lot more assertions can be put as per the requirements. Karate has some very good in built functions for
asserting the nested JSON's
TEST CASE-2(This example shows a simple smoke test on all the wsk user functions)
Feature: This feature file will test all the wsk functions
Background:
* configure ssl = true
* def nameSpace = 'guest'
* def params = '?blocking=true&result=false'
* def scriptcode = call
read('classpath:com/karate/openwhisk/functions/hello-world.js')
* def base64encoding =
read('classpath:com/karate/openwhisk/utils/base64.js')
Scenario: TC01-As a user I want to all the wsk functions available to the
user and check if they give the proper response
# Get User Auth
* def getNSCreds = call
read('classpath:com/karate/openwhisk/wskadmin/get-user.feature')
{nameSpace:'#(nameSpace)'}
* def result = getNSCreds.result
* def Auth = base64encoding(result)
* print "Got the Creds for the guest user"
* print Auth
# Create an Action .Create an action for the above defined guest name
#* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',
actionName:'Dammyyy'}
* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* print actionName
* print "Successfully Created an action"
# Get Action Details
* def actionDetails = call
read('classpath:com/karate/openwhisk/wskactions/get-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
* print "Successfully got the action details"
#Invoke Action
* def invokeAction = call
read('classpath:com/karate/openwhisk/wskactions/invoke-action.feature')
{params:'#(params)',requestBody:'',nameSpace:'#(nameSpace)'
,Auth:'#(Auth)',actionName:'#(actionName)'}
* def actID = invokeAction.activationId
* print = "Successfully invoked the action"
* def webhooks = callonce
read('classpath:com/karate/openwhisk/utils/sleep.feature') {sheepCount:&#
Further test suite improvements
Hello all, in an effort to improve the OpenWhisk test suite I'd like to propose some changes which I described in my comment on https://github.com/apache/incubator-openwhisk/issues/3737 (as a follow up on previous changes) This would be again an incremental improvement to the structure of the test suite and increase the test coverage of the system test suite itself. Any comments/suggestions are welcome. When there's an agreement on which changes to do I can go forward and implement them. Thanks in advance, Martin Gencur Quality Engineering, Red Hat
Re: Proposal on a future architecture of OpenWhisk
On 18.7.2018 14:41, Markus Thoemmes wrote: Hi Martin, thanks for the great questions :) thinking about scalability and the edge case. When there are not enough containers and new controllers are being created, and all of them redirect traffic to the controllers with containers, doesn't it mean overloading the available containers a lot? I'm curious how we throttle the traffic in this case. True, the first few requests will overload the controller that owns the very first container. That one will request new containers immediately, which will then be distributed to all existing Controllers by the ContainerManager. An interesting wrinkle here is, that you'd want the overloading requests to be completed by the Controllers that sent it to the "single-owning-Controller". Ah, got it. So it is a pretty common scenario. Scaling out controllers and containers. I thought this is a case where we reach a limit of created containers and no more containers can be created. What we could do here is: Controller0 owns ContainerA1 Controller1 relays requests for A to Controller0 Controller0 has more requests than it can handle, so it requests additional containers. All requests coming from Controller1 will be completed with a predefined message (for example "HTTP 503 overloaded" with a specific header say "X-Return-To-Sender-By: Controller0") Controller1 recognizes this as "okay, I'll wait for containers to appear", which will eventually happen (because Controller0 has already requested them) so it can route and complete those requests on its own. Controller1 will now no longer relay requests to Controller0 but will request containers itself (acknowledging that Controller0 is already overloaded). Yeah, I think it makes sense. I guess the other approach would be to block creating new controllers when there are no containers available as long as we don't want to overload the existing containers. And keep the overflowing workload in Kafka as well. Right, the second possibility is to use a pub/sub (not necessarily Kafka) queue between Controllers. Controller0 subscribes to a topic for action A because it owns a container for it. Controller1 doesn't own a container (yet) and publishes a message as overflow to topic A. The wrinkle in this case is, that Controller0 can't complete the request but needs to send it back to Controller1 (where the HTTP connection is opened from the client). Does that make sense? I was rather thinking about blocking the creation of Controller1 in this case and responding to the client that the system is overloaded. But the first approach seems better because it's a pretty common use case (not reaching the limit of created containers). Thanks! Martin Cheers, Markus
Re: Proposal on a future architecture of OpenWhisk
Hi Markus, thinking about scalability and the edge case. When there are not enough containers and new controllers are being created, and all of them redirect traffic to the controllers with containers, doesn't it mean overloading the available containers a lot? I'm curious how we throttle the traffic in this case. I guess the other approach would be to block creating new controllers when there are no containers available as long as we don't want to overload the existing containers. And keep the overflowing workload in Kafka as well. Thanks, Martin Gencur QE, Red Hat On 13.7.2018 19:29, Markus Thoemmes wrote: Hello OpenWhiskers, I just published a proposal on a potential future architecture for OpenWhisk that aligns deployments with and without an underlying container orchestrator like Mesos or Kubernetes. It also incooperates some of the proposals that are already out there and tries to give a holistic view of where we want OpenWhisk to go to in the near future. It's designed to keep the APIs stable but is very invasive in its changes under the hood. This proposal is the outcome of a lot of discussions with fellow colleagues and community members. It is based on experience with the problems the current architecture has. Moreover it aims to remove friction with the deployment topologies on top of a container orchestrator. Feedback is very very very welcome! The proposal has some gaps and generally does not go into much detail implementationwise. I'd love to see all those gaps filled by the community! Find the proposal here: https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture Cheers, Markus
