[incubator-openwhisk-deploy-kube] branch master updated: fixes for apigateway; restructure deployment to eliminate cycles (#133)

[rabbah]

This is an automated email from the ASF dual-hosted git repository.

rabbah pushed a commit to branch master
in repository 
https://gitbox.apache.org/repos/asf/incubator-openwhisk-deploy-kube.git


The following commit(s) were added to refs/heads/master by this push:
 new d5dfa34  fixes for apigateway; restructure deployment to eliminate 
cycles (#133)
d5dfa34 is described below

commit d5dfa34dcd0e6bef4dd73dda3b920f7d070d05a9
Author: David Grove 
AuthorDate: Wed Jan 3 14:12:15 2018 -0500

fixes for apigateway; restructure deployment to eliminate cycles (#133)

* fixes for apigateway; restructure deployment to eliminate cycles

Several small fixes to get apigateway functionality working including
adding an ingress definition for apigateway.

Restructure deployment so that all services and ingresses get created
during initial cluster configuration.  This breaks circular
dependencies since it makes the service host/port and ingress
information available before any pods are deployed.

Closes #109.

* use PUBLIC_GATEWAY_URL for apigateway

Take advantage of new PUBLIC_GATEWAY_URL support added to
incubator-openwhisk-apigateway to get full control over the base URL
used by the apigateway.  Part of the fix for allowing both openwhisk
and apigateway to be exposed as services from a single host using URL
rewriting in a front-end proxy.
---
 README.md |  18 --
 kubernetes/apigateway/README.md   |   7 +++
 kubernetes/apigateway/apigateway.yml  |  26 ++---
 kubernetes/cluster-setup/README.md|   6 ++
 kubernetes/cluster-setup/services.yml | 100 ++
 kubernetes/controller/README.md   |   8 +--
 kubernetes/controller/controller.yml  |  16 --
 kubernetes/couchdb/README.md  |   5 +-
 kubernetes/couchdb/couchdb.yml|   2 +
 kubernetes/ingress/README.md  |  40 ++
 kubernetes/ingress/ingress-ibm.yml|   8 ++-
 kubernetes/ingress/ingress-simple.yml |   5 +-
 kubernetes/kafka/kafka.yml|  16 --
 kubernetes/nginx/nginx.yml|  23 
 kubernetes/zookeeper/zookeeper.yml|  22 
 tools/travis/build.sh |  38 +++--
 16 files changed, 200 insertions(+), 140 deletions(-)

diff --git a/README.md b/README.md
index ddf2c80..0dd241c 100644
--- a/README.md
+++ b/README.md
@@ -38,9 +38,17 @@ You can also provision a Kubernetes cluster from a cloud 
provider, subject to th
 
 # Configuring OpenWhisk
 
+The first time you deploy OpenWhisk on Kubernetes, we recommend
+following the steps below manually so you can inspect the results and
+debug your setup.  After you are confident that OpenWhisk deploys
+smoothly on your cluster, you might find it useful to drive your
+deployments using the script [build.sh](tools/travis/build.sh) that we
+use to deploy OpenWhisk on Kubernetes for our Travis CI testing.
+
 ## Initial Cluster Configuration
 
 * Follow the steps for initial [Cluster 
Setup](kubernetes/cluster-setup/README.md)
+* Configure your [Ingresses](kubernetes/ingress/README.md), including 
configuring the wsk CLI.
 
 ## Configure or Deploy CouchDB
 
@@ -62,11 +70,13 @@ directory tree. Follow the instructions for each step in 
order.
 * Deploy [Zookeeper](kubernetes/zookeeper/README.md)
 * Deploy [Kafka](kubernetes/kafka/README.md)
 * Deploy [Controller](kubernetes/controller/README.md)
-* Deploy [Nginx](kubernetes/nginx/README.md)
-* Deploy [Ingress](kubernetes/ingress/README.md), including configuring the 
wsk CLI.
 * Deploy [Invoker](kubernetes/invoker/README.md)
-* Deploy [RouteMgmt](kubernetes/routemgmt/README.md)
-* Deploy [Package Catalog](kubernetes/openwhisk-catalog/README.md)
+* Deploy [Nginx](kubernetes/nginx/README.md)
+
+## Install system actions and the openwhisk catalog
+
+* Install [RouteMgmt](kubernetes/routemgmt/README.md)
+* Install [Package Catalog](kubernetes/openwhisk-catalog/README.md)
 
 ## Verify
 
diff --git a/kubernetes/apigateway/README.md b/kubernetes/apigateway/README.md
index 91db8e2..672016a 100644
--- a/kubernetes/apigateway/README.md
+++ b/kubernetes/apigateway/README.md
@@ -8,3 +8,10 @@ To deploy the ApiGateway, you only need to run the following 
command:
 ```
 kubectl apply -f apigateway.yml
 ```
+
+Note: The URL returned from `wsk api create` may contain a spurious
+:8080 due to its assumption about the meaning of PUBLIC_MANAGEDURL_HOST.
+Working on a fix to the upstream incubator-openwhisk-apigateway project
+to weaken the assumption that the API URL is constructed by concatenating
+PUBLIC_MANAGEDURL_HOST:PUBLIC_MANAGEDURL_PORT as this is not always
+appropriate for kube-based deployments.
diff --git a/kubernetes/apigateway/apigateway.yml 
b/kubernetes/apigateway/apigateway.yml
index 6d1fa3c..98d8d3b 100644
--- a/kubernetes/apigateway/apigateway.yml
+++ b/kubernetes/apigateway/apigateway.yml
@@ 

[incubator-openwhisk] branch master updated: Optional package (#3128)

[dubeejw]

This is an automated email from the ASF dual-hosted git repository.

dubeejw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git


The following commit(s) were added to refs/heads/master by this push:
 new 4285cc2  Optional package (#3128)
4285cc2 is described below

commit 4285cc2791d56a486d0c52ca92b6f51abe3026d9
Author: Belinda Vennam 
AuthorDate: Wed Jan 3 10:18:04 2018 -0600

Optional package (#3128)

* adding variable for toggling optional packages
---
 ansible/tasks/installOpenwhiskCatalog.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ansible/tasks/installOpenwhiskCatalog.yml 
b/ansible/tasks/installOpenwhiskCatalog.yml
index 699fd7c..73c3cbe 100644
--- a/ansible/tasks/installOpenwhiskCatalog.yml
+++ b/ansible/tasks/installOpenwhiskCatalog.yml
@@ -18,6 +18,7 @@
 
 - set_fact:
 skip_catalog_install="{{ item.value.skip | default(false) }}"
+optional_package_skip="{{ item.value.optional_package_skip | default(true) 
}}"
 
 - name: "ensure catalog_location directory exists"
   file:
@@ -32,7 +33,7 @@
 version: "{{ version }}"
 
 - name: install the catalog from the catalog location
-  shell: ./installCatalog.sh {{ catalog_auth_key }} {{ api_host }} {{ cli.path 
}} chdir="{{ catalog_location }}/packages"
+  shell: ./installCatalog.sh {{ catalog_auth_key }} {{ api_host }} {{ cli.path 
}} chdir="{{ catalog_location }}/packages" "{{ optional_package_skip }}"
   environment:
 OPENWHISK_HOME: "{{ openwhisk_home }}"
   when: skip_catalog_install == false

-- 
To stop receiving notification emails like this one, please contact
['"commits@openwhisk.apache.org" '].

[incubator-openwhisk-client-go] branch master updated: Skip testing whether the HTTP body indicates a whisk failure result while command is 'wsk action invoke NNN --result' (#52)

[dubeejw]

This is an automated email from the ASF dual-hosted git repository.

dubeejw pushed a commit to branch master
in repository 
https://gitbox.apache.org/repos/asf/incubator-openwhisk-client-go.git


The following commit(s) were added to refs/heads/master by this push:
 new d3b5495  Skip testing whether the HTTP body indicates a whisk failure 
result while command is 'wsk action invoke NNN --result' (#52)
d3b5495 is described below

commit d3b5495baa386ebaeaeef46e4f18fdd85758dc14
Author: jiangpch 
AuthorDate: Wed Jan 3 22:32:27 2018 +0800

Skip testing whether the HTTP body indicates a whisk failure result while 
command is 'wsk action invoke NNN --result' (#52)

Closes: #51
---
 whisk/client.go | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/whisk/client.go b/whisk/client.go
index 8b978ae..b32af63 100644
--- a/whisk/client.go
+++ b/whisk/client.go
@@ -374,10 +374,13 @@ func (c *Client) Do(req *http.Request, v interface{}, 
ExitWithErrorOnTimeout boo
 // Handle 0. HTTP Success + Body indicating a whisk failure result
 //   NOTE: Need to ignore activation records send in response to 'wsk get 
activation NNN` as
 // these will report the same original error giving the appearance 
that the command failed.
+// Need to ignore `wsk action invoke NNN --result` too, otherwise 
action whose result is sth likes
+// '{"response": {"key": "value"}}' will return an error to such 
command.
 if (IsHttpRespSuccess(resp) &&  // 
HTTP Status == 200
 data!=nil &&// 
HTTP response body exists
 v != nil &&
 !strings.Contains(reflect.TypeOf(v).String(), "Activation") &&  // 
Request is not `wsk activation get`
+!(req.URL.Query().Get("result") == "true") &&   // 
Request is not `wsk action invoke NNN --result`
 !IsResponseResultSuccess(data)) {   // 
HTTP response body has Whisk error result
 Debug(DbgInfo, "Got successful HTTP; but activation response reports 
an error\n")
 return parseErrorResponse(resp, data, v)

-- 
To stop receiving notification emails like this one, please contact
['"commits@openwhisk.apache.org" '].

[incubator-openwhisk] branch master updated: Separate container removal from job completion. (#3132)

[dubeejw]

This is an automated email from the ASF dual-hosted git repository.

dubeejw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git


The following commit(s) were added to refs/heads/master by this push:
 new 94d5f3f  Separate container removal from job completion. (#3132)
94d5f3f is described below

commit 94d5f3f2a631ebf53d7718f8af8d6a20163fb89b
Author: rodric rabbah 
AuthorDate: Wed Jan 3 08:38:06 2018 -0500

Separate container removal from job completion. (#3132)

The current handshake between the container proxy and the container
pool conflates container removal and job completion. The former does
not necessarily indicate there is free capacity in the pool since
there are conditions under which the job is rescheduled (e.g., resume
failure, container aging, container is reclaimed).
---
 .../whisk/core/containerpool/ContainerPool.scala | 17 +
 .../whisk/core/containerpool/ContainerProxy.scala| 20 +++-
 .../core/containerpool/test/ContainerPoolTests.scala | 14 ++
 .../containerpool/test/ContainerProxyTests.scala |  6 +++---
 4 files changed, 49 insertions(+), 8 deletions(-)

diff --git 
a/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.scala 
b/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.scala
index 1fa6b3a..61172dd 100644
--- a/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.scala
+++ b/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.scala
@@ -81,6 +81,10 @@ class ContainerPool(childFactory: ActorRefFactory => 
ActorRef,
 
   def receive: Receive = {
 // A job to run on a container
+//
+// Run messages are received either via the feed or from child containers 
which cannot process
+// their requests and send them back to the pool for rescheduling (this 
may happen if "docker" operations
+// fail for example, or a container has aged and was destroying itself 
when a new request was assigned)
 case r: Run =>
   val container = if (busyPool.size < maxActiveContainers) {
 // Schedule a job to a warm container
@@ -110,6 +114,9 @@ class ContainerPool(childFactory: ActorRefFactory => 
ActorRef,
   freePool = freePool - actor
   actor ! r // forwards the run request to the container
 case None =>
+  // this can also happen if createContainer fails to start a new 
container, or
+  // if a job is rescheduled but the container it was allocated to has 
not yet destroyed itself
+  // (and a new container would over commit the pool)
   logging.error(this, "Rescheduling Run message, too many message in 
the pool")(r.msg.transid)
   self ! r
   }
@@ -131,8 +138,18 @@ class ContainerPool(childFactory: ActorRefFactory => 
ActorRef,
   freePool = freePool - sender()
   busyPool.get(sender()).foreach { _ =>
 busyPool = busyPool - sender()
+// container was busy, so there is capacity to accept another job 
request
 feed ! MessageFeed.Processed
   }
+
+// This message is received for one of these reasons:
+// 1. Container errored while resuming a warm container, could not process 
the job, and sent the job back
+// 2. The container aged, is destroying itself, and was assigned a job 
which it had to send back
+// 3. The container aged and is destroying itself
+// Update the free/busy lists but no message is sent to the feed since 
there is no change in capacity yet
+case RescheduleJob =>
+  freePool = freePool - sender()
+  busyPool = busyPool - sender()
   }
 
   /** Creates a new container and updates state accordingly. */
diff --git 
a/core/invoker/src/main/scala/whisk/core/containerpool/ContainerProxy.scala 
b/core/invoker/src/main/scala/whisk/core/containerpool/ContainerProxy.scala
index dd4336e..c3f89c5 100644
--- a/core/invoker/src/main/scala/whisk/core/containerpool/ContainerProxy.scala
+++ b/core/invoker/src/main/scala/whisk/core/containerpool/ContainerProxy.scala
@@ -67,7 +67,8 @@ case object Remove
 // Events sent by the actor
 case class NeedWork(data: ContainerData)
 case object ContainerPaused
-case object ContainerRemoved
+case object ContainerRemoved // when container is destroyed
+case object RescheduleJob // job is sent back to parent and could not be 
processed because container is being destroyed
 
 /**
  * A proxy that wraps a Container. It is used to keep track of the lifecycle
@@ -101,6 +102,7 @@ class ContainerProxy(
 with Stash {
   implicit val ec = context.system.dispatcher
   implicit val logging = new AkkaLogging(context.system.log)
+  var rescheduleJob = false // true iff actor receives a job but cannot 
process it because actor will destroy itself
 
   startWith(Uninitialized, NoData())
 
@@ -248,7 +250,9 @@ class ContainerProxy(
   // Sending the message to self on a failure will 

[incubator-openwhisk] branch master updated: Refine error message on failure to fetch action in invoker. (#3139)

[dubeejw]

This is an automated email from the ASF dual-hosted git repository.

dubeejw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git


The following commit(s) were added to refs/heads/master by this push:
 new eb20606  Refine error message on failure to fetch action in invoker. 
(#3139)
eb20606 is described below

commit eb206069b88dd1e103f7b9fbf809820afc8d2943
Author: rodric rabbah 
AuthorDate: Wed Jan 3 08:35:23 2018 -0500

Refine error message on failure to fetch action in invoker. (#3139)
---
 common/scala/src/main/scala/whisk/http/ErrorResponse.scala |  1 +
 .../src/main/scala/whisk/core/invoker/InvokerReactive.scala| 10 +++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/common/scala/src/main/scala/whisk/http/ErrorResponse.scala 
b/common/scala/src/main/scala/whisk/http/ErrorResponse.scala
index 9ce5c45..8dcfe20 100644
--- a/common/scala/src/main/scala/whisk/http/ErrorResponse.scala
+++ b/common/scala/src/main/scala/whisk/http/ErrorResponse.scala
@@ -187,6 +187,7 @@ object Messages {
 
   val actionRemovedWhileInvoking = "Action could not be found or may have been 
deleted."
   val actionMismatchWhileInvoking = "Action version is not compatible and 
cannot be invoked."
+  val actionFetchErrorWhileInvoking = "Action could not be fetched."
 }
 
 /** Replaces rejections with Json object containing cause and transaction id. 
*/
diff --git 
a/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala 
b/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
index 1a772b9..c885c85 100644
--- a/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
+++ b/core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
@@ -47,7 +47,7 @@ import whisk.core.containerpool.ContainerProxy
 import whisk.core.containerpool.PrewarmingConfig
 import whisk.core.containerpool.Run
 import whisk.core.containerpool.logging.LogStoreProvider
-import whisk.core.database.NoDocumentException
+import whisk.core.database._
 import whisk.core.entity._
 import whisk.core.entity.size._
 import whisk.http.Messages
@@ -201,8 +201,12 @@ class InvokerReactive(config: WhiskConfig, instance: 
InstanceId, producer: Messa
   // making this an application error. All other errors are 
considered system
   // errors and should cause the invoker to be considered 
unhealthy.
   val response = t match {
-case _: NoDocumentException => 
ActivationResponse.applicationError(Messages.actionRemovedWhileInvoking)
-case _  => 
ActivationResponse.whiskError(Messages.actionMismatchWhileInvoking)
+case _: NoDocumentException =>
+  
ActivationResponse.applicationError(Messages.actionRemovedWhileInvoking)
+case _: DocumentTypeMismatchException | _: DocumentUnreadable 
=>
+  
ActivationResponse.whiskError(Messages.actionMismatchWhileInvoking)
+case _ =>
+  
ActivationResponse.whiskError(Messages.actionFetchErrorWhileInvoking)
   }
   val now = Instant.now
   val causedBy = if (msg.causedBySequence) {

-- 
To stop receiving notification emails like this one, please contact
['"commits@openwhisk.apache.org" '].

[incubator-openwhisk] branch master updated: Enforce WhiskAction equality mod DocRevision. (#3121)

[dubeejw]

This is an automated email from the ASF dual-hosted git repository.

dubeejw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git


The following commit(s) were added to refs/heads/master by this push:
 new 6de2694  Enforce WhiskAction equality mod DocRevision. (#3121)
6de2694 is described below

commit 6de269440b8ccc20bb1e21de0f81e8a727921d56
Author: rodric rabbah 
AuthorDate: Wed Jan 3 08:24:00 2018 -0500

Enforce WhiskAction equality mod DocRevision. (#3121)
---
 .../main/scala/whisk/core/connector/Message.scala  |  9 -
 .../main/scala/whisk/core/entity/WhiskAction.scala |  5 +
 .../containerpool/test/ContainerPoolTests.scala| 15 ++
 .../scala/whisk/core/entity/test/SchemaTests.scala | 23 --
 4 files changed, 41 insertions(+), 11 deletions(-)

diff --git a/common/scala/src/main/scala/whisk/core/connector/Message.scala 
b/common/scala/src/main/scala/whisk/core/connector/Message.scala
index c7cff9f..d79ba26 100644
--- a/common/scala/src/main/scala/whisk/core/connector/Message.scala
+++ b/common/scala/src/main/scala/whisk/core/connector/Message.scala
@@ -58,15 +58,6 @@ case class ActivationMessage(override val transid: 
TransactionId,
  cause: Option[ActivationId] = None)
 extends Message {
 
-  def meta =
-JsObject("meta" -> {
-  cause map { c =>
-JsObject(c.toJsObject.fields ++ activationId.toJsObject.fields)
-  } getOrElse {
-activationId.toJsObject
-  }
-})
-
   override def serialize = ActivationMessage.serdes.write(this).compactPrint
 
   override def toString = {
diff --git a/common/scala/src/main/scala/whisk/core/entity/WhiskAction.scala 
b/common/scala/src/main/scala/whisk/core/entity/WhiskAction.scala
index afd85cc..4aaddbd 100644
--- a/common/scala/src/main/scala/whisk/core/entity/WhiskAction.scala
+++ b/common/scala/src/main/scala/whisk/core/entity/WhiskAction.scala
@@ -216,6 +216,11 @@ case class WhiskActionMetaData(namespace: EntityPath,
  *
  * exec is typed to CodeExec to guarantee executability by an Invoker.
  *
+ * Note: Two actions are equal regardless of their DocRevision if there is one.
+ * The invoker uses action equality when matching actions to warm containers.
+ * That means creating an action, invoking it, then 
deleting/recreating/reinvoking
+ * it will reuse the previous container. The delete/recreate restores the 
SemVer to 0.0.1.
+ *
  * @param namespace the namespace for the action
  * @param name the name of the action
  * @param exec the action executable details
diff --git 
a/tests/src/test/scala/whisk/core/containerpool/test/ContainerPoolTests.scala 
b/tests/src/test/scala/whisk/core/containerpool/test/ContainerPoolTests.scala
index a607f5b..dcfe5e4 100644
--- 
a/tests/src/test/scala/whisk/core/containerpool/test/ContainerPoolTests.scala
+++ 
b/tests/src/test/scala/whisk/core/containerpool/test/ContainerPoolTests.scala
@@ -90,6 +90,7 @@ class ContainerPoolTests
 
   val runMessage = createRunMessage(action, invocationNamespace)
   val runMessageDifferentAction = createRunMessage(differentAction, 
invocationNamespace)
+  val runMessageDifferentVersion = 
createRunMessage(action.copy().revision(DocRevision("v2")), invocationNamespace)
   val runMessageDifferentNamespace = createRunMessage(action, 
differentInvocationNamespace)
   val runMessageDifferentEverything = createRunMessage(differentAction, 
differentInvocationNamespace)
 
@@ -133,6 +134,20 @@ class ContainerPoolTests
 containers(1).expectNoMsg(100.milliseconds)
   }
 
+  it should "reuse a warm container when action is the same even if revision 
changes" in within(timeout) {
+val (containers, factory) = testContainers(2)
+val feed = TestProbe()
+val pool = system.actorOf(ContainerPool.props(factory, 2, 2, feed.ref))
+
+pool ! runMessage
+containers(0).expectMsg(runMessage)
+containers(0).send(pool, NeedWork(warmedData()))
+
+pool ! runMessageDifferentVersion
+containers(0).expectMsg(runMessageDifferentVersion)
+containers(1).expectNoMsg(100.milliseconds)
+  }
+
   it should "create a container if it cannot find a matching container" in 
within(timeout) {
 val (containers, factory) = testContainers(2)
 val feed = TestProbe()
diff --git a/tests/src/test/scala/whisk/core/entity/test/SchemaTests.scala 
b/tests/src/test/scala/whisk/core/entity/test/SchemaTests.scala
index dd3fa04..504db38 100644
--- a/tests/src/test/scala/whisk/core/entity/test/SchemaTests.scala
+++ b/tests/src/test/scala/whisk/core/entity/test/SchemaTests.scala
@@ -26,17 +26,16 @@ import scala.language.postfixOps
 import scala.language.reflectiveCalls
 import scala.util.Failure
 import scala.util.Try
-
 import org.junit.runner.RunWith
 import org.scalatest.BeforeAndAfter
 import org.scalatest.FlatSpec
 import org.scalatest.Matchers
 import org.scalatest.junit.JUnitRunner
-
 import spray.json._
 import