2020-02-06 00:13:55 UTC - Tom Barber: well yeah there is an issue here because it deployed but then says the virtualenv is missing stuff https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948035140800?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:14:03 UTC - Tom Barber: they aren't "large", 12mb https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948043141000?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:14:11 UTC - Tom Barber: but something is dying on the EKS side and kiling it https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948051141200?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:14:44 UTC - Tom Barber: I also made the EKS nodes larger earlier in case that had an effect but doesn't seem to have done the job https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948084141400?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:17:13 UTC - Tom Barber: this evening I've got stuff like:
``` Failed to deploy function (pixlize-serverless-dev-saveSession) due to error: Unknown Error From API: write EPIPE``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948233141600?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:22:03 UTC - Tom Barber: ah ha: ```export AWS_CLIENT_TIMEOUT=300000 sls deploy``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948523141800?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:22:26 UTC - Tom Barber: worked that time around, might be the solution I'm looking for, I'll deploy a few times and see if it holds up https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580948546142100?thread_ts=1580917776.131400&cid=C3TPCAQG1 ---- 2020-02-06 00:47:33 UTC - Kostis Kaffes: I was able to solve this issue by adding new ansible rules that: 1. Store the container image to a file 2. Copy the file to the remote host 3. Recreate the container image from the file in the remote host https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580950053142300?thread_ts=1580944629.140600&cid=C3TPCAQG1 ---- 2020-02-06 02:57:39 UTC - Saahitya E: Hey all.~What should the "metrics_kamon_statsd_host" property be set to in the ansible setup when using the integrated statsd+grafana dashboard docker image to test whether metrics are being sent. Thanks in advance.~ Got it working. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580957859145300 ---- 2020-02-06 06:03:51 UTC - Nikhil: Hi guys, I am trying to use standalone controller on mac for local development and while I was able to install my actions and use curl to test it, I am unable to invoke and post to other actions from my main action. Note that this works in cloud deployment. When I look at my activation list, I just see 1 entry while it supposed to be 3. I started my standalone controller with `bash <(curl -sL <https://s.apache.org/openwhisk.sh>)` and then when it didn't work, I started the controller as a java process. I did not see any controller logs w.r.t to the second action invocation. Probably an issue with docker networking? I did a nslookup of the `action-nodejs-v10:nightly` container and this is the output ```$ docker exec -it d362932da681 nslookup host.docker.internal Server: 192.168.65.1 Address: 192.168.65.1#53 Non-authoritative answer: Name: host.docker.internal Address: 192.168.65.2``` I replaced host.docker.internal with `192.168.65.2` and that didn't help either. This is my docker version. ```$ docker --version Docker version 19.03.5, build 633a0ea``` When I login to the nodejs container and run curl manually, I am able to invoke the action. Any ideas as to how to debug this further? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580969031157800?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 08:46:56 UTC - Rodric Rabbah: @chetanm is it possible to override the api host in the config file? There is a helper function to read a host property. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580978816159600?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 10:57:50 UTC - Justin Halsall: Does anyone know what the openwhisk local disk space limit is? I saw it in a presentation once and forgot it :confused: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580986670161000?thread_ts=1580986670.161000&cid=C3TPCAQG1 ---- 2020-02-06 11:02:20 UTC - Markus Thömmes: none I think https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580986940161100?thread_ts=1580986670.161000&cid=C3TPCAQG1 ---- 2020-02-06 11:04:58 UTC - Justin Halsall: ooowh nice :slightly_smiling_face: gonna be splitting some videos into frames and sending that through to object store in that case! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580987098161300?thread_ts=1580986670.161000&cid=C3TPCAQG1 ---- 2020-02-06 11:05:20 UTC - Justin Halsall: Thanks Markus! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580987120161500?thread_ts=1580986670.161000&cid=C3TPCAQG1 ---- 2020-02-06 11:52:57 UTC - Tom Barber: anyone know how to put a signed cert on the end of the helm managed API? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580989977162100 ---- 2020-02-06 11:53:11 UTC - Tom Barber: ```whisk: ingress: type: LoadBalancer annotations: <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: 0.0.0.0/0 <http://service.beta.kubernetes.io/aws-load-balancer-ssl-cert|service.beta.kubernetes.io/aws-load-balancer-ssl-cert>: arn:aws:acm:us-east-1:963058736014:certificate/3e513c96-b9c6-4d2f-a72f-63f3435e81de``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580989991162400 ---- 2020-02-06 11:53:46 UTC - Tom Barber: I updated the load balancer ssl cert with the arn on the official one and tore the whole thing down but it still reports the self signed one https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990026163200 ---- 2020-02-06 11:55:16 UTC - Tom Barber: but I also don't spot those annotations in a service anywhere https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990116163600 ---- 2020-02-06 11:56:47 UTC - Tom Barber: *Issuer:* *.owdev-nginx.openwhisk.svc.cluster.local https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990207163800 ---- 2020-02-06 11:58:05 UTC - Tom Barber: ```apiVersion: v1 kind: Service metadata: creationTimestamp: "2020-02-06T11:30:20Z" labels: app: owdev-openwhisk chart: openwhisk-0.1.10 heritage: Tiller name: owdev-nginx release: owdev name: owdev-nginx namespace: openwhisk resourceVersion: "132770" selfLink: /api/v1/namespaces/openwhisk/services/owdev-nginx uid: 0ebb4c65-48d4-11ea-bbc7-12b20bb5dcf3 spec: clusterIP: 10.100.80.190 externalTrafficPolicy: Cluster ports: - name: http nodePort: 31234 port: 80 protocol: TCP targetPort: 80 - name: https-api nodePort: 31068 port: 443 protocol: TCP targetPort: 443 selector: name: owdev-nginx sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - hostname: a<>-<http://1548559070.us-east-1.elb.amazonaws.com|1548559070.us-east-1.elb.amazonaws.com>``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990285164200 ---- 2020-02-06 11:58:19 UTC - Tom Barber: :man-shrugging: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990299164400 ---- 2020-02-06 12:03:24 UTC - Tom Barber: hmm looks if I edit the service and paste those annotation lines it, it picks up the correct certificate https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580990604164900 ---- 2020-02-06 12:31:31 UTC - Alvaro Lopez: Hi, I have a couple of questions https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580992291165400 ---- 2020-02-06 12:31:56 UTC - Alvaro Lopez: I wonder if it is possible to create "warm" containers for blackbox actions https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580992316166000?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 12:32:53 UTC - Alvaro Lopez: so, if I define 2 docker actions, I would like to have them warmed as it is done with other actions https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580992373167000 ---- 2020-02-06 12:34:46 UTC - drholmie: Hey there! There is a way to warm blackbox containers. Following from a previous thread on the same you can define your own action in the format of a runtime in the runtimes.jaon file. More can be found here. <https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580451852055200?thread_ts=1580451852.055200&cid=C3TPCAQG1|https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580451852055200?thread_ts=1580451852.055200&cid=C3TPCAQG1> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580992486168100?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 12:38:48 UTC - Alvaro Lopez: cool, I will give it a try https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580992728168400?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 12:43:46 UTC - drholmie: Hope it works! Just make sure to keep your blackbox containers base image an openwhisk supported one and to not perform any `COPY` or install something from a script in your dockerfile. It tends to fail the deployment (made that mistake too many times :sweat_smile:) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580993026168800?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:36:49 UTC - Alvaro Lopez: Cool, it seems to work https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996209169000?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:37:15 UTC - drholmie: Yay! https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996235169200?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:37:39 UTC - Alvaro Lopez: But (always a but :wink: ) my scenario is a bit different I am afraid https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996259169400?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:37:55 UTC - Alvaro Lopez: I would like to have *all* Docker actions pre-warmed https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996275169600?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:38:11 UTC - Alvaro Lopez: i.e. whenever I define a new black box action, I would like to have it prewarmed https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996291169800?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:39:20 UTC - Tom Barber: I've filed my ELB issue in Github, it feels like a bug although if anyone can prove otherwise I'm happy with that. When I've got some other stuff working, I'll patch whatever template is required and submit a PR https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996360170900 ---- 2020-02-06 13:39:34 UTC - Alvaro Lopez: I can make this solution to work, i.e. I add a new action type to the `runtimes.json` , before I create the new action https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996374171000?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:39:50 UTC - Alvaro Lopez: of that specific type https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996390171200?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:40:08 UTC - Alvaro Lopez: but updating the `runtimes.json` implies a restart of the controller and invoker, right? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996408171400?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:40:24 UTC - drholmie: Ah interesting! Although hope it doesn't take too long then when deploying. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996424171600?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:40:32 UTC - drholmie: Yup it does restart it https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996432171800?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:40:41 UTC - drholmie: As far as I can tell https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996441172000?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:41:10 UTC - Alvaro Lopez: nevertheless, I can cope with that for the time being https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996470172200?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:41:29 UTC - Alvaro Lopez: you made my day :wink: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996489172400?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 13:41:55 UTC - drholmie: Awesome! Hope things work out. Always happy to help :) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580996515172600?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 14:26:52 UTC - chetanm: Logically it should have worked at least with the direct jar mode of invocation . Have not tried the recent docker version so maybe some issue https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999212172800?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:27:16 UTC - chetanm: @Rodric Rabbah which part you are referring to override https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999236173000?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:28:06 UTC - chetanm: I believe flow here is to invoke an action via openwhisk client lib from within function which uses the api host passed via env https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999286173200?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:31:40 UTC - Rodric Rabbah: I grepped for `.internal` and found this https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999500173400?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:32:12 UTC - Rodric Rabbah: ``` def getLocalHostInternalName(): String = { sys.props .get("whisk.standalone.host.internal") .getOrElse( if (SystemUtils.IS_OS_MAC || SystemUtils.IS_OS_WINDOWS) "host.docker.internal" else hostIpLinux) }``` I assumed (without looking further) that this value is being used to set `__OW_API_HOST` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999532173600?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:33:21 UTC - Rodric Rabbah: to rule out additional docker networking quirks, I was thinking that Nikhil could specify his 192. address directly for the controller host and see if actions can invoke each other. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999601173800?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 14:33:35 UTC - Rodric Rabbah: (I recall that I ad issues resolving `.internal`from the api gateway) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1580999615174000?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 15:49:42 UTC - chetanm: got it which i think now possible by specifying system prop `whisk.standalone.host.internal`. But looking at Nikhil comment > I replaced host.docker.internal with `192.168.65.2` and that didn’t help either. i am not sure if it would work. but can be tried https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581004182174200?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 16:16:21 UTC - Nikhil: Yes. I tried replacing .internal with 192. and had my controller bind to that IP and that didn’t help. Also the apigateway issue was a dns resolver issue, however in my case nslookup works. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581005781176800?thread_ts=1580969031.157800&cid=C3TPCAQG1 ---- 2020-02-06 19:27:37 UTC - Andy Steed: Any suggestion on how to restart or retry the automated travis build for a given PR ? It would appear that mine failed due to a transient failure unrelated to the PR contents (nevermind was able to find someone on our team who was able to help) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581017257177900?thread_ts=1581017257.177900&cid=C3TPCAQG1 ---- 2020-02-06 19:59:11 UTC - Rodric Rabbah: Usually closing and reopening does the trick https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019151178600?thread_ts=1581017257.177900&cid=C3TPCAQG1 ---- 2020-02-06 20:04:01 UTC - Tom Barber: This is a new one that I managed to uncover in testing today https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019441179100 ---- 2020-02-06 20:04:06 UTC - Tom Barber: ```[2020-02-06T14:07:30.887Z] [INFO] [#tid_LO6333MEbRlQaMTEZVcVNRWfiPCCvc1W] [KubernetesContainer] sending initialization to ContainerId(wskowdev-invoker-00-65-guest-pixlizeserverlessdevdatasetsummary) ContainerAddress(192.168.180.206, 8080) [marker:invoker_activationInit_start:3205] Exception in thread "pool-4-thread-173" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332)``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019446179500 ---- 2020-02-06 20:04:13 UTC - Tom Barber: thats in the invoker logs https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019453179700 ---- 2020-02-06 20:04:28 UTC - Tom Barber: the invoker is now unavailable, should I assume the heap space relates to the invoker? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019468180200 ---- 2020-02-06 20:04:35 UTC - Tom Barber: not the action being run https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581019475180500 ---- 2020-02-06 20:18:32 UTC - Tom Barber: broke another invoker i'm making a habit of this https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581020312181000 ---- 2020-02-06 20:27:11 UTC - Dave Grove: the invoker pod is currently lacking a `livenessProbe` . The controller will detect that an invoker is down and stop sending it work (should see that in the controller logs with the invoker being marked as unhealthy), but there isn’t an auto restart of the invoker pod by kubernetes via a failed livenessProbe. There really should be one https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581020831183600 ---- 2020-02-06 20:35:43 UTC - Tom Barber: let me ask a completely unanswerable and subjective question https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581021343184100 ---- 2020-02-06 20:36:19 UTC - Tom Barber: if you were going to launch an openwhisk cluster to support a webapp which is going to get demoed by an unknown (but not very large) group of people..... :wink: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581021379184900 ---- 2020-02-06 20:36:37 UTC - Tom Barber: how much Xmx would you give a controller and invoker realistically? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581021397185400 ---- 2020-02-06 20:37:10 UTC - Tom Barber: cause they launch with like 512mb and 1024mb by default I believe, but if you wanted to ensure things don't go boom, whats a sensible level? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581021430186100 ---- 2020-02-06 20:37:34 UTC - Tom Barber: like should I be looking at 10's of GB like I would with some data processing pipeline? or just a few gigs? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581021454186600 ---- 2020-02-06 20:56:56 UTC - Dave Grove: @Rodric Rabbah probably has a more informed opinion for moderate-sized deployments. I’d give each controller several gigs. Each invoker needs to have enough memory to handle the code, inputs and outputs (including logs) for all the actions it is managing simultaneously. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581022616189300 ---- 2020-02-06 20:57:49 UTC - Tom Barber: I work in research, we live by uninformed decisions... sounds good to me :wink: yay : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581022669189800 ---- 2020-02-06 21:46:29 UTC - Bilal: If you have a lot of blackbox actions, don't forget to change this setting `whisk.loadbalancer.blackbox-fraction` <https://github.com/apache/openwhisk-deploy-kube/blob/master/docs/k8s-custom-build-cluster-scaleup.md#error-only-single-invoker-instance-being-used-to-handle-all-the-workload> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581025589190000?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 21:46:52 UTC - Bilal: It might be an issue you run into when scaling up with multiple invokers. This is from the openwhisk-deploy-kube repo, but I believe it's relevant to all Openwhisk. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581025612190300?thread_ts=1580992316.166000&cid=C3TPCAQG1 ---- 2020-02-06 23:43:45 UTC - Tom Barber: If you want to prewarm some python containers, you update runtimes.json, have I deduced that correctly? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581032625191100 ---- 2020-02-06 23:48:19 UTC - Tom Barber: ... yes it is white_check_mark : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1581032899191300 ----
