2019-11-26 00:12:22 UTC - Ali Tariq: thanks! i will trying increasing the number of invokers. p.s. every node has 64GB of RAM (and i don't have any other processes or workloads running, so that should be enough) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574727142440400?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 00:52:03 UTC - Ali Tariq: i tried to increase invoker replicas to `5` , but it still exhibits the same behavior. Getting the response `<Response [503]> b'{\n "code": "CHWqKpYaj5P8Sb396pejMZIceN0s2Trt",\n "error": "The server is currently unavailable (because it is overloaded or down for maintenance)."\n}'` for every new request. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574729523440600?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 00:53:05 UTC - Ali Tariq: last few lines from controller logs. It doesn't specify the actual problem, is there a way to increase the logging level to identify the problem? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574729585441000?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 03:05:45 UTC - Ali Tariq: i found the timestamp from the controller logs where invoker status changes to `unhealthy` . Then parsing the invoker logs around that timestamp, i found something. I don't know why but for some reason it fails to create pods ... any insights? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574737545441500?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 03:25:36 UTC - Ali Tariq: the above error further affects the creation of `whisksystem-invokerhealthtestaction0` pods - the error shown keeps repeating (few lines shown in the snippet) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574738736441900?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 12:40:42 UTC - chetanm: @Tyson Norris @dan mcweeney For KCF if above errors are seen then is there some timeout settings which can be tweaked https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574772042442300?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 13:32:54 UTC - Dave Grove: Log processing at the invoker is often a bottleneck for the KubernetesContainerFactory. Try disabling invoker log processing by adding the snippet below to your mycluster.yaml https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574775174442500?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 13:33:07 UTC - Dave Grove: ```invoker: options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider"``` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574775187442700?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 13:34:16 UTC - Dave Grove: In general, you need to offload log processing from the invoker to a node-level log store provider if you are trying to push a large load through the system. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574775256442900?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 14:17:53 UTC - volo: in order to be able to create/deploy functions in openwhisk would we need docker repo/images like for example for openfaas? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574777873443800 ---- 2019-11-26 14:50:42 UTC - Shawn Black: You can do the docker if you want — <https://github.com/apache/openwhisk-runtime-docker> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574779842444500 ---- 2019-11-26 14:50:50 UTC - Shawn Black: BUT OpenWhisk does support numerous languages https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574779850444800 ---- 2019-11-26 14:51:12 UTC - Shawn Black: <https://openwhisk.apache.org/documentation.html> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574779872445000 ---- 2019-11-26 14:51:57 UTC - Shawn Black: Officially: .NET, Go, Java, JS, PHP, Python, Ruby, Swift. Experimental: Ballerina and Rust https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574779917445700 ---- 2019-11-26 16:18:29 UTC - Rodric Rabbah: Further, You do NOT need docker to create functions. The openwhisk programming model is similar to aws lambda in this regard. You package functions as zip files or individual function files. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574785109447800 ---- 2019-11-26 16:20:04 UTC - Rodric Rabbah: You can even use a binary directly in some cases without docker to run as a function.
You can run docker containers as functions still and this is useful when you need to bundle packages and dependencies that cant be done in a zip file, or when you want to run something in a language not supported natively. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574785204450500 ---- 2019-11-26 19:07:28 UTC - Markus Thömmes: can we unblock log publishing from the CI test runs? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574795248451000 ---- 2019-11-26 19:43:23 UTC - Dave Grove: log uploading is failing (hasn’t worked in a long time), but the log upload script still exits with 0, so its “harmless” https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797403451700 ---- 2019-11-26 19:43:44 UTC - Dave Grove: other than not having the logs to inspect when actually needed. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797424452000 ---- 2019-11-26 19:44:15 UTC - Markus Thömmes: well that's kinda my point :smile: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797455452300 ---- 2019-11-26 19:44:45 UTC - Markus Thömmes: I cannot for the life of me get Openwhisk setup locally :confused: I'd need a vehicle to debug failing CI properly which these logs would give me https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797485453100 ---- 2019-11-26 19:47:40 UTC - Rodric Rabbah: We need to check what’s happening in the relay app. I don’t have access to it anymore and I think it was moved to another account but have no idea really. It would take a bit of effort - do you want to volunteer :yay: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797660455800 ---- 2019-11-26 19:48:05 UTC - Dave Grove: sure it isn’t just that box is full again? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797685456400 ---- 2019-11-26 19:48:24 UTC - Rodric Rabbah: :man-shrugging: I can clean that up as a first try https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797704457100 ---- 2019-11-26 19:48:49 UTC - Rodric Rabbah: for some vague reasons I recall the relay itself wasn’t working https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797729458000 ---- 2019-11-26 19:49:01 UTC - Dave Grove: there are actually a few uploads from October and they are all small files. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797741458200 ---- 2019-11-26 19:49:17 UTC - Rodric Rabbah: Ok then maybe it is easy https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797757458500 ---- 2019-11-26 19:49:23 UTC - Markus Thömmes: :pray: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797763458800 ---- 2019-11-26 19:49:24 UTC - Rodric Rabbah: I’ll try now https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797764458900 ---- 2019-11-26 19:49:31 UTC - Markus Thömmes: safe me from python hell pls https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797771459200 ---- 2019-11-26 19:49:42 UTC - Dave Grove: is there any way to bulk delete files in box? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797782459500 ---- 2019-11-26 19:49:56 UTC - Rodric Rabbah: Yes. Drop the whole folder :) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574797796459900 ---- 2019-11-26 19:54:32 UTC - Rodric Rabbah: not confident it’s working from these dates but we are at capacity https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798072460000 ---- 2019-11-26 19:55:14 UTC - Dave Grove: right. the thing I was noticing was that the last few files uploaded are all tiny. so maybe they just fit. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798114460800 ---- 2019-11-26 19:55:17 UTC - Dave Grove: guess we can find out. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798117461000 ---- 2019-11-26 19:59:23 UTC - Rodric Rabbah: wiped content just waiting for box to purge https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798363461300 ---- 2019-11-26 19:59:48 UTC - Rodric Rabbah: reclaimed 0.4GB / 10.0GB for far :man-facepalming: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798388461600 ---- 2019-11-26 20:00:11 UTC - Markus Thömmes: that'll do for me :smile: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798411461800 ---- 2019-11-26 20:00:25 UTC - Markus Thömmes: is the link pasted at the end of the build or do I need to know a magic URL? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798425462200 ---- 2019-11-26 20:00:39 UTC - Rodric Rabbah: should be pinned but this is it <https://openwhisk.app.box.com/v/travis-logs> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574798439462500 ---- 2019-11-26 20:10:48 UTC - Rodric Rabbah: :confused: hasn’t gc’ed the rest, not sure how long that takes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574799048462900 ---- 2019-11-26 20:10:54 UTC - Rodric Rabbah: but the folder is empty and consumes 0B https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574799054463200 ---- 2019-11-26 20:11:05 UTC - Markus Thömmes: eventual consistency :yay: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574799065463400 ---- 2019-11-26 20:11:21 UTC - Rodric Rabbah: did you kick a new travis job? curious to see if the upload works https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574799081463700 ---- 2019-11-26 20:11:30 UTC - Markus Thömmes: I did, takes a while tho +1 : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574799090463900 ---- 2019-11-26 20:36:18 UTC - Markus Thömmes: it worked, thanks @Rodric Rabbah pray : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574800578464200 ---- 2019-11-26 21:25:56 UTC - Tyson Norris: I haven’t seen these `KubernetesClientTimeoutException` errors from fabric8 client - check your pod status, and make sure your invoker pod has access to api server? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574803556464500?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 23:50:28 UTC - Ali Tariq: @Dave Grove i tried disabling the logs like you suggested, it did not help much. I was looking at the logs from controller - i have `5` invoker replicas but only one of them is being consistently used (see the attached snippet - invoker4 keeps repeating). Its like all the load is being handled by a single invoker - p.s. i still got the `unavailable or down for maintenance` response. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574812228464700?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 23:53:53 UTC - Ali Tariq: @Tyson Norris i double checked the `invoker-pod` status and its always `running` . How can i check if it has access to api-server? ... Like i said, its only down for sometime (5-10min), after which its again able to service requests. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574812433465100?thread_ts=1574633042.430600&cid=C3TPCAQG1 ---- 2019-11-26 23:57:18 UTC - Ali Tariq: Please note that i am using `whisk:ingress:type:NodePort` & `k8s:persistence:enabled:false` in my `cluster.yaml` . https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1574812638465300?thread_ts=1574633042.430600&cid=C3TPCAQG1 ----
