2020-05-12 02:54:45 UTC - Rodric Rabbah: this is not known afaik - send to the dev list might get wider audience
ill check with our team https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589252085253700?thread_ts=1589241292.253600&cid=C3TPCAQG1 ---- 2020-05-12 05:40:24 UTC - Ning yougang: hm.. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589262024254300?thread_ts=1588920832.236500&cid=C3TPCAQG1 ---- 2020-05-12 05:40:33 UTC - Ning yougang: I didn't use the apigway https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589262033254500?thread_ts=1588920832.236500&cid=C3TPCAQG1 ---- 2020-05-12 05:40:46 UTC - Ning yougang: Did it solved now? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589262046254700?thread_ts=1588920832.236500&cid=C3TPCAQG1 ---- 2020-05-12 12:26:02 UTC - Rodric Rabbah: Can you share a test case? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589286362255200?thread_ts=1589241292.253600&cid=C3TPCAQG1 ---- 2020-05-12 14:38:40 UTC - Rodric Rabbah: @Brendan Doyle i asked my team - there is a “feature” in the invoker which dilates the execution time before the container is paused/reclaimed https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589294320255600?thread_ts=1589241292.253600&cid=C3TPCAQG1 ---- 2020-05-12 14:39:46 UTC - Rodric Rabbah: it has to do with the log limits and expected time to drain the logs - if you have a default log limit set and you haven’t changed that, it could be part of the reason (default log limit is 10MB) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589294386255800?thread_ts=1589241292.253600&cid=C3TPCAQG1 ---- 2020-05-12 15:14:17 UTC - Nima Mahmoudi: Nope, I am using the web actions for now, but the issue still persists. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589296457256000?thread_ts=1588920832.236500&cid=C3TPCAQG1 ---- 2020-05-12 16:52:33 UTC - Brendan Doyle: Yea I was going to say I found this issue, don't know if it ever was fixed but sounds like similar of a problem. The only other thing I can think of is the docker daemon is frozen and can't pause the container. Our customer already fixed their function so it's hard for me to reproduce, but I can get the code and see what they were doing with logging <https://github.com/apache/openwhisk/issues/4298> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589302353256200?thread_ts=1589241292.253600&cid=C3TPCAQG1 ---- 2020-05-12 17:06:43 UTC - Kostis Kaffes: When would an Invoker evict a “warm” container for a specific action? It it only when its memory capacity is reached? white_check_mark : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589303203257200 ---- 2020-05-12 17:30:53 UTC - Rodric Rabbah: there is also a default duration after which the container is evoked even if no capacity is needed - that’s configurable +1 : Kostis Kaffes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589304653258500 ---- 2020-05-12 18:56:30 UTC - Kostis Kaffes: Thanks! Three quick follow-ups: 1. Which of the two places takes precedence for memory capacity, <https://github.com/apache/openwhisk/blob/master/core/invoker/src/main/resources/application.conf#L60> or <https://github.com/apache/openwhisk/blob/master/ansible/group_vars/all#L194>? 2. Which reclamation policy is used? LRU? 3. Could you point me to where the default duration is defined? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589309790260200?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 21:32:31 UTC - Kostis Kaffes: One more question, if there is already a running container for a specific action that is serving an activation and another activation from the same function arrives at an Invoker, does the Invoker need to fetch the action code from couchdb again? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589319151263700?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 22:15:29 UTC - Brendan Doyle: Depends if there is a warm container available. If there is, it's reused for the second activation https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589321729263800?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 22:16:22 UTC - Kostis Kaffes: If the warm container is not available? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589321782264000?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 22:23:23 UTC - Brendan Doyle: Every cold start will fetch the action code thankyou : Kostis Kaffes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589322203264200?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:11:53 UTC - Lixiang Ao: has anyone seen this problem: a very simple python action works fine w/256M memory, but w/ 128M memory, it doesn’t work: > The action did not initialize and exited unexpectedly. -1 : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325113266200?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:13:32 UTC - Rodric Rabbah: action code is cached in the invoker +1 : Brendan Doyle https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325212266300?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:14:02 UTC - Rodric Rabbah: it’s a short ttl and the cache uses soft pointers so if the gc needs to reclaim heap space code can get evicted from the cache https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325242266500?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:14:08 UTC - Rodric Rabbah: i think default is 5 mins https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325248266700?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:15:06 UTC - Rodric Rabbah: ansible (environment variables) override https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325306267100?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 23:15:12 UTC - Brendan Doyle: ^ yea I misspoke. cold container still needs to get the code but might not need to go to couchdb https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325312267400?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:16:32 UTC - Rodric Rabbah: for your 3rd q: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325392267600?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 23:16:33 UTC - Rodric Rabbah: <https://github.com/apache/openwhisk/blob/ffef0a90dc41c296bc287b6877b75bac7518d4ce/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerProxy.scala#L851> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325393267800?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 23:16:52 UTC - Rodric Rabbah: <https://github.com/apache/openwhisk/blob/fb6c779da576e9e4d32a6d7501a4d2031ab1dbf3/core/invoker/src/main/resources/application.conf#L138-L144> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325412268000?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 23:22:49 UTC - Kostis Kaffes: Thanks both of you! +1 : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325769268200?thread_ts=1589319151.263700&cid=C3TPCAQG1 ---- 2020-05-12 23:25:01 UTC - Kostis Kaffes: :thankyou: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589325901268700?thread_ts=1589309790.260200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:09 UTC - Lixiang Ao: @Rodric Rabbah any problem with the question? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326029269100?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:26 UTC - Rodric Rabbah: no https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326046269300?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:34 UTC - Rodric Rabbah: thumbs down as in i have not seen that https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326054269500?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:44 UTC - Lixiang Ao: oh, got it https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326064269700?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:46 UTC - Lixiang Ao: sorry https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326066269900?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:27:56 UTC - Rodric Rabbah: np sorry if that was confusing https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326076270100?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:28:02 UTC - Rodric Rabbah: the code is identical? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326082270300?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:28:13 UTC - Rodric Rabbah: if the action ran out of memory you’d get a different error https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326093270500?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:28:18 UTC - Lixiang Ao: I think it’s about the action size https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326098270700?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:28:33 UTC - Lixiang Ao: I remove a 32M binary then it works https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326113270900?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:28:50 UTC - Lixiang Ao: I think it’s out of memory https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326130271100?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:29:04 UTC - Kostis Kaffes: There is a 48M default limit for the action code, right? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326144271300?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:29:21 UTC - Lixiang Ao: but slightly surprising tho, because it’s 32M code vs 128M available https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326161271500?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:30:10 UTC - Rodric Rabbah: are you fetching the code dynamically into the function? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326210271700?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:31:21 UTC - Lixiang Ao: no, I included the 32M binary in the zip https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326281271900?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:32:48 UTC - Rodric Rabbah: you can try what’s suggested here https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326368272200?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:32:48 UTC - Rodric Rabbah: <https://medium.com/openwhisk/advanced-debugging-of-openwhisk-actions-518414636932> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326368272400?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:32:49 UTC - Rodric Rabbah: basically run the runtime container locally https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326369272600?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:32:49 UTC - Rodric Rabbah: `docker run -p 8080:8080 <image name>` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326369272800?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:33:11 UTC - Rodric Rabbah: `/path/to/openwhisk/tools/actionProxy/invoke.py init <file.zip>` https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326391273100?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:34:07 UTC - Rodric Rabbah: see if that works - then you can limit the memory on the container see if you get more insight - it’s possible that you’re exhausting some limit but if it’s memory the invoker sets an error message that says “exceeded memory” or such https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326447273300?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:35:49 UTC - Rodric Rabbah: <https://github.com/apache/openwhisk/blob/a3f41e665794919bb16f06bd15ac24b6ba1f79e0/common/scala/src/main/scala/org/apache/openwhisk/http/ErrorResponse.scala#L146> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326549273500?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:36:18 UTC - Lixiang Ao: ye that’s the confusing part… it didn’t specifically say https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326578273700?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:36:35 UTC - Lixiang Ao: I’ll try locally https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326595273900?thread_ts=1589325113.266200&cid=C3TPCAQG1 ---- 2020-05-12 23:36:41 UTC - Lixiang Ao: thank you https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589326601274100?thread_ts=1589325113.266200&cid=C3TPCAQG1 ----
