2020-05-20 00:45:45 UTC - Lixiang Ao: I followed <https://medium.com/@ansjin/openwhisk-deployment-on-a-kubernetes-cluster-7fd3fc2f3726> to deploy OW on Kubernetes. It doesn’t use kops tho https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589935545336800?thread_ts=1589861729.336600&cid=C3TPCAQG1 ---- 2020-05-20 00:49:09 UTC - Lixiang Ao: Has anyone successfully visualize cold/warm starts in Grafana? My Grafana just plots some random lines. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589935749338600 ---- 2020-05-20 00:49:14 UTC - Lixiang Ao: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589935754338700?thread_ts=1589935754.338700&cid=C3TPCAQG1 ---- 2020-05-20 00:49:42 UTC - Lixiang Ao: other metrics work https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589935782339300 ---- 2020-05-20 00:57:24 UTC - Brendan Doyle: the dynamic pre-warm pr got me interested in pre warm containers. Is there any documentation on how to configure pre-warm containers statically as is, I'm having trouble finding any https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589936244340600?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:01:14 UTC - Lixiang Ao: example here <https://github.com/apache/openwhisk/blob/154faf985262f4fd0e06ca55b0fc96b6e37c3965/ansible/files/runtimes.json#L58-L63> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589936474340700?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:04:38 UTC - Brendan Doyle: thanks didn't realize it was just something you define in the runtime manifest https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589936678340900?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:05:38 UTC - Brendan Doyle: is it possible to create pre warmed containers for different memory profiles for the same runtime? white_check_mark : Rodric Rabbah https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589936738341100?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:20:53 UTC - Brendan Doyle: also curious on what happens once all of the preWarmed containers are used and those warm containers are evicted? Are new preWarmed containers created so there's always x containers running for that runtime? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589937653341300?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:22:30 UTC - Rodric Rabbah: Prewarmed containers are called stem cells. When a stem cell is used another is generated. So the pool of stem cells is replenished. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589937750343100?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:23:03 UTC - Rodric Rabbah: Looks like a possible regression https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589937783343700?thread_ts=1589935754.338700&cid=C3TPCAQG1 ---- 2020-05-20 01:23:23 UTC - Brendan Doyle: What happens if the invoker is at capacity though? It doesn't create additional preWarmed containers does it? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589937803343900?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:29:31 UTC - Rodric Rabbah: I don’t recall the heuristic has changed so I’d need to look at that part of the code to refresh my memory https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938171346000?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:29:58 UTC - Rodric Rabbah: Maybe @Tyson Norris or @Ning yougang know faster https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938198346700?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:30:31 UTC - Brendan Doyle: cool yea my only concern would be potentially overprovisioning and our host runs out of memory https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938231346900?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:31:38 UTC - Brendan Doyle: Although our container pool is like 33% of our host memory so if that is the case over provisioning a little shouldn't be a big deal https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938298347100?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:38:17 UTC - Ning yougang: here is a configuration(or example) for the prewarm kind: <https://github.com/apache/openwhisk/pull/4871/files#diff-b3aad3cfa7e78187bab586c8c438461bR62> this pr just adjust prewarm containers dynamically via reactive configuration, if prewarm expired, removed them, if cold start happened, create some prewarmed containers accordingly https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938697347400?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:41:42 UTC - Ning yougang: if don't config "reactive", keep the same logic as before. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589938902347700?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:44:02 UTC - Brendan Doyle: do you happen to know if the invoker has warm containers filling the entire container pool, will the configuration still create new pre warm containers and essentially over provision the container pool? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589939042347900?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:48:15 UTC - Lixiang Ao: I’ll create an issue https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589939295348100?thread_ts=1589935754.338700&cid=C3TPCAQG1 ---- 2020-05-20 01:56:50 UTC - Ning yougang: 1. is it possible to create pre warmed containers for different memory profiles for the same runtime? Yes, you can see `runtimes.json` (e.g. for same kind: nodejs:N, may have 256 MB and 512 MB)
2. also curious on what happens once all of the preWarmed containers are used and those warm containers are evicted? yes, if 1 prewarmed container is used, a new prewarmed container will be created, at the same time, previous old prewarmed container is moved from `prewarmedPool` to `busyPool` 3. Are new preWarmed containers created so there's always x containers running for that runtime? Just for in upsteam codes, the prewarmed container number for kind/memory is stable. After applied this pr and config "reactive", may be not, because the prewarmed container will be deleted due to expiration and create again for next schedule happened. 4. What happens if the invoker is at capacity though? It doesn't create additional preWarmed containers does it? Currently, for upsteam or this pr, doesn't take memory used by prewarm account into along with userMemory as well, i think we can implement in subquent pr. Regarding `It doesn't create additional preWarmed containers does it?`, if doesn't apply this pr, the prewarm contaienr number is stale as above said, if applied, the prewarm contaienr number is in a range, you can see, minCount, maxCount in reactive. 4. do you happen to know if the invoker has warm containers filling the entire container pool, will the configuration still create new pre warm containers and essentially over provision the container pool? I didn't consider your said problem in this pr, as i already said, i will add another pr to consider prewarm container memory as used memory https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589939810348300?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 01:59:34 UTC - Brendan Doyle: thanks for the detailed answer. very helpful for understanding how this works. Having some documentation on how preWarmed containers work I think would be a great addition https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589939974348700?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 02:00:21 UTC - Brendan Doyle: also your pr is great. dynamically preparing resources is an awesome feature https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589940021348900?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 02:00:59 UTC - Ning yougang: ^^ https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589940059349100?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 02:02:26 UTC - Ning yougang: for doucment, i will add it for next pr. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589940146349300?thread_ts=1589936244.340600&cid=C3TPCAQG1 ---- 2020-05-20 02:02:52 UTC - Ning yougang: Include prewarm container's concept, its effect and reactive configuration as well. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1589940172349500?thread_ts=1589936244.340600&cid=C3TPCAQG1 ----
