Re: Allow temporal unused codes to include a big change.
Thank you, folks, for the feedbacks.
I will take them into account.
Since a few guys will participate in the contribution, we will add a prefix
[scheduler] to the title of PRs and add "scheduler" label.
It will let reviewers know they are for a new component and may include
some intermediate codes.
Best regards
Dominic
2019년 7월 5일 (금) 오후 10:05, Markus Thömmes 님이 작성:
> Continuous integration is a crucial aspect of a system used in production
> like OpenWhisk.
>
> As such, let's bite the bullet by merging things incrementally into master.
> That will also force you to think about migration on each step you take,
> which in the long run will pay off big time. Accumulating a huge change
> will never be review- nor mergeable confidently in the end.
>
> +1 to incremental changes.
>
> Am Fr., 5. Juli 2019 um 08:54 Uhr schrieb Martin Henke <
> martin.he...@web.de
> >:
>
> > Strongly + 1 to put core changes behind feature flags so that they can be
> > rolled out (and removed if necessary )
> > in a controlled way.
> > Tests should be running successfully for having the feature flag enabled
> > or disabled.
> > In the near past it took a considerable amount of my time to adapt test
> > cases for some of the features
> > introduced in the last time to make them work in all cases.
> >
> > Regards,
> > Martin
> >
> >
> >
> > > On 5. Jul 2019, at 06:59, Chetan Mehrotra
> > wrote:
> > >
> > > +1 to commit incrementally to master.
> > >
> > > If the changes touch the core logic then we can possibly have them
> > > backed by feature flags and have them disabled by default and enabled
> > > for test case. Further it would be preferable that whatever we commit
> > > (at any stage) it should have required test coverage. This would
> > > ensures that the sub parts of work in progress bigger feature are
> > > backed by unit tests.
> > >
> > > regards
> > > Chetan Mehrotra
> > >
> > > On Thu, Jul 4, 2019 at 9:35 PM Dominic Kim
> wrote:
> > >>
> > >> Hello, whiskers.
> > >>
> > >> I am trying to contribute this:
> > >> https://github.com/apache/incubator-openwhisk/pull/4532
> > >>
> > >> At first, I tried to open a base branch and merge all incremental PRs
> > into
> > >> the branch.
> > >> And finally mege the base branch into the master after the
> > implementation
> > >> is done.
> > >> Surely, reviewers would review all the subsequent PRs and it will be
> > based
> > >> on SPI and disabled by default at first, there would be no big issue I
> > >> think.
> > >>
> > >> And Markus suggested to incrementally merge all PRs into the master
> > instead
> > >> of having a big base branch.
> > >>
> >
> https://github.com/apache/incubator-openwhisk/pull/4532#issuecomment-508519119
> > >>
> > >> With the former, we don't need to include temporal unused codes but
> > need to
> > >> include a big possibly disruptive PR at once.
> > >> With the latter, there will be some temporal unused components in the
> > >> master at some point, but we can make sure merged codes do not induce
> > any
> > >> issue. And actually the latter is easier for me as well : )
> > >>
> > >> If we all agree with including the temporal unused codes in the
> master,
> > I
> > >> am happy to work in the way Markus suggested.
> > >>
> > >> One example of a temporal code is this:
> > >>
> >
> https://github.com/apache/incubator-openwhisk/pull/4532/files#diff-1d7110b32c507a6ef4ac956c287e77ebR24
> > >>
> > >> Since there is no other component, the "scheduler" cannot initialize
> all
> > >> components in the main function and there is only `println("Hello")`
> in
> > the
> > >> initial version of the main function.
> > >>
> > >>
> > >> Please let me know your thoughts.
> > >>
> > >> Best regards
> > >> Dominic
> >
> >
>
Re: Fine-grained permission
Thank you for the good point, Martin. I will take it into account. Best regards Dominic 2019년 7월 5일 (금) 오후 10:10, Martin Henke 님이 작성: > Dominic, > > I would very much like to have this change introduced in a backward > compatible fashion. > > In Detail: > The default permissions should either keep the same behaviour as with the > current code > or better be configurable via Ansible in a way that he system behaves the > same as before. > If you decide for a schema change, the code should be handle existing > actions with the existing behaviour. > > With being backward compatible there would be no need to introduce this > change in a new EntitlementProvider implementation but can (and should) > be placed in the (default) LocalEntitlement provider to have consistent > code in the default configuration for Action creation (adding the access > rights) and entitlement checking (reading the access rights). > > Thank you for driving this forward., > Martin > > > > > On 4. Jul 2019, at 17:12, Dominic Kim wrote: > > > > Thank you for the feedback, James. > > > > I am thinking of implementing it with annotations. > > For example, we would add a new annotation reserved by the system. > > When an action is created, the default permission will be specified via > the > > annotation. > > > > When any request comes, a new EntitlementProvider will look for the > > annotation and reject the request based on it. > > > > This is to minimize the action schema change. > > But if it's ok to bear with the schema change, I am ok with adding one > more > > field in an action other than annotations or parameters. > > > > 2019년 7월 4일 (목) 오후 10:07, James Thomas 님이 작성: > > > >> Protecting accidental overwritting or deletion of actions would be a > great > >> feature. I like the suggestion and approach of using Unix-style > >> permissions. > >> > >> On Thu, 4 Jul 2019 at 07:25, Dominic Kim wrote: > >> > >>> Recently I discussed this: > >>> https://github.com/apache/incubator-openwhisk/pull/4058 with my > >>> colleagues. > >>> That PR is to add a feature to protect actions from deletion by > mistake. > >>> That is a good suggestion and I think we can also include more > >> generalized > >>> way to handle the issue. > >>> > >>> For example, what we can expect about permission are as follows. > >>> > >>> 1. Action protection. > >>> 2. Hide codes from the shared package. > >>> > >>> I am a bit faint but IIRC, Rodric suggested linux-like permission > >>> management. > >>> > >>> Regarding number 1, we can achieve it with the permission, "Read / not > >>> Write / Execute". > >>> And with regared to number 2, we can also achieve it with the > permission, > >>> "not Read / not Write (this is the default of shared package action) / > >>> Execute". > >>> > >>> If we apply linux-like permission to these cases, we can have two > >> different > >>> permission flags, one for owners, the other for users of shared > packages. > >>> Then actions can have permission information such as "71" or "51". > >>> So "71" would mean the owner of an action can do "read/write/execute" > it > >>> but the one who uses the shared action would be able to do "not > read/not > >>> write/execute". > >>> "51" would mean the owner can do "read/not write/execute". > >>> > >>> There might be more cases, but I believe we can deal with them in the > >> same > >>> way. > >>> Any feedback or idea on this would be appreciated. > >>> > >>> Thanks > >>> Best regards, > >>> Dominic > >>> > >> > >> > >> -- > >> Regards, > >> James Thomas > >> > >
Re: [RESULT] [VOTE] Release Apache OpenWhisk Catalog (v0.10.0-incubating, rc1)
This vote is now closed. It passed with 4 +1 votes (Rabbah, Santana, Kim, Thomas) and no other votes. I will go ahead with the IPMC vote next. Mentors, please vote on that thread to help us get our 3 binding +1 votes. --dave On 2019/06/30 19:34:26, "David P Grove" wrote: > > > Hi, > > This is a call to vote on releasing version 0.10.0-incubating release > candidate rc1 of the following project module with artifacts built from the > Git repositories and commit IDs listed below. > > * OpenWhisk Catalog: cb8d5c73 > https://github.com/apache/incubator-openwhisk-catalog/commits/cb8d5c73 > > https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz > > https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz.asc > > https://dist.apache.org/repos/dist/dev/incubator/openwhisk/apache-openwhisk-0.10.0-incubating-rc1/openwhisk-catalog-0.10.0-incubating-sources.tar.gz.sha512 > > This release is comprised of source code distribution only. > > You can use this UNIX script to download the release and verify the > checklist below: > https://gitbox.apache.org/repos/asf?p=incubator-openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=HEAD > > Usage: > curl -s " > https://gitbox.apache.org/repos/asf?p=incubator-openwhisk-release.git;a=blob_plain;f=tools/rcverify.sh;hb=HEAD > " -o rcverify.sh > chmod +x rcverify.sh > rcverify.sh openwhisk-catalog 'OpenWhisk Catalog' 0.10.0-incubating rc1 > > Please vote to approve this release: > > [ ] +1 Approve the release > [ ] 0 Don't care > [ ] -1 Don't release, because ... > > Release verification checklist for reference: > [ ] Download links are valid. > [ ] Checksums and PGP signatures are valid. > [ ] DISCLAIMER is included. > [ ] Source code artifacts have correct names matching the current > release. > [ ] LICENSE and NOTICE files are correct for each OpenWhisk repository. > [ ] All files have license headers as specified by OpenWhisk project > policy [1]. > [ ] No compiled archives bundled in source archive. > > This majority vote is open for at least 72 hours. > > > [1] > https://github.com/apache/incubator-openwhisk-release/blob/master/docs/license_compliance.md >
Re: dockerhub "latest" update
Great progress! On Thu, 4 Jul 2019 at 17:50, David P Grove wrote: > > > > TL;DR: I changed `latest` on virtually all images yesterday. Some issues > in downstream repo CI/CD. Will work through them this weekend. > > In more detail, OpenWhisk has 50 repositories on dockerhub. 36 of them > are active (still being published/used). > > Of the active 36: > + for 25 of these repos, `latest` now corresponds to the image built from > the most recent Apache release (and the release is part of our current > "wave") > > + for 6 of these repos (5 runtimes + 1 kube-only image), there has not > been an official release and I removed the latest tag. I think for these 6 > the lack of latest has no real impact. > > + for 4 repos (ow-utils, invoker, controller, and scala), the last > official release was 9+ months ago, so I removed latest instead of setting > it to 0.9.0-incubating. This is exposing places we need to adjust in > downstream gitrepos. > > + the 36th repo is wskdeploy. I did not do anything here yet (I think we > may be able to eliminate this image entirely. If not, we need to modernize > its setup to tag with git hash commits). > > Gory details. Appended is the list of repos with a note of what latest is > now pointing to. (A few deprecated repos still have a latest tag for now > because they only contain one image called latest). > > ---dave > > > Active Images > - > > # Runtimes > action-ballerina-v0.990.2 -- NO RELEASE; REMOVED LATEST > actionloop-rust-v1.34 -- NO RELEASE; REMOVED LATEST > > actionloop-golang-v1.12 -- NO RELEASE; REMOVED LATEST > actionloop-java-v8 -- NO RELEASE; REMOVED LATEST > actionloop-ruby-v2.6 -- NO RELEASE; REMOVED LATEST > > actionloop-v2 -- DONE (1.13.0-incubating) > dockerskeleton -- DONE (1.13.0-incubating) > > action-dotnet-v2.2 -- DONE (1.13.0-incubating) > > actionloop-golang-v1.11 -- DONE (1.13.0-incubating) > > java8action -- DONE (1.13.0-incubating) > > nodejs6action -- DONE (1.14.0-incubating) > nodejsactionbase -- DONE (1.14.0-incubating) > action-nodejs-v8 -- DONE (1.14.0-incubating) > action-nodejs-v10 -- DONE (1.14.0-incubating) > action-nodejs-v12 -- DONE (1.14.0-incubating) > > action-php-v7.1 -- DONE (1.13.0-incubating) > action-php-v7.2 -- DONE (1.13.0-incubating) > action-php-v7.3 -- DONE (1.13.0-incubating) > > python2action -- DONE (1.13.0-incubating) > actionloop-python-v3.7 -- DONE (1.13.0-incubating) > python3aiaction -- DONE (1.13.0-incubating) > python3action -- DONE (1.13.0-incubating) > > action-ruby-v2.5 -- DONE (1.13.0-incubating) > > action-swift-v4.1 -- DONE (1.13.0-incubating) > action-swift-v4.2 -- DONE (1.13.0-incubating) > > # Core System > ow-utils -- REMOVED LATEST > invoker -- REMOVED LATEST > controller -- REMOVED LATEST > scala -- REMOVED LATEST > apigateway -- DONE (0.10.0-incubating) > > alarmprovider -- DONE (2.0.0-incubating) > cloudantprovider -- DONE (2.0.0-incubating) > kafkaprovider -- DONE (2.0.0-incubating) > > kube-invoker-agent -- NO RELEASE; REMOVED LATEST > > > # MISC > example -- DONE (1.13.0-incubating) > > wskdeploy TODO: Do we really need this image??? > > > Deprecated Images > --- > > # Old runtimes > > actionloop -- NO RELEASE; REMOVED LATEST > action-ballerina-v0.975 -- -- NO RELEASE; REMOVED LATEST > actionloop-golang-v1.10 -- tagged with master, no latest tag > action-swift-v3.1.1 -- NO RELEASE; REMOVED LATEST > action-swift-v4.0 -- NO RELEASE; REMOVED LATEST > swift3action -- NO RELEASE; REMOVED LATEST > javaaction > > # Samples > thumbnail > asciiart > spellcheck > > # Misc Stuff > ansible-runner -- NO RELEASE; REMOVED LATEST > script-runner -- NO RELEASE; REMOVED LATEST > couchdb-catalog > couchdb-snapshot -- NO RELEASE; REMOVED LATEST -- Matt Sicker
Re: Fine-grained permission
Dominic, I would very much like to have this change introduced in a backward compatible fashion. In Detail: The default permissions should either keep the same behaviour as with the current code or better be configurable via Ansible in a way that he system behaves the same as before. If you decide for a schema change, the code should be handle existing actions with the existing behaviour. With being backward compatible there would be no need to introduce this change in a new EntitlementProvider implementation but can (and should) be placed in the (default) LocalEntitlement provider to have consistent code in the default configuration for Action creation (adding the access rights) and entitlement checking (reading the access rights). Thank you for driving this forward., Martin > On 4. Jul 2019, at 17:12, Dominic Kim wrote: > > Thank you for the feedback, James. > > I am thinking of implementing it with annotations. > For example, we would add a new annotation reserved by the system. > When an action is created, the default permission will be specified via the > annotation. > > When any request comes, a new EntitlementProvider will look for the > annotation and reject the request based on it. > > This is to minimize the action schema change. > But if it's ok to bear with the schema change, I am ok with adding one more > field in an action other than annotations or parameters. > > 2019년 7월 4일 (목) 오후 10:07, James Thomas 님이 작성: > >> Protecting accidental overwritting or deletion of actions would be a great >> feature. I like the suggestion and approach of using Unix-style >> permissions. >> >> On Thu, 4 Jul 2019 at 07:25, Dominic Kim wrote: >> >>> Recently I discussed this: >>> https://github.com/apache/incubator-openwhisk/pull/4058 with my >>> colleagues. >>> That PR is to add a feature to protect actions from deletion by mistake. >>> That is a good suggestion and I think we can also include more >> generalized >>> way to handle the issue. >>> >>> For example, what we can expect about permission are as follows. >>> >>> 1. Action protection. >>> 2. Hide codes from the shared package. >>> >>> I am a bit faint but IIRC, Rodric suggested linux-like permission >>> management. >>> >>> Regarding number 1, we can achieve it with the permission, "Read / not >>> Write / Execute". >>> And with regared to number 2, we can also achieve it with the permission, >>> "not Read / not Write (this is the default of shared package action) / >>> Execute". >>> >>> If we apply linux-like permission to these cases, we can have two >> different >>> permission flags, one for owners, the other for users of shared packages. >>> Then actions can have permission information such as "71" or "51". >>> So "71" would mean the owner of an action can do "read/write/execute" it >>> but the one who uses the shared action would be able to do "not read/not >>> write/execute". >>> "51" would mean the owner can do "read/not write/execute". >>> >>> There might be more cases, but I believe we can deal with them in the >> same >>> way. >>> Any feedback or idea on this would be appreciated. >>> >>> Thanks >>> Best regards, >>> Dominic >>> >> >> >> -- >> Regards, >> James Thomas >>
Re: Allow temporal unused codes to include a big change.
Continuous integration is a crucial aspect of a system used in production
like OpenWhisk.
As such, let's bite the bullet by merging things incrementally into master.
That will also force you to think about migration on each step you take,
which in the long run will pay off big time. Accumulating a huge change
will never be review- nor mergeable confidently in the end.
+1 to incremental changes.
Am Fr., 5. Juli 2019 um 08:54 Uhr schrieb Martin Henke :
> Strongly + 1 to put core changes behind feature flags so that they can be
> rolled out (and removed if necessary )
> in a controlled way.
> Tests should be running successfully for having the feature flag enabled
> or disabled.
> In the near past it took a considerable amount of my time to adapt test
> cases for some of the features
> introduced in the last time to make them work in all cases.
>
> Regards,
> Martin
>
>
>
> > On 5. Jul 2019, at 06:59, Chetan Mehrotra
> wrote:
> >
> > +1 to commit incrementally to master.
> >
> > If the changes touch the core logic then we can possibly have them
> > backed by feature flags and have them disabled by default and enabled
> > for test case. Further it would be preferable that whatever we commit
> > (at any stage) it should have required test coverage. This would
> > ensures that the sub parts of work in progress bigger feature are
> > backed by unit tests.
> >
> > regards
> > Chetan Mehrotra
> >
> > On Thu, Jul 4, 2019 at 9:35 PM Dominic Kim wrote:
> >>
> >> Hello, whiskers.
> >>
> >> I am trying to contribute this:
> >> https://github.com/apache/incubator-openwhisk/pull/4532
> >>
> >> At first, I tried to open a base branch and merge all incremental PRs
> into
> >> the branch.
> >> And finally mege the base branch into the master after the
> implementation
> >> is done.
> >> Surely, reviewers would review all the subsequent PRs and it will be
> based
> >> on SPI and disabled by default at first, there would be no big issue I
> >> think.
> >>
> >> And Markus suggested to incrementally merge all PRs into the master
> instead
> >> of having a big base branch.
> >>
> https://github.com/apache/incubator-openwhisk/pull/4532#issuecomment-508519119
> >>
> >> With the former, we don't need to include temporal unused codes but
> need to
> >> include a big possibly disruptive PR at once.
> >> With the latter, there will be some temporal unused components in the
> >> master at some point, but we can make sure merged codes do not induce
> any
> >> issue. And actually the latter is easier for me as well : )
> >>
> >> If we all agree with including the temporal unused codes in the master,
> I
> >> am happy to work in the way Markus suggested.
> >>
> >> One example of a temporal code is this:
> >>
> https://github.com/apache/incubator-openwhisk/pull/4532/files#diff-1d7110b32c507a6ef4ac956c287e77ebR24
> >>
> >> Since there is no other component, the "scheduler" cannot initialize all
> >> components in the main function and there is only `println("Hello")` in
> the
> >> initial version of the main function.
> >>
> >>
> >> Please let me know your thoughts.
> >>
> >> Best regards
> >> Dominic
>
>
[slack-digest] [2019-07-04] #general
2019-07-04 03:47:04 UTC - Dominic Kim: In Jenkins slaves, there are few
containers including registry.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562212024196700
2019-07-04 03:47:08 UTC - Dominic Kim: ```
root@openwhisk-vm1-he-de ~ # docker ps
CONTAINER IDIMAGE COMMAND
CREATED STATUS PORTS
NAMES
d029867ba796openwhisk/action-nodejs-v10:nightly "/bin/sh -c 'node
..." 12 hours agoUp 12 hours 8080/tcp
wsk0_23_guest_a11562166966366
7db1ced2d58eopenwhisk/action-nodejs-v10:nightly "/bin/sh -c 'node
..." 12 hours agoUp 12 hours 8080/tcp
wsk0_22_guest_a11562166963567
dca1ac7f32fdopenwhisk/action-nodejs-v10:nightly "/bin/sh -c 'node
..." 12 hours agoUp 12 hours 8080/tcp
wsk0_21_prewarm_nodejs10
e7deca6ef1b1openwhisk/action-nodejs-v10:nightly "/bin/sh -c 'node
..." 12 hours agoUp 12 hours 8080/tcp
wsk0_20_prewarm_nodejs10
c5e7622853ebregistry:2"/entrypoint.sh
/e..." 13 hours agoUp 13 hours 0.0.0.0:444-444/tcp,
5000/tcp registry
```
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562212028196900
2019-07-04 03:47:16 UTC - Dominic Kim: ```
root@openwhisk-vm2-he-de ~ # docker ps
CONTAINER IDIMAGE COMMAND CREATED
STATUS PORTSNAMES
383dceb8292fregistry:2 "/entrypoint.sh /e..." 36 hours ago
Up 36 hours 0.0.0.0:444-444/tcp, 5000/tcp registry
```
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562212036197200
2019-07-04 03:47:34 UTC - Dominic Kim: I am curious whether I can just stop
them and upgrade the docker.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562212054197600
2019-07-04 04:51:32 UTC - Dominic Kim: I would remove all containers.
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562215892198100
2019-07-04 05:10:30 UTC - Dominic Kim: It's done.
It would be great if the system sanity is confirmed by @Vincent Hou.
+1 : Jiang PengCheng
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562217030199200?thread_ts=1562217030.199200=C3TPCAQG1
2019-07-04 09:03:57 UTC - James Thomas: I’m seeing Travis errors this morning….
```
TASK [controller : (re)start controller] ***
Thursday 04 July 2019 08:50:51 + (0:00:00.034) 0:00:44.516 *
fatal: [controller0]: FAILED! = {"changed": false, "msg": "Error pulling
image openwhisk/controller:latest - 404 Client Error: Not Found (\"manifest for
openwhisk/controller:latest not found\")"}
Error pulling image openwhisk/controller:latest - 404 Client Error: Not
Found ("manifest for openwhisk/controller:latest not found")
``` in the openwhisk-client-js repo
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p156223103720
2019-07-04 09:04:16 UTC - James Thomas: I’m guessing this is maybe related to
the work @Dave Grove has been doing on cleaning up the docker tags?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562231056200600
2019-07-04 09:05:21 UTC - James Thomas: seems to be happening in other places
too:
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562231121201200
2019-07-04 09:05:24 UTC - Jiang PengCheng: seems yes, should use the `nightly`
tag
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562231124201300
2019-07-04 13:31:28 UTC - Pepi Paraskevoulakou: hey guys i am new on in the
community, id like to ask about whisk, isn't project of ibm anymore right? as i
saw in the website whisk is incubator project of Apache right?
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562247088203900
2019-07-04 13:33:32 UTC - Pepi Paraskevoulakou: <#C3TPCAQG1|general> just
because i was reading an e-book titled : "Developing Serveless applications - a
practical introduction of Apache Openwhisk- compliments of IBM"
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562247212205400?thread_ts=1562247212.205400=C3TPCAQG1
2019-07-04 13:33:37 UTC - Markus Thömmes: @Pepi Paraskevoulakou that is
correct. Some IBMers contribute to the project but it is under the governance
of the Apache Foundation
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562247217205600
2019-07-04 13:46:54 UTC - Pepi Paraskevoulakou: thank you @Markus Thömmes !
https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1562248014206200
2019-07-04 13:50:02 UTC - mmarth: fyi there is also
Re: Re: OpenWhisk invoker overloads - "Rescheduling Run message"
Hello Dominic, thanks for your detailed response. I guess your understanding is right - just this small correction: > So the main issue here is there are too many "Rescheduling Run" messages in invokers? It's not the main issue to see these log entries in the invoker. This is just the indication that something is going wrong in the invoker - more activations are waiting to be processed than the ContainerPool can currently serve. Actually, there are different reasons why "Rescheduling Run message" log entries can show up in the invoker: 1. Controllers send too many activations to an invoker. 2. In the invoker, the container pool sends a Run message to a container proxy but the container proxy fails to process it properly and hands it back to the container pool. Examples: a Run message arrives while the proxy is already removing the container; if concurrency>1, the proxy buffers Run messages and returns them in failure situations. Although I'm not 100% sure, I see more indications for reason 1 in our logs than for reason 2. Regarding hypothesis "#controllers * getInvokerSlot(invoker user memory size) > invoker user memory size": I can rule out this hypothesis in our environments. We have "#controllers * getInvokerSlot(invoker user memory size) = invoker user memory size". I provided PR [1] to be sure about that. Regarding hypothesis "invoker simply pulls too many Run messages from MessageFeed". I think the part you described is perfectly right. The questions remains why controllers send too many Run messages or a Run message with an activation that is larger than free memory capacity currently available in the pool. The load balancer has a memory book-keeping for all of its invoker shards (memory size determined by getInvokerSlot()) so the load balancer is supposed to only schedule an activation to an invoker if the required memory does not exceed controller's shard of the invoker. Even if resulting Run messages arrive on the invoker in a changed order, the invoker's shard free memory should be sufficient. Do you see a considerable number of "Rescheduling Run message" log entries in your environments? [1] https://github.com/apache/incubator-openwhisk/pull/4520 Mit freundlichen Grüßen / Regards, Sven Lange-Last Senior Software Engineer IBM Cloud Functions Apache OpenWhisk E-mail: sven.lange-l...@de.ibm.com Find me on: Schoenaicher Str. 220 Boeblingen, 71032 Germany IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Re: Allow temporal unused codes to include a big change.
Strongly + 1 to put core changes behind feature flags so that they can be
rolled out (and removed if necessary )
in a controlled way.
Tests should be running successfully for having the feature flag enabled or
disabled.
In the near past it took a considerable amount of my time to adapt test cases
for some of the features
introduced in the last time to make them work in all cases.
Regards,
Martin
> On 5. Jul 2019, at 06:59, Chetan Mehrotra wrote:
>
> +1 to commit incrementally to master.
>
> If the changes touch the core logic then we can possibly have them
> backed by feature flags and have them disabled by default and enabled
> for test case. Further it would be preferable that whatever we commit
> (at any stage) it should have required test coverage. This would
> ensures that the sub parts of work in progress bigger feature are
> backed by unit tests.
>
> regards
> Chetan Mehrotra
>
> On Thu, Jul 4, 2019 at 9:35 PM Dominic Kim wrote:
>>
>> Hello, whiskers.
>>
>> I am trying to contribute this:
>> https://github.com/apache/incubator-openwhisk/pull/4532
>>
>> At first, I tried to open a base branch and merge all incremental PRs into
>> the branch.
>> And finally mege the base branch into the master after the implementation
>> is done.
>> Surely, reviewers would review all the subsequent PRs and it will be based
>> on SPI and disabled by default at first, there would be no big issue I
>> think.
>>
>> And Markus suggested to incrementally merge all PRs into the master instead
>> of having a big base branch.
>> https://github.com/apache/incubator-openwhisk/pull/4532#issuecomment-508519119
>>
>> With the former, we don't need to include temporal unused codes but need to
>> include a big possibly disruptive PR at once.
>> With the latter, there will be some temporal unused components in the
>> master at some point, but we can make sure merged codes do not induce any
>> issue. And actually the latter is easier for me as well : )
>>
>> If we all agree with including the temporal unused codes in the master, I
>> am happy to work in the way Markus suggested.
>>
>> One example of a temporal code is this:
>> https://github.com/apache/incubator-openwhisk/pull/4532/files#diff-1d7110b32c507a6ef4ac956c287e77ebR24
>>
>> Since there is no other component, the "scheduler" cannot initialize all
>> components in the main function and there is only `println("Hello")` in the
>> initial version of the main function.
>>
>>
>> Please let me know your thoughts.
>>
>> Best regards
>> Dominic
