Re: Free Travis-CI will go away for open source projects
Dave, that sounds like we do not have the problem I was anticipating when reading the news. Thank you for taking care. Regards,. Martin > On 29. Nov 2020, at 18:03, David P Grove wrote: > > > > > Carlos Santana wrote on 11/20/2020 10:23:37 AM: >> >> Our current usage of Travis for OpenWhisk we use the ASF foundation >> account, and Infra pays some amount $ to able to support so many builds > by >> many Apache projects. >> > > I just read through the thread on builds@a.o on Travis migration. > > TL;DR - ASF is migrating projects from .org to .com and continuing to pay > for additional build resources for ASF projects. Some of our repos are > already migrated to travis-ci.com. We just need to figure out which of our > active repos aren't migrated yet and ask infra to migrate them. Then we > can continue as is. > > --dave
Free Travis-CI will go away for open source projects
Hi, free Travis usage will be ending for open source projects end of the year. See: https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing Open source projects will migrated to trial accounts in travis-ci.com with some free budget. > For those of you who have been building on public repositories (on > travis-ci.com, with no paid subscription), we will upgrade you to our trial > (free) >plan with a 10K credit allotment (which allows around 1000 minutes in > a Linux environment). It looks like our OW projects have to find other alternatives like GitHub Actions. Kind regards, Martin
Re: JDK 11
Rodric, UUID creation was just an example from the past where workarounds were needed to circumvent waiting for enough entropy for the random numbers used for UUID creation. As said we did no further investigation. Yes. OpenJ9 is Apache licensed and distributed via AdoptOpenJDK. There are no further changes needed. Regards, Martin > Am 28.01.2020 um 18:09 schrieb Rodric Rabbah : > > Thanks Martin for sharing the results with the community - That's a steep > performance degradation. > Open J9 is Apache licensed. Are other changes needed? > > You hinted at uuid - do you know that's implicated or a guess? > > -r > >> On Tue, Jan 28, 2020 at 8:05 AM Martin Henke wrote: >> >> Folks, >> >> as indicated, we performed performance tests of the OpenJDK upgrade PR >> https://github.com/apache/openwhisk/pull/4706 >> in our performance test environment (we are using the exact set of Gatling >> tests that are contributed to Openwhisk). >> >> Though our first impressions looked good, we found severe degradations in >> the latency and warm invocation tests afterwards. >> >> Latency tests dropped to around 50% of the JDK8 results, warm invocation >> to a staggering 25%. >> These results were stable with using the original and the updated Gatling >> image and switching to parallelGC on OpenJDK11. >> >> Since GC does not seem to matter, the degradation might be caused by a >> changed >> implementation in the OpenJDK 11 (e.g. UUID generation). >> We did not perform any deeper research. >> >> Next we tried to change to use the OpenJ9 JDK11 instead of OpenJDK. >> Change in common/scala/Dockerfile: >> " >> -FROM adoptopenjdk/openjdk11:x86_64-alpine-jdk-11.0.3_7 >> +FROM >> adoptopenjdk/openjdk11-openj9:x86_64-alpine-jdk-11.0.5_10_openj9-0.17.0 >> " >> With OpenJ9 in default configuration all performance metrics were looking >> good again. >> We did not see any degredation anymore. >> >> In the light of these test results, I herewith would like to start the >> discussion to base the JDK update PR >> to openJ9 JDK11. >> >> Kind Regards, >> Martin >>
Re: JDK 11
Folks, as indicated, we performed performance tests of the OpenJDK upgrade PR https://github.com/apache/openwhisk/pull/4706 in our performance test environment (we are using the exact set of Gatling tests that are contributed to Openwhisk). Though our first impressions looked good, we found severe degradations in the latency and warm invocation tests afterwards. Latency tests dropped to around 50% of the JDK8 results, warm invocation to a staggering 25%. These results were stable with using the original and the updated Gatling image and switching to parallelGC on OpenJDK11. Since GC does not seem to matter, the degradation might be caused by a changed implementation in the OpenJDK 11 (e.g. UUID generation). We did not perform any deeper research. Next we tried to change to use the OpenJ9 JDK11 instead of OpenJDK. Change in common/scala/Dockerfile: " -FROM adoptopenjdk/openjdk11:x86_64-alpine-jdk-11.0.3_7 +FROM adoptopenjdk/openjdk11-openj9:x86_64-alpine-jdk-11.0.5_10_openj9-0.17.0 " With OpenJ9 in default configuration all performance metrics were looking good again. We did not see any degredation anymore. In the light of these test results, I herewith would like to start the discussion to base the JDK update PR to openJ9 JDK11. Kind Regards, Martin On 2019/12/23 01:05:20, Rodric Rabbah wrote: > Thanks Martin. What I’m most interested in is knowing what other than the > Gatling tests which we are running in Travis and can run in Jenkins if > anything you’re doing differently so that the project can replicate and > automate it where necessary. > > > -r> > > > On Dec 22, 2019, at 2:34 PM, Martin Henke wrote:> > > > > > Rodric,> > > > > > I forgot to mention. When we find a degradation (and I honestly do not hope > > to see one) and we find the reason, we will of course share the information > > needed to setup tests in the Apache environment to catch those. > > > > > > Regards,> > > Martin> > > > > >> Am 22.12.2019 um 20:29 schrieb Martin Henke :> > >> > > >> Rodric,> > >> > > >> It seems to be natural that the IBM environment differs in machine > >> configuration and network bandwidth and any other parameters from any > >> other OW installation.> > >> > > >> Nevertheless I see value for the project to test the JDK upgrade > >> additionally in our env to identify potential problems. As said, we saw > >> performance degredations before, which were not caught in the Apache > >> environments.> > >> > > >> Regards,> > >> Martin> > >> > > >> P.S I am away from keyboard for the next time and might be slow in > >> answering.> > >> > > >>>> Am 22.12.2019 um 19:33 schrieb Rodric Rabbah :> > >>> > > >>> Thanks Martin. Can you share the environment details/how it’s different > >>> from the Jenkins job Vincent set up at one point for the project - so > >>> that any testing important to the project can be done in apache land. > > >>> > > >>> -r> > >>> > > >>>> On Dec 22, 2019, at 9:35 AM, Martin Henke wrote:> > >>>> > > >>>> Rodric,> > >>>> > > >>>> there are no special tests beside the ones that were already contributed > >>>> by Christian from our team. > > >>>> > > >>>> The unknowns are given by our environment. > > >>>> > > >>>> Regards,> > >>>> Martin> > >>>> > > >>>> Von meinem iPhone gesendet> > >>>> > > >>>>> Am 21.12.2019 um 12:45 schrieb Rodric Rabbah :> > >>>>> > > >>>>> Martin thanks for the input. Do you or others plan to contribute any > >>>>> of the> > >>>>> performance testing that you're doing/will do to the project - and > >>>>> what> > >>>>> would it take to facilitate this so that we are also increasing the > >>>>> project> > >>>>> velocity?> > >>>>> > > >>>>> -r> > >>>>> > > >>>>>>> On Sat, Dec 21, 2019 at 5:08 AM Martin Henke wrote:> > >>>>>> > > >>>>>> Rodric,> > >>>>>> > > >>>>>> thank you for taking on this dangling task.> >
Re: JDK 11
Rodric, I forgot to mention. When we find a degradation (and I honestly do not hope to see one) and we find the reason, we will of course share the information needed to setup tests in the Apache environment to catch those. Regards, Martin > Am 22.12.2019 um 20:29 schrieb Martin Henke : > > Rodric, > > It seems to be natural that the IBM environment differs in machine > configuration and network bandwidth and any other parameters from any other > OW installation. > > Nevertheless I see value for the project to test the JDK upgrade additionally > in our env to identify potential problems. As said, we saw performance > degredations before, which were not caught in the Apache environments. > > Regards, > Martin > > P.S I am away from keyboard for the next time and might be slow in answering. > >> Am 22.12.2019 um 19:33 schrieb Rodric Rabbah : >> >> Thanks Martin. Can you share the environment details/how it’s different >> from the Jenkins job Vincent set up at one point for the project - so that >> any testing important to the project can be done in apache land. >> >> -r >> >>> On Dec 22, 2019, at 9:35 AM, Martin Henke wrote: >>> >>> Rodric, >>> >>> there are no special tests beside the ones that were already contributed by >>> Christian from our team. >>> >>> The unknowns are given by our environment. >>> >>> Regards, >>> Martin >>> >>> Von meinem iPhone gesendet >>> >>>> Am 21.12.2019 um 12:45 schrieb Rodric Rabbah : >>>> >>>> Martin thanks for the input. Do you or others plan to contribute any of >>>> the >>>> performance testing that you're doing/will do to the project - and what >>>> would it take to facilitate this so that we are also increasing the project >>>> velocity? >>>> >>>> -r >>>> >>>>>> On Sat, Dec 21, 2019 at 5:08 AM Martin Henke wrote: >>>>> >>>>> Rodric, >>>>> >>>>> thank you for taking on this dangling task. >>>>> >>>>> Since we saw some severe performance impacts when we tried to move OW to >>>>> other Java 8 JDK versions >>>>> before, we would like to run the final change through our performance >>>>> tests. >>>>> >>>>> Therefore I would like to ask for a short deferral until midth of January >>>>> when people have returned >>>>> from their Christmas holidays (you know that we Germans tend to have long >>>>> holidays :-)). >>>>> >>>>> Regards, >>>>> Martin >>>>> >>>>> >>>>>> On 2019/12/20 21:56:12, Rodric Rabbah wrote: >>>>>> This PR https://github.com/apache/openwhisk/pull/4706 move openwhisk >>>>> to> >>>>>> using Java 11 for building and running the controller/invoker >>>>> containers.> >>>>>> It has been open since Oct 30 with no comments since Nov 14. It was >>>>> covered> >>>>>> at the Dec 4 community call> >>>>>> >>>>> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-12-04+OW+Tech+Interchange+Meeting+Notes> >>>>> >>>>>> and there was agreement to merge the PR.> >>>>>> >>>>>> Accordingly, I will merge it next week.> >>>>>> >>>>>> -r> >>>>>> >>> >
Re: JDK 11
Rodric, It seems to be natural that the IBM environment differs in machine configuration and network bandwidth and any other parameters from any other OW installation. Nevertheless I see value for the project to test the JDK upgrade additionally in our env to identify potential problems. As said, we saw performance degredations before, which were not caught in the Apache environments. Regards, Martin P.S I am away from keyboard for the next time and might be slow in answering. > Am 22.12.2019 um 19:33 schrieb Rodric Rabbah : > > Thanks Martin. Can you share the environment details/how it’s different from > the Jenkins job Vincent set up at one point for the project - so that any > testing important to the project can be done in apache land. > > -r > >> On Dec 22, 2019, at 9:35 AM, Martin Henke wrote: >> >> Rodric, >> >> there are no special tests beside the ones that were already contributed by >> Christian from our team. >> >> The unknowns are given by our environment. >> >> Regards, >> Martin >> >> Von meinem iPhone gesendet >> >>> Am 21.12.2019 um 12:45 schrieb Rodric Rabbah : >>> >>> Martin thanks for the input. Do you or others plan to contribute any of the >>> performance testing that you're doing/will do to the project - and what >>> would it take to facilitate this so that we are also increasing the project >>> velocity? >>> >>> -r >>> >>>>> On Sat, Dec 21, 2019 at 5:08 AM Martin Henke wrote: >>>> >>>> Rodric, >>>> >>>> thank you for taking on this dangling task. >>>> >>>> Since we saw some severe performance impacts when we tried to move OW to >>>> other Java 8 JDK versions >>>> before, we would like to run the final change through our performance >>>> tests. >>>> >>>> Therefore I would like to ask for a short deferral until midth of January >>>> when people have returned >>>> from their Christmas holidays (you know that we Germans tend to have long >>>> holidays :-)). >>>> >>>> Regards, >>>> Martin >>>> >>>> >>>>> On 2019/12/20 21:56:12, Rodric Rabbah wrote: >>>>> This PR https://github.com/apache/openwhisk/pull/4706 move openwhisk >>>> to> >>>>> using Java 11 for building and running the controller/invoker >>>> containers.> >>>>> It has been open since Oct 30 with no comments since Nov 14. It was >>>> covered> >>>>> at the Dec 4 community call> >>>>> >>>> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-12-04+OW+Tech+Interchange+Meeting+Notes> >>>> >>>>> and there was agreement to merge the PR.> >>>>> >>>>> Accordingly, I will merge it next week.> >>>>> >>>>> -r> >>>>> >>
Re: JDK 11
Rodric, there are no special tests beside the ones that were already contributed by Christian from our team. The unknowns are given by our environment. Regards, Martin Von meinem iPhone gesendet > Am 21.12.2019 um 12:45 schrieb Rodric Rabbah : > > Martin thanks for the input. Do you or others plan to contribute any of the > performance testing that you're doing/will do to the project - and what > would it take to facilitate this so that we are also increasing the project > velocity? > > -r > >> On Sat, Dec 21, 2019 at 5:08 AM Martin Henke wrote: >> >> Rodric, >> >> thank you for taking on this dangling task. >> >> Since we saw some severe performance impacts when we tried to move OW to >> other Java 8 JDK versions >> before, we would like to run the final change through our performance >> tests. >> >> Therefore I would like to ask for a short deferral until midth of January >> when people have returned >> from their Christmas holidays (you know that we Germans tend to have long >> holidays :-)). >> >> Regards, >> Martin >> >> >>> On 2019/12/20 21:56:12, Rodric Rabbah wrote: >>> This PR https://github.com/apache/openwhisk/pull/4706 move openwhisk >> to> >>> using Java 11 for building and running the controller/invoker >> containers.> >>> It has been open since Oct 30 with no comments since Nov 14. It was >> covered> >>> at the Dec 4 community call> >>> >> https://cwiki.apache.org/confluence/display/OPENWHISK/2019-12-04+OW+Tech+Interchange+Meeting+Notes> >> >>> and there was agreement to merge the PR.> >>> >>> Accordingly, I will merge it next week.> >>> >>> -r> >>>
Re: JDK 11
Rodric, thank you for taking on this dangling task. Since we saw some severe performance impacts when we tried to move OW to other Java 8 JDK versions before, we would like to run the final change through our performance tests. Therefore I would like to ask for a short deferral until midth of January when people have returned from their Christmas holidays (you know that we Germans tend to have long holidays :-)). Regards, Martin On 2019/12/20 21:56:12, Rodric Rabbah wrote: > This PR https://github.com/apache/openwhisk/pull/4706 move openwhisk to> > using Java 11 for building and running the controller/invoker containers.> > It has been open since Oct 30 with no comments since Nov 14. It was covered> > at the Dec 4 community call> > https://cwiki.apache.org/confluence/display/OPENWHISK/2019-12-04+OW+Tech+Interchange+Meeting+Notes> > > and there was agreement to merge the PR.> > > Accordingly, I will merge it next week.> > > -r> >
Re: removing "projections" on web actions
Rodric, Carlos, I am fine too with the cut of date on December 1st. This will allow us to deprecate this feature in a coordinated way. Thanks for your help, Martin > Am 31.08.2019 um 15:29 schrieb Carlos Santana : > > > I was never a fan of projections, very clever trick but the idea that an the > path to an endpoint was controlling a filter for the http response gives me > the heebie-jeebies :-) > > On the other side feels like a form of graphQL > > From experience don’t do the feature flag, and don’t try to implement in > another way. > > If you want to invest effort instead show an example/lib of using whisk > actions as front workers for graphQL might be more useful similar to > Cloudflare [1] when talking about projections and filtering responses. > > I like the idea of a cut off date. Do a /remind in Slack or a Whisk alarm > trigger :-) > > [1] https://developers.cloudflare.com/workers/templates/boilerplates/graphql/ > > - Carlos Santana > @csantanapr > > On Aug 30, 2019, at 9:53 AM, Rodric Rabbah wrote: > >>> Assuming no one wants to bother with a feature flag for a deprecated >> feature, let's set a concrete date on which the PR will be merged. My >> suggestion is December 1, 2019. That gives ample time for downstream >> operators to do their deprecation process. >> >> I don't want to bother with a feature flag. I think the feature as >> implemented is unwieldy and kudos to those that did get it to work. Fine >> with me to defer it's already been open 4 months. >> >> That said, I have thought that we could provide projection capabilities but >> using X- headers (an argument for keeping some of the code around). I am >> not sure/convinced this feature in general though is useful. >> >> -r
Re: removing "projections" on web actions
Rodric, Chetan, Carlos, after checking our (IBM Functions) production logs we see minor usage of the projection feature (.text and .json). That means that we (IBM Functions) need to notify these customers to make them adapt their action code and/or URLs to the removed feature. I would like therefore ask either to guard the code removal with a feature flag, or to delay the projection code removal (best for another 3-4 month). Regards, Martin P.S: Interesting side note: Though projections on “.http” got removed two years ago, we confusingly saw A LOT calls of the form ".http/foo/bar”. Research showed that those customers are accessing the “foo/bar” section of the path via the "message.__ow_path" property in their action code to be used as additional parameters. > On 23. Aug 2019, at 18:04, Rodric Rabbah wrote: > > This PR https://github.com/apache/openwhisk/pull/4592 removes the > documentation for projecting results of actions via the web invoke path. > > This was previously discussed > https://github.com/apache/openwhisk/issues/4352 and there is a pending PR > to also remove the functionality but I thought removing the documentation > is a good first step that should not be controversial. > > This feature is cumbersome to use in practice and for meaningful > applications. > Please comment on the PR if you have input otherwise will progress by > silent assent. > > -r
Re: Passing TransactionId as part of action invocation
Chetan, from an operational point of view I have some fear that we will confuse the user by making the transaction id visible as a second id besides the activation id. Some will certainly use it to fetch activation records and fail, which will lead to questions. Any thoughts from your side ? Regards, Martin > On 20. Aug 2019, at 12:32, Chetan Mehrotra wrote: > > I created a separate thread to discuss how to store such metadata related > to activation. > > Current open PR #4586 only enables exposing the transactionId to env. It > does not make any attempt to store the transactionId currently. Once we > decide how such data should be stored then I can open PR for the same > > Chetan Mehrotra > > > On Mon, Aug 19, 2019 at 8:47 AM Rodric Rabbah wrote: > >> Yes indeed. Your pr already open I think is fine as is. >> >> -r >> >> On Aug 19, 2019, at 11:36 AM, Chetan Mehrotra >> wrote: >> That’s true. Time for api/v2... >>> >>> This is now becoming a rabbit hole! What option should we use without >> going >>> for v2? >>> >>> 1. Introduce a new "meta" sub document >>> 2. OR Change annotations to flat map while storing but transform that to >>> array based structure while returning to client >>> >>> Chetan Mehrotra >>> >>> On Mon, Aug 19, 2019 at 7:15 AM Rodric Rabbah wrote: > However changing them now would cause compatibility > issue with various tooling out there which may be interpreting the > annotation per current design That’s true. Time for api/v2... 😅 >>
Re: Using Rust for the KnativeWhisk controller
Michele, Two thoughts: 1) For writing a controller in Knative I recommend to choose Go instead of Rust (even when I like Rust more). With Go you can leverage the fantastic Operator SDK from Redhat which makes writing controllers fairly simple (I had my first one up and running in under an hour). Link: https://github.com/operator-framework/operator-sdk The getting started guide is a good starting point: https://github.com/operator-framework/getting-started It also addresses the lifecycle of an controller with the Lifecycle Manager: https://github.com/operator-framework/operator-lifecycle-manager (I have not yet used this myself) 2) I think we should clearly separate the Knative work from Openwhisk and stay there with a strict Scala only policy (with the existing exceptions). Introducing more languages would in my opinion lead to maintenance problems and the waste of build up skills. Regards, Martin > On 15. Jul 2019, at 11:58, Michele Sciabarra wrote: > > Hello all, > > In my efforts to work a Kanative Whisk, I reached the point where I have a > kit with tekton-pipelines, knatve-serving building an actionlooop based > runtime. Now I need to implement a controller, in order to use the existing > wsk tooling. > > I know there is a prototype kwsk implementation made by redhat, written in > Go but looks like it is obsolete and unfinished, and to the best of my > knowledge, abandoned. > > I would like to resume the effort of writing an updated controller. I > actually already have a prototype using the Julia language. Julia is really > awesome, Python simplicity and Go speed, but I feed the community would > disagree on using Julia. Of course if I am wrong... let me know because that > would be my preferred choice. > > However, I feel that, given our Scala background, Rust would be a much > better choice for the KnativeWhisk controller. So I propose to use Rust for > the KwhiskController. > > What does the community think of the proposal? > > > -- > Michele Sciabarra > mich...@sciabarra.com
Re: Fine-grained permission
Dominic, I would very much like to have this change introduced in a backward compatible fashion. In Detail: The default permissions should either keep the same behaviour as with the current code or better be configurable via Ansible in a way that he system behaves the same as before. If you decide for a schema change, the code should be handle existing actions with the existing behaviour. With being backward compatible there would be no need to introduce this change in a new EntitlementProvider implementation but can (and should) be placed in the (default) LocalEntitlement provider to have consistent code in the default configuration for Action creation (adding the access rights) and entitlement checking (reading the access rights). Thank you for driving this forward., Martin > On 4. Jul 2019, at 17:12, Dominic Kim wrote: > > Thank you for the feedback, James. > > I am thinking of implementing it with annotations. > For example, we would add a new annotation reserved by the system. > When an action is created, the default permission will be specified via the > annotation. > > When any request comes, a new EntitlementProvider will look for the > annotation and reject the request based on it. > > This is to minimize the action schema change. > But if it's ok to bear with the schema change, I am ok with adding one more > field in an action other than annotations or parameters. > > 2019년 7월 4일 (목) 오후 10:07, James Thomas 님이 작성: > >> Protecting accidental overwritting or deletion of actions would be a great >> feature. I like the suggestion and approach of using Unix-style >> permissions. >> >> On Thu, 4 Jul 2019 at 07:25, Dominic Kim wrote: >> >>> Recently I discussed this: >>> https://github.com/apache/incubator-openwhisk/pull/4058 with my >>> colleagues. >>> That PR is to add a feature to protect actions from deletion by mistake. >>> That is a good suggestion and I think we can also include more >> generalized >>> way to handle the issue. >>> >>> For example, what we can expect about permission are as follows. >>> >>> 1. Action protection. >>> 2. Hide codes from the shared package. >>> >>> I am a bit faint but IIRC, Rodric suggested linux-like permission >>> management. >>> >>> Regarding number 1, we can achieve it with the permission, "Read / not >>> Write / Execute". >>> And with regared to number 2, we can also achieve it with the permission, >>> "not Read / not Write (this is the default of shared package action) / >>> Execute". >>> >>> If we apply linux-like permission to these cases, we can have two >> different >>> permission flags, one for owners, the other for users of shared packages. >>> Then actions can have permission information such as "71" or "51". >>> So "71" would mean the owner of an action can do "read/write/execute" it >>> but the one who uses the shared action would be able to do "not read/not >>> write/execute". >>> "51" would mean the owner can do "read/not write/execute". >>> >>> There might be more cases, but I believe we can deal with them in the >> same >>> way. >>> Any feedback or idea on this would be appreciated. >>> >>> Thanks >>> Best regards, >>> Dominic >>> >> >> >> -- >> Regards, >> James Thomas >>
Re: Allow temporal unused codes to include a big change.
Strongly + 1 to put core changes behind feature flags so that they can be
rolled out (and removed if necessary )
in a controlled way.
Tests should be running successfully for having the feature flag enabled or
disabled.
In the near past it took a considerable amount of my time to adapt test cases
for some of the features
introduced in the last time to make them work in all cases.
Regards,
Martin
> On 5. Jul 2019, at 06:59, Chetan Mehrotra wrote:
>
> +1 to commit incrementally to master.
>
> If the changes touch the core logic then we can possibly have them
> backed by feature flags and have them disabled by default and enabled
> for test case. Further it would be preferable that whatever we commit
> (at any stage) it should have required test coverage. This would
> ensures that the sub parts of work in progress bigger feature are
> backed by unit tests.
>
> regards
> Chetan Mehrotra
>
> On Thu, Jul 4, 2019 at 9:35 PM Dominic Kim wrote:
>>
>> Hello, whiskers.
>>
>> I am trying to contribute this:
>> https://github.com/apache/incubator-openwhisk/pull/4532
>>
>> At first, I tried to open a base branch and merge all incremental PRs into
>> the branch.
>> And finally mege the base branch into the master after the implementation
>> is done.
>> Surely, reviewers would review all the subsequent PRs and it will be based
>> on SPI and disabled by default at first, there would be no big issue I
>> think.
>>
>> And Markus suggested to incrementally merge all PRs into the master instead
>> of having a big base branch.
>> https://github.com/apache/incubator-openwhisk/pull/4532#issuecomment-508519119
>>
>> With the former, we don't need to include temporal unused codes but need to
>> include a big possibly disruptive PR at once.
>> With the latter, there will be some temporal unused components in the
>> master at some point, but we can make sure merged codes do not induce any
>> issue. And actually the latter is easier for me as well : )
>>
>> If we all agree with including the temporal unused codes in the master, I
>> am happy to work in the way Markus suggested.
>>
>> One example of a temporal code is this:
>> https://github.com/apache/incubator-openwhisk/pull/4532/files#diff-1d7110b32c507a6ef4ac956c287e77ebR24
>>
>> Since there is no other component, the "scheduler" cannot initialize all
>> components in the main function and there is only `println("Hello")` in the
>> initial version of the main function.
>>
>>
>> Please let me know your thoughts.
>>
>> Best regards
>> Dominic
Re: Re: A plan to (re) implement OpenWhisk on top of Knative
Hi Matt, I fully agree with your thoughts. We should definitely (stepwise) aim for an unified model for all runtimes. Regards, Martin > On 3. Jul 2019, at 16:43, Matt Rutkowski wrote: > > Hi Martin, > > It is my belief that Michele, now freshly returned from book editing > (congrats), was going to implement the same interface for pre/post > processing requests that we implemented for NodeJS. This would allow us a > path to support this across all language runtimes as well as formalize our > runtime contract that aligns with a Knative Service approach and then also > allows us to better explore some of the use cases being discussed on > another thread from Rodric. I believe that we should look at smaller > steps towards that alignment like removing the need for the "init" > entrypoint and allowing access to parameters either by env. vars. or > function arguments allowing both compat. with 12-factor app approach, as > well as attempting to encourage a functional programming model where you > should ideally be unaware of any system environment. > > Kind regards, > Matt > > > > > From: Martin Henke > To: dev@openwhisk.apache.org > Date: 07/03/2019 09:29 AM > Subject:[EXTERNAL] Re: A plan to (re) implement OpenWhisk on top > of Knative > > > > Michele, > > FYI: Sugandha and myself published a Medium blog article which describes > to build Nodejs10 images using Tekton and run it on Knative > (based on the work from Priti). > It might be on interest in the context of your Actionloop related Knative > work. > > Link: > https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40sugandha.agrawal18_build-2Dknative-2Dservice-2Dwith-2Dtekton-2Dand-2Dapache-2Dopenwhisk-2Dnode-2Djs-2Druntime-2Df660bbc3a11e&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=bl7bwPaoiSE6fi0fyVHNC9bNUnh1ZUUlfl3lwUZbcH0&s=IQBGPfSTPiAjKhIlsqKDpAjoJBNd7By1YnjxOIx2454&e= > > > > Regards, > Martin > > > On 2019/05/20 15:00:02, "Michele Sciabarra" wrote: >> Ok great, I see the discussion is starting to bring ideas.> >> >> Yes my goal is basically to run existing actions in Knative, create and > invoke. And possibile retain the ability of an action to invoke another > action.> >> >> I understand the different way they expose services, so I am rethinking > the idea of using a "work-alike" path. > >> >> If it is needed we can add it with an ingress but it may be not > necessary in the initial implementation.> >> >> Also I checked a bit ML and discussions and I see this Tekton thing that > should be the preferred way.> >> >> Not sure if I understand the relation with the current Build API > documented in the website. Is Tekton "compatible" or it has a different > API?> >> >> >> -- > >> Michele Sciabarra> >> mich...@sciabarra.com> >> >> - Original message -> >> From: "Markus Thömmes" > >> To: dev@openwhisk.apache.org> >> Subject: Re: A plan to (re) implement OpenWhisk on top of Knative> >> Date: Monday, May 20, 2019 4:50 PM> >> >> Good discussion, thanks!> >> >> Can we try to define what the desired end-goal is here? I'm a bit > unclear> >> what resembling the OpenWhisk API actually buys us.> >> >> To me, the desired end-state would be to run OpenWhisk actions as-is on > a> >> Knative cluster (similar to OpenFaaS' and Azure's integration). There's > no> >> good way for us to provide the full API without spinning up a control > plane> >> and we can only handle so much via the CLI. So to me, the end-goal > looks> >> like:> >> >> 1. *wsk action create* actually doing all the pieces necessary to run a> > >> piece of code on Knative.> >> 2. *wsk action invoke* doing some HTTP call under the hood to "invoke" > that> >> action. The action should be reachable via a sensible URL. If we really> > >> want to keep the API surface (as I said, I'm dubious here) we can also > do> >> that via ingress level abstractions (like VirtualService).> >> >> Cheers,> >> Markus> >> >> Am Mo., 20. Mai 2019 um 15:33 Uhr schrieb Martin Henke > >>> :> >> >>>> >>>> On 20. May 2019, at 14:55, Michele Sciabarra > >>> wrote:> >>>>> >>>>> Michele,> >>>>
Re: A plan to (re) implement OpenWhisk on top of Knative
Michele, FYI: Sugandha and myself published a Medium blog article which describes to build Nodejs10 images using Tekton and run it on Knative (based on the work from Priti). It might be on interest in the context of your Actionloop related Knative work. Link: https://medium.com/@sugandha.agrawal18/build-knative-service-with-tekton-and-apache-openwhisk-node-js-runtime-f660bbc3a11e Regards, Martin On 2019/05/20 15:00:02, "Michele Sciabarra" wrote: > Ok great, I see the discussion is starting to bring ideas.> > > Yes my goal is basically to run existing actions in Knative, create and > invoke. And possibile retain the ability of an action to invoke another > action.> > > I understand the different way they expose services, so I am rethinking the > idea of using a "work-alike" path. > > > If it is needed we can add it with an ingress but it may be not necessary in > the initial implementation.> > > Also I checked a bit ML and discussions and I see this Tekton thing that > should be the preferred way.> > > Not sure if I understand the relation with the current Build API documented > in the website. Is Tekton "compatible" or it has a different API?> > > > -- > > Michele Sciabarra> > mich...@sciabarra.com> > > - Original message -> > From: "Markus Thömmes" > > To: dev@openwhisk.apache.org> > Subject: Re: A plan to (re) implement OpenWhisk on top of Knative> > Date: Monday, May 20, 2019 4:50 PM> > > Good discussion, thanks!> > > Can we try to define what the desired end-goal is here? I'm a bit unclear> > what resembling the OpenWhisk API actually buys us.> > > To me, the desired end-state would be to run OpenWhisk actions as-is on a> > Knative cluster (similar to OpenFaaS' and Azure's integration). There's no> > good way for us to provide the full API without spinning up a control plane> > and we can only handle so much via the CLI. So to me, the end-goal looks> > like:> > > 1. *wsk action create* actually doing all the pieces necessary to run a> > piece of code on Knative.> > 2. *wsk action invoke* doing some HTTP call under the hood to "invoke" that> > action. The action should be reachable via a sensible URL. If we really> > want to keep the API surface (as I said, I'm dubious here) we can also do> > that via ingress level abstractions (like VirtualService).> > > Cheers,> > Markus> > > Am Mo., 20. Mai 2019 um 15:33 Uhr schrieb Martin Henke > >:> > > >> > > > On 20. May 2019, at 14:55, Michele Sciabarra > > > wrote:> > > >> > > >> Michele,> > > >> > > >> I like the idea to make the ActionLoop based runtimes to be runnable on> > > Knative.> > > >>> > > >> My thoughts on this:> > > >> - I second Markus concern to implement the invocation API onto Knative> > > instead of just using Knative service syntax.> > > > Can you elaborate this? I do not understand.> > >> > > Knative service syntax:https:// > > action)>../> > > OW invocation https://> > > /api/v1/namespaces//actions/> > >> > > (I personally so no worth in inventing a distinct API for OW images, but> > > as said I would see that as a valid optional feature)> > >> > > >> > > >> - I would have concerns to make it dependent on Gloo which is kind of a> > > minority choice for Knative load balancing> > > > I do not think it will be hard to setup a test also using Istio, I do> > > not want to be limited to Gloo.> > >> > > I just wanted to prevent that Gloo gets a “official” prerequisite for an> > > “official” OW on Knative flow.> > > It is of course free to you to use what ever you want to do in your> > > prototype.> > >> > > > - In my opinion the goal should be to have some uniform behaviour for> > > ActionLoop based runtimes> > > >> and other ones like the adapted NodeJS runtimes demonstrated by Matt> > > and Priti> > > > As much as I can tell the current implementation is just the building> > > and exposing the "/init" and "/run" but I can be wrong.> > > > The build can be of course reused, so it continues the effort. For the> > > frontend, from the documentation I think Matt wants to add a proxy, while > > I> > > would like to implemeent the "invocati
Re: oauth token verification in the controller
Rodric, IBM Cloud Functions hasn’t implemented OAuth, but JWT based bearer token based authentication based on the IBM Cloud IAM system. The first problem we encountered was that our bearer token did not provide a namespace related context (which is only configured in the IAM system). The only obvious namespace related information is contained in the URI (but unfortunately not always when looking at the _ default). The next problem was the need to construct the Identity object (of which the namespace information is a key component) fully in the authorization code to not break the API handling (which brings us back to the missing namespace information). We overcame the problems by asking the user to provide missing namespace information via additional headers and by retrieving additional authorization related data from proprietiary data stores. There were more problems in the realm to provide the meaningful error messages. Nevertheless our implementation proves that OW is already able to handle JWT based bearer tokens and provide them to be used in outbound calls. Regards, Martin > Am 21.06.2019 um 13:23 schrieb Rodric Rabbah : > > I'm curious if anyone has thought about or implemented an oauth based > authentication mechanism in the controller. I've thought about replacing > the subject authentication with oauth and think it would not be a lot of > work to do although it does have some wider implications. > > -r
Re: Accept NodeJs syntax quirks
Rodric, I am fine with closing #4514 and merging #136. Regards, Martin > Am 18.06.2019 um 18:01 schrieb Rodric Rabbah : > > Hey Martin - the refactoring was not intended to be semantic changing. > Thanks for reporting the bug. I prefer to not accept #4514 and instead > address this by restoring the behavior as noted in #136. I did add a test > in the latter inspired by the code that broke in the former. > > -r > >> On Tue, Jun 18, 2019 at 11:49 AM Martin Henke wrote: >> >> Rodrics latest updates to the nodeJs runtime uncovered that some of our >> NodeJs test actions are using variables in the global scope (by omitting >> the let or var keywords). The related tests are now failing in the Travis >> build. >> >> I opened https://github.com/apache/incubator-openwhisk/pull/4514 to fix >> the syntax of those actions. >> While this PR would fix the build problem, there would be the danger of >> breaking existing customer actions with the runtime changes. >> >> Therefore Rodric additionally opened a PR in the NodeJs runtime to allow >> these syntax quirks again. >> https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/136 >> >> I do suggest to fix the problem by accepting his runtime PR and only >> optionally merge my core PR with the syntax changes >> after we see the problem go away. >> >> Regards, >> Martin
Accept NodeJs syntax quirks
Rodrics latest updates to the nodeJs runtime uncovered that some of our NodeJs test actions are using variables in the global scope (by omitting the let or var keywords). The related tests are now failing in the Travis build. I opened https://github.com/apache/incubator-openwhisk/pull/4514 to fix the syntax of those actions. While this PR would fix the build problem, there would be the danger of breaking existing customer actions with the runtime changes. Therefore Rodric additionally opened a PR in the NodeJs runtime to allow these syntax quirks again. https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/136 I do suggest to fix the problem by accepting his runtime PR and only optionally merge my core PR with the syntax changes after we see the problem go away. Regards, Martin
Re: A plan to (re) implement OpenWhisk on top of Knative
Tekton is different but very very similar. You get a lot of DejaVus. No big learning Curve. Martin > On 20. May 2019, at 17:00, Michele Sciabarra wrote: > > Ok great, I see the discussion is starting to bring ideas. > > Yes my goal is basically to run existing actions in Knative, create and > invoke. And possibile retain the ability of an action to invoke another > action. > > I understand the different way they expose services, so I am rethinking the > idea of using a "work-alike" path. > > If it is needed we can add it with an ingress but it may be not necessary in > the initial implementation. > > Also I checked a bit ML and discussions and I see this Tekton thing that > should be the preferred way. > > Not sure if I understand the relation with the current Build API documented > in the website. Is Tekton "compatible" or it has a different API? > > > -- > Michele Sciabarra > mich...@sciabarra.com > > - Original message - > From: "Markus Thömmes" > To: dev@openwhisk.apache.org > Subject: Re: A plan to (re) implement OpenWhisk on top of Knative > Date: Monday, May 20, 2019 4:50 PM > > Good discussion, thanks! > > Can we try to define what the desired end-goal is here? I'm a bit unclear > what resembling the OpenWhisk API actually buys us. > > To me, the desired end-state would be to run OpenWhisk actions as-is on a > Knative cluster (similar to OpenFaaS' and Azure's integration). There's no > good way for us to provide the full API without spinning up a control plane > and we can only handle so much via the CLI. So to me, the end-goal looks > like: > > 1. *wsk action create* actually doing all the pieces necessary to run a > piece of code on Knative. > 2. *wsk action invoke* doing some HTTP call under the hood to "invoke" that > action. The action should be reachable via a sensible URL. If we really > want to keep the API surface (as I said, I'm dubious here) we can also do > that via ingress level abstractions (like VirtualService). > > Cheers, > Markus > > Am Mo., 20. Mai 2019 um 15:33 Uhr schrieb Martin Henke > : > >> >>> On 20. May 2019, at 14:55, Michele Sciabarra >> wrote: >>> >>>> Michele, >>> >>>> I like the idea to make the ActionLoop based runtimes to be runnable on >> Knative. >>>> >>>> My thoughts on this: >>>> - I second Markus concern to implement the invocation API onto Knative >> instead of just using Knative service syntax. >>> Can you elaborate this? I do not understand. >> >> Knative service syntax:https://> action)>../ >> OW invocation https:// >> /api/v1/namespaces//actions/ >> >> (I personally so no worth in inventing a distinct API for OW images, but >> as said I would see that as a valid optional feature) >> >>> >>>> - I would have concerns to make it dependent on Gloo which is kind of a >> minority choice for Knative load balancing >>> I do not think it will be hard to setup a test also using Istio, I do >> not want to be limited to Gloo. >> >> I just wanted to prevent that Gloo gets a “official” prerequisite for an >> “official” OW on Knative flow. >> It is of course free to you to use what ever you want to do in your >> prototype. >> >>> - In my opinion the goal should be to have some uniform behaviour for >> ActionLoop based runtimes >>>> and other ones like the adapted NodeJS runtimes demonstrated by Matt >> and Priti >>> As much as I can tell the current implementation is just the building >> and exposing the "/init" and "/run" but I can be wrong. >>> The build can be of course reused, so it continues the effort. For the >> frontend, from the documentation I think Matt wants to add a proxy, while I >> would like to implemeent the "invocation" straight in the runtime. This is >> open to discussion, but of course it is better to reach an agreement. >> >> Also in the work of Priti and Matt the invocation goes directly to the >> runtime. The action code is either passed with the call (not yet tested by >> me) or set via environment variable in the docker build. >> >>> >>>> - As Knative Build seems be on a dead end I would propose to target >> Tekton as the build system (which developed as kind of >successor out of >> Knative) >>> >>> If Knative build is dead then it would be a bit unfair that they change >> it as the scope of the Knative project! >
Re: A plan to (re) implement OpenWhisk on top of Knative
> On 20. May 2019, at 14:55, Michele Sciabarra wrote: > >> Michele, > >> I like the idea to make the ActionLoop based runtimes to be runnable on >> Knative. >> >> My thoughts on this: >> - I second Markus concern to implement the invocation API onto Knative >> instead of just using Knative service syntax. > Can you elaborate this? I do not understand. Knative service syntax:https://../ OW invocation https:///api/v1/namespaces//actions/ (I personally so no worth in inventing a distinct API for OW images, but as said I would see that as a valid optional feature) > >> - I would have concerns to make it dependent on Gloo which is kind of a >> minority choice for Knative load balancing > I do not think it will be hard to setup a test also using Istio, I do not > want to be limited to Gloo. I just wanted to prevent that Gloo gets a “official” prerequisite for an “official” OW on Knative flow. It is of course free to you to use what ever you want to do in your prototype. > - In my opinion the goal should be to have some uniform behaviour for > ActionLoop based runtimes >> and other ones like the adapted NodeJS runtimes demonstrated by Matt and >> Priti > As much as I can tell the current implementation is just the building and > exposing the "/init" and "/run" but I can be wrong. > The build can be of course reused, so it continues the effort. For the > frontend, from the documentation I think Matt wants to add a proxy, while I > would like to implemeent the "invocation" straight in the runtime. This is > open to discussion, but of course it is better to reach an agreement. Also in the work of Priti and Matt the invocation goes directly to the runtime. The action code is either passed with the call (not yet tested by me) or set via environment variable in the docker build. > >> - As Knative Build seems be on a dead end I would propose to target Tekton >> as the build system (which developed as kind of >successor out of Knative) > > If Knative build is dead then it would be a bit unfair that they change it as > the scope of the Knative project! > It looks like the goal is to setup some standards! And I would be very > disappointed to know that. Tekton evolved out of Knative Build (or more correct out of Knative Pipelines) but is very similar to the Knative build. Flows can easily be ported from one to the other, If we target Tekton build we would target the platform were the Knative build team is focusing on. But again feel free to use whatever platform for your prototype work. > At this stage the build is the more interesting thing, and it could be even > imported in main openwhisk to speed up deployment. > I have already baked it in the ActionLoop runtimes (with precompilation). > Also if we use Tekton, where is the Knative standard then? What is the point? > We can build our own system instead of "Knativizing" it... > >> Maybe it would be a good solution to tackle two things independently. >> 1) Design and implement a common protocol of building, running and calling >> OW runtimes on Knative >> 2) Implement the OW invocation API on top of Knative as an additional option >> for those who have the need to expose it. > > On this, for my personal approach at building things, I want something that > works and it is complete and useful. A "MVP”. Cool. Just go on. > So I do not plan to split the effort. Version 0.1 must be a minimal working > subset of OpenWhisk on Knative. > Because otherwise there will be incomplete useless inusable pieces around > (see for example kwsk). > > It does not mean that things cannot be modular, nor that everyone must but to > me "openwhisk-knative" must be a single repo with all the pieces to make > something where you can download is and deploy in a kubernetes cluster and be > able to deploy simple actions. When this works, we can improve incrementally > and split it but keeping it working. > >> I would looking forward to work with you on the first work item. > Great but I see now more details to discuss before we can start. Most > notably I need to understand how I can build on top of Mark and Priti work > and continue their work. ANd I can even probably recover some of the code of > kwsk as they implemented some openwhisk api, that I want now in the runtime. > I do not want to stop you in any way. My hope is that the action loop runtimes and the “other ones” do expose the same behaviour when being called. So that the users is not surprised when calling different actions in different languages. And behaving the same way might also mean to adapt the “other languages” to the same behaviour as the action loop based ones. They just should be uniform to be used. When your prototype is accessible it would be a good point of time to discuss this. As said I very much like your effort. > >> On 20. May 2019, at 08:55, Michele Sciabarra wrote: >> >> I have an idea for implementing a prototype of OpenWhisk o
Re: A plan to (re) implement OpenWhisk on top of Knative
Bertrand, I am not directly involved in Knative Build or Tekton so I might be wrong on my dead end sentence. For what I got ,I think It was realised that a build pipeline on Kube and the task to build docker containers from a given source is a general problem to solve and in most ways independent from Knative. The good side on this that step is that Jenkins X (the next version of Jenkins) is embracing Tekton already as a runtime. So some would probably get a decent Build UI. Regards, Martin > On 20. May 2019, at 14:17, Bertrand Delacretaz wrote: > > On Mon, May 20, 2019 at 2:07 PM Martin Henke wrote: >> ...As Knative Build seems be on a dead end... > > Wow, already? That stuff seems to be competing with JavaScript > frameworks in terms of short lifetimes these days ;-) > > -Bertrand
Re: A plan to (re) implement OpenWhisk on top of Knative
Michele, I like the idea to make the ActionLoop based runtimes to be runnable on Knative. My thoughts on this: - I second Markus concern to implement the invocation API onto Knative instead of just using Knative service syntax. - I would have concerns to make it dependent on Gloo which is kind of a minority choice for Knative load balancing - In my opinion the goal should be to have some uniform behaviour for ActionLoop based runtimes and other ones like the adapted NodeJS runtimes demonstrated by Matt and Priti - As Knative Build seems be on a dead end I would propose to target Tekton as the build system (which developed as kind of successor out of Knative) Maybe it would be a good solution to tackle two things independently. 1) Design and implement a common protocol of building, running and calling OW runtimes on Knative 2) Implement the OW invocation API on top of Knative as an additional option for those who have the need to expose it. I would looking forward to work with you on the first work item. Regards, Martin > On 20. May 2019, at 08:55, Michele Sciabarra wrote: > > >>> I have an idea for implementing a prototype of OpenWhisk on top of Knative. >>> My basic ideas are: do not use any proxy, forwarding or adapter: extend >>> the runtime to support the REST call and expose them as ingress. And use a >>> wrapper on top of `kubectl` to generate all the needed components. > >> Does this tie into the work that Matt was doing to the runtimes to make >> them runnable on Knative? Is this lined up with that at all? > Actually yes. He suggested I can investigate how to migrate ActionLoop to > port many other languages to Knative. > Also he recommended I add my idea and this is what I am doing. Current code > is, if I am not wrong, a Knative build of the nodejs runtime. > > There has been a number of attempts and proposal to move forward OpenWhisk. > My idea that to succeed we need something small but that just works. This is > my idea to be able to implement in the shorter time frame possible an actual > subset of OpenWhisk that works and it is truly built on top of Knative. So I > am putting the thing a bit further than Matt work. > > >>> My goal is to have a functional work-alike of OpenWhisk built on top of >>> Knative, using ActionLoop as a foundation. I will extend ActionLoop to >>> support the required REST calls of OpenWhisk. >> >>> I also want to create tool, I will call `wskn`. This tool will initially >>> just a python script, a wrapper on top of `kubectl` as it will generate >>> kubernetes descriptors. >> Why not build this into "wsk" itself? The Azure Functions CLI as an example >> supports multiple deployment types like this in one CLI. > > When it will works, yes, of course. But to start, what I really need is a > prototype that can generate kubernetes descripttors to feed to kubectl, so a > simplee, quick and ditry, separate tool (that I will keep together the > runtime) is all I need for now. > >>> >>> It will support initially just the the action creation and invocation, and >>> only synchronous (blocking) behaviour, as all the request will go straight >>> to the runtimes. Hopefully also a subset of `package` and `activation`. >>> Again triggers, rules, asynchronous for later. >>> >>> The idea is that you will be able to create actions and web actions that >>> can run existing OpenWhisk actions, at least those with blocking behaviour >>> that run with ActionLoop (Go, Java, Python, PHP, Swift, Rust, Ruby, >>> Crystal...) >>> >>> Implementation. >>> == >>> >>> This is how I plan to implement it. >>> >>> At this stage I want to use just Knative Serving and Knative Build, using >>> Gloo for the ingress part. I also plan to install a local Docker registry >>> Kubernetes registry, so we do not have to use DockerHub for everything. All >>> of this can be done with existing command line tools in a few minutes in >>> any running Kubernetes deployment. >>> > >> Why specifying Gloo here? Do you need anything specific from Gloo itself? >> If not I'd propose to just keep it on a Knative Serving API surface level. > I want to build it on top of Knative serving, full stop. Currently, > installing Gloo is pretty easy and is more lightweight than Istio so I will > use it for my first implementation. > >>> >>> When I create an action, it will use Knative build that will work roughly >>> in this way: >>> >>> - create a configmap with the action code >>> - build the actin using ActionLoop precompilation feature that will return >>> a zip file including all the needed to run the action >>> - create a new docker image extending the runtime with the new zip, using >>> Kaanico >>> - push the image in the local registry >> This feels like a fairly heavyweight process, we should be able to come up >> with a way to circumvent zipping entirely. Maybe the runtime can detect >> that the unzipped content is already there and skip the unzip step? > > Actually
Re: Extending Authentication and Entitlement - Heads up
Andy, I just opened the pull request for the entitlement SPI (Please see: https://github.com/apache/incubator-openwhisk/pull/3822) As you can see it implements the existing EntitlementProvider interface as SPI without changing it. I think it would be still easy to adapt it according to your proposed changes. Regards, Martin
Re: Proposal to integrate IAM use cases with the OpenWhisk platform
Hi Vinuri, good to see that IAM integration seems to be a feature that is also needed by other parties. As announced on this list yesterday I am also working on integrating Openwhisk into an existing IAM system. Please see my last reply on the " Extending Authentication and Entitlement - Heads up” thread for details on the approach I am taking. As you can see there, we plan to introduce an extension point for entitlement providers that would be the point of control to get answers on the “who is allowed to do what on the namespace” questions. In an ideal world I would also here see the point to answer questions regarding namespace creation/update/deletion Unfortunately there is no API for this use cases. So maybe we should take this a starting point for the dicussion to add this “missing” API = crud for namespaces) before adding additional plug points. Moving this functionality to the server side would also make the client more lite-weight. Personally I do not see much value in replicating functionality that already exist in the IAM systems like adding namespaces to roles/users and giving access rights on them to the tooling. Additionally I do not see how this can be done for different systems in a common way. Please let me know what you think about my comments. Lets keep in discussion. Regards, Martin > On 20. Jun 2018, at 13:25, Vinuri Perera wrote: > > Hi Team, > > I'm looking into improving the openwhisk platform to support some common > IAM usecases which might be needed by enterprises. This will integrate IAM > capabilities to the platform and also to the wsk CLI. > > *##Problem* > > The requirement is to control access of users to the OpenWhisk platform, > who are already in the enterprise userstore. The reason is, most > organizations have their well-defined userstores, groups, roles etc > available and they want to plug it to this system. > > The current wskadmin tool is capable of generating namespaces. But, it > cannot map any roles to that namespace and make decisions on whether to > allow access to a certain namespace or not (authorize). > > Integration with an Identity Provider (IDP) is required in this aspect > where the IDP manage the userstores and authenticate/authorize users so we > maintain the namespace-role mapping in OpenWhisk side. > > > *##Approach* > > Approach for this is similar to the API Gateway, where we package the > required functions separately and deploy as web actions. These functions > can modify to integrate with any identity provider available to manage and > control access. OpenWhisk CLI will require to modify accordingly to cater > the identified use cases. > > > > *## Identified use-cases and suggested updates to the CLI* > > > 1. Creating the roles with namespace mapping. > > Admin users should define the required roles for the namespace. e.g - > Users who contain the 'manager' role should be able to access the > 'userManagement' namespace. > >*CLI Command* > >wsk namespace create [Namespace] [Roles] > > 2. Users generate namespace tokens and login to the system. > > Any user should be able to generate an access token for namespaces if > they belong to any of the roles that have been mapped to the namespace. The > decision will be taken by the IDP by authenticating and authorizing the > user against the roles mapped to the namespace > > There are two parts to this command, if the user does not contain the > access token for the namespace it will generate the required token after > verifying from the IDP and set the auth property for the user. If user > already contain the access token for the requested namespace this command > will set the existing auth token for the user. > > *CLI Command* > > wsk login [Username] [Namespace] > > Should prompt for the password > > 3. Revoking user namespace token > > Admin user should be able to revoke user namespace tokens. This action is > only available the admin user, CLI will check if the current namespace is > in the system namespace and handle the restrictions for accessing the > action. The action itself will verify the authorization to perform the > action from the IDP. > > * CLI Command* > > wsk user revoke [Username] [Namespace] > > > > > Above is the high-level description of the proposed solution for > integrating some of the IAM use cases with the OpenWhisk platform. > > Please Let me know if you have any comments and suggestions or an already > working/discussed solution. > > > Best Regards, > > ~Vinuri~
Re: Extending Authentication and Entitlement - Heads up
Andy, sorry for answering late. I was some days out and then dragged into other work. Let me try to explain the things we want to do with our IAM integration. After having the needed extension points available (as explained in my last mail) we plan to implement an authentication directive that will additionally to the standard implementation be able to accept bearer tokens and validate them against a list of valid public keys and certain content requirements. Since the authentication directive is the point to create a user identity, our implementation will reach out to other systems to create a “complete" user identity (e.g. containing limits and the default namespace). The created user identity will contain an AuthKey variation with IAM credentials for the user and the namespace This AuthKey will be passed to user/system containers and can be used to create triggers or to reach out to other services. For the entitlement use case we plan to implement a a provider that checks the tuple of access method, user identity and accessed resource (which is in our case the namespace) against an external policy system. Concerning the linked PR, I am totally fine to make the adapted entitlement provider the SPI interface. I do not see any need for big adaption effort regardless of order the PRs will go in. I hope that answers your questions. Regards, Martin > On 13. Jun 2018, at 01:22, Andy Steed wrote: > > Hi Martin! > > As Rodric mentioned, I've been working in Entitlements regarding namespace > limits. I'm also interested in support for IAM integration. Do you have any > use case(s) that you could share regarding what you're looking to support? > Similarly, do you have any details on additional entitlement controls you're > may be looking to expose (both short and long term) made available via the > new IAM provider? I have a task to support for Adobe's IAM provider and would > like to see how your plans potentially map onto that. > > Cheers, > Andy > > On 6/12/18, 8:18 AM, "Rodric Rabbah" wrote: > >> The first change will be to introduce an SPI to exchange the existing >EntitlementProvider with an alternative >implementation. Since the EntitlementProvider already is implemented like a >SPI-like interface >this change is very straightforward. > >Since IAM integration is of general interest and applicable for others who >deploy and manage OpenWhisk, is the current interface sufficient? For >example, this PR [1] from Andy Steed was steered toward Entitlement checks. >Similarly, the current and future limits applied to a namespace may be >equally fitting (where today they are tied to the subject records). So we >should think about how to support both the short term and long term. > >[1] > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3661&data=02%7C01%7Candsteed%40adobe.com%7Cc0f8898085ff4120cf5008d5d077c968%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636644135257283014&sdata=ojlxOA3cRPopJXrhX9Pe%2FUmJ76uSMvGonlbe5%2BQsQ4M%3D&reserved=0 > >> First the REST API will be enabled to read other authentication formats >and tokens >(e.g. bearer tokens), second there has to be the ability added to pass >different authentication information >to the user actions. > >This is great and long overdue - Are you envisioning that we can then also >attach different tokens with resources in a more fine grained capacity than >today (which is an entire namespace)? > >-r > >
Extending Authentication and Entitlement - Heads up
Hi, as some of you might have noticed with my last commit so please take this mail as a heads-up. I am on the road to introduce extensibility for the authentication and entitlement in Openwhisk. The changes are motivated by the need to integrate Openwhisk tighter into an existing (but unfortunately partly proprietary) identity and management system used in the IBM cloud. The first change will be to introduce an SPI to exchange the existing EntitlementProvider with an alternative implementation. Since the EntitlementProvider already is implemented like a SPI-like interface this change is very straightforward. The authentication changes will address two areas. First the REST API will be enabled to read other authentication formats and tokens (e.g. bearer tokens), second there has to be the ability added to pass different authentication information to the user actions. I plan to address this with introducing an SPI to swap the AuthorizationDirective in the RestApi and adding a mechanism to transport variant information in the authentication key to the invoker. All changes are designed to be transparent to the existing authentication and entitlement implementations using the subject db. I will open pull request for all these changes in the next days. Feel free to comment now to this mail or later to the pull requests. Kind regards, Martin
Tech Interchange Meeting Today
This is another kind reminder for our bi-weekly Tech Interchange Meeting today. The (updated) agenda up to now is: - Introduction of new attendees - Asking around for notable changes/updates - Updates on the graduation / release mgmt. process for OpenWhisk by Vincent S Hou [20 min] - License header auditing in the release process by Ying Chun Guo [10 min] - Sharding Loadbalancer changes by Markus Thömmes [10 min] - Find and confirm moderator for next meeting Feb 28th @all: Please contact me if you have another topic that should show up on the agenda. @Matt: Can you please add the agenda to the CWiki. Details: What: Apache OpenWhisk "Tech. Interchange" (bi-weekly) Zoom Meeting When: @ 11:00am EDT, 8am PDT, 3pm GMT, 5pm CEST , 3pm UTC Where: https://zoom.us/my/asfopenwhisk Regards, Martin
Tech Interchange Meeting February 14th
Hello, this is a kind reminder for our bi-weekly Tech Interchange Meeting tomorrow. The proposed agenda up to now is: - Introduction of new attendees - Asking around for notable changes/updates - Updates on the graduation / release mgmt. process for OpenWhisk by Vincent S Hou [20 min] - Sharding Loadbalancer changes by Markus Thömmes [10 min] - Find and confirm moderator for next meeting Feb 28th @all: Please contact me if you have another topic that should show up on the agenda. @Matt: Can you please the agenda to the CWiki. Details: What: Apache OpenWhisk "Tech. Interchange" (bi-weekly) Zoom Meeting When: @ 11:00am EDT, 8am PDT, 3pm GMT, 5pm CEST , 3pm UTC Where: https://zoom.us/my/asfopenwhisk Regards, Martin
Log level adjustments
Hi, we have showed in recent dev calls that the amount of logs the controllers and invoker write out to the log files does harm the performance under high load severely. To reduce that burden we introduced the new metric infrastructure based on Kamon which is already delivered (but switched off by default). Additionally we now pushed PR https://github.com/apache/incubator-openwhisk/pull/2960 which reassigns the level of the remaining log messages. Both changes reduce the amount of messages logged at INFO level (which is the default) to about a 5th of the original size. Our tests show that the performance under load improves by at least a factor of two. In the PR we aimed for better error logging and to still ensure maintainability of the system. We would like to emphasize that with that change it is more important to have a monitoring infrastructure in place (as a starter the document ‘metrics.md’ in the repo explains how to test the Kamon and statsd based metrics in your local environment). Please leave your feedback as comment to the PR or as answer to this message. Kind regards, Vadim and Martin
Re: Replace log markers with metrics sent to statsd
Tyson,
thank you for your comments.
Since Vadim is away today I will try to answer.
I fully agree that we should keep the logging back-end discussion
separate. Therefore I am with you to tackle this in a first PR that
changes the logging implementation to Logback (with the console appender
configured, leaving the asynchronicity solely to Akka).
Interested parties could easily adapt the log configuration if needed.
Orthogonal to this discussion is the need to reduce log output.
While I mostly agree with your comments above I see the need to move
(most) metrics out of the log path as a an integral (or even
prerequisite) part of the 'log reduction' work that has to be tackled
early. Tightly coupled part is to adjust log levels carefully to allow a
more restricted 'info' level logging (we probably even have to add
messages back that were written as metrics before)
My take on the metrics vs tracing discussion is that metrics fulfill the
need to consistently watch the behavior of the system on a more general
level in opposite be able to deep dive in the behavior of single work
threads of the systems (at best to be switched on and off as needed).
As you said, most counter related metrics fall in the first category but
also timing information might fall in that category if the aggregation
level is 'high enough' (e.g to monitor "the average request time of one
controller").
Having that said (please excuse the length of this answer) I propose to
follow this path:
- replace logging implementation with Logback
- reduce log generation by
- write metrics with Kamon
- adjust and reduce existing log levels and messages
- look into tracing as follow-up
Kind Regards,
Martin
On 27.09.17 19:44, Tyson Norris wrote:
Thanks Vadmin!
A few comments:
- RE: async logging: I thought the akka logger was already async? This is
different than the logback async appender. I’m OK with using the async
appender, but people should know that it by default is configured to ignore
events once its buffer is full, so its possible to get into a situation where
log messages “go missing” if faced with sufficient traffic.
- RE: kamon: It’s not clear to me that kamon supports timing data for general
methods, although it looks like it keeps timing data for akka message
processing.
Currently the log markers in OW keep timing data for arbitrary blocks of code
(unrelated to akka). It looks like to get “time in method” metrics you will
still need to measure the time, and then record that time using kamon.
I know Sandeep is looking into using the timing/tracing aspect as part of this
PR https://github.com/apache/incubator-openwhisk/pull/2282.
It may be interesting to consider kamon TraceContext for this, but it is also
somewhat invasive - seems like it requires wrapping all Future usage as in:
Tracer.withNewContext("sample-trace") {
Future {}
}
Is there a way to use this without this wrapping?
BTW - I agree that any “counter” type metrics should be replaced and kamon
seems good for this, I’m just not sure about the timing markers.
- RE: first steps: Does changing log library or appender config imply the need
to change the current marker/timing impl? I think a few logical steps are:
- replace logging impl (with logback)
- reduce log generation
- replace log marker counter impl (with kamon?)
- replace logging approach (with TraceContext or similar, if an option even
exists, with the goal being to NOT pass whisk Logging around everywhere)
(I’m really not sure about the last one, but it would be nice to a less
invasive way to emit logs with context)
Thanks
Tyson
On Sep 27, 2017, at 7:39 AM, Vadim Raskin
mailto:raskinva...@gmail.com>> wrote:
Hi OpenWhisk community,
while doing our profiling tests on the controller we have found that the
way we log and the logging per se make a huge impact on the controller
thoughput. In a short, the more parallel activations you have, the more
blocking threads waiting for the logger will be there. After doing several
experiments we found a couple of action items:
* replace the logging library (e.g. to logback), use an async appender and
increase the size of the LogAppender buffer.
* reduce the number of logs that we generate.
Since the first action item will only provide a partial relief and postpone
the point when we start observing the degradation, reducing the number of
logs seems to be a more plausible way to go.
One of the first points I’d like to address are the metrics that we append
to log messages in a form of markers.
The initial idea is to send them directly to statsd or some other backend.
For example Kamon library is a good candidate:
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fkamon.io&data=02%7C01%7C%7Cfa62bc6102964a52294908d505b59165%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636421199745973146&sdata=K8azfpdoqTeRzQrhB1LhBY76z5DrTSGyg7DuBmLMJZ8%3D&reserved=0
Here is the issue I’ve opened some time ago.
https://na01.safel
Hi, my name is... (continued)
My name is Martin Henke. I have joined the Cloud Functions team of IBM a short while ago. Before joining this team I have worked in several Cloud related products and project in IBM. I am looking forward to contribute to the Apache OpenWhisk project and to work with the amazing people on this list. Regards, Martin P.S: I have filled out the ICLA with my private email Address, therefore I am using this address now. You can also reach me also via my company address: martin.he...@de.ibm.com
Hi, my name is... (continued)
My name is Martin Henke. I have joined the Cloud Functions team of IBM a short while ago. Before joining this team I have worked in several Cloud related products and project in IBM. I am looking forward to contribute to the Apache OpenWhisk project and to work with the amazing people on this list. Regards, Martin P.S: I have filled out the ICLA with my private email address, therefore I am using this address now. You can also reach me also via my company email address: martin.he...@de.ibm.com
