Re: sorry for the github notification flood

[Michael Marth]

Thanks a lot, Rodric, for making the effort!

On 18.12.19, 19:23, "Carlos Santana"  wrote:

Thanks Rodric I got them all this morning :-)

Thanks for the holiday cleaning 粒 

- Carlos Santana
@csantanapr

> On Dec 18, 2019, at 11:59 AM, Chetan Mehrotra  
wrote:
> 
> Thanks Rodric for this long overdue cleanup!!.
> 
> Having a set of valid and non stale issues would be helpful
> 
> 
>> On Wed, Dec 18, 2019, 9:35 PM Rodric Rabbah  wrote:
>> 
>> if you subscribe to github notifications for our project, i'm sorry.
>> i should have sent a warning that I was going to go through a lot of our
>> our open issues and close ones that are fixed, stale, or won't fix.
>> 
>> good news is that i'm done for now and reduced the issue count by 150 or 
so
>> (largely because we've addressed the substance but were not very good 
about
>> closing).
>> 
>> bad news now that i started, i might keep going to address the remaining
>> 300 issues.
>> 
>> -r
>> 

Re: A direction to take in a new component, "the scheduler"

[Michael Marth]

Sven, all,

That's a good call-out.
In consequence to that: how about we discuss larger changes with
a) a test script that simulates the types of load under discussion (and at the 
same time makes it clear which types of load are ignored for now) and
b) a declaration of the KPIs one seeks to improve. Could be performance, 
scalability, cold start latency, cost, etc. This also makes it clear which 
other KPIs are disregarded or assumed to be ok to deteriorate in the proposal.
I think that might make it easier to discuss the relative merits of a change.

Cheers
Michael


On 18.07.19, 19:49, "Sven Lange-Last"  wrote:

Hello Dominic and other discussion participants,

we are in the great situation that Dominic (style95) volunteers to create 
and verify an alternative OpenWhisk architecture that may solve some of 
the problems that adopters of OpenWhisk may observe when running 
production systems with high utilization. I really appreciate that Dominic 
already invested and is willing to invest so much time.

While I encourage Dominic to experiment with a different architecture, my 
point of view is that it's in the interest of the project as well as all 
adopters running OpenWhisk-based production systems to keep the project 
stable. We need an approach where we can continue to evolve the existing 
architecture and provide bug fixes without being slowed down or affected 
by the work on or problems with the new architecture. At the same time, we 
need a playground where we can work on the new architecture effectively 
without negatively affecting the existing architecture.

For me, this means:

* We need feature flags / switches as suggested by Dominic and others that 
keep the existing architecture enabled by default.

* The standard deployment only configures and installs what is needed for 
the existing architecture. Installing extra components like etcd takes 
more time so that development, build and test cycles take longer. In 
addition, extra components consume more resources so that brittle system 
tests for the existing architecture may fail more often.

* We need a good separation between tests for existing and new 
architecture. System tests are often brittle and need special resources 
like etcd. Such tests for the new architecture must not be part of the 
existing gradle suites. I think that stable, fast-running unit tests can 
be part of existing suites.

* The Travis checks for pull requests must not contain system tests or 
other brittle / long-running tests of the new architecture. We already 
have some brittle tests today that occasionally fail Travis. As Dominic 
suggested, extra build / deployment / test jobs that run separately would 
be a great idea.

I would like to add a different aspect...

My experience is that the observed problems of an OpenWhisk based system 
really depend on utilization and the workload mix (memory limits, action 
duration, action container image, number of distinct actions, ...). And I 
would expect that people running a few activations will have totally 
different observations than those people running a system with hundreds of 
invokers and millions of activations with a diverse workload mix. And 
based on the observed problems, different contributors will probably try 
to solve different problems and set different priorities.

I had a great discussion with Dominic on these aspects. It turned out that 
he observes different problems than I do when running a large production 
environment. I'm very interested in Dominic's proposal - at the same time, 
my impression is that it will not necessarily address the problems I see. 
Still, we can learn a lot from Dominic's work.

At some point in time, the OpenWhisk community probably needs to decide 
whether to keep the existing architecture and evolve it - or to replace it 
with the suggested architecture if it proves useful. My impression is that 
Dominic only received limited feedback on his proposal and I'm guilty of 
providing my feedback much too late. We should discuss the problems we 
observe with running OpenWhisk more openly so that contributors can align 
their work with the need of others.


Mit freundlichen Grüßen / Regards,

Sven Lange-Last
Senior Software Engineer
IBM Cloud Functions
Apache OpenWhisk


E-mail: sven.lange-l...@de.ibm.com
Find me on:  


Schoenaicher Str. 220
Boeblingen, 71032
Germany




IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
HRB 243294

Re: Welcome new Committer Rob Allen

[Michael Marth]

+1!!
Congrats, Rob!

On 07.01.19, 18:52, "James Thomas"  wrote:

Awesome news and well deserved!

On Mon, 7 Jan 2019 at 17:28, Justin Halsall  wrote:

> Thats great news! Congrats Rob!
>
> Sent from my iPhone
>
> > On Jan 7, 2019, at 12:22 PM, Carlos Santana 
> wrote:
> >
> > OpenWhiskers,
> >
> > Yes! Another new Committer!
> >
> > Based on his ongoing and valuable contributions to the project, the
> > OpenWhisk PPMC has elected Rob Allen as a Committer and he has
> > accepted the invitation.
> >
> > Please join me in welcoming him!
> >
> > Regards,
> > -- Carlos
> > 
>


-- 
Regards,
James Thomas

Re: Donation of ibm-functions/composer to Apache OpenWhisk

[Michael Marth]

That's awesome, Dave! Thank you and IBM for this contribution.

Cheers
Michael

On 28.09.18, 19:02, "David P Grove"  wrote:



IBM Research would like to donate the Composer code in
ibm-functions/composer to the Apache OpenWhisk incubator project.  The code
is already open source under the Apache 2.0 license.

Unless there are objections raised here on the dev list, I will plan to
submit the infra tickets on Monday to create an
apache/incubator-openwhisk-composer git repo, enable Travis on it, etc. and
we will migrate the code next week.

--dave

Re: Asking opinions about "Learning OpenWhisk" a book from O'Reilly I am writing

[Michael Marth]

Hi Michele,

Congratulations for getting an O'Reilly contract. That's awesome!
Also, this list is totally the right place for this topic (IMO).

I have a little comment about the TOC: the way I read it your intended audience 
are OW users (action developers), not developers of OW itself. Is that right? 
However, even for the former group it is often helpful to know how their action 
code actually gets executed. So maybe a chapter about the internal architecture 
of OW would be helpful.

My2c
Michael


On 19.09.18, 18:32, "Michele Sciabarra"  wrote:

Hello all, 

I am not sure if it is appropriate to talk of this on the mailing list or 
not. If not please let me know and I will stop immediately and apologize for 
this post. I assume for now it is acceptable (because I saw similar discussions 
on other mailing lists on the Apache Group), so I post this.

As some community members already know,  I wrote some chapters (6) of a 
book on OpenWhisk (so you know what you did when I disappeared for a few months 
:)), then I was close to release it as open source and I asked what to do of 
them on the Slack channel.  In a sense the book was complete and ready to 
release.

To my surprise, instead of recommending to release the book as Open Source 
I was told instead that publishing it with a prestigious editor would have been 
better. 

So I dared to propose the book to the (IMHO) most prestigious technical 
publisher I know, O'Reilly, and guess what, the book was approved! 

The chapters I wrote so far focus on Javascript. However, I was recommended 
(by Carlos and Rodric) not to talk only of Javascript. There is a lot of stuff 
on Javascript people are looking for learning about other languages.

So I ended up with a plan to cover also Python and Go.
This is the planned TOC so far. I am asking for opinions on it:

TItle: Learning OpenWhisk

Part1: Introducing Serverless Development  in JavaScript

- Serverless and OpenWhisk Architecture
- A Simple Serverless Application in JavaScript
- OpenWhisk CLI and JavaScript API
- Common Patterns in OpenWhisk 
- Integration Patterns in OpenWhis
- Testing OpenWhisk Applications

Part 2: Advanced Serverless Development in Python and GoLang

- Using Python in OpenWhisk
- Using Databases in OpenWhisk
- Creating an Alexa Skill in Python
- Using GoLang in OpenWhisk
- Using Message Queues in OpenWhisk
- Creating a Slackbot in GoLang

Appendixes
- Deployment with wskdeploy
- Installing OpenWhisk in Kubernetes

The key concern is if a similar TOC is acceptable and making the best 
compromise, or the book could be too wide (and hard to read) for the potential 
audience. 

In my opinion, it should be a good compromise between completeness without 
requiring too many skills. But here I am open to hearing other opinions. For 
example, I thought to stick only to javascript but then I would lose more 
advanced aspects that can be of interests to many developers. 

Note I can share freely drafts or chapters of the book with members of the 
community is interested. Contact me privately. But keep in mind the publisher 
give me only 10 free copies so I cannot promise too many printed free copies :) 
:) :)


PS: the Animal!  You know each O'Reilly book has an animal in the cover. 
After a lot of thinking, I proposed an animal not yet used but I think it fits 
a lot OpenWhisk: the Pagurus (https://en.wikipedia.org/wiki/Pagurus). Do you 
like the idea?

-- 
  Michele Sciabarra
  mich...@sciabarra.com

Re: Prototyping for a future architecture

[Michael Marth]

Hi Markus,

IMHO what you propose below is a rather severe change in scope of this 
discussion and effort.
Up until so far this was about _evolving_ the OW architecture. We have not 
explicitly discussed it, but one could assume that it is at least feasible to 
gradually adopt the new architecture. So there would be a smooth path between 
the current state of the code base and a future one.

Your proposal below breaks this assumption somewhat (by proposing a new repo 
instead of a branch - which will inevitably make the 2 code bases drift apart) 
as well as explicitly by suggesting a new implementation language. Especially 
the latter would create a schism between OW-now and OW-future.
This schism has implications like the perception of OW-now being deprecated, 
the _possibility_ of no clean upgrade path, the immediate split of the 
community between *-now and *-future and of course carries the risk of the 
version 2 syndrome.

I would propose to implement the future architecture in a branch and in Scala 
first. If it turns out to be good, then subsequent experiments can show or 
not-show if a switch of language is of additional value. That would allow to 
make a decision based on data rather than anything else.

My2c
Michael


On 28.08.18, 14:26, "Markus Thömmes"  wrote:

Hi all,

Am Mo., 27. Aug. 2018 um 20:04 Uhr schrieb David P Grove :

>
>
>
> "Markus Thömmes"  wrote on 08/23/2018 04:19:33
> PM:
>
> >
> > Key point I want to make is: At some point we'll have to start to
> prototype
> > things out and see if our assumptions actually hold water. For example,
> my
> > assumption on a work-stealing backend is pretty much in the air.
> >
> > My proposal for going forward would be:
> > 1. Create a playground for the implementation of some parts of the 
system
> > (a new repository?)
> > 2. Let's build some of the things that are uncontroversial and 
absolutely
> > needed in any case (ContainerRouter, ContainerManager).
> > 3. Play around with them, hook them up in different ways, see what works
> > and what doesn't.
> >
> > Some things will need some testing out to see the scale that the
> components
> > can operate at. These things will narrow or widen the solution space for
> > the more controversial topics around how to distribute containers in the
> > system, how to balance between the routers, work-stealing queue: yes/no
> etc.
> >
> > Having some simple components fleshed out could encourage innovation and
> > creates some facts that we need to focus things into a good direction.
> >
> > What do you think? Too early to start with this and/or the wrong way of
> > doing it?
> >
>
> +1 for starting to prototype.  It's been a good discussion and I think
> we've identified some things that we know we don't know, so time to
> experiment and find out.
>
>
> Not sure what the best logistics are for this.  Would like the work to be
> at Apache (community visibility).  I'm not sure if the best way is a new
> repo or an experimental branch of the main repo (we could dial down the
> level of testing on the branch to make it less cumbersome?).  The branch 
is
> attractive to me because it might make it easier to keep in synch with the
> components we aren't changing.
>

I actually think we should generate a new repository for this. Opening PRs
etc. will then not clutter the "main" repository and we don't need to worry
about breaking anything.

If nobody objects I'm going to get "incubator-openwhisk-protoype" generated
and will create a rough outline in the repository (different folders for
different parts of the system).

One more basic question (and this is going to be controversial): Most of
the componentry in the execution layer will have to be build anew. I'd like
to switch the implementation language of at least the ContainerRouter to
Golang. Why? Because scale/performance matters a lot for them. As Dave
mentioned on multiple occasions, it will greatly matter how many containers
such a Router can handle under load and that scale will define the
implementation alternatives we will have at hand. I therefore would like to
optimise these Routers for minimal overhead. We could go even lower level,
but I guess that'd be at a kinda big maintenance cost.

I can envision the ContainerManager to be simpler to implement in Golang as
well, at least for some of the deployment alternatives (Kubernetes comes to
mind). The Golang based clients seemed superior to me vs. clients in any
other language.

As all of the communication will probably be HTTP only anyway, these
implementations should be swappable anytime.

Comments very welcome! Let me know your opinions.

Cheers,
Markus

Re: Proposal on a future architecture of OpenWhisk

[Michael Marth]

Thanks, Markus!

On 15.08.18, 16:30, "Markus Thömmes"  wrote:

Hi Michael,

loosing/adding a shard is essentially reconciled by the ContainerManager.
As it keeps track of all the ContainerRouters in the system, it can also
observe one going down/crashing or one coming up and joining the "cluster".

If one Router leaves the cluster, the ContainerManager knows which
containers where "managed" by that router and redistributes them across the
Routers left in the system.
If one Router joins the cluster, we can try to rebalance containers to take
load off existing ones. Precise algorithm to be defined but the primitives
should be in place to be able to do that.

Does that answer the question?

Cheers,
Markus

Am Mi., 15. Aug. 2018 um 16:18 Uhr schrieb Michael Marth
:

> Markus,
>
> I agree with your preference of making the state sharded instead of
> distributed. (not only for the scalability reasons you quote but also for
> operational concerns).
> What are your thoughts about losing a shard (planned or crashed) or adding
> a shard?
>
> Michael
>
>
> On 15.08.18, 09:58, "Markus Thömmes"  wrote:
>
> Hi Dragos,
>
> thanks for your questions, good discussion :)
>
> Am Di., 14. Aug. 2018 um 23:42 Uhr schrieb Dragos Dascalita Haut
> :
>
> > Markus, I appreciate the enhancements you mentioned in the wiki, and
> I'm
> > very much inline with the ideas you brought in there.
> >
> >
> >
> > "...having the ContainerManager be a cluster singleton..."
> >
> > I was just in process to reply with the same idea :)
> >
> > In addition, I was thinking we can leverage Akka Distributed Data
> [1] to
> > keep all ContainerRouter actors eventually consistent. When creating
> a new
> > container, the ContainerManager can write with a consistency
> "WriteAll"; it
> > would be a little slower but it would improve consistency.
> >
>
> I think we need to quantify "a little slower". Note that "WriteAll"
> becomes
> slower and slower the more actors you add to the cluster. Scalability
> is at
> question then.
>
> Of course scalability is also at question if we make the
> ContainerManager a
> singleton. The ContainerManager has a 1:1 relationship to the
> Kubernetes/Mesos scheduler. Do we know how those are distributed? I
> think
> the Kubernetes scheduler is a singleton, but I'll need to doublecheck
> on
> that.
>
> I can see the possibility to move the ContainerManager into each
> Router and
> have them communicate with each other to shard in the same way I'm
> proposing. As Dave is hitting on the very same points, I get the
> feeling we
> should/could breakout that specific discussion if we can agree on some
> basic premises of the design (see my answers on the thread with Dave).
> WDYT?
>
>
> >
> >
> > The "edge-case" isn't clear to me b/c I'm coming from the assumption
> that
> > it doesn't matter which ContainerRouter handles the next request,
> given
> > that all actors have the same data. Maybe you can help me understand
> better
> > the edge-case ?
> >
>
> ContainerRouters do not have the same state specifically. The
> live-concurrency on a container is potentially very fast changing 
data.
> Sharing that across a potentially unbounded number of routers is not
> viable
> performance wise.
>
> Hence the premise is to manage that state locally and essentially
> shard the
> list of available containers between all routers, so each of them can
> keep
> its respective state local.
>
>
> >
> >
> > Re Knative approach, can you expand why the execution layer/data
> plane
> > would be replaced entirely by Knative serving ? I think knative
> serving
> > handles very well some cases like API requests, but it's not
> designed to
> > guarantee concurrency restrictions like "1 request at a time per
> container"
> > - something that AI Actions need.
> >
>
 

Re: Proposal on a future architecture of OpenWhisk

[Michael Marth]

Markus,

I agree with your preference of making the state sharded instead of 
distributed. (not only for the scalability reasons you quote but also for 
operational concerns).
What are your thoughts about losing a shard (planned or crashed) or adding a 
shard?

Michael


On 15.08.18, 09:58, "Markus Thömmes"  wrote:

Hi Dragos,

thanks for your questions, good discussion :)

Am Di., 14. Aug. 2018 um 23:42 Uhr schrieb Dragos Dascalita Haut
:

> Markus, I appreciate the enhancements you mentioned in the wiki, and I'm
> very much inline with the ideas you brought in there.
>
>
>
> "...having the ContainerManager be a cluster singleton..."
>
> I was just in process to reply with the same idea :)
>
> In addition, I was thinking we can leverage Akka Distributed Data [1] to
> keep all ContainerRouter actors eventually consistent. When creating a new
> container, the ContainerManager can write with a consistency "WriteAll"; 
it
> would be a little slower but it would improve consistency.
>

I think we need to quantify "a little slower". Note that "WriteAll" becomes
slower and slower the more actors you add to the cluster. Scalability is at
question then.

Of course scalability is also at question if we make the ContainerManager a
singleton. The ContainerManager has a 1:1 relationship to the
Kubernetes/Mesos scheduler. Do we know how those are distributed? I think
the Kubernetes scheduler is a singleton, but I'll need to doublecheck on
that.

I can see the possibility to move the ContainerManager into each Router and
have them communicate with each other to shard in the same way I'm
proposing. As Dave is hitting on the very same points, I get the feeling we
should/could breakout that specific discussion if we can agree on some
basic premises of the design (see my answers on the thread with Dave). WDYT?


>
>
> The "edge-case" isn't clear to me b/c I'm coming from the assumption that
> it doesn't matter which ContainerRouter handles the next request, given
> that all actors have the same data. Maybe you can help me understand 
better
> the edge-case ?
>

ContainerRouters do not have the same state specifically. The
live-concurrency on a container is potentially very fast changing data.
Sharing that across a potentially unbounded number of routers is not viable
performance wise.

Hence the premise is to manage that state locally and essentially shard the
list of available containers between all routers, so each of them can keep
its respective state local.


>
>
> Re Knative approach, can you expand why the execution layer/data plane
> would be replaced entirely by Knative serving ? I think knative serving
> handles very well some cases like API requests, but it's not designed to
> guarantee concurrency restrictions like "1 request at a time per 
container"
> - something that AI Actions need.
>

You are right... today! I'm not saying Knative is necessarily a superior
backend for OpenWhisk as it stands today. All I'm saying is that from an
architecture point-of-view, Knative serving replaces all of the concerns
that the execution layer has.


>
>
> Thanks,
>
> dragos
>
>
> [1] - https://doc.akka.io/docs/akka/2.5/distributed-data.html
>
>
> 
> From: David P Grove 
> Sent: Tuesday, August 14, 2018 2:15:13 PM
> To: dev@openwhisk.apache.org
> Subject: Re: Proposal on a future architecture of OpenWhisk
>
>
>
>
> "Markus Thömmes"  wrote on 08/14/2018 10:06:49
> AM:
> >
> > I just published a revision on the initial proposal I made. I still owe 
a
> > lot of sequence diagrams for the container distribution, sorry for 
taking
> > so long on that, I'm working on it.
> >
> > I did include a clear seperation of concerns into the proposal, where
> > user-facing abstractions and the execution (loadbalacing, scaling) of
> > functions are loosely coupled. That enables us to exchange the execution
> > system while not changing anything in the Controllers at all (to an
> > extent). The interface to talk to the execution layer is HTTP.
> >
>
> Nice writeup!
>
> For me, the part of the design I'm wondering about is the separation of 
the
> ContainerManager and the ContainerRouter and having the ContainerManager 
by
> a cluster singleton. With Kubernetes blinders on, it seems more natural to
> me to fuse the ContainerManager into each of the ContainerRouter instances
> (since there is very little to the ContainerManager except (a) talking to
> Kubernetes and (b) keeping track of which Containers it has handed out to
> which ContainerRouters -- a task which is 

Re: MiniWhisk: what you think?

[Michael Marth]

Michele,

Fyi 
https://github.com/apache/incubator-openwhisk-devtools/tree/master/node-local
(obviously does not cover your Go use case - but since you asked what else is 
around...)

Cheers
Michael

On 27.07.18, 00:46, "Michele Sciabarra"  wrote:

Indeed after thinking about my idea was to try to extend the invoker.py
to serve urls, and add a —watch feature so if files are modified then
they are zipped and sent to the runtime as /init. Since the go runtime
compiles already I think it is better to leave the compilation to the
runtime. Another interesting feature of the goproxy is that it supports
the “unnecessary” multiple initializations so it should ne able to do
incremental compilation at least for go code.Another interesting feature 
that I am ‘stealing’ from the openwhisk
shell is support for debuggers. I will try to make available delve if I
can. I suppose it is just enough to be able to run the  client
executable with delve and expose the debugger port.  I think the
openwhisk shell is awesome for node but I want something specific for go
that works with vscode, and let me run tests on the code as I develop
it. And also the go debugger now.I plan to write the miniwhisk for now in 
Python, as part of the examples
for go. Then eventually rewrite it in go if it proves interesting enough
to became a standalone tool.
Thoughts?

--
  Michele Sciabarra
  mich...@sciabarra.com



On Thu, Jul 26, 2018, at 8:54 PM, Rodric Rabbah wrote:
>
> > My approach was to implement the OpenWhisk platform API
> > using a stub server that would execute the actions using Docker in
> > the host> > system.
>
> You really don’t need this though - look at the invoker.py
> script. That> is enough IMO, either extending that or copying that into a 
new
> executable (go or node).
>
> -r

Re: should we enable signed commits on our github repos?

[Michael Marth]

+1 to the hurdle. Even in complicated projects people (like me) like to fix 
typos in READMEs


On 10.07.18, 17:46, "Rob Allen"  wrote:


Personally, I only sign tags on the OSS projects I lead. 

If you do it on a per-commit basis, it's yet another hurdle that a 
contributor has to go through. That may not be a consideration for OpenWhisk as 
it already is a complicated project for the inexperienced to contribute to.

Regards,

Rob

> On 10 Jul 2018, at 16:41, Rodric Rabbah  wrote:
> 
> Who knows why we haven't enabled signed commits on the apache repos -
> should we require all commits to be signed?
> 
> -r
> 
> Ref: https://help.github.com/articles/signing-commits-using-gpg/

Re: Please review our project's draft Apache Incubator June board report

[Michael Marth]

Hi,

I have a small update on the below. Will add to the original thread.

Michael

On 06.06.18, 15:24, "Bertrand Delacretaz"  wrote:

Hi,

On Tue, Jun 5, 2018 at 10:28 PM, Matt Rutkowski  wrote:
> ...We recently had a public (and private threads) around "models" for
> donation at Apache which went no where...

IIUC you mean donating resources for automated testing?

The ASF now has a Targeted Sponsors [1] program meant specifically for that.

If you think that's not useful, or if there are obstacles, I'm happy
to help figure out how to fix that. Ideally here or on private lists
if really required.

-Bertrand

[1] http://www.apache.org/foundation/thanks.html

Re: How to best run non-local tests in ASF (was: Performance tests for OpenWhisk)

[Michael Marth]

A quick update on this:

Carlos (and others) has been working on the mechanics of getting distributed 
tests to run on hardware that is managed by Infra. For this purpose we should 
have a couple of Infra-managed VMs that are dedicated to OpenWhisk. Bertrand 
has gotten me in touch with the ASF fundraising people to figure out the 
mechanics of how to transfer a donation directed at this purpose.
Will keep you posted as there's progress.



On 09.04.18, 11:50, "Bertrand Delacretaz"  wrote:

On Sun, Apr 8, 2018 at 3:36 AM, Carlos Santana  wrote:
> ...Michael could you direct the inquiries on the infra list?...

I think it makes sense to keep this thread here for now and dig deeper
on the (type of) solution that Luciano has been explaining.

Note that http://apache.org/foundation/thanks.html now includes a
"targeted sponsors" section where companies can be recognized as
sponsors by donating cloud services, among other options. That might
be an additional incentive for someone to donate computing resources
for this purpose.

I'm happy to make connections with the ASF's VP of Fundraising if
needed to facilitate things.

-Bertrand

[discuss] long running actions

[Michael Marth]

Hi all,

taking this discussion from Slack to here:

What would be the best way to support long running actions, especially actions 
for which the duration is unknown when the action is started.
An example is e.g. a blackbox container that transcodes a video. The main issue 
is that OW limits the amount of time an action can run.

In my mind, one aspect should also be that there must be a way to inspect 
progress and health of such a long running action. Maybe using the action logs 
is a simple way to do this.

Couple of comments that were made on Slack already:
Markus: “it should be possible to make the upper bound of the runtime to be 
settable to “Inf”. Would “only” need adjustment in the HttpClient used towards 
the container to support an infinite timeout” [1]
Rodric: “callbacks more suitable then” [2]
James: “would think you want some timeout (however large) to stop an action 
which never returns.” [3]

Interested in your thoughts.

Michael

[1] https://openwhisk-team.slack.com/archives/C3TPCAQG1/p152768478110
[2] https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1527685597000100
[3] https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1527689801000253

Re: AttachmentStore - Handling concurrent attachment updates

[Michael Marth]

Hi Chetan,



My2c: making the attachments immutable will yield great benefits as you write 
below:



1. Proper handling of concurrent updates

2. Simplified caching of attachments as immutable objects can be cached 
easily



On #2: with immutable attachments caching becomes trivial which will help with 
more distributed deployments (across different data centers).



Great proposal

Michael









On 26/03/18 05:58, "Chetan Mehrotra"  wrote:



Last week I had a Slack call with Rodric around AttachmentStore PR and

as part of that we discussed the problem around handling of concurrent

updates of attachments. Details below are based on that discussion.



As of now CouchDB can detect concurrent updates of attachment due to

inbuilt MVCC support. However most of the Object Stores (like S3/IBM

COS, Azure Blob Store etc) which are to be used for new

AttachmentStore implementation does not provide any conditional update

and are designed more for immutable storage.



Consider an Action Update sequence which is currently done in 2 parts



1. Update the document

2. Upload the attachment



Now consider an AttachmentStore implementation (as per PR #3453

design) which stores attachment content against a key like



   whiskentity//



Where



1. whiskaction - Key prefix to store attachments related to Whisk entities

2.  - Document Id with which the attachment is being attached

3.  - Name of attachment like `jarfile`



Object Stores are optimized for direct key lookup and also allows

searches based on key prefix. Hence the use of such a format which

allows direct attachment lookup for readAttachment and all attachments

related to specific doc for deleteAttachments



Now consider following flow



1. thread 1: updates the document and succeeds

2. thread 2: updates the document (based on thread 1) and succeeds

3. thread 2: attaches i.e. writes an attachment to the AttachmentStore

4. thread 1: attaches



This would result in a race condition where in the end attachment

meant for document state at #1 gets linked to document at state #2. To

handle such cases we should switch to immutable attachment design



A - Proposal - Use Immutable Attachments





In current flow we perform an "update" of existing attachment with a

given name. For e.g. currently action update flow is like



1. Put document with attachment info



"exec": {

"kind": "java",

"code": {

  "attachmentName": "jarfile",

  "attachmentType": "application/java-archive"

},

"binary": true,

"main": "Hello"

  }



2. Attach the attachment with name set to value of `attachmentName`



Instead of that we should allow `ArtifactStore` (which in turn rely on

AttachmentStore) to generate the name and then save that name against

`attachmentName`. So proposed flow is



1. Upload the attachment and have ArtifactStore return a generated name



  protected[core] def attach(doc: DocInfo, contentType: ContentType,

docStream: Source[ByteString, _])(

implicit transid: TransactionId): Future[(DocInfo, AttachmentName)]



2. Then update the document with attachmentName set to name returned

in previous step



3. Then delete the old attachment after #2 completes successfully



With this approach the attachments would be immutable and that would enable



1. Proper handling of concurrent updates

2. Simplified caching of attachments as immutable objects can be cached 
easily



B - Orphaned Blob Garbage Collection

--



With above approach there is a possiblity that some action update flow

may end up in between leaving some orphan blob instances in Object

stores. To clean them up we can implement a garbage collection login

as part of wskadmin



Please share your feedback about the new proposal. I would start work

on a PR for new proposal so that its easier to discuss specific

semantics. Once this work is done we can come back to AttachmentStore

PR and implement that as per newer flow



Chetan Mehrotra

Re: Upcoming OpenWhisk releases

[Michael Marth]

Hi Carlos,

Re the 1.0.0 in "1.0.0-incubating" below: I cannot recall a discussion on the 
actual version number for the first release (sorry, in case I missed it).
I wonder whether we intend to follow SemVer. If yes, then releasing as 1.0 
would mean we would not change the external APIs in an incompatible way (unless 
we go to 2.0, etc). Is this the common understanding?

Thanks for sharing your thoughts on this
Michael



On 15/03/18 12:17, "Carlos Santana"  wrote:

Yes I agree Bertram 100%

I have being following the Github issues and PRs in the issues and commits 
mailing lists 

No other discussions are going about release other than what’s in the 
release repo that Vincent and Daisy are attacking and the discussions that both 
are bringing to the dev mailing list. 

And yes I agree I think we are very close on automation to start kicking 
out our first 1.0.0-incubating RC1 :-)


- Carlos Santana
@csantanapr

> On Mar 15, 2018, at 7:07 AM, Bertrand Delacretaz  
wrote:
> 
> Hi,
> 
> AFAICS releases have been discussed in your calls, with my incubation
> mentor hat on I'd like to stress the importance of discussing those
> here or at least exposing pointers to discussions here if they are
> happening somewhere else, like in issue tracker tickets.
> 
> I am consciously not following the Slack discussions as these are
> meant to be throwaway and important stuff needs to be discussed here.
> I suppose other mentors do the same so bringing these discussions here
> will help gather feedback early and often to help OpenWhisk do good
> Apache releases soon!
> 
> Using a [mentors] subject line header is a good way to raise the
> mentors attention when needed.
> 
> HTH,
> -Bertrand

Re: Using bash-based actions

[Michael Marth]

Hi Erez,

Fwiw some issues on bash support with more info:
https://github.com/apache/incubator-openwhisk/issues/2927
https://github.com/apache/incubator-openwhisk/pull/3138

Michael



On 04/03/18 13:00, "Carlos Santana"  wrote:

Hi Herez

Perl and Bash Actions are supported is just that we need to improve the
documentation to make it more explicit I’m working on improving docker
Actions documentation this week :-)

Try this:

wsk action update fooBash foo.sh —native

Or

wsk action update fooPerl foo.pl —native

It can be any script file the action will chmod +x and then executed.

For the second question I don’t know what you mean for logs.
Any text to stdout and stderr before the last line with the json string
result are capture in te logs in the activation record

Fell free to open an issue with more info on how to reproduce your problem.

— Carlos
PS: my employer is also IBM. :-)

On Sun, Mar 4, 2018 at 5:56 AM Erez Hadad  wrote:

> Hi folks,
>
> I'm working with technical people from different organizations outside my
> employer (IBM), and we're testing OpenWhisk as a mechanism for integrating
> our different tools and services into a single heterogeneous event-driven
> programming model.
> On the face of it, OW seems like a great fit.
> However, when it comes down to implementation, we often hit a case where
> the easiest way to delegate action execution to a custom tool is by
> writing a simple bash script. This is where things get complicated.
> To the best of my understanding (please correct me otherwise), there is no
> first-class support for bash runtime in OW.
> So, it's down to creating either a "native" action using a zip, or a
> custom docker action. Both options are quite more cumbersome than using
> native runtimes.
>
> So this is my first question - would it make sense to have a native bash
> runtime in OW? especially given that some of the function that OW
> seemingly provides is glue-code between existing tools and services (where
> bash also shines).
>
> As a second issue, we explored the native action. It seems that output is
> not captured in the action logs when including ssh calls that invoke bash
>
> My second question - does anyone have a different experience with this
> that they can share? (maybe this problem is not related to OW?)
>
> Regards,
> -- Erez Hadad
>
> Erez Hadad, PhD
> Cloud System Technologies
> IBM Research - Haifa
> email: er...@il.ibm.com
> phone: +972-4-829-6509
>
>
>
>

Re: Using Apache Creadur to audit in the release process

[Michael Marth]

Hi Daisy,

Tangential question (sorry): is there a wiki or GH page describing the overall 
plan for releases? I am especially interested in the Travis part, but would be 
great to see the full picture.

Thanks!
Michael



On 25/01/18 15:04, "Ying Chun Guo"  wrote:

Hi, all

As we are setting up the release process, I'm investigating how Apache 
Creadur[1] - the auditing tools - can help us in the release process. This 
email describes what I found and what I propose. We can discuss together.

First of all, we need to understand audit is very important in a Apache 
release process. "every ASF release MUST comply with ASF licensing policy. This 
requirement is of utmost importance and an audit SHOULD be performed before any 
full release is created.", described by Apache Release Policy[2]. Apache 
Creadur is such audit tooling to help us.

Apache Creadur includes three projects:
- Apache Rat audits license headers. It will check if files have Apache 
License or not, and generate a report.
- Apache Tentacles helps to audit in bulk components uploaded to a staging 
repository. It will check if there is a LICENSE and NOTICE files under each 
archived source package and compiled package. A HTML report will be generated.
- Apache Whisker will generate a correct legal documentation if a package 
bundles code under several licenses. 

I propose to use:
- Apache Rat to check license headers during the release of the source 
package. We can develop a program to auto 'read' the report generated by Rat. 
If the report doesn't find any issues, the release can be continued. Or else, 
it will be stopped and errors will be returned.
- Apache Tentacles to check if every archived package has a LICENSE and a 
NOTICE file. The check need to be done both in the release of the source 
package and the release of the compiled package after the artifacts are 
uploading to a staging repository. Similar as Rat report, we will develop 
program to auto "read" the report and decide whether there are issues.

Apache Whisker is not relevant to us up to now, because we don't have codes 
under none Apache licenses. ( Correct me if I'm wrong ). In the future, we may 
need it.

Let me know if you have any comments and suggestions to the audit process 
and tooling.

Best regards
Daisy Guo

[1] http://creadur.apache.org
[2] http://www.apache.org/legal/release-policy.html#licensing

[jobs] OpenWhisk engineering position at Adobe in San Jose

[Michael Marth]

Hey there,

as per the subject: my team at Adobe is looking [1] for a software engineer to 
work on OpenWhisk (for our I/O Runtime project [2]).

Cheers
Michael

[1] 
https://adobe.wd5.myworkdayjobs.com/external_university/job/San-Jose/Software-Engineer-for-Serverless-Runtime_57259
[2] https://www.adobe.io/apis/cloudplatform/runtime.html

Re: Decomposition of OpenWhisk components and repositories

[Michael Marth]

Hi Carlos,

Thanks for sharing that!
Slide 4 made me wonder on the distinction between “maintained” repos and 
“experimental” repos. I totally agree that this distinction exists in practice, 
but I wondered how you got to the list. And I should add: I don’t dispute the 
repos you put on the “experimental” list. My question was whether the list of 
experimental repos is common knowledge (and if not: how we can make it so)

Cheers
Michael

On 07/12/17 04:57, "Carlos Santana"  wrote:

This week I gave a talk internal in IBM to illustrate the work recently
done in braking openwhisk components into multiple repositories, and how we
integrate them in our downstream deployment.

Some slides I think are useful to share with the community it gives a map
where are components located.

Also shows how the CLI code was finally migrated to it's own repo, and
manual synchronization stopped.

I uploaded the slides to the wiki [1]

Let me know if the link works and you are able to download.

There is also an old issue [2] I just closed that contains one of the main
slides.

[1]

https://cwiki.apache.org/confluence/download/attachments/74689638/Whisk-Component-Repos-Public.pdf?api=v2
[2] https://github.com/apache/incubator-openwhisk/issues/422

-- Carlos

list etiquette: job postings?

[Michael Marth]

Hi,

as you might (or not) be aware there is a #jobs channel in the OW Slack. 
However, that channel only has 35 members which made me wonder how people feel 
about OpenWhisk job postings on this list?
Obviously, the posting would need to be clearly relevant to OpenWhisk and 
should have a subject tag like [jobs] so they can be easily filtered.
My view is that such postings might be good for building the OW community, but 
could also understand if people object and want this list for dev discussions 
only.

Thoughts?

Cheers
Michael

Re: Call for Agenda of this weeks OpenWhisk “Tech. Interchange” meeting

[Michael Marth]

Carlos, Matt,

I will not be able to join, but I am very interested in the releases. Would it 
be possible for you to share the update on list either before or after the call?

TIA!
Michael

On 23/10/17 18:42, "Jeremias Werner"  wrote:

Hi,

This week we will have the next OpenWhisk "Tech. Interchange" meeting and I 
volunteered to prepare the Agenda. 
We already got a few topics:

1. Introduction/New faces on the call? (5 min)
2. Asking around for notable changes/updates?(5 min)
3. Topics:
   3.1 Feedback and work on trigger management operations - James Thomas, 
Carlos Santana, Jason Peterson (15min)
   3.2 Discussion for trigger activations - Rodric Rabbah (15 min)
   3.3 Demo of Kamon.io integration for monitoring in controller/invoker  - 
Martin Henke, Vadim Raskin (15 min)
   3.4 Short update on release process - Carlos Santana / Matt Ruttkowski 
(10 min)
4. Find and confirm moderator for next meeting (2 min)

Carlos, Rodric, would that be work for you?

All, pls feel free to bring up any topics you want to discuss and we can 
adjust the agenda as needed. 

Looking forward to a vibrant discussion.

Kind regards,

Jeremias Werner

Re: Propose weekly "Technical Exchange" video meeting for OpenWhisk

[Michael Marth]

Hi Isabel,

Happy to share my personal experience. My background (prior to OpenWhisk) is 
with ASF projects that use only the email list and occasional meetups.
Overall, my assessment is that these calls are a welcome addition. I do like 
them, primarily because it allows to put faces to the names and make the 
interactions more personal.
Due to conflicting meetings I can rarely attend in person, though. I catch up 
with the recordings – but don’t feel excluded by that (others might feel 
differently about this).
It’s good that there are meeting notes and I do read them quickly before 
watching the recording. For my taste they could even be less verbose, I find 
them hard to follow. For me it would also work just to list the topics that 
were discussed.
I think what remains to be true is that matters of substance that are discussed 
in the calls need to be transferred to the list for deeper, async discussion. 
From what I can tell this is happening.

My2c
Michael


On 11/10/17 13:15, "isabel.drostfr...@gmail.com on behalf of Isabel 
Drost-Fromm"  wrote:

On 2017-06-10 09:37, Isabel Drost-Fromm  wrote:
> On Fri, Jun 09, 2017 at 07:48:40PM -0400, Rodric Rabbah wrote:>
> Please also make sure to track the impact these sessions have on your
project's reports.

You've been running the video sessions plus summary for a while now.

I'm really interested in a retrospective on what the impact on the
community is. Do people not on the call feel like they can follow the
project? Did you manage to engage new contributors through these sessions?
How did you manage to incorporate feedback from those not participating?


Isabel

Re: Signalling production-state of an action

[Michael Marth]

Hi Carlos,

Thanks for the feedback.
“_” as a separator was just an example, could be anything else as well.

But IIUC your comment
“Like a boolean that the user wants a certain namespace to be treated as  
"test/debug"”
you hint at an idea I like better: letting users apply properties to 
namespaces. (not just by naming convention, but by storing these properties in 
the DB).
Did I get that right?



On 01/09/17 19:43, "Carlos Santana" <csantan...@gmail.com> wrote:

We (IBM) in our in production environment today might have some hard
constraint to support this.
And we already took the liberty to use `_` in the namespace to mean
something else.

For example in our case the namespace is composed of user global account
information, in our case we leverage CloudFoundry system or Organization
and Space.
Where NameSpace = $Organization + "_" + $Space

So we already have in production users with namespaces with "_" in their
production namespaces.

If using the value of the namespace is an option then if namespace ends
with `_*test` but we need to check if we have users using a namespace like
this today I think that would be somewhat difficult.
In our case use would do something like `carlos_app1` and
`carlos_app1test`, we will tell folks to create a new space that ends with
`test`


Other options:
Something on the specific entity (like a action annotation).
We could add something to the subjects DB, and the controller can tag
something to the activation when sending to kafka for the invoker.
Like a boolean that the user wants a certain namespace to be treated as
"test/debug"

I was thinking a secondary OW API Key, but this brakes down for Web Actions
as they are public

No concrete feedback but worth the discussion, just wanted to give feedback
on the convention of using `_`

--Carlos Santana
    

On Thu, Aug 31, 2017 at 7:42 AM Michael Marth <mma...@adobe.com.invalid>
wrote:

> Hi,
>
> in yesterday’s tech exchange Rodric mentioned that OW does not „know“ if
> an action is in production or still in development stage.
> The context of that comment was that we could consider to treat the
> activation log collection separately (depending on the 
in-production-state).
>
> In our OW deployment we have discussed a somewhat related topic:
> How would our users move their actions from a development state to a
> production state. Consider the user’s application to consider of a couple
> of actions. Once the actions actually are deployed in production, then how
> would the users edit and test new versions of these actions and deploy the
> new versions into production.
> (note: this is a separate topic from the production state and CI/CD
> pipeline of the OW system itself)
>
> The concept we came up with is based on a namespace convention:
> If a user is provisioned with the namespace e.g. “mynamespace” then that
> user can create additional arbitrary namespaces separated by an 
underscore,
> e.g. mynamespace_dev, mynamespace_mytest123, etc. We would treat 
everything
> with an underscore as non-production.
>
> In case you wonder why we actually care: Knowing which actions are in
> production and which ones are in development allows us to make decisions 
on
> how the activation logs are collected and displayed, on alerting, on the
> events/feeds we pipe into these actions etc.
>
> I wanted to present our thoughts here to get feedback on:
> Are there better approaches we could look into?
> Is there an interest for a mechanism in OW itself to represent production
> state (or simple adoption of the convention described above)?
>
> Michael
>

Signalling production-state of an action

[Michael Marth]

Hi,

in yesterday’s tech exchange Rodric mentioned that OW does not „know“ if an 
action is in production or still in development stage.
The context of that comment was that we could consider to treat the activation 
log collection separately (depending on the in-production-state).

In our OW deployment we have discussed a somewhat related topic:
How would our users move their actions from a development state to a production 
state. Consider the user’s application to consider of a couple of actions. Once 
the actions actually are deployed in production, then how would the users edit 
and test new versions of these actions and deploy the new versions into 
production.
(note: this is a separate topic from the production state and CI/CD pipeline of 
the OW system itself)

The concept we came up with is based on a namespace convention:
If a user is provisioned with the namespace e.g. “mynamespace” then that user 
can create additional arbitrary namespaces separated by an underscore, e.g. 
mynamespace_dev, mynamespace_mytest123, etc. We would treat everything with an 
underscore as non-production.

In case you wonder why we actually care: Knowing which actions are in 
production and which ones are in development allows us to make decisions on how 
the activation logs are collected and displayed, on alerting, on the 
events/feeds we pipe into these actions etc.

I wanted to present our thoughts here to get feedback on:
Are there better approaches we could look into?
Is there an interest for a mechanism in OW itself to represent production state 
(or simple adoption of the convention described above)?

Michael

Re: Tech Interchange posted for 2017-08-30

[Michael Marth]

Thanks, Matt!
I watched the recording yesterday. At the end there was a question whether 
these calls are useful. Answer from my POV: they are useful and a welcome 
addition to email-based discussions. (and I should mention that I was not 
entirely convinced before the calls started)
Would love to see them continue.



On 30/08/17 20:16, "Carlos Santana"  wrote:

As always Matt great notes

On Wed, Aug 30, 2017 at 12:45 PM Matt Rutkowski  wrote:

> Youtube: https://youtu.be/oiuzssJlBXo
>
>
>
>
> From:   "Matt Rutkowski" 
> To: d...@openwhisk.incubator.apache.org
> Date:   08/30/2017 11:14 AM
> Subject:Tech Interchange posted for 2017-08-30
>
>
>
> Notes from today's meeting on CWIKI...
>
>
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_OPENWHISK_2017-2D08-2D30-2BOW-2BTech-2BInterchange-2B-2D-2BMeeting-2BNotes=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw=OFyAmIFMSwRsbBCg9wpguE7FcPwvzpjXStlxKbptAf8=Y326rg7ZSt8UzRATnyVdVaRRvC0Mz8aUGT9w5El9K3o=
>
>
> video to follow shortly
>
> Kind regards,
> Matt
>
>
>
>
>
>
>

Releases of rOpenWhisk (was: Updating Package and Language versions for a kind)

[Michael Marth]

James, all,

“If OpenWhisk did start to produce "releases""

I had it in my backlog to ask this - are we ready to do releases? I think we 
are, but wondered if something is holding us back that I am not aware of…

Michael





On 13/07/17 11:02, "James Thomas"  wrote:

>Good ideas Rob. I had a similar issue when looking at the Swift runtime
>recently.
>https://lists.apache.org/thread.html/fa01baca7d97d08c855abfc69fc17a23e038115fcfc4f2a31d650fa1@%3Cdev.openwhisk.apache.org%3E
>
>Would it be possible to have a scheduled upgrade process for installed
>modules? Once every four, six or eight weeks? If OpenWhisk did start to
>produce "releases", it could tie in with that.
>
>I'd guess that most people using the built-in packages are more kicking the
>tires than building production apps. Once you start being a production app,
>you'll want to explicitly bundle and control your app dependencies. I'd +1
>on being more aggressive with upgrading module versions.
>
>I'd like to have a Github issue to follow for this, I find it easier than
>the mailing list.
>
>On 13 July 2017 at 09:33, Rob Allen  wrote:
>
>> Hi all,
>>
>> On the PHP PR, @rr [commented] [1]:
>>
>> > The built in packages are convenient - less zip files for the initial
>> ramp up. But it creates a maintenance issue: when do you pick up updates to
>> the packages (minor/patch level only?) and not break existing actions using
>> the "kind". That is: is the kind itself semantically versioned?
>>
>> This applies to all kinds and so probably should be discussed project
>> level and ideally we should document how this is handled.
>>
>> There are two things here:
>>
>> 1. The language runtimes release patch updates for minor versions. e.g.
>> PHP `7.1.7` will become `7.1.8` next month with a small number of bug fixes
>> including crashers and possibly security fixes.
>>
>> 2. Each kind bindles a number of packages via the language's standard
>> package management system: Swift Package Manager for Swift, NPM for NodeJs,
>> etc. The projects that produce these packages update them with new versions
>> minor and patch versions.
>>
>> The tension is obviously between keeping updated for fixes vs the risk of
>> breaks due to a project's inability to keep BC between patch versions. e.g.
>> the NodeJS kind comes with the `async v2.1.4` package. However `v2.1.5` of
>> that package fixes a stack overflow issue. Should our actions have that
>> fix? Closer to home, the NodeJS kind ships with `OpenWhisk v3.3.2`, but the
>> latest is `v3.6.0` which is needed for non-experimental API Gateway support…
>>
>> Some questions:
>>
>> 1. Should we update the language runtime for a kind for a patch level
>> change (e.g. update the current NodeJS:6 kind from `6.9.1` to the latest
>> `6.9.5`?
>> 2. Should we ever update the language runtime for a kind for a minor level
>> change (e.g. update the current NodeJS:6 kind from `6.9.1` to the latest
>> `6.11.1`?
>> 3. Should we ever update the packages in a kind to the latest patch level
>> or minor level?
>> 4. What's our policy when a security issue is published for a language or
>> a package that we ship in a non-deprecated kind?
>>
>> Whatever the answers are, I think we should document them clearly
>> somewhere.
>>
>>
>> Also, I've started this conversation as a mailing list topic as it's a
>> "policy" thing. Given my previous comments on mailing lists, should I also
>> create a GitHub issue prefixed with "Discussion" to provide more visibility
>> in order to garner wider community input?
>>
>>
>> Regards,
>>
>> Rob...
>>
>> [1]: https://github.com/apache/incubator-openwhisk/pull/2415#
>> issuecomment-314716101 > incubator-openwhisk/pull/2415#issuecomment-314716101>
>
>
>
>
>-- 
>Regards,
>James Thomas

Re: Improving support for UI driven use cases

[Michael Marth]

Hi Alex,

That is a very interesting question.
If the programming model and guarantees that are exposed to developers involve 
guarantees on amount of memory then I still see two options:
- reserve the full capacity (i.e. Current model)
- or “overbook” a container. (not exactly the spirit of the current proposal 
but would lead to similar results)
This leads into a more product-management-like discussion if asking the 
developers to specify the amount of RAM they desire is a good thing in the 
first place. In the spirit of “devs shall not care about infra” it might be 
preferable not even make devs think about that and just execute the code with 
just enough RAM (or whatever resources are needed).
I mean you can look at the fact that some serverless providers expose RAM, etc 
to the developers as actually breaking the abstraction and working against the 
core value prop.
TBH I am not sure if there is a “right” way to look at this topic. Might depend 
on circumstances of the OW deployment.

Michael






On 05/07/17 17:45, "Alex Glikson" <glik...@il.ibm.com> wrote:

>Once different 'flavors' of pools/invokers are supported, one could 
>implement whatever policy for resource allocation and/or isolation and/or 
>load balancing they want in an invoker (or group of invokers) - without 
>necessarily affecting the 'core' of OpenWhisk, as long as the programming 
>model remains the same.
>However, with containers handling multiple requests, I am not sure that 
>the latter will be still true -- in particular, whether the developer can 
>still assume dedicated resource allocation per action invocation 
>(primarily memory), or we would also need to surface heterogeneous 
>'flavors' of resources allocated for an action (which might be perceived 
>as a natural and good thing - or maybe the opposite, given that we are 
>trying to make the developer unaware of infrastructure).
>
>Regards,
>Alex
>
>
>
>
>From:   "Michael M Behrendt" <michaelbehre...@de.ibm.com>
>To: dev@openwhisk.apache.org
>Date:   05/07/2017 05:58 PM
>Subject:Re: Improving support for UI driven use cases
>
>
>
>
>
>Hi Michael/Rodric,
>
>I'm struggling to understand how a separate invoker pool helps us avoiding
>to implement traditional autoscaling if we process multiple activations as
>threads within a shared process. Can you pls elaborate / provide an
>example?
>
>Sent from my iPhone
>
>> On 5. Jul 2017, at 16:53, Michael Marth <mma...@adobe.com.INVALID> 
>wrote:
>>
>> Michael B,
>> Re your question: exactly what Rodric said :)
>>
>>
>>
>>> On 05/07/17 12:32, "Rodric Rabbah" <rod...@gmail.com> wrote:
>>>
>>> The issue at hand is precisely because there isn't any autoscaling of
>capacity (N invokers provide M containers per invoker). Once all those
>slots are consumed any new requests are queued - as previously discussed.
>>>
>>> Adding more density per vm is one way of providing additional capacity
>over finite resources. This is the essence of the initial proposal.
>>>
>>> As noted in previous discussions on this topic, this should be viewed 
>as
>managing a different resource pool (and not the same pool of containers as
>ephemeral actions). Once you buy into that, generalization to other
>resource pools becomes natural.
>>>
>>> Going further, serverless becomes the new PaaS.
>>>
>>> -r
>>>
>>>> On Jul 5, 2017, at 6:11 AM, Michael M Behrendt
><michaelbehre...@de.ibm.com> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> thanks for the feedback -- glad you like my stmt re value prop :-)
>>>>
>>>> I might not yet have fully gotten my head around Steve's proposal --
>what
>>>> are your thoughts on how this would help avoiding the reimplementation
>of
>>>> an autoscaling / feedback loop mechanism, as we know it from more
>>>> traditional runtime platforms?
>>>>
>>>>
>>>> Thanks & best regards
>>>> Michael
>>>>
>>>>
>>>>
>>>> From:   Michael Marth <mma...@adobe.com.INVALID>
>>>> To: "dev@openwhisk.apache.org" <dev@openwhisk.apache.org>
>>>> Date:   07/05/2017 11:25 AM
>>>> Subject:Re: Improving support for UI driven use cases
>>>>
>>>>
>>>>
>>>> Hi Michael,
>>>>
>>>> Totally agree with your statement
>>>> ?value prop of serverless is that folks don't have to care about that"
>>>>
>>>> Again, the proposal at ha

Re: Improving support for UI driven use cases

[Michael Marth]

Hi Michael,

To make sure we mean the same thing with the word “autoscaling” in the context 
of this thread and in the context of OpenWhisk: I refer to the (automated) 
increase/decrease of the VMs that run the action containers.
Is that what you also refer to?

If so, then the proposal at hand is orthogonal to autoscaling. At its core it 
is about increasing the density of executing actions within one container and 
in that sense independent of how many containers, VMs, etc there are in the 
system or how the system is shrunk/grown.

In practical terms there is still a connection between proposal and scaling the 
VMs: if the density of executing actions is increased by orders of magnitude 
then the topic of scaling the VMs becomes a much less pressing topic (at least 
for the types of workload I described previously). But this practical 
consideration should not be mistaken for this being a discussion of autoscaling.

Please let me know if I misunderstood your use of the term autoscaling or if 
the above does not explain well.

Thanks!
Michael 




On 05/07/17 16:57, "Michael M Behrendt" <michaelbehre...@de.ibm.com> wrote:

>
>
>Hi Michael/Rodric,
>
>I'm struggling to understand how a separate invoker pool helps us avoiding
>to implement traditional autoscaling if we process multiple activations as
>threads within a shared process. Can you pls elaborate / provide an
>example?
>
>Sent from my iPhone
>
>> On 5. Jul 2017, at 16:53, Michael Marth <mma...@adobe.com.INVALID> wrote:
>>
>> Michael B,
>> Re your question: exactly what Rodric said :)
>>
>>
>>
>>> On 05/07/17 12:32, "Rodric Rabbah" <rod...@gmail.com> wrote:
>>>
>>> The issue at hand is precisely because there isn't any autoscaling of
>capacity (N invokers provide M containers per invoker). Once all those
>slots are consumed any new requests are queued - as previously discussed.
>>>
>>> Adding more density per vm is one way of providing additional capacity
>over finite resources. This is the essence of the initial proposal.
>>>
>>> As noted in previous discussions on this topic, this should be viewed as
>managing a different resource pool (and not the same pool of containers as
>ephemeral actions). Once you buy into that, generalization to other
>resource pools becomes natural.
>>>
>>> Going further, serverless becomes the new PaaS.
>>>
>>> -r
>>>
>>>> On Jul 5, 2017, at 6:11 AM, Michael M Behrendt
><michaelbehre...@de.ibm.com> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> thanks for the feedback -- glad you like my stmt re value prop :-)
>>>>
>>>> I might not yet have fully gotten my head around Steve's proposal --
>what
>>>> are your thoughts on how this would help avoiding the reimplementation
>of
>>>> an autoscaling / feedback loop mechanism, as we know it from more
>>>> traditional runtime platforms?
>>>>
>>>>
>>>> Thanks & best regards
>>>> Michael
>>>>
>>>>
>>>>
>>>> From:   Michael Marth <mma...@adobe.com.INVALID>
>>>> To: "dev@openwhisk.apache.org" <dev@openwhisk.apache.org>
>>>> Date:   07/05/2017 11:25 AM
>>>> Subject:Re: Improving support for UI driven use cases
>>>>
>>>>
>>>>
>>>> Hi Michael,
>>>>
>>>> Totally agree with your statement
>>>> ?value prop of serverless is that folks don't have to care about that"
>>>>
>>>> Again, the proposal at hand does not intend to change that at all. On
>the
>>>> contrary - in our mind it?s a requirement that the developer should not
>
>>>> change or that internals of the execution engines get exposed.
>>>>
>>>> I find Stephen?s comment about generalising the runtime behaviour very
>>>> exciting. It could open the door to very different types of workloads
>>>> (like training Tensorflow or running Spark jobs), but with the same
>value
>>>> prop: users do not have to care about the managing resources/servers.
>And
>>>> for providers of OW systems all the OW goodies would still apply (e.g.
>>>> running untrusted code). Moreover, if we split the Invoker into
>different
>>>> specialised Invokers then those different specialised workloads could
>live
>>>> independently from each other (in terms of code as well as resource
>>>> allocation in deployments).
>>>> You can probably tell I am

Re: Improving support for UI driven use cases

[Michael Marth]

Hi Michael,

Totally agree with your statement
“value prop of serverless is that folks don't have to care about that"

Again, the proposal at hand does not intend to change that at all. On the 
contrary - in our mind it’s a requirement that the developer should not change 
or that internals of the execution engines get exposed.

I find Stephen’s comment about generalising the runtime behaviour very 
exciting. It could open the door to very different types of workloads (like 
training Tensorflow or running Spark jobs), but with the same value prop: users 
do not have to care about the managing resources/servers. And for providers of 
OW systems all the OW goodies would still apply (e.g. running untrusted code). 
Moreover, if we split the Invoker into different specialised Invokers then 
those different specialised workloads could live independently from each other 
(in terms of code as well as resource allocation in deployments).
You can probably tell I am really excited about Stephen's idea :) I think it 
would be a great step forward in increasing the use cases for OW.

Cheers
Michael





On 04/07/17 20:15, "Michael M Behrendt"  wrote:

>Hi Dragos,
>
>> What stops
>> Openwhisk to be smart in observing the response times, CPU consumption,
>> memory consumption of the running containers ? 
>
>What are your thoughts on how this approach would be different from the many 
>IaaS- and PaaS-centric autoscaling solutions that have been built over the 
>last years? All of them require relatively complex policies (eg scale based on 
>cpu or mem utilization, end-user response time, etc.? What are the thresholds 
>for when to add/remove capacity?), and a value prop of serverless is that 
>folks don't have to care about that.
>
>we should discuss more during the call, but wanted to get this out as food for 
>thought.
>
>Sent from my iPhone
>
>On 4. Jul 2017, at 18:50, Dascalita Dragos  wrote:
>
>>> How could a developer understand how many requests per container to set
>> 
>> James, this is a good point, along with the other points in your email.
>> 
>> I think the developer doesn't need to know this info actually. What stops
>> Openwhisk to be smart in observing the response times, CPU consumption,
>> memory consumption of the running containers ? Doing so it could learn
>> automatically how many concurrent requests 1 action can handle. It might be
>> easier to solve this problem efficiently, instead of the other problem
>> which pushes the entire system to its limits when a couple of actions get a
>> lot of traffic.
>> 
>> 
>> 
>>> On Mon, Jul 3, 2017 at 10:08 AM James Thomas  wrote:
>>> 
>>> +1 on Markus' points about "crash safety" and "scaling". I can understand
>>> the reasons behind exploring this change but from a developer experience
>>> point of view this adds introduces a large amount of complexity to the
>>> programming model.
>>> 
>>> If I have a concurrent container serving 100 requests and one of the
>>> requests triggers a fatal error how does that affect the other requests?
>>> Tearing down the entire runtime environment will destroy all those
>>> requests.
>>> 
>>> How could a developer understand how many requests per container to set
>>> without a manual trial and error process? It also means you have to start
>>> considering things like race conditions or other challenges of concurrent
>>> code execution. This makes debugging and monitoring also more challenging.
>>> 
>>> Looking at the other serverless providers, I've not seen this featured
>>> requested before. Developers generally ask AWS to raise the concurrent
>>> invocations limit for their application. This keeps the platform doing the
>>> hard task of managing resources and being efficient and allows them to use
>>> the same programming model.
>>> 
 On 2 July 2017 at 11:05, Markus Thömmes  wrote:
 
 ...
 
>>> 
 
>>> To Rodric's points I think there are two topics to speak about and discuss:
 
 1. The programming model: The current model encourages users to break
 their actions apart in "functions" that take payload and return payload.
 Having a deployment model outlined could as noted encourage users to use
 OpenWhisk as a way to rapidly deploy/undeploy their usual webserver based
 applications. The current model is nice in that it solves a lot of
>>> problems
 for the customer in terms of scalability and "crash safeness".
 
 2. Raw throughput of our deployment model: Setting the concerns aside I
 think it is valid to explore concurrent invocations of actions on the
>>> same
 container. This does not necessarily mean that users start to deploy
 monolithic apps as noted above, but it certainly could. Keeping our
 JSON-in/JSON-out at least for now though, could encourage users to
>>> continue
 to think in functions. Having a toggle per action which is disabled by
 

Re: Improving support for UI driven use cases

[Michael Marth]

I like that approach a lot!




On 04/07/17 16:05, "Stephen Fink" <fink.step...@gmail.com> wrote:

>Hi all,
>
>I’ve been lurking a bit on this thread, but haven’t had time to fully digest 
>all the issues.
>
>I’d suggest that the first step is to support “multiple heterogeneous resource 
>pools”, where a resource pool is a set of invokers managed by a load balancer. 
> There are lots of reasons we may want to support invokers with different 
>flavors:  long-running actions, invokers in a VPN, invokers with GPUs,  
>invokers with big memory, invokers which support concurrent execution, etc…  . 
> If we had a general way to plug in a new resource pool, folks could feel 
>free to experiment with any new flavors they like without having to debate the 
>implications on other flavors.
>
>I tend to doubt that there is a “one size fits all” solution here, so I’d 
>suggest we bite the bullet and engineer for heterogeneity.
>
>SJF
>
>
>> On Jul 4, 2017, at 9:55 AM, Michael Marth <mma...@adobe.com.INVALID> wrote:
>> 
>> Hi Jeremias, all,
>> 
>> Tyson and Dragos are travelling this week, so that I don’t know by when they 
>> get to respond. I have worked with them on this topic, so let me jump in and 
>> comment until they are able to reply.
>> 
>> From my POV having a call like you suggest is a really good idea. Let’s wait 
>> for Tyson & Dragos to chime in to find a date.
>> 
>> As you mention the discussion so far was jumping across different topics, 
>> especially the use case, the problem to be solved and the proposed solution. 
>> In preparation of the call I think we can clarify use case and problem on 
>> the list. Here’s my view:
>> 
>> Use Case
>> 
>> For us the use case can be summarised with “dynamic, high performance 
>> websites/mobile apps”. This implies:
>> 1 High concurrency, i.e. Many requests coming in at the same time
>> 2 The code to be executed is the same code across these different requests 
>> (as opposed to a long tail distribution of many different actions being 
>> executed concurrently). In our case “many” would mean “hundreds” or a few 
>> thousand.
>> 3 The latency (time to start execution) matters, because human users are 
>> waiting for the response. Ideally, in these order of magnitudes of 
>> concurrent requests the latency should not change much.
>> 
>> All 3 requirements need to be satisfied for this use case.
>> In the discussion so far it was mentioned that there are other use cases 
>> which might have similar requirements. That’s great and I do not want to 
>> rule them out, obviously. The above is just to make it clear from where we 
>> are coming from.
>> 
>> At this point I would like to mention that it is my understanding that this 
>> use case is within OpenWhisk’s strike zone, i.e. Something that we all think 
>> is reasonable to support. Please speak up if you disagree.
>> 
>> The Problem
>> 
>> One can look at the problem in two ways:
>> Either you keep the resources of the OW system constant (i.e. No scaling). 
>> In that case latency increases very quickly as demonstrated by Tyson’s tests.
>> Or you increase the system’s capacity. In that case the amount of machines 
>> to satisfy this use case quickly becomes prohibitively expensive to run for 
>> the OW operator – where expensive is defined as “compared to traditional web 
>> servers” (in our case a standard Node.js server). Meaning, you need 100-1000 
>> concurrent action containers to serve what can be served by 1 or 2 Node.js 
>> containers.
>> 
>> Of course, the proposed solution is not a fundamental “fix” for the above. 
>> It would only move the needle ~2 orders of magnitude – so that the current 
>> problem would not be a problem in reality anymore (and simply remain as a 
>> theoretical problem). For me that would be good enough.
>> 
>> The solution approach
>> 
>> Would not like to comment on the proposed solution’s details (and leave that 
>> to Dragos and Tyson). However, it was mentioned that the approach would 
>> change the programming model for users:
>> Our mindset and approach was that we explicitly do not want  to change how 
>> OpenWhisk exposes itself to users. Meaning, users should still be able to 
>> use NPMs, etc  - i.e. This would be an internal implementation detail that 
>> is not visible for users. (we can make things more explicit to users and 
>> e.g. Have them requests a special concurrent runtime if we wish to do so – 
>> so far we tried to make it transparent to users, though).
>> 
>

Re: Improving support for UI driven use cases

[Michael Marth]

Hi Jeremias, all,

Tyson and Dragos are travelling this week, so that I don’t know by when they 
get to respond. I have worked with them on this topic, so let me jump in and 
comment until they are able to reply.

From my POV having a call like you suggest is a really good idea. Let’s wait 
for Tyson & Dragos to chime in to find a date.

As you mention the discussion so far was jumping across different topics, 
especially the use case, the problem to be solved and the proposed solution. In 
preparation of the call I think we can clarify use case and problem on the 
list. Here’s my view:

Use Case

For us the use case can be summarised with “dynamic, high performance 
websites/mobile apps”. This implies:
1 High concurrency, i.e. Many requests coming in at the same time
2 The code to be executed is the same code across these different requests (as 
opposed to a long tail distribution of many different actions being executed 
concurrently). In our case “many” would mean “hundreds” or a few thousand.
3 The latency (time to start execution) matters, because human users are 
waiting for the response. Ideally, in these order of magnitudes of concurrent 
requests the latency should not change much.

All 3 requirements need to be satisfied for this use case.
In the discussion so far it was mentioned that there are other use cases which 
might have similar requirements. That’s great and I do not want to rule them 
out, obviously. The above is just to make it clear from where we are coming 
from.

At this point I would like to mention that it is my understanding that this use 
case is within OpenWhisk’s strike zone, i.e. Something that we all think is 
reasonable to support. Please speak up if you disagree.

The Problem

One can look at the problem in two ways:
Either you keep the resources of the OW system constant (i.e. No scaling). In 
that case latency increases very quickly as demonstrated by Tyson’s tests.
Or you increase the system’s capacity. In that case the amount of machines to 
satisfy this use case quickly becomes prohibitively expensive to run for the OW 
operator – where expensive is defined as “compared to traditional web servers” 
(in our case a standard Node.js server). Meaning, you need 100-1000 concurrent 
action containers to serve what can be served by 1 or 2 Node.js containers.

Of course, the proposed solution is not a fundamental “fix” for the above. It 
would only move the needle ~2 orders of magnitude – so that the current problem 
would not be a problem in reality anymore (and simply remain as a theoretical 
problem). For me that would be good enough.

The solution approach

Would not like to comment on the proposed solution’s details (and leave that to 
Dragos and Tyson). However, it was mentioned that the approach would change the 
programming model for users:
Our mindset and approach was that we explicitly do not want  to change how 
OpenWhisk exposes itself to users. Meaning, users should still be able to use 
NPMs, etc  - i.e. This would be an internal implementation detail that is not 
visible for users. (we can make things more explicit to users and e.g. Have 
them requests a special concurrent runtime if we wish to do so – so far we 
tried to make it transparent to users, though).

Many thanks
Michael



On 03/07/17 14:48, "Jeremias Werner" 
> wrote:

Hi

Thanks for the write-up and the proposal. I think this is a nice idea and
sounds like a nice way of increasing throughput. Reading through the thread
it feels like there are different topics/problems mixed-up and the
discussion is becoming very complex already.

Therefore I would like to suggest that we streamline the discussion a bit,
maybe in a zoom.us session where we first give Tyson and Dragos the chance
to walk through the proposal and clarify questions of the audience. Once we
are all on the same page we could think of a discussion about the benefits
(improved throughput, latency) vs. challanges (resource sharing, crash
model, container lifetime, programming model) on the core of the proposal:
running multiple activations in a single user container. Once we have a
common understanding on that part we could step-up in the architecture and
discuss what's needed on higher components like invoker/load-balancer to
get this integrated.

(I said zoom.us session since I liked the one we had a few weeks ago. It
was efficient and interactive. If you like I could volunteer to setup the
session and/or writing the script/summary)

what do you think?

Many thanks in advance!

Jeremias


On Sun, Jul 2, 2017 at 5:43 PM, Rodric Rabbah 
> wrote:

You're discounting with event driven all use cases that are still latency
sensitive because they complete a response by call back or actuation at
completion. IoT, chatbots, notifications, all examples in addition to ui
which are latency sensitive and having uniform expectations on queuing time
is 

Re: Adding OpenTracing support

[Michael Marth]

Hi Juca,

I was in discussions with Sandeep before he created the PR for Zipkin support, 
so I can give some background info:
As a part of better understanding and improving the performance characteristics 
of OW we were simply looking for a way to profile the whole system. Zipkin 
seemed (still seems) to be fit for that job. So from our perspective 
plugability was not much of a concern, we “simply” wanted to get to the data. 

HTH
Michael




On 14/06/17 11:56, "Juraci Paixão Kröhling"  wrote:

>Hey there,
>
>I work on the Hawkular project and I'm currently checking if I could be 
>helpful in adding distributed tracing capabilities into OpenWhisk.
>
>I see that there's a PR already for Zipkin[1] and I'm wondering if there 
>is a reason for that choice, or if OpenTracing could be evaluated. The 
>main advantage, from my point of view, is that OpenTracing allows the 
>actual implementation to be pluggable.
>
>If OpenTracing would be a valid solution, I could work on a PoC for that.
>
>1 - https://github.com/apache/incubator-openwhisk/pull/2282
>
>Best,
>- Juca.

Re: Performance tests for OpenWhisk

[Michael Marth]

Markus,

Quick update: sent the below to users@infra. So far no reaction. The archive is 
here [1] but Bertrand tells me only ASF member have access  - for whatever 
reason.

Michael

[1] 
https://lists.apache.org/thread.html/70999f9233dac9b416ef9dedc97c0ef196a938c05d6a407b94ba3479@%3Cusers.infra.apache.org%3E


On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth 
<mma...@adobe.com<mailto:mma...@adobe.com>> wrote:
Dear Infra team,

I am enquiring on behalf of the OpenWhisk project (currently in Incubator)
[1].

We would like to periodically run performance tests on a distributed
environment (OpenWhisk typically runs on more than 1 machine). So we are
basically looking for an ability to spin up/tear down a number of (virtual)
machines and exclusively use them for a certain amount of time (so that the
VMs are not shared and the performance test results are comparable over
time).
The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.

I would like to find out if there is an ASF-supported mechanism to do that.
For example, can Infra provide such infrastructure? Or is there a cloud
provider (like Azure) that might sponsor such efforts with VMs? Or maybe
there is an established way for commercial companies that are interested in
an ASF project to sponsor (fund) such tests?

If none of the above exists, then it would also be helpful for us to get to
know how other projects run such sort of tests.

Thanks a lot!
Michael


[1]
https://lists.apache.org/thread.html/b66ab5b438f2db5cdc8c5f5eabece201b4ad090058fa3a9a3bd09d12@%3Cdev.openwhisk.apache.org%3E




From: Markus Thömmes <markusthoem...@me.com<mailto:markusthoem...@me.com>>
Reply-To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" 
<dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>>
Date: Wednesday 26 April 2017 12:59
To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" 
<dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>>
Subject: Re: Performance tests for OpenWhisk

Hi Michael,

yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with 4+ 
cores and 8GB memory. One VM would host the management stack while one would be 
dedicated to an Invoker only. That way we could assert single-invoker 
performance the easiest.

Thanks for helping!

Cheers,
Markus

Am 26. April 2017 um 11:36 schrieb Michael Marth 
<mma...@adobe.com<mailto:mma...@adobe.com>>:

Markus,

Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.

Let me know
Michael



On 26/04/17 07:52, "Bertrand Delacretaz" 
<bdelacre...@apache.org<mailto:bdelacre...@apache.org>> wrote:

Hi Michael,

On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth 
<mma...@adobe.com<mailto:mma...@adobe.com>> wrote:
...Maybe our mentors can chime in. Has this been discussed in the ASF board or 
so?...

Best would be to ask the ASF infrastructure team via
us...@infra.apache.org<mailto:us...@infra.apache.org> - briefly describe what 
you need to see what's
possible.

-Bertrand

Re: concurrent requests on actions

[Michael Marth]

Hi Tyson,

10x more throughput, i.e. Being able to run OW at 1/10 of the cost - definitely 
worth looking into :)

Like Rodric mentioned before I figured some features might become more complex 
to implement, like billing, log collection, etc. But given such a huge 
advancement on throughput that would be worth it IMHO.
One thing I wonder about, though, is resilience against rogue actions. If an 
action is blocking (in the Node-sense, not the OW-sense), would that not block 
Node’s event loop and thus block other actions in that container? One could 
argue, though, that this rogue action would only block other executions of 
itself, not affect other actions or customers. WDYT?

Michael




On 01/05/17 17:54, "Tyson Norris"  wrote:

>Hi All -
>I created this issue some time ago to discuss concurrent requests on actions: 
>[1] Some people mentioned discussing on the mailing list so I wanted to start 
>that discussion.
>
>I’ve been doing some testing against this branch with Markus’s work on the new 
>container pool: [2]
>I believe there are a few open PRs in upstream related to this work, but this 
>seemed like a reasonable place to test against a variety of the reactive 
>invoker and pool changes - I’d be interested to hear if anyone disagrees.
>
>Recently I ran some tests
>- with “throughput.sh” in [3] using concurrency of 10 (it will also be 
>interesting to test with the --rps option in loadtest...)
>- using a change that checks actions for an annotation “max-concurrent” (in 
>case there is some reason actions want to enforce current behavior of strict 
>serial invocation per container?)
>- when scheduling an actions against the pool, if there is a currently “busy” 
>container with this action, AND the annotation is present for this action, AND 
>concurrent requests < max-concurrent, the this container is used to invoke the 
>action
>
>Below is a summary (approx 10x throughput with concurrent requests) and I 
>would like to get some feedback on:
>- what are the cases for having actions that require container isolation per 
>request? node is a good example that should NOT need this, but maybe there are 
>cases where it is more important, e.g. if there are cases where stateful 
>actions are used?
>- log collection approach: I have not attempted to resolve log collection 
>issues; I would expect that revising the log sentinel marker to include the 
>activation ID would help, and logs stored with the activation would include 
>interleaved activations in some cases (which should be expected with 
>concurrent request processing?), and require some different logic to process 
>logs after an activation completes (e.g. logs emitted at the start of an 
>activation may have already been collected as part of another activation’s log 
>collection, etc).
>- advice on creating a PR to discuss this in more detail - should I wait for 
>more of the container pooling changes to get to master? Or submit a PR to 
>Markus’s new-containerpool branch?
>
>Thanks
>Tyson
>
>Summary of loadtest report with max-concurrent ENABLED (I used 1, but this 
>limit wasn’t reached):
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Target URL:  
>https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughputConcurrent?blocking=true
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Max requests:1
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Concurrency level:   10
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Agent:   keepalive
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Completed requests:  1
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Total errors:0
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Total time:  
>241.900480915 s
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Requests per second: 41
>[Sat Apr 29 2017 16:32:37 GMT+ (UTC)] INFO Mean latency:241.7 ms
>
>Summary of loadtest report with max-concurrent DISABLED:
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Target URL:  
>https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Max requests:1
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Concurrency level:   10
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Agent:   keepalive
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Completed requests:  1
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Total errors:19
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Total time:  
>2770.658048791 s
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Requests per second: 4
>[Sat Apr 29 2017 19:21:51 GMT+ (UTC)] INFO Mean latency:2767.3 ms
>
>
>
>
>
>[1] https://github.com/openwhisk/openwhisk/issues/2026
>[2] https://github.com/markusthoemmes/openwhisk/tree/new-containerpool
>[3] 

Re: Performance tests for OpenWhisk

[Michael Marth]

Markus,

Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.

Let me know
Michael



On 26/04/17 07:52, "Bertrand Delacretaz" <bdelacre...@apache.org> wrote:

>Hi Michael,
>
>On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth <mma...@adobe.com> wrote:
>> ...Maybe our mentors can chime in. Has this been discussed in the ASF board 
>> or so?...
>
>Best would be to ask the ASF infrastructure team via
>us...@infra.apache.org - briefly describe what you need to see what's
>possible.
>
>-Bertrand

Re: Save activations in a new activations DB

[Michael Marth]

Hi Christian,

Sorry to chime in late - I was out.
Recently, I had also been thinking about splitting more static configure data 
(like the action) from highly transactional data like the activations. My 
reason was, however, to have an easier  way forward to multi-datacenter 
deployments.

Regardless of the motivation for the split into activation-db and "static-db" I 
would like to make this comment:
Now, that we split out the activation-db into its own separate API we should 
take the opportunity to design this API (interface) that truly allows pluggable 
implementations. I am particularly interested in an Elastic Search impl for the 
activation-db (as I believe that ES lends itself well to 
activations-workloads). There might be other interesting impls. Point being: 
let's iterate a bit over the interface to make sure it can be implemented in 
non-CouchDB deployments. I am convinced this will be helpful in the future in 
order to evolve OW.
In the light of the above it would be useful to design the activation-db 
interface with a mindset that does not assume that activation-db and static-db 
are the same physical storage or even the same storage technology. Mentioning 
this to avoid assumptions about ID-semantics, joins, etc.

Wdyt?
Michael

Sent from a mobile device
_
From: apa...@cbickel.de
Sent: Wednesday, April 12, 2017 1:37 PM
Subject: Re: Save activations in a new activations DB
To: >


Hi OpenWhisk developers,

the first Pull Request I mentioned in the last mail 
(https://github.com/openwhisk/openwhisk/pull/2123) is merged now.
If you use one of the three environments in open, the default will still be, 
that activations are saved in the whisks-db. But now you can set that flag to 
true.
The next Pull request is opened now: 
https://github.com/openwhisk/openwhisk/pull/2134
It sets the default of the three environments to use the activations-db. It 
also requires this variable in your environment. It still can be set to true or 
false.

If you have some OpenWhisk deployments, I'd suggest to set the variable to 
false, do your migration and set it to true afterwards.
The migration is described here: 
https://gist.github.com/cbickel/37e651965781b27de245eac0ce831a53

My next, and last, PR will be to remove the ability to use that flag.

If you have any problems with your migration or concerns just reach out to me.

Greetings Christian


April 10, 2017 2:56 PM, apa...@cbickel.de wrote:

> Hi OpenWhisk developers,
>
> some weeks ago, I started working on some database performance and one item 
> affects the
> whisks-database.
> All actions, rules, triggers, packages and activations are stored in this 
> database. Because of all
> the activations, this database becomes really huge over time.
> To keep the database with the artifacts, created by the users (actions, ...), 
> small we had the idea
> to put all activations into another database. The activations-db.
>
> To solve the problem of the migration of existing OpenWhisk deployments, my 
> proposal is to make the
> split configurable with ansible.
> PR 2123 (https://github.com/openwhisk/openwhisk/pull/2123) creates this flag. 
> By default, the
> whisks database will be used to store activations.
> After some time, I'll open another PR, that changes the default to use the 
> activations-db.
> Again, after some time, I'll open a third PR to remove this flag again. By 
> default, all activations
> will go into the activations-db.
> The reason for removing this flag again is, that it will be easier to 
> maintain only one
> configuration: the configuration to use two seperate dbs.
> And we think, that all owners of OpenWhisk deployments agree that it would be 
> good to split
> configuration data of the user (actions, ...) and logs (activations, ...).
>
> If you need to migrate your deployment, it would work with the steps on the 
> following document:
> https://gist.github.com/cbickel/37e651965781b27de245eac0ce831a53
> (https://gist.github.com/cbickel/37e651965781b27de245eac0ce831a53)
>
> Does anyone have any concerns about this change or any comments?
>
> Greetings Christian

Re: factoring out "runtimes" into deployment configuration

[Michael Marth]

Hi Rodric,

I think that’s great.

Only comment (more to Matt than you):
If the runtimes are dynamic by deployment (and of course over time) this 
probably should be reflected in the packaging format spec [1].
(see also previous comment [2]:)

- line 252: not sure if the list of runtime names should be in the normative
section of a spec, because (as you also write) this list will fluctuate between
deployments and over time. OTOH there should be established well known names for
well-known runtimes. Maybe point to a community-curated, separate markdown file
(where new entries get a PR that can be voted upon)?

Cheers
Michael

[1] 
https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/specification/openwhisk_v0.8.pdf
[2] http://markmail.org/message/pa35wxxl52tvbfxc



On 05/03/17 17:39, "Rodric Rabbah" > 
wrote:

I've been thinking about how we can add more action runtimes more easily -
without changing the backend (API, controller, invoker, CLI). One way I'm
leaning toward and started prototyping to get a better understanding of the
feasibility is to strip all runtime types from the backend (and CLI) and
encode the information about what action runtimes the system supports
through configuration parameters.

This will make it possible to decouple deploying the core components from
managing the action runtime images. An example of encoding the runtime
information in a deployment configuration is
https://github.com/openwhisk/openwhisk/pull/1948.

In support of this general direction (even if we settle on a different
approach), I have increasingly flattened the "Exec" type hierarchy in the
backend[1-3]

[1] https://github.com/openwhisk/openwhisk/pull/1938
[2] https://github.com/openwhisk/openwhisk/pull/1942
[3] https://github.com/openwhisk/openwhisk/pull/1911

A sketch of how to use the runtime manifest can be seen in this WIP commit:
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f

I can see how using this we can add new versions of Node.js, python:3, and
other handy images by curating a list of these in a separate process.

Feedback, as always, solicited and appreciated.

-r

Re: Passing a context object to actions

[Michael Marth]

Hi Rodric,

Thanks for passing the link! Very interesting.

IIUC the discussion then the consensus for that particular issue is summarised 
in [1], but this is not implemented, yet. Is this correct?

Also, I have a question about the solution in [1]
(Happy to post it into that issue, if that’s the better place to have the 
discussion) 
I wonder why the parameter *values* are show up in the sequence definition at 
*creation* time of the sequence in
“wsk action create sequence S —action A1 P11 V11 P12 V12 —action A2 P21 V21 P22 
V23"
That seems to not allow for setting parameter values at invocation time - or do 
I misinterpret the proposal?

Thanks!
Michael

[1] https://github.com/openwhisk/openwhisk/issues/116#issuecomment-229976950




On 17/01/17 13:55, "Rodric Rabbah" <rod...@gmail.com> wrote:

>You're raising a different issue that is orthogonal to final/immutable 
>parameters. For sequences and parameter scoping see this discussion: 
>https://github.com/openwhisk/openwhisk/issues/116
>
>-r
>
>> On Jan 17, 2017, at 7:47 AM, Michael Marth <mma...@adobe.com> wrote:
>> 
>> Hi Rodric, all,
>> 
>> Final parameters sound nice. I think they probably solve the challenge that 
>> Dragos outlined originally.
>> However, I am not sure if they are the ideal solution for OW in general.
>> 
>> Let me re-cap the issue:
>> * assume OW deployments where an API gateway authenticates incoming requests 
>> and sets e.g. the userId as a header that gets forwarded to the action
>> * the subsequently invoked action’s code wants to read that header (userId)
>> * if a sequence was invoked by the request each action in the sequence must 
>> be able to read that header (and be sure that it has not been tampered with 
>> by previous actions in the sequence)
>> 
>> IIUC final parameters solve this challenge provided that all involved 
>> parties (gateway deployment and all actions in the sequence) agree on the 
>> name and semantics of that header containing the userId. That is not a 
>> problem as long as these actions always get deployed into one particular OW 
>> deployment. However, I think it breaks compatibility of actions when moved 
>> between different OW deployments.
>> I see the downsides of changing the method signature that you mentioned, 
>> Rodric. Maybe there’s another way? (like environment variables that can be 
>> read from the function or globally accessible objects, etc)
>> Is this already solved in Bluemix?
>> 
>> Cheers
>> Michael
>> 
>> 
>> 
>>> On 11/01/17 23:17, "Rodric Rabbah" <rod...@gmail.com> wrote:
>>> 
>>> +1 to @sjfink: passing large objects inline should probably be adopted as
>>> an anti-pattern.
>>> 
>>>> I'm also adding an extra thought to our thread: if we want to communicate
>>> a "user_id" and "app_id" to the actions, but any action can edit the
>>> incoming event, how would we make sure that some important fields like
>>> these ones can't be overwritten by other actions in a sequence and they can
>>> be securely passed through and trusted ?
>>> 
>>> @ddascal <ddas...@adobe.com> I just opened this PR that prototypes the idea
>>> of "final" parameters that might be applicable. This came up in discussion
>>> with Steve where he suggested that any defined parameter (if it has a
>>> value) should be considered final. This would mean that parameters on a
>>> package that have a default value cannot be overridden by a binding or an
>>> action in the package. I implemented a variant of this [1] that applies to
>>> certain openwhisk actions where an action may carry a "final" annotation
>>> which prevents an incoming request from overriding any of its predefined
>>> parameters.
>>> 
>>> In a more general perspective, we can give up on default parameters in
>>> favor of final parameters everywhere. Or a compromise where parameters may
>>> have defaults and may be overriden from package to binding to action but
>>> not from invoke time parameters (as in the pull request).
>>> 
>>> -r
>>> 
>>> [1] https://github.com/openwhisk/openwhisk/pull/1710

Re: Passing a context object to actions

[Michael Marth]

Thanks Matt - will have a look and report back here if anything comes up.

Cheers
Michael



On 11/01/17 22:23, "Matt Rutkowski" <mrutk...@us.ibm.com> wrote:

>Hi Michael,
>
>I should chime in here...
>
>Part of the "packaging specification" work we are doing (under the 
>openwhisk-wskdeploy tool/repo) seeks to provide a YAML descriptor of OW 
>packages (incl. Actions, Feeds, Triggers, Event Providers, etc.) including 
>a description of an actions expected inputs and outputs (optionally typed 
>and constrained).  This would be used by tooling (and validation) 
>components in order to confirm if Actions that are composed (currently 
>sequenced) are compatible.  We have discussed the possibility of also 
>using the information to provide inter-sequence (platform provided) 
>"massaging" of types/names in order for a more automated way to connect 
>Actions without forcing a clone/fork/rewrite.
>
>In addition, we provide a manifest file that tells the tool where to 
>"pull" input values from, provide credentials for external services (as 
>well as define their signature and location/protocol).  Perhaps you could 
>review this work and provide feedback?
>
>Kind regards,
>Matt 
>
>
>
>From:   Michael Marth <mma...@adobe.com>
>To: "dev@openwhisk.apache.org" <dev@openwhisk.apache.org>
>Date:   01/11/2017 08:51 AM
>Subject:Re: Passing a context object to actions
>
>
>
>Hi Rodric,
>
>Re
>
>This has come up before. The reason the signature of actions is dictionary
>-> dictionary is for action composition. If your signature is (dictionary,
>dictionary) -> dictionary, you cannot compose actions into a sequence for
>example. One of the tenants of the underlying programming model is that
>composition, as with any programming language, is a fundamental feature.
>
>This is an interesting aspect. Do you happen to have a link to these 
>previous discussions? Would be great to read up on the arguments.
>
>Thanks!
>Michael
>
>
>On 07/01/17 03:13, "Rodric Rabbah" <rod...@gmail.com<
>mailto:rod...@gmail.com>> wrote:
>
>   * The fact that we need to unpack the request body and change it has an
>impact on performance
>
>Why? Let the event payload be E, and the request context C, you can pass 
>to
>your action a new dictionary { “event” -> E, “context” -> C }. The gateway
>doesn’t need to decode E and can treat it as an opaque value. Can you
>clarify why the request body must be unpacked?
>
>   * If we want to send binary payloads in a special format, the API
>Gateway or some other process in the middle has to know how to
>decode/encode that payload/Content-Type. So we're limited to what these
>intermediary processes know.
>
>See above, you don’t need to add to the event payload, you can box and
>decoration in the new object.
>
>The same thing happens with default action parameters; they get merged
>with the event. Basically besides the event itself there's no other way to
>configure the action with extra params or pass it a context.
>
>OpenWhisk actions, as functions, have the signature f: dictionary ->
>dictionary. You can modify the dictionary any way you like. So if your 
>want
>to encode a function that receives three parameters g(E, C, P) where P are
>bound params, isn’t this equivalent to a dictionary with three properties
>for each argument? The mapping is isomorphic.
>
>AWS Lambda allows a similar mechanism to pass a "context" to a function.
>In OpenWhisk actions the main method signature would allow 2 params
>instead of 1 (as it is today) and it could look like: function main(event,
>context)
>
>This has come up before. The reason the signature of actions is dictionary
>-> dictionary is for action composition. If your signature is (dictionary,
>dictionary) -> dictionary, you cannot compose actions into a sequence for
>example. One of the tenants of the underlying programming model is that
>composition, as with any programming language, is a fundamental feature.
>
>And really what is a context other than just an added property? Can you
>show me a case where boxing an opaque value into another dictionary 
>doesn’t
>give you the feature you want?
>
>If you start going down the road of changing the action signature, then I
>think you have to buy into a much more disruptive idea, where the 
>signature
>of a method implies legal vs illegal compositions, and perhaps even admit
>typed parameters.
>
>-r
>
>
>
>
>

Re: Passing a context object to actions

[Michael Marth]

Hi Rodric,

Re

This has come up before. The reason the signature of actions is dictionary
-> dictionary is for action composition. If your signature is (dictionary,
dictionary) -> dictionary, you cannot compose actions into a sequence for
example. One of the tenants of the underlying programming model is that
composition, as with any programming language, is a fundamental feature.

This is an interesting aspect. Do you happen to have a link to these previous 
discussions? Would be great to read up on the arguments.

Thanks!
Michael


On 07/01/17 03:13, "Rodric Rabbah" > 
wrote:

   * The fact that we need to unpack the request body and change it has an
impact on performance

Why? Let the event payload be E, and the request context C, you can pass to
your action a new dictionary { “event” -> E, “context” -> C }. The gateway
doesn’t need to decode E and can treat it as an opaque value. Can you
clarify why the request body must be unpacked?

   * If we want to send binary payloads in a special format, the API
Gateway or some other process in the middle has to know how to
decode/encode that payload/Content-Type. So we're limited to what these
intermediary processes know.

See above, you don’t need to add to the event payload, you can box and
decoration in the new object.

The same thing happens with default action parameters; they get merged
with the event. Basically besides the event itself there's no other way to
configure the action with extra params or pass it a context.

OpenWhisk actions, as functions, have the signature f: dictionary ->
dictionary. You can modify the dictionary any way you like. So if your want
to encode a function that receives three parameters g(E, C, P) where P are
bound params, isn’t this equivalent to a dictionary with three properties
for each argument? The mapping is isomorphic.

AWS Lambda allows a similar mechanism to pass a "context" to a function.
In OpenWhisk actions the main method signature would allow 2 params
instead of 1 (as it is today) and it could look like: function main(event,
context)

This has come up before. The reason the signature of actions is dictionary
-> dictionary is for action composition. If your signature is (dictionary,
dictionary) -> dictionary, you cannot compose actions into a sequence for
example. One of the tenants of the underlying programming model is that
composition, as with any programming language, is a fundamental feature.

And really what is a context other than just an added property? Can you
show me a case where boxing an opaque value into another dictionary doesn’t
give you the feature you want?

If you start going down the road of changing the action signature, then I
think you have to buy into a much more disruptive idea, where the signature
of a method implies legal vs illegal compositions, and perhaps even admit
typed parameters.

-r