Re: detection of base64 encoded code fields

[Markus Thömmes]

I agree, I don't think it's possible to accurately tell base64 from some
random string without input from the user.

Am Sa., 18. Apr. 2020 um 23:46 Uhr schrieb Rodric Rabbah :

> Actually my suggested decoding step doesn't work either. I don't think
> there's a sound way to record the binary property without cooperation from
> the clients.
>
> -r
>
> On Sat, Apr 18, 2020 at 4:38 PM Rodric Rabbah  wrote:
>
> > Hello OpenWhiskers.
> >
> > When an action is created or updated, the controller detects if the
> "code"
> > part of the HTTP Request is base64 encoded or a plain text string. The
> > current heuristic for detecting if the code is base64 encoded is buggy.
> It
> > uses a regex match
> >
> >
> ^([A-Za-z0-9+\/]{4})*([A-Za-z0-9+\/]{4}|[A-Za-z0-9+\/]{3}=|[A-Za-z0-9+\/]{2}==)$
> >
> > which will match a 4-character string such as "abcd" or "true". This is
> > outlined in an issue https://github.com/apache/openwhisk/issues/4884.
> >
> > We used to attempt to base64 decode the code string to determine if it
> was
> > base 64 encoded but this was replaced with the regex match. For the most
> > part the heuristic is OK but this is not completely sound.
> >
> > I am soliciting feedback on the following schema change to the API. The
> > `binary` property on of the exec field is currently not required and
> > ignored by the controller. This forces the controller to always try to
> > determine the value of this property.
> >
> > Instead, I'm suggesting we modify the schema to accept the binary
> > property. If present it is used as is and it is the caller/client
> > responsibility to set it accordingly. In the absence of this property in
> > the HTTP request, for backward compatibility, augment the heuristic to
> > perform the expensive decoding step as a last step in the confirmation
> > process.
> >
> > For actions already in the database (previously created actions), the
> > `binary` property is already set in the document in the db and can be
> used
> > as is, instead of recomputing the value.
> >
> > In essence this is a step toward moving the responsibility to the clients
> > instead of the server.
> > Thoughts? In the absence of input, I will open a PR to implement the
> > change in this general direction. Thanks for reading this far.
> >
> > -r
> >
> >
>

Re: discuss whether need to consider prewarmedPool as consumed memory resources

[Markus Thömmes]

Hi,

this does mean, that the prewarm pool's capacity cannot be reclaimed by
other actions should the need arise, right? I think that's problematic if
that notion isn't carried through into loadbalancing itself as AFAIK the
controller currently does not take the prewarm pool into account at all.

Which kind of a workload pattern would that benefit? Got any benchmark
and/or just data showcasing that?

Cheers,
Markus

Am Mo., 30. März 2020 um 10:13 Uhr schrieb 甯尤刚 :

> Hi, guys
> ​
> I wrote a pr here: https://github.com/apache/openwhisk/pull/4870
> this pr's target: make prewarmedPool as `already consumed memory
> resources`, this pr has an benefit that if consider prewarmedPool as
> consumed memory resource, can aovid to do  much `cold start a container`
> when the time of `space is consumed finished` is coming
> ​
> Have any suggestion?
>

Re: JDK 11

[Markus Thömmes]

FWIW, the UUID entropy workaround where not JVM specific IIRC but OS
dependent, as we switched from random to urandom as a source of entropy.

I don't feel strongly for either OpenJDK vs. OpenJ9, but it'd be nice to
understand where the degradation comes from and thus make a deliberate
decision to choose one over the other based on that research. If anything,
that might be very valuable input for the OpenJDK community to fix this
regression, maybe? That research could also help us better understand where
the bottlenecks of our system are and thus point us towards improvements to
be made.

We could of course switch to OpenJ9 to bypass the degradation if we want to
move to JDK11 quickly.

Thanks for sharing the results,
Markus

Am Di., 28. Jan. 2020 um 18:24 Uhr schrieb Martin Henke :

> Rodric,
>
> UUID creation was just an example from the past where workarounds were
> needed to circumvent waiting for enough entropy for the random numbers used
> for UUID creation. As said we did no further investigation.
>
> Yes. OpenJ9 is Apache licensed and distributed via AdoptOpenJDK. There are
> no further changes needed.
>
> Regards,
> Martin
>
> > Am 28.01.2020 um 18:09 schrieb Rodric Rabbah :
> >
> > Thanks Martin for sharing the results with the community - That's a
> steep
> > performance degradation.
> > Open J9 is Apache licensed. Are other changes needed?
> >
> > You hinted at uuid - do you know that's implicated or a guess?
> >
> > -r
> >
> >> On Tue, Jan 28, 2020 at 8:05 AM Martin Henke 
> wrote:
> >>
> >> Folks,
> >>
> >> as indicated, we performed performance tests of the OpenJDK upgrade PR
> >> https://github.com/apache/openwhisk/pull/4706
> >> in our performance test environment (we are using the exact set of
> Gatling
> >> tests that are contributed to Openwhisk).
> >>
> >> Though our first impressions looked good, we found severe degradations
> in
> >> the latency and warm invocation tests afterwards.
> >>
> >> Latency tests dropped to around 50% of the JDK8 results, warm invocation
> >> to a staggering 25%.
> >> These results were stable with using the original and the updated
> Gatling
> >> image and switching to parallelGC on OpenJDK11.
> >>
> >> Since GC does not seem to matter,  the degradation might be caused by a
> >> changed
> >> implementation in the  OpenJDK 11 (e.g. UUID generation).
> >> We did not perform any deeper research.
> >>
> >> Next we tried to change to use the OpenJ9 JDK11 instead of OpenJDK.
> >> Change in common/scala/Dockerfile:
> >> "
> >> -FROM adoptopenjdk/openjdk11:x86_64-alpine-jdk-11.0.3_7
> >> +FROM
> >> adoptopenjdk/openjdk11-openj9:x86_64-alpine-jdk-11.0.5_10_openj9-0.17.0
> >> "
> >> With OpenJ9 in default configuration all performance metrics were
> looking
> >> good again.
> >> We did not see any degredation anymore.
> >>
> >> In the light of these test results, I herewith would like to start the
> >> discussion to base the JDK update PR
> >> to openJ9 JDK11.
> >>
> >> Kind Regards,
> >> Martin
> >>
>
>

Re: OpenWhisk project goodies

[Markus Thömmes]

Thanks for taking this on you Rodric. For new contributors it can really
make all the difference to be welcomed into a community like this. Kudos to
you!

Happy holidays!

Am Mi., 18. Dez. 2019 um 17:16 Uhr schrieb Rodric Rabbah :

> As a project and community we have a lot to celebrate this year and be
> thankful for. I'm thankful for all the new contributors and contributions,
> for the community's recognition of new committers, and for all the work
> that went into the project graduation.
>
> On Slack yesterday I asked if new community members who haven't received
> any project swag would like some goodies (a coffee  or tea mug, t-shirt or
> hoodie, stickers...) and ended up sending out a bunch of gifts. It's a
> small token of appreciation for our community.
>
> This message is to reach out to other community members not on slack who'd
> like one of these goodies. Email me directly or message me on slack. You'll
> find items https://www.redbubble.com/ (search for our project name to see
> what's available and if you only get one result, click it and scroll down
> to see more).
>
> Happy holidays and my best wishes to everyone!
>
> -r
>

Re: Road to Scala 2.13

[Markus Thömmes]

Hey all,

here's a set of potentially downstream breaking changes:
https://github.com/apache/openwhisk/pull/4736: Note how I changed "|
string" to "| lower" in the ansible scripts. Pureconfig seems to be more
picky when it comes to Boolean values and only allows "true" and "false"
there. Python usually produces "True" and "False", which caused things to
break.
https://github.com/apache/openwhisk/pull/4749: This updates the
akka-management parts of the system which potentially has an impact on
setups that use Kubernetes or Mesos API to build the controller cluster.

Am Do., 21. Nov. 2019 um 15:10 Uhr schrieb Carlos Santana <
csantan...@gmail.com>:

> Woot +1
>
> - Carlos Santana
> @csantanapr
>
> > On Nov 21, 2019, at 5:58 AM, David P Grove  wrote:
> >
> > 
> >
> >
> > "Markus Thömmes"  wrote on 11/21/2019
> 02:03:30
> > AM:
> >>>
> >>> One thing to be careful of is that many downstream repos (runtime,
> >>> providers) depend on the test suite from the core repo.  We had a
> > couple
> >>> rounds of breakage in the last few months where a test suite change got
> > a
> >>> clean core travis run, but still broke all the downstream repos leading
> > to
> >>> some hasty fixing and/or reverting.
> >>>
> >>
> >> I hear you! Is there a good way to verify stuff works across all
> >> repositories or will I have to manually go through all of them and check
> > if
> >> my changes have any impact?
> >>
> >>
> >
> > Not going to be perfect, but I think all of the breakages we had would
> have
> > been caught by building even one downstream repo against the modified
> core
> > since the test suite is used more or less the same way in all the runtime
> > repos.
> >
> > --dave
>

Re: Pool slot was shut down

[Markus Thömmes]

The fix you shared seems to be only logging related so it seems that these
logs are harmless? If we don't see any detrimental effects to actual usage
I'd vote for waiting for the fix to be released to just shut down the noisy
logging. If we see actual errors though, we of course need to look at
reverting the change (and potential collapsing changes too).

Sorry for the noise.

Am Mi., 27. Nov. 2019 um 22:06 Uhr schrieb David P Grove :

> Doing a little googling, I can see an issue [1] in akka-http with a merged
> fix [2] that looks like it might be related.   Also looks like akka-http
> 10.1.11 is about a week out from release [3].
>
> Not clear to me if that means we should revert our akka-http version bump
> until 10.1.11 is out and try again then or not.  Markus?
>
> --dave
>
> [1] https://github.com/akka/akka-http/issues/2728
> [2] https://github.com/akka/akka-http/pull/2816
> [3] https://github.com/akka/akka-http/issues/2836
>
> "Ruediger Maass"  wrote on 11/27/2019 03:37:07
> PM:
> >
> > Since a few days I observe many entries in the logs of our builds like
> > this
> >
> > [2019-11-25T09:30:56.217Z] [ERROR] Outgoing request stream error
> >
> > akka.http.impl.engine.client.pool.NewHostConnectionPool
> > $HostConnectionPoolStage$$anon$1$Slot$$anon$2:
> > Pool slot was shut down
> >
> > When looking at green Openwhisk builds (travis) I also observe these
> > problems - e.g,.in
> > https://urldefense.proofpoint.com/v2/url?
> >
>
> u=https-3A__travis-2Dci.org_apache_openwhisk_jobs_616576483-3Futm-5Fmedium-3Dnotification-26utm-5Fsource-3Dgithub-5Fstatus&d=DwIFAg&c=jf_iaSHvJObTbx-
>
> > siA1ZOg&r=Fe4FicGBU_20P2yihxV-
> >
>
> apaNSFb6BSj6AlkptSF2gMk&m=mwUYQarkVTPbD6tBpHhyHglEMwOXnzknlvWyQn8Wqw4&s=s8p5T1wfxxv4ljEwFWrXMEONyMEY94h9GG3MmpWbgnY&e=
>
> >
> > Something going wrong here?
> >
> > Thanks, Ruediger
> >
> >
> >
>

Re: Road to Scala 2.13

[Markus Thömmes]

Am Mi., 20. Nov. 2019 um 19:52 Uhr schrieb David P Grove :

>
> "Markus Thömmes"  wrote on 11/20/2019 10:44:21
> AM:
> >
> > Hey fellow Openwhiskers,
> >
> > I somewhat fancy a little bit of Scala action and wanted to give updating
> > Openwhisk to Scala 2.13 a shot. They have reworked quite a bit in Scala
> > 2.13 and keeping track with the versions might be very valuable,
> especially
> > if Dotty comes around eventually.
> >
> > To pull that off, we'll need to bump quite a few libraries, some of them
> > even in their major version. I plan to send each of those major upgrades
> as
> > individual PRs, so we can pinpoint potential issues via git bisecting if
> > necessary.
> 
> >
> > We'll see if there are more dragons, but I'm happy to work through them
> to
> > some extent :). Is anybody concerned with doing this move in general and
> > with the process laid out specifically?
> >
>
> Generally sounds good; happy to have someone pushing on chasing language
> versions :)
>
> One thing to be careful of is that many downstream repos (runtime,
> providers) depend on the test suite from the core repo.  We had a couple
> rounds of breakage in the last few months where a test suite change got a
> clean core travis run, but still broke all the downstream repos leading to
> some hasty fixing and/or reverting.
>

I hear you! Is there a good way to verify stuff works across all
repositories or will I have to manually go through all of them and check if
my changes have any impact?


>
> -dave
>

Road to Scala 2.13

[Markus Thömmes]

Hey fellow Openwhiskers,

I somewhat fancy a little bit of Scala action and wanted to give updating
Openwhisk to Scala 2.13 a shot. They have reworked quite a bit in Scala
2.13 and keeping track with the versions might be very valuable, especially
if Dotty comes around eventually.

To pull that off, we'll need to bump quite a few libraries, some of them
even in their major version. I plan to send each of those major upgrades as
individual PRs, so we can pinpoint potential issues via git bisecting if
necessary.

kamon-statsd and rxscala are two libraries where there are no 2.13
artifacts published yet. RxScala meanwhile has support of 2.13 in its
repository, so we'd need to wait on a new release there. kamon-statsd has
an open issue to support 2.13, we might need to go ahead and nudge a bit
there, see https://github.com/kamon-io/kamon-statsd/issues/9. Do we even
need the statds integration anymore at this point? @Sven Lange-Last might
know.

We'll see if there are more dragons, but I'm happy to work through them to
some extent :). Is anybody concerned with doing this move in general and
with the process laid out specifically?

Cheers,
Markus

Re: Extend the SPI entitlement interface

[Markus Thömmes]

Hi Steffen,

the interface is already Future[Boolean], why is it not sufficient to fail
the future if the network calls fail for any reason in this case? After
all, that's what the Future's failure mode is all about. Or asked
differently: What states does a failed Future represent today?

Cheers,
Markus

Am So., 3. Nov. 2019 um 17:27 Uhr schrieb Steffen Rost :

> The current implementation of the entitlement SPI allows providers to
> return only true (entitled/valid) or false (not entitled/invalid) as
> result of an entitlement request. While this is sufficient if the
> entitlement request is performed offline it doesn't allow to signal that
> the entitlement request couldn't be performed at all when calling an
> external entitlement service failed because of a network error or
> unavailability of the service itself. In order to address these cases we
> propose to change the entitlement SPI interface from Future[Boolean] to
> Future[Option[Boolean]] (for implementation details please follow the link
> under [1])
>
> In our IBM Functions service we use IAM (Identity and Access Management).
> For IAM enabled namespaces an IAM/Bearer token is passed on each request
> which is checked for access allowance on the requested namespace resources
> using an external policy system. If the request succeeds an user identity
> is created which is passed to user/system containers and can be used to
> create triggers or to reach out to other services.
>
> A problem arises if the entitlement request to the external policy system
> cannot be performed. As mentioned before this could happen due to a
> temporary network outage, the unavailability of the external service or
> the service decides to throttle the request due to availability issues. In
> any of these cases the SPI provider can currently only return invalid
> which is interpreted as 403/Forbidden even though the request would be
> perfectly valid if it could only have been done. In our system this
> problem caused recently many customer complains as triggers were disabled
> because of those false positives.
>
> With the proposed change our IAM entitlement SPI implementation could
> return none in such cases to signal that the entitlement couldn't be done
> allowing the caller to act more appropriate, for example do not disable
> the trigger.
>
> I will open a pull request for the proposed interface change. Feel free to
> comment now to this mail or to the pull request.
>
>
> [1]
>
> https://github.com/apache/openwhisk/blob/7062549035eb45f376653ce2433fcbe70e6fd806/core/controller/src/main/scala/org/apache/openwhisk/core/entitlement/Entitlement.scala#L180-L189
>
>
> Mit freundlichen Gruessen / Kind regards
> Steffen Rost
>
> --
> IBM Cloud Functions Development
> Phone +49-7031-16-4841 (Fax: -3545)
> E-Mail: sr...@de.ibm.com
>
> --
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Matthias Hartmann -- Geschäftsführung:
> Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
> HRB 243294
>
>

Re: Action health checks

[Markus Thömmes]

Yes, I used the word "retry" here to mean "reschedule to another
container", just like you would if the healthiness probe failed.

A word of caution: TCP probes might be behaving strangely in a container
setting. They sometimes accept connections even though nothing is listening
and stuff like that.

Am Mi., 30. Okt. 2019 um 16:34 Uhr schrieb Tyson Norris
:

> I don't think "retry" is the right handling for warm connection failures -
> if a connection cannot be made due to container crash/removal, it won't
> suddenly come back. I would instead treat it as a "reschedule", where the
> failure routes the activation back to ContainerPool, to be scheduled to a
> different container. I'm not sure how distinct we can be on detecting
> contrainer failure vs temporary network issue that may or may not resolve
> on its own, so I would treat them the same, and assume the container is
> gone.
>
> So for this PR, is there any objection to:
> - for prewarm, use the tcp connection for monitoring outside of activation
> workflow
> - for warm, handle it as a case of retry, where request *connection*
> failure only for /run, will be handled by way of rescheduling back to
> ContainerPool (/init should already be handled by retry for a time period).
>
> Thanks!
> Tyson
>
> On 10/30/19, 7:03 AM, "Markus Thömmes"  wrote:
>
> Increasing latency would be my biggest concern here as well. With a
> health
> ping, we can't even be sure that a container is still healthy for the
> "real
> request". To guarantee that, I'd still propose to have a look at the
> possible failure modes and implement a retry mechanism on them. If you
> get
> a "connection refused" error, I'm fairly certain that it can be retried
> without harm. In fact, any error where we can guarantee that we haven't
> actually reached the container can be safely retried in the described
> way.
>
> Pre-warmed containers indeed are somewhat of a different story. A
> health
> ping as mentioned here would for sure help there, be it just a TCP
> probe or
> even a full-fledged /health call. I'd be fine with either way in this
> case
> as it doesn't affect the critical path.
>
> Am Di., 29. Okt. 2019 um 18:00 Uhr schrieb Tyson Norris
> :
>
> > By "critical path" you mean the path during action invocation?
> > The current PR only introduces latency on that path for the case of a
> > Paused container changing to Running state (once per transition from
> Paused
> > -> Running).
> > In case it isn't clear, this change does not affect any retry (or
> lack of
> > retry) behavior.
> >
> > Thanks
> > Tyson
> >
> > On 10/29/19, 9:38 AM, "Rodric Rabbah"  wrote:
> >
> > as a longer term point to consider, i think the current model of
> "best
> > effort at most once" was the wrong design point - if we embraced
> > failure
> > and just retried (at least once), then failure at this level
> would
> > lead to
> > retries which is reasonable.
> >
> > if we added a third health route or introduced a health check,
> would we
> > increase the critical path?
> >
> > -r
> >
> > On Tue, Oct 29, 2019 at 12:29 PM David P Grove <
> gro...@us.ibm.com>
> > wrote:
> >
> > > Tyson Norris  wrote on 10/28/2019
> > 11:17:50 AM:
> > > > I'm curious to know what other
> > > > folks think about "generic active probing from invoker" vs
> "docker/
> > > > mesos/k8s specific integrations for reacting to container
> > failures"?
> > > >
> > >
> > > From a pure maintenance and testing perspective I think a
> single
> > common
> > > mechanism would be best if we can do it with acceptable runtime
> > overhead.
> > >
> > > --dave
> > >
> >
> >
> >
>
>
>

Re: Action health checks

[Markus Thömmes]

Increasing latency would be my biggest concern here as well. With a health
ping, we can't even be sure that a container is still healthy for the "real
request". To guarantee that, I'd still propose to have a look at the
possible failure modes and implement a retry mechanism on them. If you get
a "connection refused" error, I'm fairly certain that it can be retried
without harm. In fact, any error where we can guarantee that we haven't
actually reached the container can be safely retried in the described way.

Pre-warmed containers indeed are somewhat of a different story. A health
ping as mentioned here would for sure help there, be it just a TCP probe or
even a full-fledged /health call. I'd be fine with either way in this case
as it doesn't affect the critical path.

Am Di., 29. Okt. 2019 um 18:00 Uhr schrieb Tyson Norris
:

> By "critical path" you mean the path during action invocation?
> The current PR only introduces latency on that path for the case of a
> Paused container changing to Running state (once per transition from Paused
> -> Running).
> In case it isn't clear, this change does not affect any retry (or lack of
> retry) behavior.
>
> Thanks
> Tyson
>
> On 10/29/19, 9:38 AM, "Rodric Rabbah"  wrote:
>
> as a longer term point to consider, i think the current model of "best
> effort at most once" was the wrong design point - if we embraced
> failure
> and just retried (at least once), then failure at this level would
> lead to
> retries which is reasonable.
>
> if we added a third health route or introduced a health check, would we
> increase the critical path?
>
> -r
>
> On Tue, Oct 29, 2019 at 12:29 PM David P Grove 
> wrote:
>
> > Tyson Norris  wrote on 10/28/2019
> 11:17:50 AM:
> > > I'm curious to know what other
> > > folks think about "generic active probing from invoker" vs "docker/
> > > mesos/k8s specific integrations for reacting to container
> failures"?
> > >
> >
> > From a pure maintenance and testing perspective I think a single
> common
> > mechanism would be best if we can do it with acceptable runtime
> overhead.
> >
> > --dave
> >
>
>
>

Re: Action health checks

[Markus Thömmes]

Heya,

thanks for the elaborate proposal.

Do you have any more information on why these containers are dying off in
the first place? In the case of Kubernetes/Mesos I could imagine we might
want to keep the Invoker's state consistent by checking it against the
respective API repeatedly. On Kubernetes for instance, you could setup an
informer that'd inform you about any state changes on the pods that this
Invoker has spawned. If a prewarm container dies this way, we can simply
remove it from the Invoker's bookkeeping and trigger a backfill.

Secondly, could we potentially fold this check into the HTTP requests
themselves? If we get a "connection refused" on an action that we knew
worked before, we can safely retry. There should be a set of exceptions
that our HTTP clients should surface that should be safe for us to retry in
the invoker anyway. The only addition you'd need in this case is an
enhancement on the ContainerProxy's state machine I believe, that allows
for such a retrying use-case. The "connection refused" use-case I mentioned
should be equivalent to the TCP probe you're doing now.

WDYT?

Cheers,
Markus

Am So., 27. Okt. 2019 um 02:56 Uhr schrieb Tyson Norris
:

> Hi Whiskers –
> We periodically have an unfortunate problem where a docker container (or
> worse, many of them) dies off unexpectedly, outside of HTTP usage from
> invoker. In these cases, prewarm or warm containers may still have
> references at the Invoker, and eventually if an activation arrives that
> matches those container references, the HTTP workflow starts and fails
> immediately since the node is not listening anymore, resulting in failed
> activations. Or, any even worse situation, can be when a container failed
> earlier, and a new container, initialized with a different action is
> initialized on the same host and port (more likely a problem for k8s/mesos
> cluster usage).
>
> To mitigate these issues, I put together a health check process [1] from
> invoker to action containers, where we can test
>
>   *   prewarm containers periodically to verify they are still
> operational, and
>   *   warm containers immediately after resuming them (before HTTP
> requests are sent)
> In case of prewarm failure, we should backfill the prewarms to the
> specified config count.
> In case of warm failure, the activation is rescheduled to ContainerPool,
> which typically would either route to a different prewarm, or start a new
> cold container.
>
> The test ping is in the form of tcp connection only, since we otherwise
> need to update the HTTP protocol implemented by all runtimes. This test is
> good enough for the worst case of “container has gone missing”, but cannot
> test for more subtle problems like “/run endpoint is broken”. There could
> be other checks to increase the quality of test we add in the future, but
> most of this I think requires expanding the HTTP protocol and state managed
> at the container, and I wanted to get something working for basic
> functionality to start with.
>
> Let me know if you have opinions about this, and we can discuss  here or
> in the PR.
> Thanks
> Tyson
>
> [1] https://github.com/apache/openwhisk/pull/4698
>

Re: A direction to take in a new component, "the scheduler"

[Markus Thömmes]

Hi,

this sounds a lot like the "OpenWhisk 2.0" discussion we had in an older
thread where I proposed an architectural overhaul. See this thread for
context:
https://lists.apache.org/thread.html/d19306dd976138a153f48d32c5a55f2853e4b8ff405fc46f7260e905@%3Cdev.openwhisk.apache.org%3E

There have been strong opinions in the past against overhauling.

Cheers,
Markus

Am Do., 18. Juli 2019 um 13:42 Uhr schrieb Dominic Kim :

> Recently I discussed the direction and a safe way to add a new component
> with Sven Lange-Last.
> Let me share the discussion results.
> More feedbacks and critique are welcomed.
>
> Since the implementation includes a breaking architectural change, it comes
> with a risk.
> It must not break any existing downstream system as well as the upstream
> pipelines.
> So all implementations should be disabled by default along with proper
> switches.
>
> The new change would require many iterations to become stable enough for a
> production system.
> It would be better to have a new separate (openwhisk) CI pipeline for the
> changes.
> All unit/system tests will only be executed on the new CI pipeline.
>
> I will defer to Sven, he may add more comments.
>
> Best regards
> Dominic
>

Re: Execute OpenWhisk action via Ignite and Firecracker VM (pr #4556)

[Markus Thömmes]

Great stuff Chetan! The quick impl. also shows our abstractions kinda work.
That makes me happy too :).

Am Di., 16. Juli 2019 um 00:56 Uhr schrieb Carlos Santana <
csantan...@gmail.com>:

> Great work Chetan
>
> This must be a new record how fast to get a POC integrated in a few days
> after ignite being announced
>
> -- Carlos
>
> On Fri, Jul 12, 2019 at 9:46 AM Chetan Mehrotra  >
> wrote:
>
> > Hi Team,
> >
> > Seeing the recent Weave Ignite announcement [1] I tried to implement a
> > quick prototype for integrating it with OpenWhisk [2].
> >
> > So far the results are good and OpenWhisk can create and launch
> > ignite/firecracker based vm for action invocation. Currently the cold
> > startup times have few issues due to late binding of node service
> > within the vm. Also the logs part does not work. However apart from
> > that other aspects work ok.
> >
> > It is a promising technology and we can possibly explore it more for
> > integrating it with OpenWhisk to provide more secure execution
> > environment.
> >
> > Chetan Mehrotra
> > [1] https://www.weave.works/blog/fire-up-your-vms-with-weave-ignite
> > [2] https://github.com/apache/incubator-openwhisk/pull/4556
> >
>
>
> --
> Carlos Santana
> 
>

Re: Using Rust for the KnativeWhisk controller

[Markus Thömmes]

I share none of these sentiments. Neither on Golang being barebones nor on
a few of the examples you mentioned making me "want to die" nor on
"absurdities", "silliness" in the languages design or on an "absolute mess"
others built with it.

I have used the "lower barrier to entry means also a proliferation of bad
code" argument myself in the past. There are other angles to look for in a
language where you want contributions outside of your team, aka a
community. That being said: The barrier of Rust does not prevent you from
writing bad code with it. The same is even more true for languages like
C++, where it's easy to make very low level mistakes.

Again: Both languages are absolutely fine for what they are. This shouldn't
be about personal preference either but a discussion about what's best for
this project specifically. Using Golang in the Kubernetes/Knative space
simply easily buys you the most mindshare by far.

Am Do., 18. Juli 2019 um 13:36 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> Sigh! Having used Go for over one year and half, and being the one who
> built the Go runtime and applied to another 5 languages, I know what is
> wrong with Go.
>
> Go is terribly bare-bone and expressing everything that is a bit complex
> is an excercise in patience. And a REST server is already too complicated
> for Go. Give a look to the mess that is the prototype kwsk.
>
> The pattern "if err != nil" everywhere is just plainly silly, you have 3
> more lines in your code, 2 of them useless, for each error. Yes of course
> if can write a function that will "panic" for each error but also this is
> not better than ignoring all the errors and panicing when you meet it.
>
> Parsing JSON is another absurdity, you have to declare a full struct when
> you just need to access to a single value and ignore the rest... or
> navigate/cast/navigate/cast the struct until you get there.
>
> If you give a look to the AWS api for example, you would like to kill
> yourself: for each function call you need to create a full struct, because
> Go does not have any optional parameter and unless you write a function
> with a ton of parameters, there are no  other options.
>
> You  have to do basically the same for any swagger generated server, and
> unfortunately the current prototype of  OpenWhisk for Knative is generated
> with swagger, and it is an absolute mess of boilerplate code and managing
> every call with a lot of verbose code involving complex structure that maps
> all the JONS of the API.
>
> I gave a look to the Project DD and I want to die the amount of messy
> boilerplate code there is. People says that Go is readable... I disagree.
> You  have to dig into the code a lot until you find what you need. All thee
> documentations are huge list of functions with very long names. Go is a
> zero structure programming language,  and gives me the feeling of PHP.
> Lower barrier to entry means also a proliferation of bad code. I believe it
> is not strange that while C++ is one of hardest language is also the
> language used by the best programmers developing the greatest apps.
>
>
> Go makes sense only in a limited ecosystem with "system level" programming
> but since unfortunately it covered the niche of being a "compiled" language
> in a world where Java is losing favour because of the mismanagement of
> Oracle, and Docker provides enough isolation where having a jvm in a
> container in a virtual machine is  total nonsense...  So there is a
> comeback to "native" code
>
> Yes of course there is a library to make it dynamic, there are code
> generator, there are tons of tools to overcome the limitations of Go... but
> there is also a solution: use a better language.
>
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>
> - Original message -
> From: "Markus Thömmes" 
> To: dev@openwhisk.apache.org
> Subject: Re: Using Rust for the KnativeWhisk controller
> Date: Thursday, July 18, 2019 12:54 PM
>
> Am Do., 18. Juli 2019 um 10:04 Uhr schrieb Michele Sciabarra <
> mich...@sciabarra.com>:
>
> > Actually, I am not going to rewrite the controller in Go.
> > There is already one here: https://github.com/projectodd/kwsk
> >
> > I proposed to use Rust instead since looks like it is growing fast, so I
> > could "prototype" in Rust to see where we get. But since the community
> does
> > not like it, I give up.
> >
>
> I just wanted to quickly get back to this: I am not proposing to not use
> Rust in this context because the community does not like it. It's rather a
> conscious decision to foster engagement in t

Re: Using Rust for the KnativeWhisk controller

[Markus Thömmes]

Am Do., 18. Juli 2019 um 10:04 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> Actually, I am not going to rewrite the controller in Go.
> There is already one here: https://github.com/projectodd/kwsk
>
> I proposed to use Rust instead since looks like it is growing fast, so I
> could "prototype" in Rust to see where we get. But since the community does
> not like it, I give up.
>

I just wanted to quickly get back to this: I am not proposing to not use
Rust in this context because the community does not like it. It's rather a
conscious decision to foster engagement in the community. Experience tells
us that Scala has already been quite a burden for many to learn. My
personal preference is of second order in this case. No negative sentiment
against Rust was intended, quite on the contrary: Designwise I prefer Rust
over Golang everyday. The reach and impact of Golang in the community this
is targetting (Knative and thus Kubernetes) is undeniable though.


>
> My goal is to port the actionloop runtime to be "init-less" and able to
> build docker images  with Tekton Pipelines that can be deployed to Knative
> Serving.
>
> I "only" need a way to get action code, send it to a git server and
> trigger a Tekton Build and then a Knative Serving deploy.
>
> As long as I can access the runtime I am done for this project. I will
> probably just use a script to do that.
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>
> - Original message -
> From: Dascalita Dragos 
> To: dev@openwhisk.apache.org
> Subject: Re: Using Rust for the KnativeWhisk controller
> Date: Wednesday, July 17, 2019 10:59 PM
>
> I’m inserting my +1 for Go, based on the community adoption of Go; my
> perception is that it found its place as the de-facto language for managing
> infrastructure. Akka is amazing for distributed programming model ...
> that’s my only argument for Scala, but this argument alone is not strong
> enough to go against the stream.
>
>
> What Controller functionality are you looking to re-implement ? The API
> used by WSK CLI ? Or more ?
>
>
>  dragos
>
>
> On Wed, Jul 17, 2019 at 3:25 PM Matt Sicker  wrote:
>
> > Ah, yes, Rust would be good for any performance-sensitive code or
> > modules needed. I haven't kept up to date with C++ since pre-'10, but
> > I'd imagine that it's easier to learn Rust than the latest version of
> > C++ at this point.
> >
> > On Wed, 17 Jul 2019 at 08:06, Michele Sciabarra 
> > wrote:
> > >
> > > Ok guys, I give up  with  rust. Will use Go.
> > >
> > >
> > > --
> > >   Michele Sciabarra
> > >   mich...@sciabarra.com
> > >
> > > - Original message -
> > > From: "Markus Thömmes" 
> > > To: dev@openwhisk.apache.org
> > > Subject: Re: Using Rust for the KnativeWhisk controller
> > > Date: Wednesday, July 17, 2019 9:16 AM
> > >
> > > +1 to all arguments. Rust's barrier of entry is considerably higher
> than
> > > that of Scala even. As much as I like the language's design, from a
> > > community attraction point-of-view we should absolutely opt for go,
> > > especially for things that are built around Kubernetes.
> > >
> > > That's of course to be taken with a grain of salt: If we have a
> use-case
> > > that requires the performance characteristics of Rust (especially that
> > of a
> > > lacking garbage collector), we should absolutely choose for the
> > > implementation. Implementing a controller however sounds like a
> > > business-as-usual REST talk-to-a-database thingy and Go is perfect for
> > that.
> > >
> > > Am Di., 16. Juli 2019 um 08:53 Uhr schrieb Martin Henke <
> > martin.he...@web.de
> > > >:
> > >
> > > > Michele,
> > > >
> > > > Two thoughts:
> > > >
> > > > 1) For writing a controller in Knative I  recommend to choose Go
> > instead
> > > > of Rust (even when I like Rust more).
> > > > With Go you can leverage the fantastic Operator SDK from Redhat which
> > > > makes writing controllers fairly
> > > > simple (I had my first one up and running in under an hour).
> > > > Link: https://github.com/operator-framework/operator-sdk
> > > >
> > > > The getting started guide is a good starting point:
> > > > https://github.com/operator-framework/getting-started
> > > >
> > > > It also addresses the lifecycle of an controller with the Lifecycl

Re: Using Rust for the KnativeWhisk controller

[Markus Thömmes]

+1 to all arguments. Rust's barrier of entry is considerably higher than
that of Scala even. As much as I like the language's design, from a
community attraction point-of-view we should absolutely opt for go,
especially for things that are built around Kubernetes.

That's of course to be taken with a grain of salt: If we have a use-case
that requires the performance characteristics of Rust (especially that of a
lacking garbage collector), we should absolutely choose for the
implementation. Implementing a controller however sounds like a
business-as-usual REST talk-to-a-database thingy and Go is perfect for that.

Am Di., 16. Juli 2019 um 08:53 Uhr schrieb Martin Henke :

> Michele,
>
> Two thoughts:
>
> 1) For writing a controller in Knative I  recommend to choose Go instead
> of Rust (even when I like Rust more).
> With Go you can leverage the fantastic Operator SDK from Redhat which
> makes writing controllers fairly
> simple (I had my first one up and running in under an hour).
> Link: https://github.com/operator-framework/operator-sdk
>
> The getting started guide is a good starting point:
> https://github.com/operator-framework/getting-started
>
> It also addresses the lifecycle of an controller with the Lifecycle
> Manager:
> https://github.com/operator-framework/operator-lifecycle-manager
> (I have not yet used this myself)
>
>
> 2) I think we should clearly separate the Knative work from Openwhisk and
> stay there with  a strict Scala only policy
> (with the existing  exceptions).
> Introducing more languages would in my opinion  lead to maintenance
> problems and the waste of  build up skills.
>
> Regards,
> Martin
>
>
> > On 15. Jul 2019, at 11:58, Michele Sciabarra 
> wrote:
> >
> > Hello all,
> >
> > In my efforts to work a Kanative Whisk, I reached the point where I have
> a kit with tekton-pipelines, knatve-serving building an actionlooop based
> runtime. Now I need to implement a controller, in order to use the existing
> wsk tooling.
> >
> > I know there is a prototype kwsk implementation made by redhat,  written
> in Go but looks like it is obsolete and unfinished, and to the best of my
> knowledge, abandoned.
> >
> > I would like to resume the effort of writing an updated controller. I
> actually already have a prototype using the Julia language. Julia is really
> awesome, Python simplicity and Go speed, but I feed the community would
> disagree on using Julia.  Of course if I am wrong... let me know because
> that would be my preferred choice.
> >
> > However, I feel that,  given our Scala background, Rust would be a much
> better choice for the KnativeWhisk controller.  So I propose to use Rust
> for the KwhiskController.
> >
> > What does the community think of the proposal?
> >
> >
> > --
> >  Michele Sciabarra
> >  mich...@sciabarra.com
>
>

Re: Maximum number of file descriptors available when running actions with managed runtimes

[Markus Thömmes]

Thanks as always for your very detailed writeup and analysis of the issue
at hand! Great job Sven!

It seems to me that we might want to look at analyzing our runtimes for the
number of file-descriptors they open and make up for that for the user. The
user should always get at least the amount of descriptors specified. I
could see us going two ways about this:

1. Determine the number of descriptors per runtime, inform the system about
them (via the manifest) and add these to the limits.
2. Configure the system with enough slack to cover all potential cases.

I can see 1. being quite brittle, given that by looking at your analysis,
it seems like even the host might have an impact here. I'm leaning towards
the 2. option. We should analyze all our runtimes and get a feeling for how
many file descriptors are needed by each. If they are all in the same
ballpark, I'd say let's add ~50fds on top of the limit and document it as
"at least X" vs "exactly X".

Cheers,
Markus

Am Mi., 10. Juli 2019 um 16:20 Uhr schrieb Sven Lange-Last <
sven.lange-l...@de.ibm.com>:

> Apache OpenWhisk limits the number of file descriptors available when
> running actions - at least, when using the Docker container factory [1].
> The soft / hard limit for the number of file descriptors that can be open
> for any process in an action container is set to 1024. This value is
> hard-coded. Since there is a system-wide limit for open file descriptors
> on a Linux system, this setting is meant to prevent a single action
> container from opening so many file descriptors that other action
> containers or the whole system are affected.
>
> Apache OpenWhisk documents this limitation in the "OpenWhisk system
> details" document [2]. End users will probably understand this document as
> a kind of promise that up to 1024 file descriptors can be opened by action
> code...
>
> While we currently have a hard-coded limit for open file descriptors in
> action containers, managed action runtime processes consume file
> descriptors to do their work. We have tests that verify a minimum /
> maximum amount of file descriptors can be opened by a default Node.js
> action [3]. These tests already consider that action code won't have the
> full limit of 1024 file descriptors available because the managed runtime
> also opens files [4].
>
> When changing the default Node.js version from 6 to 10, we noticed that
> the Node.js 10 managed runtime consumes 20 instead of 15 and adapted the
> tests accordingly [5]. While decreasing the number of available
> descriptors for action code from 1009 to 1004 is likely no problem, we
> have to be aware that decreasing that number can potentially break
> existing actions and must be considered a regression.
>
> So one purpose of this post is to make managed runtime developers aware
> that changes in the managed runtime process - like the number of file
> descriptors opened by the action process - can break existing actions.
> This also applies to the physical memory usage of the managed runtime
> process.
>
> The other purpose is to start a discussion / hear your opinion on the
> general problem that the file descriptor limit is hard coded at the moment
> and this limit will never be available for action developers because the
> managed runtime already consumes file descriptors.
>
> In addition, I wanted to inform you about a PR I opened today [6]. This PR
> further relaxes the test verifying that a minimum number of file
> descriptors is available - sorry for that. We have test systems where the
> limit tests are failing permanently. Long story in short words: Docker
> 18.09.3 seems to set up action containers differently than earlier
> versions leading to a slightly higher consumption of file descriptors. See
> the PR for all details.
>
>
> [1]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InvokerReactive.scala#L105
> [2]
>
> https://github.com/apache/incubator-openwhisk/blob/ccac7d5131a80bddeb1b6dc0c6580051c90bc264/docs/reference.md
> [3]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/tests/src/test/scala/org/apache/openwhisk/core/limits/ActionLimitsTests.scala#L366-L416
> [4]
>
> https://github.com/apache/incubator-openwhisk/blob/b0d48c87816f2e7c04ad9b9f9b4844a6e064047f/tests/src/test/scala/org/apache/openwhisk/core/limits/ActionLimitsTests.scala#L68-L69
> [5]
>
> https://github.com/apache/incubator-openwhisk/pull/4450#discussion_r277969441
> [6] https://github.com/apache/incubator-openwhisk/pull/4553
>
>
>
> Mit freundlichen Grüßen / Regards,
>
> Sven Lange-Last
> Senior Software Engineer
> IBM Cloud Functions
> Apache OpenWhisk
>
>
> E-mail: sven.lange-l...@de.ibm.com
> Find me on:
>
>
> Schoenaicher Str. 220
> Boeblingen, 71032
> Germany
>
>
>
>
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: 

Re: Allow temporal unused codes to include a big change.

[Markus Thömmes]

Continuous integration is a crucial aspect of a system used in production
like OpenWhisk.

As such, let's bite the bullet by merging things incrementally into master.
That will also force you to think about migration on each step you take,
which in the long run will pay off big time. Accumulating a huge change
will never be review- nor mergeable confidently in the end.

+1 to incremental changes.

Am Fr., 5. Juli 2019 um 08:54 Uhr schrieb Martin Henke :

> Strongly + 1 to put core changes behind feature flags so that they can be
> rolled out (and removed if necessary )
> in a controlled way.
> Tests should be running successfully for having the feature flag enabled
> or disabled.
> In the near past it took a considerable amount of my time to adapt test
> cases for some of the features
> introduced in the last time to make them work in all cases.
>
> Regards,
> Martin
>
>
>
> > On 5. Jul 2019, at 06:59, Chetan Mehrotra 
> wrote:
> >
> > +1 to commit incrementally to master.
> >
> > If the changes touch the core logic then we can possibly have them
> > backed by feature flags and have them disabled by default and enabled
> > for test case. Further it would be preferable that whatever we commit
> > (at any stage) it should have required test coverage. This would
> > ensures that the sub parts of work in progress bigger feature are
> > backed by unit tests.
> >
> > regards
> > Chetan Mehrotra
> >
> > On Thu, Jul 4, 2019 at 9:35 PM Dominic Kim  wrote:
> >>
> >> Hello, whiskers.
> >>
> >> I am trying to contribute this:
> >> https://github.com/apache/incubator-openwhisk/pull/4532
> >>
> >> At first, I tried to open a base branch and merge all incremental PRs
> into
> >> the branch.
> >> And finally mege the base branch into the master after the
> implementation
> >> is done.
> >> Surely, reviewers would review all the subsequent PRs and it will be
> based
> >> on SPI and disabled by default at first, there would be no big issue I
> >> think.
> >>
> >> And Markus suggested to incrementally merge all PRs into the master
> instead
> >> of having a big base branch.
> >>
> https://github.com/apache/incubator-openwhisk/pull/4532#issuecomment-508519119
> >>
> >> With the former, we don't need to include temporal unused codes but
> need to
> >> include a big possibly disruptive PR at once.
> >> With the latter, there will be some temporal unused components in the
> >> master at some point, but we can make sure merged codes do not induce
> any
> >> issue. And actually the latter is easier for me as well : )
> >>
> >> If we all agree with including the temporal unused codes in the master,
> I
> >> am happy to work in the way Markus suggested.
> >>
> >> One example of a temporal code is this:
> >>
> https://github.com/apache/incubator-openwhisk/pull/4532/files#diff-1d7110b32c507a6ef4ac956c287e77ebR24
> >>
> >> Since there is no other component, the "scheduler" cannot initialize all
> >> components in the main function and there is only `println("Hello")` in
> the
> >> initial version of the main function.
> >>
> >>
> >> Please let me know your thoughts.
> >>
> >> Best regards
> >> Dominic
>
>

Re: [VOTE] Apache OpenWhisk graduation to Top Level Project

[Markus Thömmes]

+1 Apache OpenWhisk should graduate.

Thanks all for the great community and opportunity to work with such smart
people!

Am Mo., 10. Juni 2019 um 17:55 Uhr schrieb Justin Halsall <
jus...@juice10.com>:

> +1 OpenWhisk is ready to graduate!
>
> > On Jun 6, 2019, at 5:11 AM, Bertrand Delacretaz 
> wrote:
> >
> > On Tue, Jun 4, 2019 at 11:26 PM Rodric Rabbah  wrote:
> >> ...Please take a minute to vote on whether or not Apache OpenWhisk
> should
> >> graduate to a Top Level Project...
> >
> > +1 with my incubation mentor hat on, OpenWhisk is ready to graduate.
> >
> > -Bertrand
> >
>
>

Re: A plan to (re) implement OpenWhisk on top of Knative

[Markus Thömmes]

For background:

Tekton has emerged out of the former Knative Build-Pipelines project. The
Build API connection will be dropped in Serving v1beta1. Tekton is the way
to go if any.

Cheers,
Markus

Am Mo., 20. Mai 2019 um 17:03 Uhr schrieb Martin Henke :

> Tekton is different but very very similar. You get a lot of DejaVus. No
> big learning Curve.
>
> Martin
>
>
> > On 20. May 2019, at 17:00, Michele Sciabarra 
> wrote:
> >
> > Ok great, I see the discussion is starting to bring ideas.
> >
> > Yes my goal is basically to run existing actions in Knative, create and
> invoke. And possibile retain the ability of an action to invoke another
> action.
> >
> > I understand the different way they expose services, so I am rethinking
> the idea of using a "work-alike" path.
> >
> > If it is needed we can add it with an ingress but it may be not
> necessary in the initial implementation.
> >
> > Also I checked a bit ML and discussions and I see this Tekton thing that
> should be the preferred way.
> >
> > Not sure if I understand the relation with the current Build API
> documented in the website. Is Tekton "compatible" or it has a different API?
> >
> >
> > --
> >  Michele Sciabarra
> >  mich...@sciabarra.com
> >
> > - Original message -
> > From: "Markus Thömmes" 
> > To: dev@openwhisk.apache.org
> > Subject: Re: A plan to (re) implement OpenWhisk on top of Knative
> > Date: Monday, May 20, 2019 4:50 PM
> >
> > Good discussion, thanks!
> >
> > Can we try to define what the desired end-goal is here? I'm a bit unclear
> > what resembling the OpenWhisk API actually buys us.
> >
> > To me, the desired end-state would be to run OpenWhisk actions as-is on a
> > Knative cluster (similar to OpenFaaS' and Azure's integration). There's
> no
> > good way for us to provide the full API without spinning up a control
> plane
> > and we can only handle so much via the CLI. So to me, the end-goal looks
> > like:
> >
> > 1. *wsk action create* actually doing all the pieces necessary to run a
> > piece of code on Knative.
> > 2. *wsk action invoke* doing some HTTP call under the hood to "invoke"
> that
> > action. The action should be reachable via a sensible URL. If we really
> > want to keep the API surface (as I said, I'm dubious here) we can also do
> > that via ingress level abstractions (like VirtualService).
> >
> > Cheers,
> > Markus
> >
> > Am Mo., 20. Mai 2019 um 15:33 Uhr schrieb Martin Henke <
> martin.he...@web.de
> >> :
> >
> >>
> >>> On 20. May 2019, at 14:55, Michele Sciabarra 
> >> wrote:
> >>>
> >>>> Michele,
> >>>
> >>>> I like the idea to make the ActionLoop based runtimes to be runnable
> on
> >> Knative.
> >>>>
> >>>> My thoughts on this:
> >>>> - I second Markus concern to implement the invocation API onto Knative
> >> instead of just using Knative service syntax.
> >>> Can you elaborate this? I do not understand.
> >>
> >> Knative service syntax:https:// >> action)>../
> >> OW invocation https://
> >> /api/v1/namespaces//actions/
> >>
> >> (I personally so no worth in inventing a distinct API for OW images, but
> >> as said I would see that as a valid optional feature)
> >>
> >>>
> >>>> - I would have concerns to make it dependent on Gloo which is kind of
> a
> >> minority choice for Knative load balancing
> >>> I do not think it will be hard to setup a test also using Istio, I do
> >> not want to be limited to Gloo.
> >>
> >> I just wanted to prevent that Gloo gets a “official” prerequisite for an
> >> “official” OW on Knative flow.
> >> It is of course free to you to use what ever you want to do in your
> >> prototype.
> >>
> >>> - In my opinion the goal should be to have some uniform behaviour for
> >> ActionLoop based runtimes
> >>>> and other ones like the adapted NodeJS runtimes demonstrated by Matt
> >> and Priti
> >>> As much as I can tell the current implementation is just the building
> >> and exposing the "/init" and "/run" but I can be wrong.
> >>> The build can be of course reused, so it continues the effort. For the
> >> frontend, from the documentation I think Matt wants to add a proxy,
> while I
> >> wo

Re: A plan to (re) implement OpenWhisk on top of Knative

[Markus Thömmes]

Good discussion, thanks!

Can we try to define what the desired end-goal is here? I'm a bit unclear
what resembling the OpenWhisk API actually buys us.

To me, the desired end-state would be to run OpenWhisk actions as-is on a
Knative cluster (similar to OpenFaaS' and Azure's integration). There's no
good way for us to provide the full API without spinning up a control plane
and we can only handle so much via the CLI. So to me, the end-goal looks
like:

1. *wsk action create* actually doing all the pieces necessary to run a
piece of code on Knative.
2. *wsk action invoke* doing some HTTP call under the hood to "invoke" that
action. The action should be reachable via a sensible URL. If we really
want to keep the API surface (as I said, I'm dubious here) we can also do
that via ingress level abstractions (like VirtualService).

Cheers,
Markus

Am Mo., 20. Mai 2019 um 15:33 Uhr schrieb Martin Henke :

>
> > On 20. May 2019, at 14:55, Michele Sciabarra 
> wrote:
> >
> >> Michele,
> >
> >> I like the idea to make the ActionLoop based runtimes to be runnable on
> Knative.
> >>
> >> My thoughts on this:
> >> - I second Markus concern to implement the invocation API onto Knative
> instead of just using Knative service syntax.
> > Can you elaborate this? I do not understand.
>
> Knative service syntax:https:// action)>../
> OW invocation https://
> /api/v1/namespaces//actions/
>
> (I personally so no worth in inventing a distinct API for OW images, but
> as said I would see that as a valid optional feature)
>
> >
> >> - I would have concerns to make it dependent on Gloo which is kind of a
> minority choice for Knative load balancing
> > I do not think it will be hard to setup a test also using Istio, I do
> not want to be limited to Gloo.
>
> I just wanted to prevent that Gloo gets a “official” prerequisite for an
> “official” OW on Knative flow.
> It is of course free to you to use what ever you want to do in your
> prototype.
>
> > - In my opinion the goal should be to have some uniform behaviour for
> ActionLoop based runtimes
> >> and other ones like the adapted NodeJS runtimes demonstrated by Matt
> and Priti
> > As much as I can tell the current implementation is just the building
> and exposing the "/init" and "/run" but I can be wrong.
> > The build can be of course reused, so it continues the effort. For the
> frontend, from the documentation I think Matt wants to add a proxy, while I
> would like to implemeent the "invocation" straight in the runtime. This is
> open to discussion, but of course it is better to reach an agreement.
>
> Also in the work of Priti and Matt the invocation goes directly to the
> runtime. The action code is either passed with the call (not yet tested by
> me) or set via environment variable in the docker build.
>
> >
> >> - As Knative Build seems be on a dead end I would propose to target
> Tekton as the build system (which developed as kind of >successor out of
> Knative)
> >
> > If Knative build is dead then it would be a bit unfair that they change
> it as the scope of the Knative project!
> > It looks like the goal is  to setup some standards! And I would be very
> disappointed to know that.
>
> Tekton evolved out of Knative Build (or more correct out of Knative
> Pipelines) but is very similar to the Knative build.
> Flows can easily be ported from one to the other,
> If we target Tekton build we would target the platform were the Knative
> build team is focusing on.
> But again feel free to use whatever platform for your prototype work.
>
> > At this stage the build is the more interesting thing, and it could be
> even imported in main openwhisk to speed up deployment.
> > I have already baked it in the ActionLoop runtimes (with precompilation).
> > Also if we use Tekton, where is the Knative standard then? What is the
> point? We can build our own system instead of "Knativizing" it...
> >
> >> Maybe it would be a good solution to tackle two things independently.
> >> 1) Design and implement a common protocol of building, running and
> calling OW runtimes on Knative
> >> 2) Implement the OW invocation API on top of Knative as an additional
> option for those who have the need to expose it.
> >
> > On this, for my personal approach at building things, I want something
> that works and it is complete and useful. A "MVP”.
>
> Cool. Just go on.
>
> > So I do not plan to split the effort. Version 0.1 must be a minimal
> working subset of OpenWhisk on Knative.
> > Because otherwise there will be incomplete useless inusable pieces
> around (see for example kwsk).
> >
> > It does not mean that things cannot be modular, nor that everyone must
> but to me "openwhisk-knative" must be a single repo with all the pieces to
> make something where you can download is and deploy in a kubernetes cluster
> and be able to deploy simple actions. When this works, we can improve
> incrementally and split it but keeping it working.
> >
> >> I would looking forward to work with you on the first

Re: A plan to (re) implement OpenWhisk on top of Knative

[Markus Thömmes]

Hi Michele,

thank you for the detailed writeup. A few thoughts inline:

Am So., 19. Mai 2019 um 19:00 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> I have an idea for implementing a prototype of OpenWhisk on top of Knative.
>
> My basic ideas are: do not use any proxy, forwarding or adapter: extend
> the runtime to support the REST call and expose them as ingress. And use a
> wrapper on top of `kubectl` to generate all the needed componennts.
>

Does this tie into the work that Matt was doing to the runtimes to make
them runnable on Knative? Is this lined up with that at all?


>
> My goal is to have a functional work-alike of OpenWhisk built on top of
> Knative, using ActionLoop as a foundation. I will extend ActionLoop to
> support the required REST calls of OpenWhisk.
>
> I also want to create tool, I will call `wskn`. This tool will initially
> just a python script, a wrapper on top of `kubectl` as it will generate
> kubernetes descriptors.
>

Why not build this into "wsk" itself? The Azure Functions CLI as an example
supports multiple deployment types like this in one CLI.


>
> It will support initially just the the action creation and invocation, and
> only synchronous (blocking) behaviour, as all the request will go straight
> to the runtimes. Hopefully also a subset of `package` and `activation`.
> Again triggers, rules, asynchronous for later.
>
> The idea is that you will be able to create actions and web actions that
> can run existing OpenWhisk actions, at least those with blocking behaviour
> that run with ActionLoop (Go, Java, Python, PHP, Swift, Rust, Ruby,
> Crystal...)
>
> Implementation.
> ==
>
> This is how I plan to implement it.
>
> At this stage I want to use just Knative Serving and Knative Build, using
> Gloo for the ingress part. I also plan to install a local Docker registry
> Kubernetes registry, so we do not have to use DockerHub for everything. All
> of this can be done with existing command line tools in a few minutes in
> any running Kubernetes deployment.
>

Why specifying Gloo here? Do you need anything specific from Gloo itself?
If not I'd propose to just keep it on a Knative Serving API surface level.


>
> When I create an action, it will use Knative build that will work roughly
> in this way:
>
> - create a configmap with the action code
> - build the actin using ActionLoop precompilation feature that will return
> a zip file including all the needed to run the action
> - create a new docker image extending the runtime with the new zip, using
> Kaanico
> - push the image in the local registry
>

This feels like a fairly heavyweight process, we should be able to come up
with a way to circumvent zipping entirely. Maybe the runtime can detect
that the unzipped content is already there and skip the unzip step?

I'm fairly hesitant on the usage of a ConfigMap for storing the action
code. It's all stored in the in-cluster etcd instance and it has a limit of
1M. This is at most a stop-gap solution to provide a PoC I think. Any ideas
on how to "productize" this?


>
> At this point you can run the action. ActionLoop will be extended to
> support invocations in the format
> "/v1/namespaces/namespace/actions/package/action".
>

Why bother reimplementing this exact path? To obtain API compatibility with
OpenWhisk as it is today?


>
> It will do all the decoding required to invoke the action with the
> expected paramenters (straight invocation thrhoug the actinloop protocol,
> not proxies).
>

Does this mean moving all of the Controller's "smartness" about incoming
and outgoing HTTP requests (see the whole WebActions for example)?


>
> Each action will then be exposed using an ingress with its specific
> invocation path.
>
> If the community agrees with this plan, I would create a repo
> `incubator-openwhisk-knative` to work on it.
>
> Thoughts?
>
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>

Re: [DISCUSS] graduation from the incubator

[Markus Thömmes]

+1.

Make it happen!

Am Di., 19. März 2019 um 07:02 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> Actually it looks to me pretty unusual that a project of this size has not
> yet graduated.
>
> --
>  Michele Sciabarra
>  mich...@sciabarra.com
>
>
>
> - Original message -
> From: David P Grove 
> To: OpenWhisk Dev 
> Subject: [DISCUSS] graduation from the incubator
> Date: Friday, March 15, 2019 11:06 PM
>
>
>
> I'd like to kick off a discussion to assess the project's readiness for
> graduation from the incubator.
>
> Per Rodric's recent stats [1], the community has developed nicely in terms
> of code contribution.
>
> We've released a number of software components following the Apache release
> process. We are in the midst of making our first "uber-release" across all
> of our sub-components (expect at least 2 voting threads next week).
>
> Overall I think the community is active. Communication on the project
> slack is frequent (avg of >160 messages a day) and is now digested daily to
> the dev list. (See [2] for stats).
>
> There are a couple procedural tasks we still need to complete, foremost
> being the formal transfer of the OpenWhisk trademarks from IBM to the ASF.
> But I think we can assume that these tasks will be completed and start
> considering graduation in parallel.
>
> Please share your thoughts,
>
> --dave
>
> [1]
>
> https://lists.apache.org/thread.html/b2217c61caad5c7a0369699d06d44e5cf688d3cba982e354a45b8c78@%3Cdev.openwhisk.apache.org%3E
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103091999
>
>

Re: [Long] I implemented websocket support for the Golang and related runtimes:

[Markus Thömmes]

Hi,

leaving a few remarks. Thanks as always for your hard work Michele, you're
cranking out a lot of stuff! Great job!

1. On the initial problem statement (and for your book): Have you
considered using an action with a very high concurrency setting for your
problem? The usual issue with these persistent connections is, that
OpenWhisk (in concurrency: 1 mode) spawns a lot of containers that each
need a persistent connection. With the support for arbitrary concurrency we
should be able to heavily alleviate that and thus focus a lot of the
traffic to very few running actions. The connection can be shared inside a
single running action. I'm not convinced that you actually need an external
service here given the feature set we already have.

2. To Carlos' point with Websockets right into the container: The issue
here is, that Websockets workloads are not request/response (or at least
there is no way of knowing if they are). That completely breaks the scaling
model that OpenWhisk uses today. We measure the amount of in-flight
requests to the action. With websockets, there is no way of measuring (and
thus controlling) the amount of workload that's being pumped into a
container. That may very much be desirable for the workload at hand, but as
of today it's not a good fit for how OpenWhisk is build.

Cheers,
M

Am Di., 22. Jan. 2019 um 13:56 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> Yes but if the patch is available it is possible to use an action runtime
> as an interim solution for providing a WebSocket server using a Kubernetes
> cluster.
>
> I did it because I am writing the chapter "messaging" of the book on
> OpenWhisk, and discovered that it is not recommended to use an action as a
> Kafka client as it can flood a Kafka server. So I implemented the
> recommended solution with a websocket server.
>
> I can, of course, provide a  solution as a separate, not official way of
> using  but the idea was to talk of the "OpenWhisk" recommended way in the
> book...
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>
> - Original message -
> From: Carlos Santana 
> To: dev@openwhisk.apache.org
> Subject: Re: [Long] I implemented websocket support for the Golang and
> related runtimes:
> Date: Tue, 22 Jan 2019 07:03:32 -0500
>
> I think there are 2 features here
>
> 1. Re use an action as is to be injected into a runtime that supports the
> websocket
>
> 2. Have OpenWhisk gain improvements by the invoker communicating with with
> user container with a websocket stream vs a 1 http req /run. It might be
> useful to show the benefits in performance specially when multiconcurrency
> is enable meaning for example having 200 in flight http connections or a
> set pool vs using websocket transport to the user container. Maybe
> something for Tyson to give some thought if he sees benefits for the
> multiconcurrency
>
> For 2, I think it requires more discussions since without the invoker
> counter part it doesn’t make sense todo the runtime first, do them together
> to find the stable API contract for the user container via websocket.
>
> - Carlos Santana
> @csantanapr
>
> > On Jan 22, 2019, at 6:23 AM, Michele Sciabarra 
> wrote:
> >
> > I guess then this feature is of interest and worth to be merged, as it
> provides a basis on the runtimes to implement websockets as a general
> feature then. Am I correct ? :)
> >
> >
> > --
> >  Michele Sciabarra
> >  mich...@sciabarra.com
> >
> > - Original message -
> > From: Carlos Santana 
> > To: dev@openwhisk.apache.org
> > Subject: Re: [Long] I implemented websocket support for the Golang and
> related runtimes:
> > Date: Tue, 22 Jan 2019 05:49:39 -0500
> >
> > Thanks Michele for the clarification
> >
> > My knative remark was only about to scale to zero, meaning when there is
> times that no one is using my app no pod/container is running at all.
> >
> > The main.go code can still be loaded via ConfigMap in Knative or without
> in Kubernetes. It still uses the same image
> >
> >
> > - Carlos Santana
> > @csantanapr
> >
> >> On Jan 21, 2019, at 1:34 PM, Michele Sciabarra 
> wrote:
> >>
> >> Actually YES this is exactly what I did.  A websocket that accept a
> websocket and writes in kafka.
> >>
> >> There is not yet a demo, but I am working on it. I am building a small
> webchat doing it.
> >>
> >> So far the demo is only this one https://hellows.sciabarra.net/ that
> uses the "hello.go" as an action turned in a websocket.
> >>
> >> If you use a websocket client you can see it is actually the action
> aswering in a websocket.
> >>
> >>> ws wss://hellows.sciabarra.net/hello
> >>> {"name":"Mike"}
> >> < {"golang-main-single":"Hello, Mike!"}
> >>
> >>> {}
> >> < {"golang-main-single":"Hello, world!"}
> >>
> >> You just need the action loop container modified and you can deploy an
> action in Kubernetes
> >>
> >> Yes this can be hooked in knative, but I did it simpler with an
> "autoinit".
> >> Code of the action is provided in a file through a confi

Re: Persistent Kafka Connections

[Markus Thömmes]

Hi,

with Tyson's concurrency > 1 support we should be able to rehash the
requirements here. A single action that supports a high concurrency on one
shared connection should fulfill what's needed by the Kafka backend teams
here.

Outscaling a backend through a lot of functions is a common problem in the
FaaS space btw @Michele. It might well worth be mentioning that in your
book as a common pitfall.

Cheers,
Markus

Am Mi., 9. Jan. 2019 um 15:58 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> This is also what I was thinking! But what I should recommend?
>
> I was thinking a crazy approach: create a NON-OPENWHISK service for
> sending messages
> for example a websocket service, use that to send messages to kafka and
> handle the answers in OpenWhisk.
>
> THere is a project that makes very simple to create websocket servers:
>
> http://websocketd.com/
>
> The nice this is that this websocket server works in a way very similar to
> the actioloop, so in principle I could write an action in the same way I do
> it in OpenWhisk  but instead of running it in OpenWhisk I can run that
> action in a Kubernetes cluster in order to serve a websocket.
>
> I wonder if I should write this in an OpenWhisk book though. Probably it
> makes sense to show the limit  of OpenWhisk and how to complement it with
> Kubernetes...
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>
> - Original message -
> From: Carlos Santana 
> To: dev@openwhisk.apache.org
> Subject: Re: Persistent Kafka Connections
> Date: Wed, 9 Jan 2019 09:32:17 -0500
>
> Is one of those things that works well for a hello world low scale
> deployment.
>
> In practice we saw a lot of actions running in parallel from multiple
> tenants and all of these even each one doing their caching were opening too
> many new socket connection to a kafka service and eventually the kafka
> service started to throttle and rate limit the number of new connections.
>
> This is the reason the code it’s there in the repo but deprecated and not
> recommended. From talking to kafka experts in ibm it was recommended from
> them that having stateless clients doing a lot of new connection it is a
> antipattern
>
> So they recommended to try to have a low number of connections open and
> high thruput for each of these connection, this means if you wan to produce
> from actions at scale and multitenant the actions would need to go thru
> some type of broker via http and this broker is a stateful service that
> maintains the socket connection to kafka service.
>
> — Carlos
>
> On Wed, Jan 9, 2019 at 8:12 AM Michele Sciabarra 
> wrote:
>
> > Hello whiskers,
> >
> > I am working on a simple webchat based on OpenWhisk  (full disclosure; it
> > is for the book on OpenWhisk for O'Reilly!) and I am using MessageHub.
> >
> > However in the documentation for the messaging package I read that the
> > /whisk.system/messaging/messageHubProduce is deprecated, and the
> > recommended way is to use a persistent connection to Kafka.
> >
> > This statement is puzzling me.  My first idea is "why do not they cache
> > the connection?", but I checked the code for messageHubProduce.py here:
> >
> >
> >
> https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/action/messageHubProduce.py
> >
> > and looks like the producer is cached, so I wonder if the deprecation of
> > messageHubProduce is still valid.
> >
> > Furthermore, I wrote an action to send messages to MessageHub with the
> > idea of implementing a cached connection, and apparently works:
> >
> >
> >
> https://github.com/learning-apache-openwhisk/chapter11-messages/blob/master/src/persistent/main/send.go
> >
> > So the question is: is this the correct way to send messages to Kafka
> > (caching connnections?) Can I recommend this practice in the book?
> >
> >
> > --
> >   Michele Sciabarra
> >   mich...@sciabarra.com
> >
> --
> Carlos Santana
> 
>

Re: the behavior of batcher

[Markus Thömmes]

Hi Jiang,

this is a fair question. The trade-off of using groupedWithin vs. batch is,
that groupedWithin **always** adds some latency to the database commands,
where batch only adds that latency on backpressure, as you've noticed.

Cheers,
Markus

Am Fr., 9. Nov. 2018 um 04:39 Uhr schrieb 蒋鹏程 :

> Hello all,​
> ​
> I just noticed that the db batcher doesn't behave as what I expected,
> seems it has a fixed size of workers, and each worker will get data from
> stream eagerly, so the batcher will try to insert 1 document/per requestat
> first, when all workers are busy, subsequent ​documents will be put into
> batches, and next free worker will process the batched documents,and so on.
> ​
> I think this behavior will not take full advantage of `bulk` API of
> database backend
> ​
> When there is 10,000 req/s, and the default size of workers is 64, let's
> assume every worker need take exact 10ms to complete its job, then there
> will be around 3250 res/s against the database theoretically:
> ​
> let's make 10ms as a time unit
> 1. in first 10ms, there are 100 documents in the stream, and 64 workers
> get 64 of them, 36 documents is left
> 2. in the next 10 ms, there are 100 + 36 documents in the stream, and 1
> worker can process all of them
> 3. the next 10ms * 2 is just like phrase 1 and phrase 2
> ​
> so the req/s against database backend will be (64+1)*50=3250
> ​
> is it better to use `groupedWithin` instead of `batch` here?
> ​
> Best Regards
> Jiang PengCheng
>

Re: Relieve CouchDB on high load

[Markus Thömmes]

Hi Christian,

given the recent work towards getting logs to a separate store to relieve
CouchDB and to make it possible to move it into a store that's more
appropriate for logging load and some other bits that are already
implemented (removing the DB based polling, being able to disable log
collection), I think it makes sense to be able to disable activation
writing as well! In that sense, yes I do agree on going forward with the
proposal  + implementation.

One thing that has me worried a bit is the way we're handling global,
namespaced and per-action limits today. I'm wondering if at some point
we'll need to consolidate those and make them cascade reliably. For this
specific knob, for example, I think it would make sense to have one at the
system level and on the action/trigger level accordingly. For some of our
limits that might be true today already, I feel like it might not be for
all though (although I admittedly haven't double-checked).

Cheers,
Markus

Am Di., 23. Okt. 2018 um 14:22 Uhr schrieb Christian Bickel <
cbic...@apache.org>:

> Hi developpers,
>
> in some performance tests in the past we've seen, that there are the
> following issues with Cloudant/CouchDB:
> - not all activations can be stored (there are error logs in the invoker,
> that storing was not possible)
> - during bursts, CouchDB needs to process each document to update all views
> in the activations-DB. If CouchDB is not able to process them immediately,
> because of a queue, calling these views returns an error or the result will
> be outdated (on calling them with stale). This has the impact, that there
> is a delay for all users of the system until their activation appear after
> calling `activation list`.
>
> To not have negative impact of some high-load users on the system the
> proposal is, to not store activations in activations-store for some
> specified namespaces.
> My proposal of an implementation is to put this flag into the
> limits-document in subjects database.
> This means, it can only be set by the administrator with wskadmin.
>
> I already opened a PR with this proposal:
> https://github.com/apache/incubator-openwhisk/pull/4078
>
> Do you agree on going forward with this proposal and implementation?
>
> Thanks a lot in advance for your feedback.
>
> Greetings
> Christian
>

Re: Promises (or equivalent) in Python and GoLang

[Markus Thömmes]

Hi Michele,

commenting only on Golang: While there are packages available that provide
a similar API surface, they have very little adoption throughout the Golang
community.

In general, the mechanism to achieve concurrent processing in Golang is
goroutines (https://tour.golang.org/concurrency/1). The mechanism to
communicate results in an asynchronous way is buffered channels (
https://tour.golang.org/concurrency/3).

Any specific behavior you're after?

Cheers,
Markus

Am Mo., 17. Sep. 2018 um 07:43 Uhr schrieb Michele Sciabarra <
mich...@sciabarra.com>:

> Hello,
>
> I checked the repositories for Python and GoLang clients, and I was
> expecting to find some API to perform asynchronous computation, much like
> Promises in Javascript.  You know, in Javascript you so
> ow.actions.invoke(...).then(...).
>
> What is the equivalent in Python, if any? Is there an equivalent in
> GoLang?
>
> --
>   Michele Sciabarra
>   mich...@sciabarra.com
>

Re: Notes+Video posted from today's Tech. Interchange call 2018-09-12

[Markus Thömmes]

Thanks Matt for putting this all together in lightning speed again! Very
much appreciated!

Am Mi., 12. Sep. 2018 um 20:07 Uhr schrieb Matt Rutkowski <
mrutk...@us.ibm.com>:

> Thanks Markus for moderating and special thanks to Tyson for leading us on
> the interesting topic of concurrent actions and all the resultant
> issues related to tracking usage for scheduling, load balancing and
> logging purposes.
>
> Video: https://www.youtube.com/watch?v=ARqcTQPaNEI&feature=youtu.be
> Notes:
>
> https://cwiki.apache.org/confluence/display/OPENWHISK/2018-09-12+OW+Tech+Interchange+-+Meeting+Notes
>
> also, thanks to Dragos for volunteering to moderate on the Sep. 26th call.
>
> Cheers!
>
> Kind regards,
> Matt
>
>
>

Re: How to pass size the max size of the request to the runtime?

[Markus Thömmes]

Heya,

I tend to agree with Felix. Buffers are usually just used as a transport
mechanism and they shouldn't need to fit the whole response at once (which
doesn't imply we don't load the response in memory, but at least that input
buffer doesn't need to be == content-length). Could you give a bit more
context on what the scan buffer does and why 64k isn't enough space to do
what we need?

Moreover, I'd rather not set any hard limits on the runtimes themselves.
The limits should be imposed by the surrounding system, the user-containers
could accept however much payload as they can fit. That reduces the amount
of configuration and testing needed. If we start imposing these limits on
every layer, we'll need to test every layer to correctly impose the limits.

Cheers,
Markus

Am Di., 11. Sep. 2018 um 15:08 Uhr schrieb Felix Meschberger
:

> Hi
>
> Holding the complete input in memory ? Sounds like a good DoS surface -
> unless you limit the input size
>
> Regards
> Felix
>
> --
> Creative typing support courtesy of my iPhone
>
> Am 11.09.2018 um 15:02 schrieb Rodric Rabbah  rod...@gmail.com>>:
>
> The http post will have a content length is that useful?
>
> -r
>
> On Sep 10, 2018, at 7:45 AM, Michele Sciabarra  > wrote:
>
> Hello, I am in the process of running the mandatory tests against the Go
> Runtime.
> In the process, I fixed a lot of bugs, because those tests revealed a
> number of details about encoding, env variables and other things that were
> not obvious to me in the first place.
>
> Now I have a problem: I am trying to pass the test that tried to send a
> one-megabyte big request to the runtime.
> Currently, it does not work because I discovered the "scan" buffer has in
> Golang a fixed size of 64k.
>
> Of course, I can increase it but I need to know how big it must be. I know
> that you can set some parameters at OpenWhisk level but I am not aware how
> a runtime can know those parameters. Most notably I need to be able to read
> the maximum size of the requests because I need to allocate a buffer at
> init time. Any hints?
>
>
> --
> Michele Sciabarra
> mich...@sciabarra.com
>

Re: How to do basic validation of custom OpenWhisk deployment

[Markus Thömmes]

Huge +1 from my end!

I always thought of the "system.*" suite to cover exactly what you're
looking for, but we've been lax with regards to splitting test-suites and
the properties needed as you're experiencing now. It'd be great to clean
that up and put tests in appropriate packages to be able to run targets
easily (like the one you need here).

Am Mi., 5. Sep. 2018 um 15:39 Uhr schrieb Chetan Mehrotra <
chetan.mehro...@gmail.com>:

> TL;DR How to test that a custom OpenWhisk deployment is working as expected
>
> The requirement is that I have a custom OpenWhisk deployment and I need to
> do basic validation of this setup like action CRUD is working or not.
> Looking at current test structure I believe it should be possible to
> validate that by running the tests under system/basic.
>
> Such a run should just need following config
>
> 1. Edge host url and port
> 2. Auth credentials for some test account
> 3. OpenWhisk home directory - To enable loading test artifacts
>
> However if I try to run the basic test currently they seem to require some
> other properties from whisk.properties. Based on trial and error below is
> minimal set of props required
>
> -
> edge.host=172.17.0.1
> edge.host.apiport=443
> testing.auth=/path/to/openwhisk/ansible/files/auth.guest
> openwhisk.home=/path/to/openwhisk
>
> # Props below should be optional
> limits.actions.sequence.maxLength=50
> whisk.ssl.challenge=openwhisk
>
> whisk.ssl.cert=/path/to/openwhisk/ansible/roles/nginx/files/openwhisk-server-cert.pem
> 
>
> Would it be fine to provide a test target which just work with initial 4
> properties? Going further have them passed inline i.e. auth credentials
> passed via env instead of first adding them to some file.
>
> Chetan Mehrotra
>

Re: Prototyping for a future architecture

[Markus Thömmes]

Hi Tzu-Chiao,

first of all: This looks amazing! As the terminology is similar: Is this a
separate design or is this based on what I've put out a few weeks ago?

In general: Everybody is eligible to join the prototyping effort, I think
that's the whole point of having that discussion here. You're very welcome
to join there and it seems you'll even be more knowledgeable on some of the
things than anybody else, so it'd be awesome to get your experience in
there!

Cheers,
Markus

Am Mi., 29. Aug. 2018 um 10:48 Uhr schrieb TzuChiao Yeh <
su3g4284zo...@gmail.com>:

> Hi,
>
> I'm wondering if someone who pay interests aside from core teams (like me)
> are eligible to join prototyping? I know it will increase the maintenance
> effort on managing all things being visible: opening and labeling issues,
> plans, assignees, etc. If yes, from my point of view, a standalone repo
> with issue tracker and pull requests system might be easier to achieve the
> visibility and communication. Will it be possible to use submodule if
> there's a code-reuse or clarifying migration path concerned?
>
> For prototyping:
> I've drafted a small prototype [1] during last months. Bare me to introduce
> some background about me: some folks might know that, I'm a master student
> and have attended google summer of code this year with openwhisk, mentored
> by Rodric and Carlos. The prototype is the submitted result for final work
> during last few months, although it's buggy, ill-designed and not-finished
> yet. However, if it's ready to start prototyping now, bare me to showoff
> this under a poor state, as the purpose of my past work, hope this help to
> explore ideas before going to deeper and collaboration. (I've asked most of
> issues I've found in previous threads and may be solved in mind between
> folks however.)
>
> [1] https://tz70s.github.io/posts/openwhisk-performance-improvement/
>
> Thanks,
> Tzu-Chiao
>
> On Wed, Aug 29, 2018 at 11:31 AM Markus Thömmes  >
> wrote:
>
> > Hi,
> >
> > okay, let's separate concerns:
> >
> > ### Golang vs. Scala vs. something else entirely: ###
> >
> > I'm okay doing it in Scala although I don't see an issue in swapping the
> > language for a component that needs to be rewritten from the ground up
> > anyway (which I'm suggesting at least the ContainerRouter is).
> >
> > The ContainerPool/ContainerProxy part of it comes to mind immediately. It
> > was built with a scale of maybe 100 containers max in mind. There are a
> few
> > performance problems with it at scale, that I can point out off the bat.
> > The changes needed to mitigate those are rather severe and while we
> **can**
> > implement those (and maybe even should if this proofs to make sense) I
> > believe it makes sense to at least experiment with a freshly written
> > component. In the new architecture, we can also greatly reduce the state
> > model to just paused/unpaused for the ContainerRouter, further
> simplifying
> > its implementation.
> >
> > The ContainerRouter's speed requirements are more close to those of
> > nginx/envoy than they are to those of the current ContainerPool/Proxy
> > implementation. As Dave mentioned, for the ContainerRouter it will be
> very
> > critical to know how many containers it can handle in reality, hence I'm
> > shooting for an as efficient implementation of its innerts as possible to
> > flesh out how far we can push the boundaries here.
> >
> > In any case: Yes we can stick to Scala here, but at least the
> > ContainerRouter might be 100% new code and we might not even be able to
> > rely on things like akka-http for its implementation for the reasons
> > mentioned above.
> >
> > ### Seperate repo vs. branch ###
> >
> > Towards Michaels points: The prototype repo is supposed to be used for
> > experimenting/prototyping and for that purpose only. As Dave pointed out
> > above, we'd want these experiments to happen with Apache visibility.
> There
> > will never be a completely working system available in that repository.
> The
> > goal of it is to find out if certain assumptions of a new design are
> > feasible or not. The goal is **not** to build productised components that
> > we can take over to the mainline.
> >
> > I always thought of that rearch as an incremental step as well and it
> > shouldn't need major changes to the API (or at least none that we are not
> > already introducing right now). I don't think we're going to see the true
> > potential of this though if we try to take the syste

Re: Prototyping for a future architecture

[Markus Thömmes]

> https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F091hdj,%2Fm%2F09gbxjr
>
> [2] - https://octoverse.github.com/
>
> On Tue, Aug 28, 2018 at 2:53 PM Rodric Rabbah  wrote:
>
> > Thanks Michael for raising these points. I share the same opinion and
> > sentiment and think a branch with a clean migration story is better and
> > makes more sense. I am not entirely convinced that the choice of language
> > itself will make the difference vs the new architecture which is quite
> > different and should in itself be more efficient.
> >
> > -r
> >
> > On Tue, Aug 28, 2018 at 4:51 PM Michael Marth 
> > wrote:
> >
> > > Hi Markus,
> > >
> > > IMHO what you propose below is a rather severe change in scope of this
> > > discussion and effort.
> > > Up until so far this was about _evolving_ the OW architecture. We have
> > not
> > > explicitly discussed it, but one could assume that it is at least
> > feasible
> > > to gradually adopt the new architecture. So there would be a smooth
> path
> > > between the current state of the code base and a future one.
> > >
> > > Your proposal below breaks this assumption somewhat (by proposing a new
> > > repo instead of a branch - which will inevitably make the 2 code bases
> > > drift apart) as well as explicitly by suggesting a new implementation
> > > language. Especially the latter would create a schism between OW-now
> and
> > > OW-future.
> > > This schism has implications like the perception of OW-now being
> > > deprecated, the _possibility_ of no clean upgrade path, the immediate
> > split
> > > of the community between *-now and *-future and of course carries the
> > risk
> > > of the version 2 syndrome.
> > >
> > > I would propose to implement the future architecture in a branch and in
> > > Scala first. If it turns out to be good, then subsequent experiments
> can
> > > show or not-show if a switch of language is of additional value. That
> > would
> > > allow to make a decision based on data rather than anything else.
> > >
> > > My2c
> > > Michael
> > >
> > >
> > > On 28.08.18, 14:26, "Markus Thömmes" 
> wrote:
> > >
> > > Hi all,
> > >
> > > Am Mo., 27. Aug. 2018 um 20:04 Uhr schrieb David P Grove <
> > > gro...@us.ibm.com
> > > >:
> > >
> > > >
> > > >
> > > >
> > > > "Markus Thömmes"  wrote on 08/23/2018
> > > 04:19:33
> > > > PM:
> > > >
> > > > >
> > > > > Key point I want to make is: At some point we'll have to start
> to
> > > > prototype
> > > > > things out and see if our assumptions actually hold water. For
> > > example,
> > > > my
> > > > > assumption on a work-stealing backend is pretty much in the
> air.
> > > > >
> > > > > My proposal for going forward would be:
> > > > > 1. Create a playground for the implementation of some parts of
> > the
> > > system
> > > > > (a new repository?)
> > > > > 2. Let's build some of the things that are uncontroversial and
> > > absolutely
> > > > > needed in any case (ContainerRouter, ContainerManager).
> > > > > 3. Play around with them, hook them up in different ways, see
> > what
> > > works
> > > > > and what doesn't.
> > > > >
> > > > > Some things will need some testing out to see the scale that
> the
> > > > components
> > > > > can operate at. These things will narrow or widen the solution
> > > space for
> > > > > the more controversial topics around how to distribute
> containers
> > > in the
> > > > > system, how to balance between the routers, work-stealing
> queue:
> > > yes/no
> > > > etc.
> > > > >
> > > > > Having some simple components fleshed out could encourage
> > > innovation and
> > > > > creates some facts that we need to focus things into a good
> > > direction.
> > > > >
> > > > > What do you think? Too early to start with this and/or the
> wrong
> > > way of
> > > > > doing it?
&g

Re: Bi-weekly Tech Interchange call tomorrow

[Markus Thömmes]

Hi Tyson,

shall we discuss the mechanics on how to start the prototype for the
future-arch discussion?

Cheers,
Markus

Am Di., 28. Aug. 2018 um 16:59 Uhr schrieb Tyson Norris
:

> Hi Whiskers!
>
> Please send any agenda items you would like to discuss at the Tech
> Interchange call tomorrow.
> Thanks
> Tyson
>
> Call details:
> Web Meeting: Tech Interchange (bi-weekly):
> - Day-Time: Wednesdays, 11AM EDT (Eastern US), 5PM CEST (Central Europe),
> 3PM UTC, 11PM CST (Beijing)
> - Zoom: https://zoom.us/my/asfopenwhisk
>

Re: Prototyping for a future architecture

[Markus Thömmes]

Hi all,

Am Mo., 27. Aug. 2018 um 20:04 Uhr schrieb David P Grove :

>
>
>
> "Markus Thömmes"  wrote on 08/23/2018 04:19:33
> PM:
>
> >
> > Key point I want to make is: At some point we'll have to start to
> prototype
> > things out and see if our assumptions actually hold water. For example,
> my
> > assumption on a work-stealing backend is pretty much in the air.
> >
> > My proposal for going forward would be:
> > 1. Create a playground for the implementation of some parts of the system
> > (a new repository?)
> > 2. Let's build some of the things that are uncontroversial and absolutely
> > needed in any case (ContainerRouter, ContainerManager).
> > 3. Play around with them, hook them up in different ways, see what works
> > and what doesn't.
> >
> > Some things will need some testing out to see the scale that the
> components
> > can operate at. These things will narrow or widen the solution space for
> > the more controversial topics around how to distribute containers in the
> > system, how to balance between the routers, work-stealing queue: yes/no
> etc.
> >
> > Having some simple components fleshed out could encourage innovation and
> > creates some facts that we need to focus things into a good direction.
> >
> > What do you think? Too early to start with this and/or the wrong way of
> > doing it?
> >
>
> +1 for starting to prototype.  It's been a good discussion and I think
> we've identified some things that we know we don't know, so time to
> experiment and find out.
>
>
> Not sure what the best logistics are for this.  Would like the work to be
> at Apache (community visibility).  I'm not sure if the best way is a new
> repo or an experimental branch of the main repo (we could dial down the
> level of testing on the branch to make it less cumbersome?).  The branch is
> attractive to me because it might make it easier to keep in synch with the
> components we aren't changing.
>

I actually think we should generate a new repository for this. Opening PRs
etc. will then not clutter the "main" repository and we don't need to worry
about breaking anything.

If nobody objects I'm going to get "incubator-openwhisk-protoype" generated
and will create a rough outline in the repository (different folders for
different parts of the system).

One more basic question (and this is going to be controversial): Most of
the componentry in the execution layer will have to be build anew. I'd like
to switch the implementation language of at least the ContainerRouter to
Golang. Why? Because scale/performance matters a lot for them. As Dave
mentioned on multiple occasions, it will greatly matter how many containers
such a Router can handle under load and that scale will define the
implementation alternatives we will have at hand. I therefore would like to
optimise these Routers for minimal overhead. We could go even lower level,
but I guess that'd be at a kinda big maintenance cost.

I can envision the ContainerManager to be simpler to implement in Golang as
well, at least for some of the deployment alternatives (Kubernetes comes to
mind). The Golang based clients seemed superior to me vs. clients in any
other language.

As all of the communication will probably be HTTP only anyway, these
implementations should be swappable anytime.

Comments very welcome! Let me know your opinions.

Cheers,
Markus

Re: [Discussion] Use TransactionID as ActivationID

[Markus Thömmes]

Thanks for bringing that up Rodric,

it seems like we could have a one stone two birds thing going on here? Do
you see an immediate reason why the proposal I made would not work? We'd
certainly need to check if any client hardcodes a fixed length for our
activation ids.

Am Di., 28. Aug. 2018 um 05:11 Uhr schrieb Dominic Kim :

> got it 👍:+1
>
> Best regards
> Dominic
>
> 2018년 8월 28일 (화) 오전 10:38, Rodric Rabbah 님이 작성:
>
> > Related discussion in this issue https://github.com/apache/
> > incubator-openwhisk/issues/3083 with a connection to open-tracing.
> >
> > -r
> >
>

Re: [Discussion] Use TransactionID as ActivationID

[Markus Thömmes]

Hi Dominic,

Am Mo., 27. Aug. 2018 um 07:41 Uhr schrieb Dominic Kim :

> Hey Markus.
>
> Yes readable activation is a great idea as it would be useful for debugging
> and some cases such as you mentioned(groupable activations.)
>
> BTW, are you suggesting to keep activation generation way in Nginx, and
> allow controllers to have parent/child activations in order to avoid
> duplicate IDs?
> More precisely, controllers will generate ID only in case of triggers and
> sequence actions.
> So if we make parent/child relationship in controllers only, it won't be
> duplicate because there is no such notion in activation generation in
> Nginx.
> Do I understand correctly?
>

Yes I think you got that right. Nginx would generate the activationId for
the initial trigger fire for example. For each rule the controller invokes
then, the controller can generate a child id for that invocation. If we can
make that readable, we can easily correlate the parent with the child,
giving us even more of a gain from this proposal.

So from my PoV: Go for it!


>
> Thanks
> Best regards
> Dominic.
>
>
> 2018년 8월 23일 (목) 오후 5:34, Markus Thömmes 님이 작성:
>
> > Hi Dominic,
> >
> > this is certainly a good thought and would simplify debugging production
> > systems as well.
> >
> > A rough and dirty idea for the uniqueness issue you mentioned:
> > What if we just append a suffix for each of the child ids (essentially
> they
> > are in a parent/child relationship). A simple example:
> >
> > Sequence action invoke, Router generated id: abc-def (used as
> TransactionId
> > and ActivationId)
> > Invocation of the first action: abc-def-1
> > Invocation of the second action: abc-def-2
> > ...
> >
> > The suffix doesn't necessarily need to be a number, but especially in the
> > case of sequences, that could greatly help observability and
> understanding
> > what's going on. It'd also help operators, because the respective
> requests
> > would be groupable (logwise) by the respective "BaseActivationId"
> (abc-def
> > in the example). As the ActivationId no longer needs to be a UUID anyway,
> > we can do whatever we want to do with it.
> >
> > If we use numbers, we need to ensure consistent generation of these of
> > course. I think though, both Triggers and Sequence invocations will only
> be
> > handled locally by one Controller, so it can hand out the numbers
> > guaranteeing they don't overlap.
> >
> > What do you think?
> >
> > Cheers,
> > Markus
> >
> > Am Do., 23. Aug. 2018 um 05:39 Uhr schrieb Dominic Kim <
> > style9...@gmail.com
> > >:
> >
> > > Dear whiskers.
> > >
> > > This is to discuss to whether we will allow using TransactionID as
> > > ActivationID.
> > > Some of discussion history is here:
> > > https://github.com/apache/incubator-openwhisk/pull/3671
> > >
> > > As that PR is merged, now ActivationID is included in response headers.
> > >
> > > Currently, TransactionID is being generated in Nginx with this change:
> > > https://github.com/apache/incubator-openwhisk/pull/3199
> > >
> > > Then, one question that Rabbah raised comes up.
> > > Why do we have similar two IDs in headers? (x-request-id,
> > > x-openwhisk-activation-id)
> > > Couldn't it be possible to use just one ID for both the cases?
> > >
> > > Basic idea is just to use TransactionID(x-request-id) as a
> ActivationID.
> > > But there are still few concerns that Christian described here:
> > >
> > >
> >
> https://github.com/apache/incubator-openwhisk/pull/3671#issuecomment-392002543
> > >
> > > And the main issue is the uniqueness of ActivationID.
> > > If we allow using TransactionID as ActivationID, ActivationID could be
> > > generated in two different places(Nginx, controller) as a controller
> also
> > > needs to generate activation id in case of triggers and sequence
> actions
> > > invocation.
> > >
> > > I want to kick off the discussion and listen to a good idea from many
> > > whiskers.
> > >
> > >
> > > Thanks
> > > Regards
> > > Dominic.
> > >
> >
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi,

Am Fr., 24. Aug. 2018 um 00:07 Uhr schrieb Tyson Norris
:

> > Router is not pulling at queue for "specific actions", just for any
> action
> > that might replace idle containers - right? This is complicated with
> > concurrency though since while a container is not idle (paused +
> > removable), it may be useable, but only if the action received is
> the same
> > as one existing warm container, and that container has concurrency
> slots
> > available for additional activations. It may be helpful to diagram
> some of
> > this stealing queue flow a bit more, I'm not seeing how it will work
> out
> > other than creating more containers than is absolutely required,
> which may
> > be ok, not sure.
> >
>
> Yes, I will diagram things out soonish, I'm a little bit narrow on time
> currently.
>
> The idea is that indeed the Router pulls for *specific* actions. This
> is a
> problem when using Kafka, but might be solvable when we don't require
> Kafka. I have to test this for feasibility though.
>
>
> Hmm OK - it's not clear how a router that is empty (not servicing any
> activations) becomes a router that is pulling for that specific action,
> when other routers pulling for that action are at capacity (so new
> containers are needed)
>

Disclaimer: All of this is very much in the idea and not part of the
original proposal.

That's where the second part of the "idea" that I had above comes in: If we
can somehow detect, that nobody is pulling (all are at capacity or have no
container), that is then the moment where we need to create new containers.

I proposed further above in the discussion that the Routers do *not* ask
for more Containers but rather that signal of the MQ (hey, nobody is having
capacity for that action apparently... so let's create capacity) could be
used to create Containers.

That's a very blunt assumption though, I don't know if that's feasible at
all. There might be a performant way of implementing just that signal in a
distributed way though (the signal being: We are out of capacity over the
whole cluster, we need more). A second idea that comes to my mind would be
to implement a shared counter (it can easily be eventually consistent,
consistency is I think not a concern here). Once that counter drops to 0 or
below 0, we need more Containers.

This is I think were the prototyping thread comes in: Personally, I don't
feel comfortable stating that any of the approaches outlined by me really
do work without having tried it out.

Prototyping for a future architecture

[Markus Thömmes]

Hi OpenWhiskers,

We've been discussing a new direction for our architecture based on a
proposal I made here:
https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture

Discussion threads:
-
https://lists.apache.org/thread.html/29289006d190b2c68451f7625c13bb8020cc8e9928db66f1b0def18e@%3Cdev.openwhisk.apache.org%3E
-
https://lists.apache.org/thread.html/02986151fb2cffb7426c0d3b207c35923bc49bf164a967d310b3bbb6@%3Cdev.openwhisk.apache.org%3E
-
https://lists.apache.org/thread.html/efec3d588e3519ce72b44f2d5ae00cabf1f6a33afaa2f08194290553@%3Cdev.openwhisk.apache.org%3E

This discussion has been veeery fruitful and I enjoyed i'm enjoying it a
lot! Thanks so far to anybody who continues to contribute here!

I think we are reaching a point where we start assuming a lot of things.
I'll attempt to diagram out my current view of things and maybe we can
already agree on some key parts of the current design as-is? Things I have
in mind that we need:

1. ContainerRouter: I think there's little debate around us needing an
efficient router, preferably with an API, that we can add/remove containers
from.
2. ContainerManager: In terms of it being the interface creating containers
seems pretty accepted as well, from what I read in the discussions.
3. A work-stealing backend: A backend (hopefully a pre-canned MQ) that
allows us to do action-specific pulling and signaling of demand through
that pulling. This one I think is more subject to a debate than the other
two.

We should also lay out some of the basic gains we want to have from that
new system, i.e.:
1. Makes streaming payloads possible.
2. Has very high warm-ratios.
3. Doesn't create (many) more containers than needed.

(These are examples, others will feel different)

Key point I want to make is: At some point we'll have to start to prototype
things out and see if our assumptions actually hold water. For example, my
assumption on a work-stealing backend is pretty much in the air.

My proposal for going forward would be:
1. Create a playground for the implementation of some parts of the system
(a new repository?)
2. Let's build some of the things that are uncontroversial and absolutely
needed in any case (ContainerRouter, ContainerManager).
3. Play around with them, hook them up in different ways, see what works
and what doesn't.

Some things will need some testing out to see the scale that the components
can operate at. These things will narrow or widen the solution space for
the more controversial topics around how to distribute containers in the
system, how to balance between the routers, work-stealing queue: yes/no etc.

Having some simple components fleshed out could encourage innovation and
creates some facts that we need to focus things into a good direction.

What do you think? Too early to start with this and/or the wrong way of
doing it?

Cheers,
Markus

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Dave,

I agree! I'll start another thread on a discussion of how/where we
prototype things to hash out some of the unknowns.

Cheers,
Markus

Am Do., 23. Aug. 2018 um 22:05 Uhr schrieb David P Grove :

>
> Related to the random vs. smart routing discussion.
>
> A key unknown that influences the design is how much load we can drive
> through a single ContainerRouter.
> + If they are highly scalable (500 to 1000 containers per router),
> then even a fairly large OpenWhisk deployment could be running with a
> handful of ContainerRouters (and smart routing is quite viable).
> + If they are less scalable (10s to 100 containers per router) then
> large deployments will be running with 50+ ContinaerRouters and smart
> routing degrades to random routing in terms of container reuse for our long
> tail workloads.
>
> --dave
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Do., 23. Aug. 2018 um 21:28 Uhr schrieb Tyson Norris
:

> >
> > And each ContainerRouter has a queue consumer that presumably pulls
> from
> > the queue constantly? Or is consumption based on something else? If
> all
> > ContainerRouters are consuming at the same rate, then while this does
> > distribute the load across ContainerRouters, it doesn't really
> guarantee
> > any similar state (number of containers, active connections, etc) at
> each
> > ContainerRouter, I think. Maybe I am missing something here?
> >
>
>
> The idea is that ContainerRouters do **not** pull from the queue
> constantly. They pull work for actions that they have idle containers
> for.
>
> Router is not pulling at queue for "specific actions", just for any action
> that might replace idle containers - right? This is complicated with
> concurrency though since while a container is not idle (paused +
> removable), it may be useable, but only if the action received is the same
> as one existing warm container, and that container has concurrency slots
> available for additional activations. It may be helpful to diagram some of
> this stealing queue flow a bit more, I'm not seeing how it will work out
> other than creating more containers than is absolutely required, which may
> be ok, not sure.
>

Yes, I will diagram things out soonish, I'm a little bit narrow on time
currently.

The idea is that indeed the Router pulls for *specific* actions. This is a
problem when using Kafka, but might be solvable when we don't require
Kafka. I have to test this for feasibility though.


>
> Similar state in terms of number of containers is done via the
> ContainerManager. Active connections should roughly even out with the
> queue
> being pulled on idle.
>
> Yeah carefully defining "idle" may be tricky, if we want to achieve
> absolute minimum containers in use for a specific action at any time.
>
>
> >
> > The edge-case here is for very slow load. It's minimizing the
> amount of
> > Containers needed. Another example:
> > Say you have 3 Routers. A request for action X comes in, goes to
> > Router1.
> > It requests a container, puts the work on the queue, nobody
> steals it,
> > as
> > soon as the Container gets ready, the work is taken from the
> queue and
> > executed. All nice and dandy.
> >
> > Important remark: The Router that requested more Containers is
> not
> > necessarily the one that's getting the Containers. We need to
> make
> > sure to
> > evenly distribute Containers across the system.
> >
> > So back to our example: What happens if requests for action X
> are made
> > one
> > after the other? Well, the layer above the Routers (something
> needs to
> > loadbalance them, be it DNS or another type of routing layer)
> isn't
> > aware
> > of the locality of the Container that we created to execute
> action X.
> > As it
> > schedules fairly randomly (round-robin in a multi-tenant system
> is
> > essentially random) the action will hit each Router once very
> soon. As
> > we're only generating one request after the other, arguably we
> only
> > want to
> > create only one container.
> >
> > That's why in this example the 2 remaining Routers with no
> container
> > get a
> > reference to Router1.
> >
> > In the case you mentioned:
> > > it seems like sending to another Router which has the
> container, but
> > may
> > not be able to use it immediately, may cause failures in some
> cases.
> >
> > I don't recall if it's in the document or in the discussion on
> the
> > dev-list: The router would respond to the proxied request with a
> 503
> > immediatly. That tells the proxying router: Oh, apparently we
> need more
> > resources. So it requests another container etc etc.
> >
> > Does that clarify that specific edge-case?
> >
> > Yes, but I would not call this an edge-case -  I think it is more of
> a
> > ramp up to maximum container reuse, and will probably dramatically
> impacted
> > by containers that do NOT support concurrency (will get a 503 when a
> single
> > activation is in flight, vs high concurrency container, which would
> cause
> > 503 only once max concurrency reached).
> > If each ContainerRouter is as likely to receive the original
> request, and
> > each is also as likely to receive the queued item from the stealing
> queue,
> > then there will be a lot of cross traffic during the ramp up from 1
> > container to  containers. E.g.
> >
> > From client:
> > Request1 -> Router 1 -> queue (no containers)
> > Request2 -> Router 2 -> queue (no containers)
> > Request3 -> Router 3 -> queue (no containers)
> > From queue:
> > Req

Re: Simplifying configuring OpenWhisk Components (#3984)

[Markus Thömmes]

Hi Chetan,

Am Do., 23. Aug. 2018 um 10:28 Uhr schrieb Chetan Mehrotra <
chetan.mehro...@gmail.com>:

> > Is it possible to layer configurations with this? For example: If I
> create
> a `database.conf` and a `controller.conf`, is there a way to mount these in
> a way that they are both read and merged by the specific component?
>
> This should be possible as of now also. If file mount option is used [1]
> then you need to
>
> 1. Mount configMap directory  `/controller/config`
> 2. Have one `application.conf` in that directory which has includes for
> `database.conf` and `controller.conf`
>

Ahhh thanks, I completely forgot about includes!


>
> Then with changes done for #3059 these files would be included in classpath
> and typesafe config logic to read all "application.conf" would get them
> included.
>
> Only issue is order is non deterministic so config override would not work.
> That can be enabled by setting system property
> -Dconfig.file=/controller/config/application.conf
> (or env CONFIG_config_file). This would ensure that any later includes
> would supersede previous includes
>

Sounds good.

How will this impact other deployment tools, like Docker Compose? I'm aware
that your change keeps the old path working, but do we envision to drop
that at some point?

Would it be valuable to have a writeup in the Wiki outlining how we'd
envision a future configuration method to work across all the deployment
methods that we have? I feel like there's lots of simplification to get
especially for cases like Mesos and/or Kubernetes, if we drop the env-var
rewriting.


>
>
> Chetan Mehrotra
> [1]
>
> https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap
>
> On Thu, Aug 23, 2018 at 1:32 PM Markus Thömmes 
> wrote:
>
> > Hi Chetan,
> >
> > good idea!
> >
> > A bit of background on why it is how it is: When I implemented the
> approach
> > we're having today, the basic thought was to be able to detect on a quite
> > granular level what changes are needed to which components, even when
> they
> > share values. For example: In the Kubernetes case, most of the env
> > variables are read out of a ConfigMap. Updating such a ConfigMap can
> > trigger a restart of the specific pods to reload the config.
> >
> > This is possible with this approach as well, in fact for something like
> > Kubernetes it's even easier (I believe you can mount the ConfigMap
> directly
> > without needing to pass it via env). What I'd like to make sure though:
> >
> > Is it possible to layer configurations with this? For example: If I
> create
> > a `database.conf` and a `controller.conf`, is there a way to mount these
> in
> > a way that they are both read and merged by the specific component?
> >
> > Cheers,
> > Markus
> >
> > Am Do., 23. Aug. 2018 um 08:36 Uhr schrieb Chetan Mehrotra <
> > chetan.mehro...@gmail.com>:
> >
> > > Hi Team,
> > >
> > > Currently we rely on env variables to configure OpenWhisk component
> like
> > > Controller and Invoker. For example setting
> > > whisk.loadbalancer.invoker-busy-threshold would need env name like
> > > "CONFIG_whisk_loadbalancer_invokerBusyThreshold".
> > >
> > > This mapping is at times tricky to get right and bit error prone
> > (specially
> > > when setting list props). It would be better if we can directly pass in
> > the
> > > config in typesafe config format [1]. This would simplify managing the
> > > config in version control and also reusing the default config directly.
> > >
> > > For this purpose I have created a PR #3984. For now this PR only enable
> > use
> > > to this proposed approach. If it looks good we can modify the default
> > > ansible setup to use the new approach in a subsequent PR.
> > >
> > > Feedback welcome!
> > >
> > > Chetan Mehrotra
> > > [1] https://github.com/lightbend/config/blob/master/HOCON.md
> > > [2] https://github.com/apache/incubator-openwhisk/pull/3984
> > >
> >
>

Re: [Discussion] Use TransactionID as ActivationID

[Markus Thömmes]

Hi Dominic,

this is certainly a good thought and would simplify debugging production
systems as well.

A rough and dirty idea for the uniqueness issue you mentioned:
What if we just append a suffix for each of the child ids (essentially they
are in a parent/child relationship). A simple example:

Sequence action invoke, Router generated id: abc-def (used as TransactionId
and ActivationId)
Invocation of the first action: abc-def-1
Invocation of the second action: abc-def-2
...

The suffix doesn't necessarily need to be a number, but especially in the
case of sequences, that could greatly help observability and understanding
what's going on. It'd also help operators, because the respective requests
would be groupable (logwise) by the respective "BaseActivationId" (abc-def
in the example). As the ActivationId no longer needs to be a UUID anyway,
we can do whatever we want to do with it.

If we use numbers, we need to ensure consistent generation of these of
course. I think though, both Triggers and Sequence invocations will only be
handled locally by one Controller, so it can hand out the numbers
guaranteeing they don't overlap.

What do you think?

Cheers,
Markus

Am Do., 23. Aug. 2018 um 05:39 Uhr schrieb Dominic Kim :

> Dear whiskers.
>
> This is to discuss to whether we will allow using TransactionID as
> ActivationID.
> Some of discussion history is here:
> https://github.com/apache/incubator-openwhisk/pull/3671
>
> As that PR is merged, now ActivationID is included in response headers.
>
> Currently, TransactionID is being generated in Nginx with this change:
> https://github.com/apache/incubator-openwhisk/pull/3199
>
> Then, one question that Rabbah raised comes up.
> Why do we have similar two IDs in headers? (x-request-id,
> x-openwhisk-activation-id)
> Couldn't it be possible to use just one ID for both the cases?
>
> Basic idea is just to use TransactionID(x-request-id) as a ActivationID.
> But there are still few concerns that Christian described here:
>
> https://github.com/apache/incubator-openwhisk/pull/3671#issuecomment-392002543
>
> And the main issue is the uniqueness of ActivationID.
> If we allow using TransactionID as ActivationID, ActivationID could be
> generated in two different places(Nginx, controller) as a controller also
> needs to generate activation id in case of triggers and sequence actions
> invocation.
>
> I want to kick off the discussion and listen to a good idea from many
> whiskers.
>
>
> Thanks
> Regards
> Dominic.
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Do., 23. Aug. 2018 um 00:33 Uhr schrieb Tyson Norris
:

> Hi - thanks for the discussion! More inline...
>
> On 8/22/18, 2:55 PM, "Markus Thömmes"  wrote:
>
> Hi Tyson,
>
> Am Mi., 22. Aug. 2018 um 23:37 Uhr schrieb Tyson Norris
> :
>
> > Hi -
> > >
> > > When exactly is the case that a ContainerRouter should put a
> blocking
> > > activation to a queue for stealing? Since a) it is not spawning
> > containers
> > > and b) it is not parsing request/response bodies, can we say
> this
> > would
> > > only happen when a ContainerRouter maxes out its incoming
> request
> > handling?
> > >
> >
> > That's exactly the idea! The work-stealing queue will only be
> used if
> > the
> > Router where to request landed cannot serve the demand right
> now. For
> > example, if it maxed out the slots it has for a certain action
> (all
> > containers are working to their full extent) it requests more
> > resources and
> > puts the request-token on the work-stealing queue.
> >
> > So to clarify, ContainerRouter "load" (which can trigger use of
> queue) is
> > mostly (only?) based on:
> > * the number of Container references
> > * the number of outstanding inbound  HTTP requests, e.g. when lots of
> > requests can be routed to the same container
> > * the number of outstand outbound HTTP requests to remote action
> > containers (assume all are remote)
> > It is unclear the order of magnitude considered for "maxed out
> slots",
> > since container refs should be simple (like ip+port, action metadata,
> > activation count, and warm state), inbound connection handling is
> basically
> > a http server, and outbound is a connection pool per action container
> > (let's presume connection reuse for the moment).
> > I think it will certainly need testing to determine these and to be
> > configurable in any case, for each of these separate stats.. Is there
> > anything else that affects the load for ContainerRouter?
> >
>
> "Overload" is determined by the availability of free slots on any
> container
> being able to serve the current action invocation (or rather the
> absence
> thereof). An example:
> Say RouterA has 2 containers for action X. Each container has an
> allowed
> concurrency of 10. On each of those 2 there are 10 active invocations
> already running (the ContainerRouter knows this, these are open
> connections
> to the containers). If another request comes in for X, we know we don't
> have capacity for it. We request more resources and offer the work we
> got
> for stealing.
>
> I don't think there are tweaks needed here. The Router keeps an
> "activeInvocations" number per container and compares that to the
> allowed
> concurrency on that container. If activeInvocations ==
> allowedConcurrency
> we're out of capacity and need more.
>
> We need a work-stealing queue here to dynamically rebalance between the
> Routers since the layer above the Routers has no idea about capacity
> and
> (at least that's my assumption) schedules randomly.
>
> I think it is confusing to say that the ContainerRouter doesn't have
> capacity for it - rather, the existing set of continers in the
> ContainerRouter don't have capacity for it. I understand now, in any case.
>

Noted, will adjust future wording on this, thanks!


> So there are a couple of active paths in ContainerRouter, still only
> considering sync/blocking activations:
> * warmpath - run immediately
> * coldpath - send to queue
>
> And each ContainerRouter has a queue consumer that presumably pulls from
> the queue constantly? Or is consumption based on something else? If all
> ContainerRouters are consuming at the same rate, then while this does
> distribute the load across ContainerRouters, it doesn't really guarantee
> any similar state (number of containers, active connections, etc) at each
> ContainerRouter, I think. Maybe I am missing something here?
>


The idea is that ContainerRouters do **not** pull from the queue
constantly. They pull work for actions that they have idle containers for.

Similar state in terms of number of containers is done via the
ContainerManager. Active connections should roughly even out with the queue
being pulled on idle.


>
>

>
>
> >
> >   

Re: Simplifying configuring OpenWhisk Components (#3984)

[Markus Thömmes]

Hi Chetan,

good idea!

A bit of background on why it is how it is: When I implemented the approach
we're having today, the basic thought was to be able to detect on a quite
granular level what changes are needed to which components, even when they
share values. For example: In the Kubernetes case, most of the env
variables are read out of a ConfigMap. Updating such a ConfigMap can
trigger a restart of the specific pods to reload the config.

This is possible with this approach as well, in fact for something like
Kubernetes it's even easier (I believe you can mount the ConfigMap directly
without needing to pass it via env). What I'd like to make sure though:

Is it possible to layer configurations with this? For example: If I create
a `database.conf` and a `controller.conf`, is there a way to mount these in
a way that they are both read and merged by the specific component?

Cheers,
Markus

Am Do., 23. Aug. 2018 um 08:36 Uhr schrieb Chetan Mehrotra <
chetan.mehro...@gmail.com>:

> Hi Team,
>
> Currently we rely on env variables to configure OpenWhisk component like
> Controller and Invoker. For example setting
> whisk.loadbalancer.invoker-busy-threshold would need env name like
> "CONFIG_whisk_loadbalancer_invokerBusyThreshold".
>
> This mapping is at times tricky to get right and bit error prone (specially
> when setting list props). It would be better if we can directly pass in the
> config in typesafe config format [1]. This would simplify managing the
> config in version control and also reusing the default config directly.
>
> For this purpose I have created a PR #3984. For now this PR only enable use
> to this proposed approach. If it looks good we can modify the default
> ansible setup to use the new approach in a subsequent PR.
>
> Feedback welcome!
>
> Chetan Mehrotra
> [1] https://github.com/lightbend/config/blob/master/HOCON.md
> [2] https://github.com/apache/incubator-openwhisk/pull/3984
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Mi., 22. Aug. 2018 um 23:37 Uhr schrieb Tyson Norris
:

> Hi -
> >
> > When exactly is the case that a ContainerRouter should put a blocking
> > activation to a queue for stealing? Since a) it is not spawning
> containers
> > and b) it is not parsing request/response bodies, can we say this
> would
> > only happen when a ContainerRouter maxes out its incoming request
> handling?
> >
>
> That's exactly the idea! The work-stealing queue will only be used if
> the
> Router where to request landed cannot serve the demand right now. For
> example, if it maxed out the slots it has for a certain action (all
> containers are working to their full extent) it requests more
> resources and
> puts the request-token on the work-stealing queue.
>
> So to clarify, ContainerRouter "load" (which can trigger use of queue) is
> mostly (only?) based on:
> * the number of Container references
> * the number of outstanding inbound  HTTP requests, e.g. when lots of
> requests can be routed to the same container
> * the number of outstand outbound HTTP requests to remote action
> containers (assume all are remote)
> It is unclear the order of magnitude considered for "maxed out slots",
> since container refs should be simple (like ip+port, action metadata,
> activation count, and warm state), inbound connection handling is basically
> a http server, and outbound is a connection pool per action container
> (let's presume connection reuse for the moment).
> I think it will certainly need testing to determine these and to be
> configurable in any case, for each of these separate stats.. Is there
> anything else that affects the load for ContainerRouter?
>

"Overload" is determined by the availability of free slots on any container
being able to serve the current action invocation (or rather the absence
thereof). An example:
Say RouterA has 2 containers for action X. Each container has an allowed
concurrency of 10. On each of those 2 there are 10 active invocations
already running (the ContainerRouter knows this, these are open connections
to the containers). If another request comes in for X, we know we don't
have capacity for it. We request more resources and offer the work we got
for stealing.

I don't think there are tweaks needed here. The Router keeps an
"activeInvocations" number per container and compares that to the allowed
concurrency on that container. If activeInvocations == allowedConcurrency
we're out of capacity and need more.

We need a work-stealing queue here to dynamically rebalance between the
Routers since the layer above the Routers has no idea about capacity and
(at least that's my assumption) schedules randomly.


>
> That request-token will then be taken by any Router that has free
> capacity
> for that action (note: this is not simple with kafka, but might be
> simpler
> with other MQ technologies). Since new resources have been requested,
> it is
> guaranteed that one Router will eventually become free.
>
> Is "requests resources" here requesting new action containers, which it
> won't be able to process itself immediately, but should startup + warm and
> be provided to "any ContainerRouter"? This makes, sense, just want to
> clarify that "resources == containers".
>

Yes, resources == containers.


>
> >
> > If ContainerManager has enough awareness of ContainerRouters'
> states, I'm
> > not sure where using a queue would be used (for redirecting to other
> > ContainerRouters) vs ContainerManager responding with a
> ContainerRouters
> > reference (instead of an action container reference) - I'm not
> following
> > the logic of the edge case in the proposal - there is mention of
> "which
> > controller the request needs to go", but maybe this is a typo and
> should
> > say ContainerRouter?
> >
>
> Indeed that's a typo, it should say ContainerRouter.
>
> The ContainerManager only knows which Router has which Container. It
> does
> not know whether the respective Router has capacity on that container
> (the
> capacity metric is very hard to share since it's ever changing).
>
> Hence, in an edge-case where there are less Containers than Routers,
> the
> ContainerManager can hand out references to the Routers it gave
> Containers
> to the Routers that have none. (This is the edge-case described in the
> proposal).
>
> I'm not sure why in this case the ContainerManager does not just create a
> new container, instead of sending to another Router? If there is some
> intended limit on "number of containers for a particular action", that
> would be a reason, but given that the ContainerManager cannot know the
> state of the existing containers, it seems like sending to another Router
> which has the container, but may not be able to use it immediately, may
> cause failures in some cases.
>

The edge-case here is for very slow load. It's minimizing the amount of
Containers neede

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Mi., 22. Aug. 2018 um 22:49 Uhr schrieb Tyson Norris
:

> Yes, agreed this makes sense, same as Carlos is saying.
>
> Let's ignore async for now, I think that one is simpler __ - does "A
> blocking request can still be put onto the work-stealing queue" mean that
> it wouldn't always be put on the queue?
>
> If there is existing warm container capacity in the ContainerRouter
> receiving the activation, ideally it would skip the queue - right?
>

Exactly, it should skip the queue whenever possible.


>
> When exactly is the case that a ContainerRouter should put a blocking
> activation to a queue for stealing? Since a) it is not spawning containers
> and b) it is not parsing request/response bodies, can we say this would
> only happen when a ContainerRouter maxes out its incoming request handling?
>

That's exactly the idea! The work-stealing queue will only be used if the
Router where to request landed cannot serve the demand right now. For
example, if it maxed out the slots it has for a certain action (all
containers are working to their full extent) it requests more resources and
puts the request-token on the work-stealing queue.

That request-token will then be taken by any Router that has free capacity
for that action (note: this is not simple with kafka, but might be simpler
with other MQ technologies). Since new resources have been requested, it is
guaranteed that one Router will eventually become free.


>
> If ContainerManager has enough awareness of ContainerRouters' states, I'm
> not sure where using a queue would be used (for redirecting to other
> ContainerRouters) vs ContainerManager responding with a ContainerRouters
> reference (instead of an action container reference) - I'm not following
> the logic of the edge case in the proposal - there is mention of "which
> controller the request needs to go", but maybe this is a typo and should
> say ContainerRouter?
>

Indeed that's a typo, it should say ContainerRouter.

The ContainerManager only knows which Router has which Container. It does
not know whether the respective Router has capacity on that container (the
capacity metric is very hard to share since it's ever changing).

Hence, in an edge-case where there are less Containers than Routers, the
ContainerManager can hand out references to the Routers it gave Containers
to the Routers that have none. (This is the edge-case described in the
proposal).
The work-stealing queue though is used to rebalance work in case one of the
Routers get overloaded.


>
> Thanks
> Tyson
>
> On 8/21/18, 1:16 AM, "Markus Thömmes"  wrote:
>
> Hi Tyson,
>
> if we take the concerns apart as I proposed above, timeouts should only
> ever be triggered after a request is scheduled as you say, that is: As
> soon
> as it's crossing the user-container mark. With the concern separation,
> it
> is plausible that blocking invocations are never buffered anywhere,
> which
> makes a lot of sense, because you cannot persist the open HTTP
> connection
> to the client anyway.
>
> To make the distinction clear: A blocking request can still be put
> onto the
> work-stealing queue to be balanced between different ContainerRouters.
>
> A blocking request though would never be written to a persistent buffer
> that's used to be able to efficiently handle async invocations and
> backpressuring them. That buffer should be entirely separate and could
> possibly be placed outside of the execution system to make the
> distinction
> more explicit. The execution system itself would then only deal with
> request-response style invocations and asynchronous invocations are
> done by
> having a seperate queue and a consumer that creates HTTP requests to
> the
> execution system.
>
> Cheers,
> Markus
>
> Am Mo., 20. Aug. 2018 um 23:30 Uhr schrieb Tyson Norris
> :
>
> > Thanks for summarizing Markus.
> >
> > Yes this is confusing in context of current system, which stores in
> kafka,
> > but not to indefinitely wait, since timeout begins immediately
> > So, I think the problem of buffering/queueing is: when does the
> timeout
> > begin? If not everything is buffered the same, their timeout should
> not
> > begin until processing begins.
> >
> > Maybe it would make sense to:
> > * always buffer (indefinitely) to queue for async, never for sync
> > * timeout for async not started till read from queue - which may be
> > delayed from time of trigger or http request
> > * this should also come with some system monitoring to indicate the

Re: Guide to conduct performance benchmark against OpenWhisk

[Markus Thömmes]

Hi Dominic,

it seems like you haven't published your writeup? The page seems empty.

Cheers,
Markus

Am Di., 21. Aug. 2018 um 12:44 Uhr schrieb Dominic Kim :

> Dear whiskers.
>
> I just wrote the guide to perform a throughput test against Openwhisk in
> cwiki.
> https://cwiki.apache.org/confluence/display/OPENWHISK/Performance+Benchmark
>
> You can easily perform throughput tests with multiple actions and
> namespaces with it.
> I hope this would be helpful for those who want to perform stress tests
> with various scenarios.
>
> Please ping me if you have any questions on it.
>
> Thanks
> Regards
> Dominic.
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

if we take the concerns apart as I proposed above, timeouts should only
ever be triggered after a request is scheduled as you say, that is: As soon
as it's crossing the user-container mark. With the concern separation, it
is plausible that blocking invocations are never buffered anywhere, which
makes a lot of sense, because you cannot persist the open HTTP connection
to the client anyway.

To make the distinction clear: A blocking request can still be put onto the
work-stealing queue to be balanced between different ContainerRouters.

A blocking request though would never be written to a persistent buffer
that's used to be able to efficiently handle async invocations and
backpressuring them. That buffer should be entirely separate and could
possibly be placed outside of the execution system to make the distinction
more explicit. The execution system itself would then only deal with
request-response style invocations and asynchronous invocations are done by
having a seperate queue and a consumer that creates HTTP requests to the
execution system.

Cheers,
Markus

Am Mo., 20. Aug. 2018 um 23:30 Uhr schrieb Tyson Norris
:

> Thanks for summarizing Markus.
>
> Yes this is confusing in context of current system, which stores in kafka,
> but not to indefinitely wait, since timeout begins immediately
> So, I think the problem of buffering/queueing is: when does the timeout
> begin? If not everything is buffered the same, their timeout should not
> begin until processing begins.
>
> Maybe it would make sense to:
> * always buffer (indefinitely) to queue for async, never for sync
> * timeout for async not started till read from queue - which may be
> delayed from time of trigger or http request
> * this should also come with some system monitoring to indicate the queue
> processing is not keeping up with some configurable max delay threshold ("I
> can’t tolerate delays of > 5 minutes", etc)
> * ContainerRouters can only pull from async queue when
> * increasing the number of pending activations won’t exceed some
> threshold (prevent excessive load of async on ContainerRouters)
> * ContainerManager is not overloaded (can still create containers,
> or has some configurable way to indicate the cluster is healthy enough to
> cope with extra processing)
>
> We could of course make this configurable so that operators can choose to:
> * treat async/sync activations the same for sync/async (the overloaded
> system fails when either ContainerManager or ContainerRouters are max
> capacity)
> * treat async/sync with preference for:
> * sync - where async is buffered for unknown period before
> processing, incoming sync traffic (or lack of)
> * async - where sync is sent to the queue, to be processed in
> order of receipt interleaved with async traffic (similar to today, I think)
>
> I think the impact here (aside from technical) is the timing difference if
> we introduce latency in side affects based on the activation being sync vs
> async.
>
> I’m also not sure prioritizing message processing between sync/async
> internally in ContainerRouter is better than just have some dedicated
> ContainerRouters that receive all async activations, and others that
> receive all sync activations, but the end result is the same, I think.
>
>
> > On Aug 19, 2018, at 4:29 AM, Markus Thömmes 
> wrote:
> >
> > Hi Tyson, Carlos,
> >
> > FWIW I should change that to no longer say "Kafka" but "buffer" or
> "message
> > queue".
> >
> > I see two use-cases for a queue here:
> > 1. What you two are alluding to: Buffering asynchronous requests because
> of
> > a different notion of "latency sensitivity" if the system is in an
> overload
> > scenario.
> > 2. As a work-stealing type balancing layer between the ContainerRouters.
> If
> > we assume round-robin/least-connected (essentially random) scheduling
> > between ContainerRouters, we will get load discrepancies between them. To
> > smoothen those out, a ContainerRouter can put the work on a queue to be
> > stolen by a Router that actually has space for that work (for example:
> > Router1 requests a new container, puts the work on the queue while it
> waits
> > for that container, Router2 already has a free container and executes the
> > action by stealing it from the queue). This does has the added complexity
> > of breaking a streaming communication between User and Container (to
> > support essentially unbounded payloads). A nasty wrinkle that might
> render
> > this design alternative invalid! We could come up with something smarter
> > here, i.e. only putting a reference to the work on the queue and the
> 

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Di., 21. Aug. 2018 um 00:20 Uhr schrieb Tyson Norris
:

>
>
> On Aug 19, 2018, at 3:59 AM, Markus Thömmes  <mailto:markusthoem...@apache.org>> wrote:
>
> Hi Tyson,
>
> Am Fr., 17. Aug. 2018 um 23:45 Uhr schrieb Tyson Norris
> mailto:tnor...@adobe.com.invalid>>:
>
>
> If the failover of the singleton is too long (I think it will be based on
> cluster size, oldest node becomes the singleton host iirc), I think we need
> to consider how containers can launch in the meantime. A first step might
> be to test out the singleton behavior in the cluster of various sizes.
>
>
> I agree this bit of design is crucial, a few thoughts:
> Pre-warm wouldn't help here, the ContainerRouters only know warm
> containers. Pre-warming is managed by the ContainerManager.
>
> Ah right
>
>
> Considering a fail-over scenario: We could consider sharing the state via
> EventSourcing. That is: All state lives inside of frequently snapshotted
> events and thus can be shared between multiple instances of the
> ContainerManager seamlessly. Alternatively, we could also think about only
> working on persisted state. That way, a cold-standby model could fly. We
> should make sure that the state is not "slightly stale" but rather both
> instances see the same state at any point in time. I believe on that
> cold-path of generating new containers, we can live with the extra-latency
> of persisting what we're doing as the path will still be dominated by the
> container creation latency.
>
> Wasn’t clear if you mean not using ClusterSingleton? To be clear in
> ClusterSingleton case there are 2 issues:
> - time it takes for akka ClusterSingletonManager to realize it needs to
> start a new actor
> - time it takes for the new actor to assume a usable state
>
> EventSourcing (or ext persistence) may help with the latter, but we will
> need to be sure the former is tolerable to start with.
> Here is an example test from akka source that may be useful (multi-jvm,
> but all local):
>
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fakka%2Fakka%2Fblob%2F009214ae07708e8144a279e71d06c4a504907e31%2Fakka-cluster-tools%2Fsrc%2Fmulti-jvm%2Fscala%2Fakka%2Fcluster%2Fsingleton%2FClusterSingletonManagerChaosSpec.scala&data=02%7C01%7Ctnorris%40adobe.com%7C63c6bb3a36724f38cc9d08d605c2ddee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636702732034251656&sdata=omVsIo%2FoD8weG4Zy%2BGX2A53ATRmylUxYCbqknu4MoeM%3D&reserved=0
>
> Some things to consider, that I don’t know details of:
> - will the size of cluster affect the singleton behavior in case of
> failure? (I think so, but not sure, and what extent); in the simple test
> above it takes ~6s for the replacement singleton to begin startup, but if
> we have 100s of nodes, I’m not sure how much time it will take. (I don’t
> think this should be hard to test, but I haven’t done it)
> - in case of hard crash, what is the singleton behavior? In graceful jvm
> termination, I know the cluster behavior is good, but there is always this
> question about how downing nodes will be handled. If this critical piece of
> the system relies on akka cluster functionality, we will need to make sure
> that the singleton can be reconstituted, both in case of graceful
> termination (restart/deployment events) and non-graceful termination (hard
> vm crash, hard container crash) . This is ignoring more complicated cases
> of extended network partitions, which will also have bad affects on many of
> the downstream systems.
>
>
> I don't think we need to be eager to consider akka-cluster to be set in
> stone here. The singleton in my mind doesn't need to be clustered at all.
> Say we have a fully shared state through persistence or event-sourcing and
> a hot-standby model, couldn't we implement the fallback through routing in
> front of the active/passive ContainerManager pair? Once one goes
> unreachable, fall back to the other.
>
>
>
> Yeah I would rather see the hot standby and deal with persistence. I don’t
> think akka clustersingleton is going to be fast enough in a high volume
> scenario.
> Either routing in front or ContainerRouters who observe the active
> (leader) status, we just have to determine that the status change is
> tolerably fast.
>
>
>
>
>
> Handover time as you say is crucial, but I'd say as it only impacts
> container creation, we could live with, let's say, 5 seconds of
> failover-downtime on this path? What's your experience been on singleton
> failover? How long did it take?
>
>
> Seconds in the simplest case, so I think we need to test it in a scaled
> case (100s of cluster nodes), as well as the hard crash case (

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Am So., 19. Aug. 2018 um 18:59 Uhr schrieb TzuChiao Yeh <
su3g4284zo...@gmail.com>:

> On Sun, Aug 19, 2018 at 7:13 PM Markus Thömmes 
> wrote:
>
> > Hi Tzu-Chiao,
> >
> > Am Sa., 18. Aug. 2018 um 06:56 Uhr schrieb TzuChiao Yeh <
> > su3g4284zo...@gmail.com>:
> >
> > > Hi Markus,
> > >
> > > Nice thoughts on separating logics in this revision! I'm not sure this
> > > question has already been clarified, sorry if duplicate.
> > >
> > > Same question on cluster singleton:
> > >
> > > I think there will be two possibilities on container deletion: 1.
> > > ContainerRouter removes it (when error or idle-state) 2.
> ContainerManager
> > > decides to remove it (i.e. clear space for new creation).
> > >
> > > For case 2, how do we ensure the safe deletion in ContainerManager?
> > > Consider if there's still a similar model on busy/free/prewarmed pool,
> it
> > > might require additional states related to containers from busy to free
> > > state, then we can safely remove it or reject if nothing found (system
> > > overloaded).
> > >
> > > By paused state or other states/message? There might be some trade-offs
> > on
> > > granularity (time-slice in scheduling) and performance bottleneck on
> > > ClusterSingleton.
> > >
>
> I'm not sure if I quite got the point, but here's an attempt on an
> > explanation:
> >
> > Yes, Container removal in case 2 is triggered from the ContainerManager.
> To
> > be able to safely remove it, it requests all ContainerRouters owning that
> > container to stop serving it and hand it back. Once it's been handed
> back,
> > the ContainerManager can safely delete it. The contract should also say:
> A
> > container must be handed back in unpaused state, so it can be deleted
> > safely. Since the ContainerRouters handle pause/unpause, they'll need to
> > stop serving the container, unpause it, remove it from their state and
> > acknowledge to the ContainerManager that they handed it back.
> >
>
> Thank you, it's clear to me.
>
>
> > There is an open question on when to consider a system to be in overflow
> > state, or rather: How to handle the edge-situation. If you cannot
> generate
> > more containers, we need to decide whether we remove another container
> (the
> > case you're describing) or if we call it quits and say "503, overloaded,
> go
> > away for now". The logic deciding this is up for discussion as well. The
> > heuristic could take into account how many resources in the whole system
> > you already own, how many resources do others own and if we want to
> decide
> > to share those fairly or not-fairly. Note that this is also very much
> > related to being able to scale the resources up in themselves (to be able
> > to generate new containers). If we assume a bounded system though, yes,
> > we'll need to find a strategy on how to handle this case. I believe with
> > the state the ContainerManager has, it can provide a more eloquent answer
> > to that question than what we can do today (nothing really, we just keep
> on
> > churning through containers).
> >
>
> I agree. An additional problem is in the case of burst requests,
> ContainerManager will "over-estimate" containers allocation, whether
> work-stealing between ContainerRouters has been enabled or not. For bounded
> system, we have better carefully handle these to avoid frequently
> creation/deletion. I'm wondering if sharing message queue between
> ContainerManager (since it's not a critical path) or any mechanism for
> checking queue size (i.e. checking kafka lags) can possibly eliminate
> this?  However, this may be only happened in short running tasks and
> throttling already being helpful.
>

Are you saying: It will over-estimate container allocation because it will
create a container for each request as they arrive if there are no
containers around currently and the actual number of containers needed
might be lower for very short running use-cases where requests arrive in
short bursts?

If so: I agree, I don't see how any system can possibly solve this without
taking the estimated runtime of each request into account though. Can you
elaborate on how your thoughts on checking queue-size etc?


>
>
> > Does that answer the question?
>
>
> > >
> > > Thanks!
> > >
> > > Tzu-Chiao
> > >
> > > On Sat, Aug 18, 2018 at 5:55 AM Tyson Norris  >
> > > wrote:
> > >
> > &g

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

I believe we should keep this as general as long as possible. We should
define the characteristics we need for each path rather than deciding on a
certain technology early on.

Am So., 19. Aug. 2018 um 16:07 Uhr schrieb Dascalita Dragos <
ddrag...@gmail.com>:

> “... FWIW I should change that to no longer say "Kafka" but "buffer" or
> "message
> queue"...”
> +1. One idea could be to use Akka Streams and let the OW operator make a
> decision on using Kafka with Akka Streams, or not [1]. This would make OW
> deployment easier, Kafka becoming optional, while opening the door for
> other connectors like AWS Kinesis, Azure Event Hub, and others (see the
> link at [1] for a more complete list of connectors )
>
> [1] - https://developer.lightbend.com/docs/alpakka/current/
> On Sun, Aug 19, 2018 at 7:30 AM Markus Thömmes 
> wrote:
>
> > Hi Tyson, Carlos,
> >
> > FWIW I should change that to no longer say "Kafka" but "buffer" or
> "message
> > queue".
> >
> > I see two use-cases for a queue here:
> > 1. What you two are alluding to: Buffering asynchronous requests because
> of
> > a different notion of "latency sensitivity" if the system is in an
> overload
> > scenario.
> > 2. As a work-stealing type balancing layer between the ContainerRouters.
> If
> > we assume round-robin/least-connected (essentially random) scheduling
> > between ContainerRouters, we will get load discrepancies between them. To
> > smoothen those out, a ContainerRouter can put the work on a queue to be
> > stolen by a Router that actually has space for that work (for example:
> > Router1 requests a new container, puts the work on the queue while it
> waits
> > for that container, Router2 already has a free container and executes the
> > action by stealing it from the queue). This does has the added complexity
> > of breaking a streaming communication between User and Container (to
> > support essentially unbounded payloads). A nasty wrinkle that might
> render
> > this design alternative invalid! We could come up with something smarter
> > here, i.e. only putting a reference to the work on the queue and the
> > stealer connects to the initial owner directly which then streams the
> > payload through to the stealer, rather than persisting it somewhere.
> >
> > It is important to note, that in this design, blocking invokes could
> > potentially gain the ability to have unbounded entities, where
> > trigger/non-blocking invokes might need to be subject to a bound here to
> be
> > able to support eventual execution efficiently.
> >
> > Personally, I'm much more torn to the work-stealing type case. It
> implies a
> > wholy different notion of using the queue though and doesn't have much to
> > do with the way we use it today, which might be confusing. It could also
> > well be the case, that work-stealing type algorithms are easier to back
> on
> > a proper MQ vs. trying to make it work on Kafka.
> >
> > It might also be important to note that those two use-cases might require
> > different technologies (buffering vs. queue-backend for work-stealing)
> and
> > could well be seperated in the design as well. For instance, buffering
> > triggers fires etc. does not necessarily need to be done on the execution
> > layer but could instead be pushed to another layer. Having the notion of
> > "async" vs "sync" in the execution layer could be benefitial for
> > loadbalancing itself though. Something worth exploring imho.
> >
> > Sorry for the wall of text, I hope this clarifies things!
> >
> > Cheers,
> > Markus
> >
> > Am Sa., 18. Aug. 2018 um 02:36 Uhr schrieb Carlos Santana <
> > csantan...@gmail.com>:
> >
> > > triggers get responded right away (202) with an activation is and then
> > > sent to the queue to be processed async same as async action invokes.
> > >
> > > I think we would keep same contract as today for this type of
> activations
> > > that are eventually process different from blocking invokes including
> we
> > > Actions were the http client hold a connection waiting for the result
> > back.
> > >
> > > - Carlos Santana
> > > @csantanapr
> > >
> > > > On Aug 17, 2018, at 6:14 PM, Tyson Norris  >
> > > wrote:
> > > >
> > > > Hi -
> > > > Separate thread regarding the proposal: what is considered for
> routing
> > > activations as overload and destined for kafka?
> > > >
> > > > In general, if kafka is not on the blocking activation path, why
> would
> > > it be used at all, if the timeouts and processing expectations of
> > blocking
> > > and non-blocking are the same?
> > > >
> > > > One case I can imagine: triggers + non-blocking invokes, but only in
> > the
> > > case where those have some different timeout characteristics. e.g. if a
> > > trigger fires an action, is there any case where the activation should
> be
> > > buffered to kafka if it will timeout same as a blocking activation?
> > > >
> > > > Sorry if I’m missing something obvious.
> > > >
> > > > Thanks
> > > > Tyson
> > > >
> > > >
> > >
> >
>

Re: Concurrency PR

[Markus Thömmes]

Hi Tyson, thanks for pushing forward on this! I'll try to get a review in
on it soon.

Am Fr., 17. Aug. 2018 um 19:04 Uhr schrieb Tyson Norris
:

> Hi -
> I have been noodling with a few tests and the akka http client and gotten
> the concurrency PR [1] to a good place, I think, so if anyone can help
> review that would be appreciated.
>
> A couple of notes:
> - akka http client has some different notion of connection reuse than the
> apache client, to address this I created a separate PR [2] which, instead
> of dissuading connection reuse, simple destroys the client (and connection
> pool) when the container is paused. (This change is not reflected in 2795
> FWIW). AFAIK the connection reuse issue only comes up with container
> pauses, so I wanted to address this where it is relevant, and not impose
> additional performance costs for concurrency cases. This client is still
> not enabled by default.
> - There was mention in the comments (for 2795) about need to handle a case
> where a container doesn’t support concurrency, but the action dev has
> enabled it at the action - this PR does NOT deal with that.
>
> To summarize, enabling concurrency requires:
> - all actions may signal that they support concurrency, so all images that
> might be used would need to support concurrency, if concurrency is enabled
> in your deployment
> - log collection must be handled outside of invoker (since invoker does
> not deal with interleaved log parsing)
> - wsk cli will require changes to allow action devs to set the concurrency
> limits on actions (current PR only exposes the OW api for doing this); I
> have a PR queued up for that [3]. (Will need another PR for the cli once
> the client-go lib is updated)
>
> To better handle the case of images that don’t support concurrency, or
> don’t support log collection from invoker, I would suggest we change the
> container protocol to allow containers to broadcast their support either
> via the /init endpoint, or via a new /info endpoint. This of course would
> not give feedback until an action is executed (as opposed to when action is
> created), but I think this is ok. I will work on a separate PR for this,
> but want to mention some thoughts here about possible approaches to address
> these known concerns.
>

Why not make this part of the runtimes manifest? Handling this as late as
actually invoking the action feels kinda weird if we can just as well know
ahead of time, that creating an action with a concurrency > 1 will not work
and should therefore forbid creation at all. Any strong reason not to
encode that information into the runtimes manifest?


>
> Thanks
> Tyson
>
>
> [1] https://github.com/apache/incubator-openwhisk/pull/2795
> [2] https://github.com/apache/incubator-openwhisk/pull/3976
> [3] https://github.com/apache/incubator-openwhisk-client-go/pull/94
>
>

Re: Kafka and Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson, Carlos,

FWIW I should change that to no longer say "Kafka" but "buffer" or "message
queue".

I see two use-cases for a queue here:
1. What you two are alluding to: Buffering asynchronous requests because of
a different notion of "latency sensitivity" if the system is in an overload
scenario.
2. As a work-stealing type balancing layer between the ContainerRouters. If
we assume round-robin/least-connected (essentially random) scheduling
between ContainerRouters, we will get load discrepancies between them. To
smoothen those out, a ContainerRouter can put the work on a queue to be
stolen by a Router that actually has space for that work (for example:
Router1 requests a new container, puts the work on the queue while it waits
for that container, Router2 already has a free container and executes the
action by stealing it from the queue). This does has the added complexity
of breaking a streaming communication between User and Container (to
support essentially unbounded payloads). A nasty wrinkle that might render
this design alternative invalid! We could come up with something smarter
here, i.e. only putting a reference to the work on the queue and the
stealer connects to the initial owner directly which then streams the
payload through to the stealer, rather than persisting it somewhere.

It is important to note, that in this design, blocking invokes could
potentially gain the ability to have unbounded entities, where
trigger/non-blocking invokes might need to be subject to a bound here to be
able to support eventual execution efficiently.

Personally, I'm much more torn to the work-stealing type case. It implies a
wholy different notion of using the queue though and doesn't have much to
do with the way we use it today, which might be confusing. It could also
well be the case, that work-stealing type algorithms are easier to back on
a proper MQ vs. trying to make it work on Kafka.

It might also be important to note that those two use-cases might require
different technologies (buffering vs. queue-backend for work-stealing) and
could well be seperated in the design as well. For instance, buffering
triggers fires etc. does not necessarily need to be done on the execution
layer but could instead be pushed to another layer. Having the notion of
"async" vs "sync" in the execution layer could be benefitial for
loadbalancing itself though. Something worth exploring imho.

Sorry for the wall of text, I hope this clarifies things!

Cheers,
Markus

Am Sa., 18. Aug. 2018 um 02:36 Uhr schrieb Carlos Santana <
csantan...@gmail.com>:

> triggers get responded right away (202) with an activation is and then
> sent to the queue to be processed async same as async action invokes.
>
> I think we would keep same contract as today for this type of activations
> that are eventually process different from blocking invokes including we
> Actions were the http client hold a connection waiting for the result back.
>
> - Carlos Santana
> @csantanapr
>
> > On Aug 17, 2018, at 6:14 PM, Tyson Norris 
> wrote:
> >
> > Hi -
> > Separate thread regarding the proposal: what is considered for routing
> activations as overload and destined for kafka?
> >
> > In general, if kafka is not on the blocking activation path, why would
> it be used at all, if the timeouts and processing expectations of blocking
> and non-blocking are the same?
> >
> > One case I can imagine: triggers + non-blocking invokes, but only in the
> case where those have some different timeout characteristics. e.g. if a
> trigger fires an action, is there any case where the activation should be
> buffered to kafka if it will timeout same as a blocking activation?
> >
> > Sorry if I’m missing something obvious.
> >
> > Thanks
> > Tyson
> >
> >
>

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tzu-Chiao,

Am Sa., 18. Aug. 2018 um 06:56 Uhr schrieb TzuChiao Yeh <
su3g4284zo...@gmail.com>:

> Hi Markus,
>
> Nice thoughts on separating logics in this revision! I'm not sure this
> question has already been clarified, sorry if duplicate.
>
> Same question on cluster singleton:
>
> I think there will be two possibilities on container deletion: 1.
> ContainerRouter removes it (when error or idle-state) 2. ContainerManager
> decides to remove it (i.e. clear space for new creation).
>
> For case 2, how do we ensure the safe deletion in ContainerManager?
> Consider if there's still a similar model on busy/free/prewarmed pool, it
> might require additional states related to containers from busy to free
> state, then we can safely remove it or reject if nothing found (system
> overloaded).
>
> By paused state or other states/message? There might be some trade-offs on
> granularity (time-slice in scheduling) and performance bottleneck on
> ClusterSingleton.
>

I'm not sure if I quite got the point, but here's an attempt on an
explanation:

Yes, Container removal in case 2 is triggered from the ContainerManager. To
be able to safely remove it, it requests all ContainerRouters owning that
container to stop serving it and hand it back. Once it's been handed back,
the ContainerManager can safely delete it. The contract should also say: A
container must be handed back in unpaused state, so it can be deleted
safely. Since the ContainerRouters handle pause/unpause, they'll need to
stop serving the container, unpause it, remove it from their state and
acknowledge to the ContainerManager that they handed it back.

There is an open question on when to consider a system to be in overflow
state, or rather: How to handle the edge-situation. If you cannot generate
more containers, we need to decide whether we remove another container (the
case you're describing) or if we call it quits and say "503, overloaded, go
away for now". The logic deciding this is up for discussion as well. The
heuristic could take into account how many resources in the whole system
you already own, how many resources do others own and if we want to decide
to share those fairly or not-fairly. Note that this is also very much
related to being able to scale the resources up in themselves (to be able
to generate new containers). If we assume a bounded system though, yes,
we'll need to find a strategy on how to handle this case. I believe with
the state the ContainerManager has, it can provide a more eloquent answer
to that question than what we can do today (nothing really, we just keep on
churning through containers).

Does that answer the question?


>
> Thanks!
>
> Tzu-Chiao
>
> On Sat, Aug 18, 2018 at 5:55 AM Tyson Norris 
> wrote:
>
> > Ugh my reply formatting got removed!!! Trying this again with some >>
> >
> > On Aug 17, 2018, at 2:45 PM, Tyson Norris  > > wrote:
> >
> >
> > If the failover of the singleton is too long (I think it will be based on
> > cluster size, oldest node becomes the singleton host iirc), I think we
> need
> > to consider how containers can launch in the meantime. A first step might
> > be to test out the singleton behavior in the cluster of various sizes.
> >
> >
> > I agree this bit of design is crucial, a few thoughts:
> > Pre-warm wouldn't help here, the ContainerRouters only know warm
> > containers. Pre-warming is managed by the ContainerManager.
> >
> >
> > >> Ah right
> >
> >
> >
> > Considering a fail-over scenario: We could consider sharing the state via
> > EventSourcing. That is: All state lives inside of frequently snapshotted
> > events and thus can be shared between multiple instances of the
> > ContainerManager seamlessly. Alternatively, we could also think about
> only
> > working on persisted state. That way, a cold-standby model could fly. We
> > should make sure that the state is not "slightly stale" but rather both
> > instances see the same state at any point in time. I believe on that
> > cold-path of generating new containers, we can live with the
> extra-latency
> > of persisting what we're doing as the path will still be dominated by the
> > container creation latency.
> >
> >
> >
> > >> Wasn’t clear if you mean not using ClusterSingleton? To be clear in
> > ClusterSingleton case there are 2 issues:
> > - time it takes for akka ClusterSingletonManager to realize it needs to
> > start a new actor
> > - time it takes for the new actor to assume a usable state
> >
> > EventSourcing (or ext persistence) may help with the latter, but we will
> > need to be sure the former is tolerable to start with.
> > Here is an example test from akka source that may be useful (multi-jvm,
> > but all local):
> >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fakka%2Fakka%2Fblob%2F009214ae07708e8144a279e71d06c4a504907e31%2Fakka-cluster-tools%2Fsrc%2Fmulti-jvm%2Fscala%2Fakka%2Fcluster%2Fsingleton%2FClusterSingletonManagerChaosSpec.scala&data=02%7

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Dave,
Am Sa., 18. Aug. 2018 um 17:01 Uhr schrieb David P Grove :

>
> [ Discussion about cluster singleton or not for the ContainerManager]
>
> fwiw, I believe for Kubernetes we do not need to attempt to deal with fault
> tolerance for the ContainerManager state ourselves.  We can use labels to
> replicate all the persistent metadata for a container (prewarm or not, the
> ContainerRouter it is assigned to) in the Kube objects representing the
> pods in Kube's etcd metadata server.  If we need to restart a
> ContainerManager, the new instance can come up "instantly" and start
> servicing requests while recovering the state of the previous instance via
> querries against etcd to discover the pre-existing containers it owned.
>

Note that there is also state it has about ContainerRouters (how many are
there and which own which containers). We could make that queryable as
well, so as soon as a fallback happens, the fallback component queries the
state of all routers to get into consistent state.

I agree we should replicate as little state as possible and in the
Kubernetes case, we already have state about containers through pods and
their labels.


>
> We'll need to validate the performance of this is acceptable (should be,
> since it is just some asynchronous labeling operations when (a) the
> container is created and (b) on the initial transition from stemcell to
> warm), but it is going to be pretty simple to implement and makes good
> usage of the underlying platform's capabilities.
>

Agreed, good point.


>
> --dave
>

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

Am Fr., 17. Aug. 2018 um 23:45 Uhr schrieb Tyson Norris
:

>
> If the failover of the singleton is too long (I think it will be based on
> cluster size, oldest node becomes the singleton host iirc), I think we need
> to consider how containers can launch in the meantime. A first step might
> be to test out the singleton behavior in the cluster of various sizes.
>
>
> I agree this bit of design is crucial, a few thoughts:
> Pre-warm wouldn't help here, the ContainerRouters only know warm
> containers. Pre-warming is managed by the ContainerManager.
>
> Ah right
>
>
> Considering a fail-over scenario: We could consider sharing the state via
> EventSourcing. That is: All state lives inside of frequently snapshotted
> events and thus can be shared between multiple instances of the
> ContainerManager seamlessly. Alternatively, we could also think about only
> working on persisted state. That way, a cold-standby model could fly. We
> should make sure that the state is not "slightly stale" but rather both
> instances see the same state at any point in time. I believe on that
> cold-path of generating new containers, we can live with the extra-latency
> of persisting what we're doing as the path will still be dominated by the
> container creation latency.
>
> Wasn’t clear if you mean not using ClusterSingleton? To be clear in
> ClusterSingleton case there are 2 issues:
> - time it takes for akka ClusterSingletonManager to realize it needs to
> start a new actor
> - time it takes for the new actor to assume a usable state
>
> EventSourcing (or ext persistence) may help with the latter, but we will
> need to be sure the former is tolerable to start with.
> Here is an example test from akka source that may be useful (multi-jvm,
> but all local):
>
> https://github.com/akka/akka/blob/009214ae07708e8144a279e71d06c4a504907e31/akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerChaosSpec.scala
>
> Some things to consider, that I don’t know details of:
> - will the size of cluster affect the singleton behavior in case of
> failure? (I think so, but not sure, and what extent); in the simple test
> above it takes ~6s for the replacement singleton to begin startup, but if
> we have 100s of nodes, I’m not sure how much time it will take. (I don’t
> think this should be hard to test, but I haven’t done it)
> - in case of hard crash, what is the singleton behavior? In graceful jvm
> termination, I know the cluster behavior is good, but there is always this
> question about how downing nodes will be handled. If this critical piece of
> the system relies on akka cluster functionality, we will need to make sure
> that the singleton can be reconstituted, both in case of graceful
> termination (restart/deployment events) and non-graceful termination (hard
> vm crash, hard container crash) . This is ignoring more complicated cases
> of extended network partitions, which will also have bad affects on many of
> the downstream systems.
>

I don't think we need to be eager to consider akka-cluster to be set in
stone here. The singleton in my mind doesn't need to be clustered at all.
Say we have a fully shared state through persistence or event-sourcing and
a hot-standby model, couldn't we implement the fallback through routing in
front of the active/passive ContainerManager pair? Once one goes
unreachable, fall back to the other.


>
>
>
> Handover time as you say is crucial, but I'd say as it only impacts
> container creation, we could live with, let's say, 5 seconds of
> failover-downtime on this path? What's your experience been on singleton
> failover? How long did it take?
>
>
> Seconds in the simplest case, so I think we need to test it in a scaled
> case (100s of cluster nodes), as well as the hard crash case (where not
> downing the node may affect the cluster state).
>
>
>
>
> On Aug 16, 2018, at 11:01 AM, Tyson Norris  >
> wrote:
>
> A couple comments on singleton:
> - use of cluster singleton will introduce a new single point of failure
> - from time of singleton node failure, to single resurrection on a
> different instance, will be an outage from the point of view of any
> ContainerRouter that does not already have a warm+free container to service
> an activation
> - resurrecting the singleton will require transferring or rebuilding the
> state when recovery occurs - in my experience this was tricky, and requires
> replicating the data (which will be slightly stale, but better than
> rebuilding from nothing); I don’t recall the handover delay (to transfer
> singleton to a new akka cluster node) when I tried last, but I think it was
> not as fast as I hoped it would be.
>
> I don’t have a great suggestion for the singleton failure case, but
> would like to consider this carefully, and discuss the ramifications (which
> may or may not be tolerable) before pursuing this particular aspect of the
> design.
>
>
> On prioritization:
> - if concurrency is 

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Tyson,

thanks for the great input!

Am Do., 16. Aug. 2018 um 23:14 Uhr schrieb Tyson Norris
:

> Thinking more about the singleton aspect, I guess this is mostly an issue
> for blackbox containers, where manifest/managed containers will mitigate at
> least some of the singleton failure delays by prewarm/stemcell containers.
>
> So in the case of singleton failure, impacts would be:
> - managed containers once prewarms are exhausted (may be improved by being
> more intelligent about prewarm pool sizing based on load etc)
> - managed containers that don’t match any prewarms (similar - if prewarm
> pool is dynamically configured based on load, this is less problem)
> - blackbox containers (no help)
>
> If the failover of the singleton is too long (I think it will be based on
> cluster size, oldest node becomes the singleton host iirc), I think we need
> to consider how containers can launch in the meantime. A first step might
> be to test out the singleton behavior in the cluster of various sizes.
>

I agree this bit of design is crucial, a few thoughts:
Pre-warm wouldn't help here, the ContainerRouters only know warm
containers. Pre-warming is managed by the ContainerManager.

Considering a fail-over scenario: We could consider sharing the state via
EventSourcing. That is: All state lives inside of frequently snapshotted
events and thus can be shared between multiple instances of the
ContainerManager seamlessly. Alternatively, we could also think about only
working on persisted state. That way, a cold-standby model could fly. We
should make sure that the state is not "slightly stale" but rather both
instances see the same state at any point in time. I believe on that
cold-path of generating new containers, we can live with the extra-latency
of persisting what we're doing as the path will still be dominated by the
container creation latency.

Handover time as you say is crucial, but I'd say as it only impacts
container creation, we could live with, let's say, 5 seconds of
failover-downtime on this path? What's your experience been on singleton
failover? How long did it take?


>
> > On Aug 16, 2018, at 11:01 AM, Tyson Norris 
> wrote:
> >
> > A couple comments on singleton:
> > - use of cluster singleton will introduce a new single point of failure
> - from time of singleton node failure, to single resurrection on a
> different instance, will be an outage from the point of view of any
> ContainerRouter that does not already have a warm+free container to service
> an activation
> > - resurrecting the singleton will require transferring or rebuilding the
> state when recovery occurs - in my experience this was tricky, and requires
> replicating the data (which will be slightly stale, but better than
> rebuilding from nothing); I don’t recall the handover delay (to transfer
> singleton to a new akka cluster node) when I tried last, but I think it was
> not as fast as I hoped it would be.
> >
> > I don’t have a great suggestion for the singleton failure case, but
> would like to consider this carefully, and discuss the ramifications (which
> may or may not be tolerable) before pursuing this particular aspect of the
> design.
> >
> >
> > On prioritization:
> > - if concurrency is enabled for an action, this is another
> prioritization aspect, of sorts - if the action supports concurrency, there
> is no reason (except for destruction coordination…) that it cannot be
> shared across shards. This could be added later, but may be worth
> considering since there is a general reuse problem where a series of
> activations that arrives at different ContainerRouters will create a new
> container in each, while they could be reused (and avoid creating new
> containers) if concurrency is tolerated in that container. This would only
> (ha ha) require changing how container destroy works, where it cannot be
> destroyed until the last ContainerRouter is done with it. And if container
> destruction is coordinated in this way to increase reuse, it would also be
> good to coordinate construction (don’t concurrently construct the same
> container for multiple containerRouters IFF a single container would enable
> concurrent activations once it is created). I’m not sure if others are
> desiring this level of container reuse, but if so, it would be worth
> considering these aspects (sharding/isolation vs sharing/coordination) as
> part of any redesign.
>

Yes, I can see where you're heading here. I think this can be generalized:

Assume intra-container concurrency C and number of ContainerRouters R.
If C > R: Shard the "slots" on this container evenly across R. The
container can only be destroyed after you receive R acknowledgements of
doing so.
If C < R: Hand out 1 slot to C Routers, point the remaining Routers to the
ones that got slots.

Concurrent creation: Batch creation requests while one container is being
created. Say you received a request for a new container that has C slots.
If there are more requests for that containe

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Michael,

loosing/adding a shard is essentially reconciled by the ContainerManager.
As it keeps track of all the ContainerRouters in the system, it can also
observe one going down/crashing or one coming up and joining the "cluster".

If one Router leaves the cluster, the ContainerManager knows which
containers where "managed" by that router and redistributes them across the
Routers left in the system.
If one Router joins the cluster, we can try to rebalance containers to take
load off existing ones. Precise algorithm to be defined but the primitives
should be in place to be able to do that.

Does that answer the question?

Cheers,
Markus

Am Mi., 15. Aug. 2018 um 16:18 Uhr schrieb Michael Marth
:

> Markus,
>
> I agree with your preference of making the state sharded instead of
> distributed. (not only for the scalability reasons you quote but also for
> operational concerns).
> What are your thoughts about losing a shard (planned or crashed) or adding
> a shard?
>
> Michael
>
>
> On 15.08.18, 09:58, "Markus Thömmes"  wrote:
>
> Hi Dragos,
>
> thanks for your questions, good discussion :)
>
> Am Di., 14. Aug. 2018 um 23:42 Uhr schrieb Dragos Dascalita Haut
> :
>
> > Markus, I appreciate the enhancements you mentioned in the wiki, and
> I'm
> > very much inline with the ideas you brought in there.
> >
> >
> >
> > "...having the ContainerManager be a cluster singleton..."
> >
> > I was just in process to reply with the same idea :)
> >
> > In addition, I was thinking we can leverage Akka Distributed Data
> [1] to
> > keep all ContainerRouter actors eventually consistent. When creating
> a new
> > container, the ContainerManager can write with a consistency
> "WriteAll"; it
> > would be a little slower but it would improve consistency.
> >
>
> I think we need to quantify "a little slower". Note that "WriteAll"
> becomes
> slower and slower the more actors you add to the cluster. Scalability
> is at
> question then.
>
> Of course scalability is also at question if we make the
> ContainerManager a
> singleton. The ContainerManager has a 1:1 relationship to the
> Kubernetes/Mesos scheduler. Do we know how those are distributed? I
> think
> the Kubernetes scheduler is a singleton, but I'll need to doublecheck
> on
> that.
>
> I can see the possibility to move the ContainerManager into each
> Router and
> have them communicate with each other to shard in the same way I'm
> proposing. As Dave is hitting on the very same points, I get the
> feeling we
> should/could breakout that specific discussion if we can agree on some
> basic premises of the design (see my answers on the thread with Dave).
> WDYT?
>
>
> >
> >
> > The "edge-case" isn't clear to me b/c I'm coming from the assumption
> that
> > it doesn't matter which ContainerRouter handles the next request,
> given
> > that all actors have the same data. Maybe you can help me understand
> better
> > the edge-case ?
> >
>
> ContainerRouters do not have the same state specifically. The
> live-concurrency on a container is potentially very fast changing data.
> Sharing that across a potentially unbounded number of routers is not
> viable
> performance wise.
>
> Hence the premise is to manage that state locally and essentially
> shard the
> list of available containers between all routers, so each of them can
> keep
> its respective state local.
>
>
> >
> >
> > Re Knative approach, can you expand why the execution layer/data
> plane
> > would be replaced entirely by Knative serving ? I think knative
> serving
> > handles very well some cases like API requests, but it's not
> designed to
> > guarantee concurrency restrictions like "1 request at a time per
> container"
> > - something that AI Actions need.
> >
>
> You are right... today! I'm not saying Knative is necessarily a
> superior
> backend for OpenWhisk as it stands today. All I'm saying is that from
> an
>     architecture point-of-view, Knative serving replaces all of the
> concerns
> that the execution layer has.
>
>
> >
> >
> > Thanks,
> >
> > dragos
> >
> >
> > [1] - https://doc.akka.io/docs/akka/2.5/distributed-data.html
> >
> >
> &g

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Bertrand,

that's indeed something I haven't thought about yet, but as you say, the
ContainerManager could support multiple backends at once and schedule a
container wherever it thinks it makes sense. From the ContainerRouter's
PoV, a container is just an IP address + Port, so that concern is
encapsulated in the ContainerManager.

Cheers,
Markus

Am Mi., 15. Aug. 2018 um 10:35 Uhr schrieb Bertrand Delacretaz <
bdelacre...@apache.org>:

> Hi,
>
> On Tue, Aug 14, 2018 at 4:07 PM Markus Thömmes
>  wrote:
> ...
> >
> https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture
> ...
>
> Very clear proposal, thank you! And thanks for bringing the discussion
> here.
>
> Is the ContainerManager meant to support multiple underlying
> orchestrators? I'm thinking of a use case where you want to segregate
> actions of a specific set of namespaces to a dedicated orchestrator.
> This can be useful for cases where people don't trust existing
> container isolation mechanisms.
>
> From my understanding it looks like this is covered, just wanted to check.
>
> -Bertrand
>
>
> -Bertrand
>

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Dragos,

thanks for your questions, good discussion :)

Am Di., 14. Aug. 2018 um 23:42 Uhr schrieb Dragos Dascalita Haut
:

> Markus, I appreciate the enhancements you mentioned in the wiki, and I'm
> very much inline with the ideas you brought in there.
>
>
>
> "...having the ContainerManager be a cluster singleton..."
>
> I was just in process to reply with the same idea :)
>
> In addition, I was thinking we can leverage Akka Distributed Data [1] to
> keep all ContainerRouter actors eventually consistent. When creating a new
> container, the ContainerManager can write with a consistency "WriteAll"; it
> would be a little slower but it would improve consistency.
>

I think we need to quantify "a little slower". Note that "WriteAll" becomes
slower and slower the more actors you add to the cluster. Scalability is at
question then.

Of course scalability is also at question if we make the ContainerManager a
singleton. The ContainerManager has a 1:1 relationship to the
Kubernetes/Mesos scheduler. Do we know how those are distributed? I think
the Kubernetes scheduler is a singleton, but I'll need to doublecheck on
that.

I can see the possibility to move the ContainerManager into each Router and
have them communicate with each other to shard in the same way I'm
proposing. As Dave is hitting on the very same points, I get the feeling we
should/could breakout that specific discussion if we can agree on some
basic premises of the design (see my answers on the thread with Dave). WDYT?


>
>
> The "edge-case" isn't clear to me b/c I'm coming from the assumption that
> it doesn't matter which ContainerRouter handles the next request, given
> that all actors have the same data. Maybe you can help me understand better
> the edge-case ?
>

ContainerRouters do not have the same state specifically. The
live-concurrency on a container is potentially very fast changing data.
Sharing that across a potentially unbounded number of routers is not viable
performance wise.

Hence the premise is to manage that state locally and essentially shard the
list of available containers between all routers, so each of them can keep
its respective state local.


>
>
> Re Knative approach, can you expand why the execution layer/data plane
> would be replaced entirely by Knative serving ? I think knative serving
> handles very well some cases like API requests, but it's not designed to
> guarantee concurrency restrictions like "1 request at a time per container"
> - something that AI Actions need.
>

You are right... today! I'm not saying Knative is necessarily a superior
backend for OpenWhisk as it stands today. All I'm saying is that from an
architecture point-of-view, Knative serving replaces all of the concerns
that the execution layer has.


>
>
> Thanks,
>
> dragos
>
>
> [1] - https://doc.akka.io/docs/akka/2.5/distributed-data.html
>
>
> 
> From: David P Grove 
> Sent: Tuesday, August 14, 2018 2:15:13 PM
> To: dev@openwhisk.apache.org
> Subject: Re: Proposal on a future architecture of OpenWhisk
>
>
>
>
> "Markus Thömmes"  wrote on 08/14/2018 10:06:49
> AM:
> >
> > I just published a revision on the initial proposal I made. I still owe a
> > lot of sequence diagrams for the container distribution, sorry for taking
> > so long on that, I'm working on it.
> >
> > I did include a clear seperation of concerns into the proposal, where
> > user-facing abstractions and the execution (loadbalacing, scaling) of
> > functions are loosely coupled. That enables us to exchange the execution
> > system while not changing anything in the Controllers at all (to an
> > extent). The interface to talk to the execution layer is HTTP.
> >
>
> Nice writeup!
>
> For me, the part of the design I'm wondering about is the separation of the
> ContainerManager and the ContainerRouter and having the ContainerManager by
> a cluster singleton. With Kubernetes blinders on, it seems more natural to
> me to fuse the ContainerManager into each of the ContainerRouter instances
> (since there is very little to the ContainerManager except (a) talking to
> Kubernetes and (b) keeping track of which Containers it has handed out to
> which ContainerRouters -- a task which is eliminated if we fuse them).
>
> The main challenge is dealing with your "edge case" where the optimal
> number of containers to create to execute a function is less than the
> number of ContainerRouters.  I suspect this is actually an important case
> to handle well for large-scale deployments of OpenWhisk.  Having 20ish
> ContainerRouters on a large cluster seems plausible, an

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi Dave,

thanks a lot for your input! Greatly appreciated.

Am Di., 14. Aug. 2018 um 23:15 Uhr schrieb David P Grove :

>
>
>
> "Markus Thömmes"  wrote on 08/14/2018 10:06:49
> AM:
> >
> > I just published a revision on the initial proposal I made. I still owe a
> > lot of sequence diagrams for the container distribution, sorry for taking
> > so long on that, I'm working on it.
> >
> > I did include a clear seperation of concerns into the proposal, where
> > user-facing abstractions and the execution (loadbalacing, scaling) of
> > functions are loosely coupled. That enables us to exchange the execution
> > system while not changing anything in the Controllers at all (to an
> > extent). The interface to talk to the execution layer is HTTP.
> >
>
> Nice writeup!
>
> For me, the part of the design I'm wondering about is the separation of the
> ContainerManager and the ContainerRouter and having the ContainerManager by
> a cluster singleton. With Kubernetes blinders on, it seems more natural to
> me to fuse the ContainerManager into each of the ContainerRouter instances
> (since there is very little to the ContainerManager except (a) talking to
> Kubernetes and (b) keeping track of which Containers it has handed out to
> which ContainerRouters -- a task which is eliminated if we fuse them).
>

As you say below, the main concern is dealing with the edge-case I laid out.


>
> The main challenge is dealing with your "edge case" where the optimal
> number of containers to create to execute a function is less than the
> number of ContainerRouters.  I suspect this is actually an important case
> to handle well for large-scale deployments of OpenWhisk.  Having 20ish
> ContainerRouters on a large cluster seems plausible, and then we'd expect a
> long tail of functions where the optimal number of container instances is
> less than 20.
>

I agree, in large scale environments that might well be an important case.


>
> I wonder if we can partially mitigate this problem by doing some amount of
> smart routing in the Controller.  For example, the first level of routing
> could be based on the kind of the action (nodejs:6, python, etc).  That
> could then vector to per-runtime ContainerRouters which dynamically
> auto-scale based on load.  Since there doesn't have to be a fixed division
> of actual execution resources to each ContainerRouter this could work.  It
> also lets easily stemcells for multiple runtimes without worrying about
> wasting too many resources.
>

The premise I wanted to keep in my proposal is that you can route
essentially random between the routers. That's also why I use the overflow
queue as a work-stealing queue essentially to balance load between the
routers if the discrepancies get too high.

My general gut-feeling as to what can work here is: Keep state local as
long as you can (the individual ContainerRouters) to make the hot-path as
fast as possible. Fall back to work-stealing (slower, more constrained),
once things get out of bands.


>
> How do you want to deal with design alternatives?  Should I be adding to
> the wiki page?  Doing something else?
>

Good question. Feels like we can break out a "Routing" Work Group out of
this? Part of my proposal was to build this out collaboratively. Maybe we
can try to find consensus on some general points (direct HTTP connection to
containers should be part of it, we'll need an overflow queue) and once/if
we agree on the general broader picture, we can break out discussions on
individual aspects of it? Would that make sense?


>
> --dave
>

Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hey OpenWhiskers,

I just published a revision on the initial proposal I made. I still owe a
lot of sequence diagrams for the container distribution, sorry for taking
so long on that, I'm working on it.

I did include a clear seperation of concerns into the proposal, where
user-facing abstractions and the execution (loadbalacing, scaling) of
functions are loosely coupled. That enables us to exchange the execution
system while not changing anything in the Controllers at all (to an
extent). The interface to talk to the execution layer is HTTP.

Wanted to get this out as a possible idea on how to incooperate Knative in
the future and how it could look like alongside other implementations.

As always, feedback is very much welcome and appreciated.
https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture

Cheers,
Markus

Re: BDD Test Cases Contribution to complement existing test cases and coverage

[Markus Thömmes]

Hi Rahul,

thanks a lot for the contribution and the work you've put into this!
Very much appreciated.

I'm not familiar with Karate or BDD at all. Could you go into a little
bit more detail on what the benefit of this approach are and why we
should consider doubling our test-coverage (and thus effort)? How do
these tests relate to the other system tests we have written in Scala
today?

Cheers,
Markus
Am Mo., 13. Aug. 2018 um 07:55 Uhr schrieb Rahul Tripathi
:
>
> Hi All,
>
> To completed the existing OpenWhisk Test I have raised this pull request 
> which has Karate Based BDD test cases along with Gatling Support for 
> performance testing. The idea is to enable the testers to contribute the user 
> workflows in the form of BDD scenarios.
> Request you to kindly review and provide your valuable feedback.
>
>
> https://github.com/apache/incubator-openwhisk/pull/3956
>
> Thanks,
> Rahul
>
>
>
>
> Thanks,
> Rahul
>
>

Re: System env vars in user containers

[Markus Thömmes]

I guess this is the famous "context" object, that some other platforms
use in their function signature? @Chetan is that what you're referring
to in option 2?
Am Mo., 6. Aug. 2018 um 10:58 Uhr schrieb Chetan Mehrotra
:
>
> > What are you thinking as an alternarive?
>
> There can be 2 ways
>
> 1. Make them part of request json itself say by placing them under
> `_ow_env` key
> 2. Or introduce a new explicit env param to the run method call
>
> #1 would be compatible. We can also introduce some convention for such
> system generated values like start with '__' or '__ow_' etc to
> avoid collision in future for such changes
>
> Chetan Mehrotra
>
>
> On Mon, Aug 6, 2018 at 2:14 PM Rodric Rabbah  wrote:
>
> >
> > > Should we reduce dependency on env variable to communicate such state?
> >
> > Oy. That’s right the activation id and deadline will conflict. What are
> > you thinking as an alternarive?
> >
> > -r

Re: System env vars in user containers

[Markus Thömmes]

Ah yeah!

Dividing /init and /run time provided parameters makes a lot of sense
to minimize potential performance impact! +1 on that.
Am Mo., 6. Aug. 2018 um 10:37 Uhr schrieb Rodric Rabbah :
>
> +1 for backward compatibility.
>
> I would also provide the env vars at init time. (Currently they’re available 
> at run time).
>
> The api host is provided as a container env var is partly historical and 
> partly because it’s the same for all containers so it was remained factored 
> out when we added the rest of the context vars.
>
> -r

Re: System env vars in user containers

[Markus Thömmes]

Hi Vadim,

thanks for bringing this up!

If I were to design a new system from the ground up, I'd definitely
introduce a new structure to have a good distinction between
environment and parameters.

Taking into consideration that we need to be backwards compatible, I'd
definitely choose option 1. It will at least enable existing workloads
(think: blackbox containers built on the skeleton) to continue working
without intervention, which is key for us.

Regarding adding "value" as well: Is there any cost when setting
environment variables, especially with very large values? Is there a
limitation on the value of environment variables? I guess we could
blacklist "value" fairly easily as well?

Cheers,
Markus
Am Mo., 6. Aug. 2018 um 10:07 Uhr schrieb Vadim Raskin :
>
> Hi,
>
>
> currently the list of system environment variables that are exposed in user
> containers is configured in several places. First in the Invoker, it
> defines api_host env var during container creation, the rest of the env
> vars is configured for each run command via passing a json object [1] as a
> payload to container’s /run endpoint. Second the exact environment
> variables to be exposed are defined in each particular runtime, see the
> example for nodejs [2].
>
>
> The problem here is that we limit the exposed env vars in several places
> (invoker, each runtime) which makes it cumbersome to introduce changes.
>
>
> One possible option to alleviate introduction of new env vars would be to
> simply expose all properties found on the root level of the json structure
> excluding the “value”. In this case all runtimes must be updated.
>
>
> Another option for the same approach would be to introduce a new structure
> on the root level called “env_vars” which will encompass all vars to be
> exposed. In this case changes for both Invoker and all runtimes would be
> required.
>
>
> Do you have any concerns here? Which option sounds better?
>
>
> Thank you,
> Vadim
>
>
> [1]
> {
> “value”:{}
> “namespace”:””
> “action_name”:””
> “activation_id”:””
> “deadline”:””
> “api_key”:”"
> }
> [2] The list of environment variables to be exposed boils down to these
> ones: namespace, action_name, activation_id, deadline, api_key.
> https://github.com/apache/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejsActionBase/src/service.js#L161

Re: Update Kafka to 2.0.0

[Markus Thömmes]

Hi Dragos,

Yes, the newer client works even on our current version, got it to
work with SSL finally as well: Check
https://github.com/apache/incubator-openwhisk/pull/3913 for da codez.

Cheers,
Markus
Am Di., 31. Juli 2018 um 03:56 Uhr schrieb Dascalita Dragos
:
>
> Hi Markus,
> Big +1 for upgrading.
>
> In my case we use blue/green deployments so that would be fine.
>
> The only potential problem is on the DCOS package for Mesos, for supporting
> the latest Kafka version; i think it should work OOTB, but until it’s
> tested I’m leaving room for unknowns. Would you know if the Kafka client
> library is able to handle both versions ?
>
> Thanks,
> Dragos
> On Mon, Jul 30, 2018 at 12:57 PM Markus Thömmes 
> wrote:
>
> > Dear OpenWhiskers,
> >
> > James Dubee, Sven Lange-Last and I have been on a Kafka-related bug
> > hunt throughout the past days and found several issues in our current
> > KafkaConsumerConnector. Several PRs have been opened since and several
> > are yet to come.
> >
> > Currently, I'm trying to make the Kafka 2.0.0 clients work with our
> > old-dated Kafka 0.11 line Kafka broker. I'm wondering if we should
> > just update the brokers to 2.0.0 as well. It has been released today
> > and is admittedly fairly young. I feel like we should start tracking
> > more recent Kafka versions though, to profit from bug and stability
> > fixes.
> >
> > One wrinkle: Today we don't support a graceful update of the Kafka
> > brokers in-place. Is any of the production deployments based on
> > OpenWhisk relying on being able to make an in-place update or is a
> > blue/green style full replacement of the whole cluster fine for
> > everybody?
> >
> > Cheers,
> > Markus
> >

Re: Tech Interchange meeting Wednesday

[Markus Thömmes]

Thanks Justin,

Yes you can put me on the agenda on knative. Consider it a shared "Ben
Browning/Markus Thömmes - Update on Knative" please!

Cheers,
Markus
Am Mo., 30. Juli 2018 um 23:39 Uhr schrieb Justin Halsall
:
>
> Hi everyone,
>
>
> Reminder that the bi-weekly call is this Wednesday (1st of August)!
> I’ll be hosting the call this week so if you have any topic you’d like to 
> talk about let me know.
>
> For this weeks schedule I already have:
>
> Matt Rutkowski & Priti Desai -> website update
>
>
> Maybe the following would also like to present?
>
> Rodric Rabbah -> CVE/Security update
> Dave Grove -> Kubernetes update
> Markus Thoemmes -> Knative update
> You! -> Let me know what you would like to present!
>
>
> Mail me or the list if you’d like to contribute.
>
> Call details
> Web Meeting: Tech Interchange (bi-weekly):
> - Day-Time: Wednesdays, 11AM EDT (Eastern US), 4PM CEST (Central Europe),
> 3PM GMT, 11PM BST (Beijing)
> - Zoom: https://zoom.us/my/asfopenwhisk
>
>
>
> Looking forward to the meeting!
>
> Cheers,
>
> Justin Halsall
>
>
>

Update Kafka to 2.0.0

[Markus Thömmes]

Dear OpenWhiskers,

James Dubee, Sven Lange-Last and I have been on a Kafka-related bug
hunt throughout the past days and found several issues in our current
KafkaConsumerConnector. Several PRs have been opened since and several
are yet to come.

Currently, I'm trying to make the Kafka 2.0.0 clients work with our
old-dated Kafka 0.11 line Kafka broker. I'm wondering if we should
just update the brokers to 2.0.0 as well. It has been released today
and is admittedly fairly young. I feel like we should start tracking
more recent Kafka versions though, to profit from bug and stability
fixes.

One wrinkle: Today we don't support a graceful update of the Kafka
brokers in-place. Is any of the production deployments based on
OpenWhisk relying on being able to make an in-place update or is a
blue/green style full replacement of the whole cluster fine for
everybody?

Cheers,
Markus

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi David,

the system is indeed dynamic and doesn't care if your workload falls
into the "heavy-load" bucket or the "light-load" bucket. In fact, it's
a form of work-stealing where an entity with no resources passes the
request to an entity with resources until that entity eventually
becomes overloaded itself.

I'm not sure the added latency here is of any importance, at least on
the high utilization case (which is your concern if I understand
correctly). As soon as the container of controller0 does no longer
suffice to serve all incoming requests concurrently, it will request
more resources immediately and return a 503 to controller1. The time
to create this container is far higher than the time it takes for the
503 to get back to controller1 so virtually no latency is added to the
request at all.

It's also important that as soon as the number of containers is >= the
number of controllers in the system, no more requests will need
proxying because every controller has at least 1 container.

I should definitly work on writing this whole protocol down and add
pictures and flow-diagrams. That should clarify even further.

Cheers,
Markus
Am Mi., 25. Juli 2018 um 19:12 Uhr schrieb David Breitgand :
>
> Hi Markus,
>
> Sure, that makes sense, but I think the question is how to optimize a
> tradeoff between the higher variance in the invocation latency (in the
> heavy utilized case) and waste of containers (in the underutilzed case).
>
> If we expect that the workloads will be volatile, switching between light
> utilization and heavy utilization, then maybe a solution should also be
> dynamic.
>
> More specifically:
> 1) Continuously determine a utilization condition by sampling incoming
> traffic
> 2) If utilization is low --> use proxying
> 3) If utilization is high --> switch proxying off
>
> If it's expected that the load will be moderate to high, I would just not
> use proxying at all and risk a bit of waste to gain performance.
>
> Maybe this also can be aligned with SLAs (which currently do not exist),
> but one can think of them as a thing of the future (in line with the
> Rodric's post
> https://medium.com/openwhisk/security-and-serverless-functions-b97618430db6
> ), so that latency/capacity can be traded off differently for different
> actions subject to SLA.
>
> Cheers.
>
> -- david
>
>
>
>
> From:   "Markus Thömmes" 
> To: dev@openwhisk.apache.org
> Date:   25/07/2018 06:20 PM
> Subject:Re: Proposal on a future architecture of OpenWhisk
>
>
>
> Hi David,
>
> note that only the first few requests in a "frequent-load" pattern
> would be proxied.
>
> Say you have 2 controllers, 1 container. controller0 owns that
> container. Heavy load comes in and hits two controllers. controller1
> will proxy the first few requests to controller0 because it doesn't
> yet own a container. controller0 will execute as much as it can and
> immediately realize it needs more containers, so it asks for them. The
> requests coming from controller1 will get rejected with a 503, stating
> that controller0 is overloaded itself and controller1 should wait
> because containers have already been requested.
>
> The ContainerManager distributes containers evenly, so even though
> controller1 hasn't asked for more containers just yet, it will get
> some due to the redistribution by the CM. It also starts asking for
> containers itself after it got the 503 by controller0.
>
> Does that make sense? This isn't in the proposal yet but has been
> discussed on this mail-thread. I'll aim to incorporate some of the
> discussions into the proposal sooner or later.
>
> Cheers,
> Markus
> Am Mi., 25. Juli 2018 um 16:54 Uhr schrieb David Breitgand
> :
> >
> > >> Hope that clarifies
> >
> > Yes, it does, thanks. But I still have a question :)
> >
> > >> if you have N controllers in the system and M
> > containers but N > M and all controllers manage their containers
> > exclusively, you'll end up with controllers not having a container to
> > manage at all.
> >
> > I am not sure how you arrive to this condition. It can only happen if
> the
> > system is very under-utilized. Suppose there is no proxying. As  you
> > write, as action invocations keep coming, every Controller will get at
> > least one request. So, it will at least once ask ContainerManager to
> > allocate a container.
> >
> > I agree that when action invocation is infrequent, it's better to have 1
> > container rather than N for that action.
> >
> > However, if an action invocation is frequent, it's the other way a

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi David,

note that only the first few requests in a "frequent-load" pattern
would be proxied.

Say you have 2 controllers, 1 container. controller0 owns that
container. Heavy load comes in and hits two controllers. controller1
will proxy the first few requests to controller0 because it doesn't
yet own a container. controller0 will execute as much as it can and
immediately realize it needs more containers, so it asks for them. The
requests coming from controller1 will get rejected with a 503, stating
that controller0 is overloaded itself and controller1 should wait
because containers have already been requested.

The ContainerManager distributes containers evenly, so even though
controller1 hasn't asked for more containers just yet, it will get
some due to the redistribution by the CM. It also starts asking for
containers itself after it got the 503 by controller0.

Does that make sense? This isn't in the proposal yet but has been
discussed on this mail-thread. I'll aim to incorporate some of the
discussions into the proposal sooner or later.

Cheers,
Markus
Am Mi., 25. Juli 2018 um 16:54 Uhr schrieb David Breitgand :
>
> >> Hope that clarifies
>
> Yes, it does, thanks. But I still have a question :)
>
> >> if you have N controllers in the system and M
> containers but N > M and all controllers manage their containers
> exclusively, you'll end up with controllers not having a container to
> manage at all.
>
> I am not sure how you arrive to this condition. It can only happen if the
> system is very under-utilized. Suppose there is no proxying. As  you
> write, as action invocations keep coming, every Controller will get at
> least one request. So, it will at least once ask ContainerManager to
> allocate a container.
>
> I agree that when action invocation is infrequent, it's better to have 1
> container rather than N for that action.
>
> However, if an action invocation is frequent, it's the other way around:
> you'd prefer having N containers rather than queueing or first going to
> ContainerManager, getting pointed to a Controller that has happens to have
> all containers for that action busy, which will result in that Controller
> again going to a ContainerManager and asking for a new container.
>
> So, how do we differentiate between the two cases? For frequently
> infrequent actions it saves, but for frequently executed actions, proxying
> will result in performance hit, will it?
>
> Thanks.
>
> -- david
>
>
>
>
> From:   "Markus Thömmes" 
> To: dev@openwhisk.apache.org
> Date:   25/07/2018 04:05 PM
> Subject:Re: Proposal on a future architecture of OpenWhisk
>
>
>
> Hi David,
>
> the problem is, that if you have N controllers in the system and M
> containers but N > M and all controllers manage their containers
> exclusively, you'll end up with controllers not having a container to
> manage at all.
> There's valid, very slow workload that needs to create only 1
> container, for example a slow cron trigger. Due to the round-robin
> nature of our front-door, eventually all controllers will get one of
> those requests at some point. Since they are by design not aware of
> the containers because they are managed by another controller they'd
> end-up asking for a newly created container. Given N controllers we'd
> always create at least N containers for any action eventually. That is
> wasteful.
>
> Instead, requests are proxied to a controller which we know manages a
> container for the given action (the ContainerManager knows that) and
> thereby bypass the need to create too many containers. If the load is
> too much to be handled by the M containers, the controllers managing
> those M will request new containers, which will get distributed to all
> controllers. Eventually, given enough load, all controllers will have
> containers to manage for each action.
>
> The ContainerManager only needs to know which controller has which
> container. It does not need to know in which state these containers
> are. If they are busy, the controller itself will request more
> resources accordingly.
>
> Hope that clarifies
>
> Cheers,
> Markus
>
> 2018-07-25 14:19 GMT+02:00 David Breitgand :
> > Hi Markus,
> >
> > I'd like to better understand the edge case.
> >
> > Citing from the wiki.
> >
> >>> Edge case: If an action only has a very small amount of containers
> > (less than there are Controllers in the system), we have a problem with
> > the method described above.
> >
> > Isn't there always at least one controller in the system? I think the
> > problem is not the number of Controllers, but r

Re: Proposal on a future architecture of OpenWhisk

[Markus Thömmes]

Hi David,

the problem is, that if you have N controllers in the system and M
containers but N > M and all controllers manage their containers
exclusively, you'll end up with controllers not having a container to
manage at all.
There's valid, very slow workload that needs to create only 1
container, for example a slow cron trigger. Due to the round-robin
nature of our front-door, eventually all controllers will get one of
those requests at some point. Since they are by design not aware of
the containers because they are managed by another controller they'd
end-up asking for a newly created container. Given N controllers we'd
always create at least N containers for any action eventually. That is
wasteful.

Instead, requests are proxied to a controller which we know manages a
container for the given action (the ContainerManager knows that) and
thereby bypass the need to create too many containers. If the load is
too much to be handled by the M containers, the controllers managing
those M will request new containers, which will get distributed to all
controllers. Eventually, given enough load, all controllers will have
containers to manage for each action.

The ContainerManager only needs to know which controller has which
container. It does not need to know in which state these containers
are. If they are busy, the controller itself will request more
resources accordingly.

Hope that clarifies

Cheers,
Markus

2018-07-25 14:19 GMT+02:00 David Breitgand :
> Hi Markus,
>
> I'd like to better understand the edge case.
>
> Citing from the wiki.
>
>>> Edge case: If an action only has a very small amount of containers
> (less than there are Controllers in the system), we have a problem with
> the method described above.
>
> Isn't there always at least one controller in the system? I think the
> problem is not the number of Controllers, but rather availability of
> prewarm containers that these Controllers control. If all containers of
> this Controller are busy at the moment, and concurrency level per
> container is 1 and the invocation hit this controller, it cannot execute
> the action immediately with one of its containers. Is that the problem
> that is being solved?
>
>>> Since the front-door schedules round-robin or least-connected, it's
> impossible to decide to which Controller the request needs to go to hit
> that has a container available.
> In this case, the other Controllers (who didn't get a container) act as a
> proxy and send the request to a Controller that actually has a container
> (maybe even via HTTP redirect). The ContainerManager decides which
> Controllers will act as a proxy in this case, since its the instance that
> distributes the containers.
>>>
>
> When reading your proposal, I was under impression that ContainerManager
> only knows about existence of containers allocated to the Controllers
> (because they asked), but ContainerManager does not know about the state
> of these containers at every given moment (i.e., whether they are being
> busy with running some action or not). I don't see Controllers updating
> ContainerManager about this in your diagrams.
>
> Thanks.
>
> -- david
>
>
>
>
> From:   "Markus Thoemmes" 
> To: dev@openwhisk.apache.org
> Date:   23/07/2018 02:21 PM
> Subject:Re: Proposal on a future architecture of OpenWhisk
>
>
>
> Hi Dominic,
>
> let's see if I can clarify the specific points one by one.
>
>>1. Docker daemon performance issue.
>>
>>...
>>
>>That's the reason why I initially thought that a Warmed state would
>>be kept
>>for more than today's behavior.
>>Today, containers would stay in the Warmed state only for 50ms, so it
>>introduces PAUSE/RESUME in case action comes with the interval of
>>more than
>>50 ms such as 1 sec.
>>This will lead to more loads on Docker daemon.
>
> You're right that the docker daemon's throughput is indeed an issue.
>
> Please note that PAUSE/RESUME are not executed via the docker daemon in
> performance
> tuned environment but rather done via runc, which does not have such a
> throughput
> issue because it's not a daemon at all. PAUSE/RESUME latencies are ~10ms
> for each
> operation.
>
> Further, the duration of the pauseGrace is not related to the overall
> architecture at
> all. Rather, it's a so narrow to safe-guard against users stealing cycles
> from the
> vendor's infrastructure. It's also a configurable value so you can tweak
> it as you
> want.
>
> The proposed architecture itself has no impact on the pauseGrace.
>
>>
>>And if the state of containers is changing like today, the state in
>>ContainerManager would be frequently changing as well.
>>This may induce a synchronization issue among controllers and, among
>>ContainerManagers(in case there would be more than one
>>ContainerManager).
>
> The ContainerManager will NOT be informed about pause/unpause state
> changes and it
> doesn't need to. I agree that such a behavior would generate serious load
> on the
> ContainerManager, but I think it's unnecessary.

Re: Knative, Kubernetes, and Apache OpenWhisk

[Markus Thömmes]

Hi Ben,

thanks for the write-up! For full disclosure to the rest of the
dev-list: I've been
involved with Knative and have contributed some bits especially to
their autoscaling
strategy.

The usual word of caution: Knative is in its very early stages (which
is actually
a good thing, because we can shape things then!). It's very exciting in that it
has the potential to drive serverless requirements into Kubernetes directly. I
don't want it go unsaid though that it's absolutely in an alpha state and not
production ready as of today. This is not to say we should not embrace it but
I want to manage expectations a little bit.

2018-07-25 3:28 GMT+02:00 Ben Browning :
>
> So, to that end, I'd like to open a discussion on whether and how we
> should rearchitect OpenWhisk to take advantage of Knative.
>
>
> To kick things off, here's my personal opinion:
>
> Kubernetes has won the container orchestration battle. Knative builds
> on top of Kubernetes and will make Kubernetes the best place to build
> serverless platforms. Apache OpenWhisk should go all-in on Knative and
> Kubernetes. We should become the best serverless platform that can be
> deployed by any cloud provider or company in their own datacenter.

I agree we should become the best serverless platform out there. I'm a little
hesitant to take "will make Kubernetes the best place to build serverless
platforms" as a fact and set in stone though.

While we should go all-in on knative and Kubernetes I don't think we should
drop support for the other deployment topologies that we support today.
Supporting at least Mesos adds to the pluralism in the field and at least in
theory could be a driver for more innovation in the future. Continue to support
"raw" VMs and/or Kubernetes directly is beneficial I believe as well.
Some of the niceties we are experimenting with continually like stemcell
containers, pre-warming etc. are way harder if not impossible under the model
that Knative sports today. This might well change but I feel like we
should keep
the door open in OpenWhisk to experiment with these optimizations ourselves.

We could in theory use a layered approach where the whole "execution
system" is an implementation detail to the "Controller" (just using today's
terms). We can then experiment with using Knative as our execution layer
but can continue to support whichever other execution layer we want to
support.

I cannot say the proposal I wrote up here
https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture
was not sparked by the inception of Knative. We could take that proposal
as a base to work out how to integrate Knative into OpenWhisk and ideally
make the "execution layer" pluggable in itself. It was kinda created with
Knative in mind but has some rough edges to sort out in that direction.

> We should take the best parts of the OpenWhisk server-side components
> and get those concepts and knowledge, even if not the direct code,
> into Knative. OpenWhisk has a lot of real-world knowledge around
> container pooling, load-balancing, warm and pre-warm containers, etc
> that Knative will need.
>
> Then, we should take the OpenWhisk client tools and extend them to
> work not just with functions and containers but also more traditional
> applications. Knative brings the concept of event-driven autoscaling
> to more than just functions and we should embrace that. We should also
> integrate Knative's concept of Builds into OpenWhisk client tools,
> which fills a gap we have today.

I agree we can widen the surface of what applications you can run via
OpenWhisk if we start to provide an "apps" API (need to be very careful
on the wording here, "apps" here means: a long-running container with a server
in itself).

It could be a great value-proposition to allow the user to run a continuum
of different workloads (functions, long-running applications) under one
API to not have to handle different auth, terms etc.

>
> Finally, we need to reach out to other serverless projects and figure
> out how we can work together. There's going to be a lot of overlapping
> and wasted effort in OpenWhisk, Riff, Kubeless, Fission, OpenFaaS, and
> so on all competing on the developer experience when we're all running
> on top of Kubernetes and Knative. OpenWhisk is already at the Apache
> Foundation which makes us a natural vendor-neutral choice for a
> combined effort.

That'd be super exciting!

Very exciting times ahead, the gears are moving!

Cheers,
Markus

Introducing sharding as an alternative for state sharing

[Markus Thömmes]

Hi folks,

we (Christian Bickel and I) just opened a pull-request for comments on 
introducing a notion of sharding instead of sharing the state between our 
controllers (loadbalancers). It also addresses a few deficiencies of the old 
loadbalancer to remove any kinds of bottlenecks there and make it as fast as 
possible.

Commit message for posterity:

The current ContainerPoolBalancer suffers a couple of problems and bottlenecks:

Inconsistent state: The data-structures keeping the state for that loadbalancer 
are not thread-safely handled, meaning there can be queuing to some invokers 
even though there is free capacity on other invokers.
Asynchronously shared state: Sharing the state is needed for a high-available 
deployment of multiple controllers and for horizontal scale in those. Said 
state-sharing makes point 1 even worse and isn't anywhere fast enough to be 
able to efficiently schedule quick bursts.
Bottlenecks: Getting the state from the outside (like for the 
ActivationThrottle) is a very costly operation (at least in the shared state 
case) and actually bottlenecks the whole invocation path. Getting the current 
state of the invokers is a second bottleneck, where one request is made to the 
corresponding actor for each invocation.
This new implementation aims to solve the problems mentioned above as follows:

All state is local: There is no shared state. Resources are managed through 
horizontal sharding. Horizontal sharding means: The invokers' slots are evenly 
divided between the loadbalancers in existence. If we deploy 2 loadbalancers 
and each invoker has 16 slots, each of the loadbalancers will have access to 8 
slots on each invoker.
Slots are given away atomically: When scheduling an activation, the slot is 
immediately assigned to that activation (implemented through Semaphores). That 
means: Even in concurrent schedules, there will not be an overload on an 
invoker as long as there is capacity left on that invoker.
Asynchronous updates of slow data: Slowly changing data, like a change in the 
invoker's state, is asynchronously handled and updated to a local version of 
the state. Querying the state is as cheap as it can be.

A few words on the implementation details:

We chose to use horizontal sharding (evenly dividing the capacity of each 
invoker) vs. vertical sharding (evenly dividing the invokers as a whole) for 
the sake of staging these changes mainly. Once we divide vertically, we'll need 
a good loadbalancing strategy in front of our controllers themselves, to keep 
unnecessary cold-starts to a minimum and maximize container reuse. By dividing 
horizontally, we maintain the same reuse policies as today and can even keep 
the same loadbalancing strategies intact. Horizontal sharding of course only 
scales so far (maybe to 4 controllers, assuming 16 slots on each invoker) but 
it will give us time to figure out good strategies for vertical sharding and 
learn along the way. For vertical sharding to work, it will also be crucial to 
have the centralized overflow queue to be able to offload work between shards 
through workstealing. All in all: Vertical sharding is a much bigger change 
than horizontal sharding.

We tried to implement everything in a single actor first, but that seemed to 
impose a bottleneck again. Note that this is very frequented code, it needs to 
be very tight. That might not match the actor model too well.

Subsuming everything: This keeps all proposed changes intact (most notably 
Tyson's parallel executions and overloading queue).

A note on the gains made by this: Our non-blocking invoke performance is now 
quite close to the raw Kafka produce performance that we have in the system 
(not that it's good in itself, but that's the next theoretical bottleneck). 
Before the changes, this was roughly bottlenecked to 4k requests per second on 
a sufficiently powerful machine. Blocking invoke performance was roughly 
doubled.

Any comments, thoughts?

Cheers,
Markus

Re: Invoker activation queueing proposal

[Markus Thömmes]

Heyho,

I ran into the same issue before and I think our scheduling code should be an 
Actor. We could microbenchmark it to assure it can happily schedule a large 
amount of actions per second to not become a bottleneck.

+1 for actorizing the LB

Cheers,
Markus

Von meinem iPhone gesendet

> Am 10.10.2017 um 13:28 schrieb Tyson Norris :
> 
> Hi - 
> Following up on this, I’ve been working on a PR. 
> 
> One issue I’ve run into (which may be problematic in other scheduling 
> scenarios) is that the scheduling in LoadBalancerService doesn’t respect the 
> new async nature of activation counting in LoadBalancerData. At least I think 
> this is a good description. 
> 
> Specifically, I am creating a test that submits activations via 
> LoadBalancer.publish, and I end up with 10 activations scheduled on invoker0, 
> even though I use an invokerBusyThreshold of 1.
> It would only occur when concurrent requests (or very short time between?) 
> arrive at the same controller, I think. (Otherwise the counts can sync up 
> quickly enough)
> I’ll work more on testing it.
> 
> Assuming this (dealing with async counters) is the problem, I’m not exactly 
> sure how to deal with it. Some options may include:
> - change LoadBalancer to an actor, so that local counter states can be easier 
> managed (these would still need to replicate, but at least locally it would 
> do the right thing) 
> - coordinate the schedule + setupActivation calls to also rely on some local 
> state for activations that should be counted but have not yet been processed 
> within LoadBalancerData
> 
> Any suggestions in this area would be great.
> 
> Thanks
> Tyson
> 
> 
> 
>> On Oct 6, 2017, at 11:04 AM, Tyson Norris  wrote:
>> 
>> With many invokers, there is less data exposed to rebalancing operations, 
>> since the invoker topics will only ever receive enough activations that can 
>> be processed “immediately", currently set to 16. The single backlog topic 
>> would only be consumed by the controller (not any invoker), and the invokers 
>> would only consumer their respective “process immediately” topic - which 
>> effectively has no, or very little, backlog - 16 max. My suggestion is that 
>> having multiple backlogs is an unnecessary problem, regardless of how many 
>> invokers there are.
>> 
>> It is worth noting the case of multiple controllers as well, where multiple 
>> controllers may be processing the same backlog topic. I don’t think this 
>> should cause any more trouble than the distributed activation counting that 
>> should be enabled via controller clustering, but it may mean that if one 
>> controller enters overflow state, it should signal that ALL controllers are 
>> now in overflow state, etc.
>> 
>> Regarding “timeout”, I would plan to use the existing timeout mechanism, 
>> where an ActivationEntry is created immediately, regardless of whether the 
>> activation is going to get processed, or get added to the backlog. At time 
>> of processing the backlog message, if the entry is timed out, throw it away. 
>> (The entry map may need to be shared in the case multiple invokers are in 
>> use, and they all consume from the same topic; alternatively, we can 
>> partition the topic so that entries are only processed by the controller 
>> that has backlogged them)
>> 
>> Yes, once invokers are saturated, and backlogging begins, I think all 
>> incoming activations should be sent straight to backlog (we already know 
>> that no invokers are available). This should not hurt overall performance 
>> anymore than it currently does, and should be better (since the first 
>> invoker available can start taking work, instead of waiting on a specific 
>> invoker to become available).
>> 
>> I’m working on a PR, I think much of these details will come out there, but 
>> in the meantime, let me know if any of this doesn’t make sense.
>> 
>> Thanks
>> Tyson
>> 
>> 
>> On Oct 5, 2017, at 2:49 PM, David P Grove 
>> mailto:gro...@us.ibm.com>> wrote:
>> 
>> 
>> I can see the value in delaying the binding of activations to invokers when 
>> the system is loaded (can't execute "immediately" on its target invoker).
>> 
>> Perhaps in ignorance, I am a little worried about the scalability of a 
>> single backlog topic. With a few hundred invokers, it seems like we'd be 
>> exposed to frequent and expensive partition rebalancing operations as 
>> invokers crash/restart. Maybe if we have N = K*M invokers, we can get away 
>> with M backlog topics each being read by K invokers. We could still get 
>> imbalance across the different backlog topics, but it might be good enough.
>> 
>> I think we'd also need to do some thinking of how to ensure that work put in 
>> a backlog topic doesn't languish there for a really long time. Once we start 
>> having work in the backlog, do we need to stop putting work in immediately 
>> topics? If we do, that could hurt overall performance. If we don't, how will 
>> the backlog topic ever get drained if most invokers are ke

Re: Invoker activation queueing proposal

[Markus Thömmes]

The overflow queue is not intended to be drained by a large scale of invokers 
but by a relatively low scale of loadbalancers! I don‘t think rebalancing is a 
problem then.

That loadbalancer basically has 2 states:

1. capacity is there: schedule everything you get and put it on immediate 
invoker queues.
2. overflow: put everything you get on the overflow queue and only schedule 
stuff that you drain from that queue to the immediate queues.

Overflow state is over once the overflow queue is empty where it can continue 
to schedule immediatly.

Von meinem iPhone gesendet

> Am 05.10.2017 um 23:49 schrieb David P Grove :
> 
> I can see the value in delaying the binding of activations to invokers when 
> the system is loaded (can't execute "immediately" on its target invoker).
> 
> Perhaps in ignorance, I am a little worried about the scalability of a single 
> backlog topic. With a few hundred invokers, it seems like we'd be exposed to 
> frequent and expensive partition rebalancing operations as invokers 
> crash/restart. Maybe if we have N = K*M invokers, we can get away with M 
> backlog topics each being read by K invokers. We could still get imbalance 
> across the different backlog topics, but it might be good enough. 
> 
> I think we'd also need to do some thinking of how to ensure that work put in 
> a backlog topic doesn't languish there for a really long time. Once we start 
> having work in the backlog, do we need to stop putting work in immediately 
> topics? If we do, that could hurt overall performance. If we don't, how will 
> the backlog topic ever get drained if most invokers are kept busy servicing 
> their immediately topics? 
> 
> --dave
> 
> Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with 
> a few about optimizing the queueing that goes on ahead of invok
> 
> From: Tyson Norris 
> To: "dev@openwhisk.apache.org" 
> Date: 10/04/2017 07:45 PM
> Subject: Invoker activation queueing proposal
> 
> 
> 
> 
> Hi -
> 
> I’ve been discussing a bit with a few about optimizing the queueing that goes 
> on ahead of invokers so that things behave more simply and predictable.
> 
> 
> 
> In short: Instead of scheduling activations to an invoker on receipt, do the 
> following:
> 
> - execute the activation "immediately" if capacity is available
> 
> - provide a single overflow topic for activations that cannot execute 
> “immediately"
> 
> - schedule from the overflow topic when capacity is available
> 
> 
> 
> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY 
> gets queued there in the case that the invoker is not fully loaded, and 
> therefore should execute it “very soon")
> 
> 
> 
> Later: it would also be good to provide more container state data from 
> invoker to controller, to get better scheduling options - e.g. if some 
> invokers can handle running more containers than other invokers, that info 
> can be used to avoid over/under-loading the invokers (currently we assume 
> each invoker can handle 16 activations, I think)
> 
> 
> 
> I put a wiki page proposal here: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Fe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk&m=UE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM&s=LD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8&e=
>  
> 
> 
> 
> WDYT?
> 
> 
> 
> Thanks
> 
> Tyson
> 
> 
> 
> 

Re: Distributed Tracing support PR

[Markus Thömmes]

Hi,

I think OpenTracing will be a great way to implement this, so we don't need to 
write SPIs for every tracing backend there is but OpenTracing already handles 
that for us.

+1

Cheers,
Markus

Am 26. September 2017 um 08:20 schrieb "sandy...@gmail.com" 
:

Hi,

sometime back I had created PR adding distributed tracing support - https://github.com/apache/incubator-openwhisk/pull/2282 
The current tracing support is based on zipkin akka-tracing library. 


Recently, Adrian had mentioned in one of comments that the same lib is not 
under active development and is probably dead. Given this I wanted to change 
the tracing support implementation and make use of OpenTracing 
(http://opentracing.io/ ).
OpenTracing will still allow to connect to a zipkin tracing server or any other 
tracing solution.

I wanted to know everyone thoughts on the use of OpenTracing.

thanks,
Sandeep Paliwal

Re: PR feedback for SPIs

[Markus Thömmes]

Hi Tyson,

first of all: Thanks for the great effort put into all of these SPI 
implementations. Here are some general comments from myself on SPIs

The ContainerFactory SPI feels natural and makes a lot of sense to support 
different deployment models. There is no doubt about that one from my side.

For the LoadBalancer SPIs though, I am quite worried that we're silently 
rebuilding all of OpenWhisk's underlying architecture without anybody really 
noticing it. It feels to me that OpenWhisk is then becoming a set of plugins 
and you cannot compare any performance data anymore, neither do different 
implementors look at the very same bottlenecks. We talked about the direct 
Loadbalancer for instance and I still feel like it is not to be conflated with 
the use-case of having concurrent requests to containers and thus higher 
density on invokers. In fact, if you feel you need a direct Loadbalancer, would 
it maybe even make sense to have that one globally instead of just for that 
specific type of actions? I'd like to keep synergy effects high in the projects 
and I strongly feel that we're missing out on those if we start to break the 
system apart and stop being opinionated about the implementations we've chosen.

As for the logstore I kinda feel the same for now, although I haven't looked 
thoroughly. It does feel like it should at least partly be unified with the 
ContainerFactory SPI since getting to logs always is a concern of the 
underlying container implementation (i.e.: Which log drivers are supported 
etc.).

What do you think?

Cheers
Markus

Am 13. September 2017 um 07:44 schrieb Tyson Norris :

Hi -
Following SPI PRs need feedback (in order) , as interested people have time:

- LoadBalancer SPI - https://github.com/apache/incubator-openwhisk/pull/2584 - 
delegates loadbalancer creation to the SPI

- Multiple LoadBalancers SPI - 
https://github.com/apache/incubator-openwhisk/pull/2668 - allows multiple load 
balancers, loaded via the previous SPI, resolved per action via 
LoadBalancerResolver; allows either single loadbalancer (current behavior) or 
kind based loadbalancer (config maps action kinds to a specific loadbalancer); 
this will allow things like latency sensitive actions routing to containers 
directly while latency insensitive actions route via Kafka, etc. Relies on 
previous

- ContainerFactory SPI - 
https://github.com/apache/incubator-openwhisk/pull/2659 - delegates container 
instantiation to an SPI; Affected areas are InvokerReactive; also moved the 
dependent classes (Container et al) to common/scala, to allow for other cases 
(e.g. controller causes container launch, etc); TBD: checking 
maxActiveContainers and maxPoolSize should be part of this SPI too, but 
currently are not. (e.g. maxActiveContainers may be dynamic with some cluster 
managers)

- LogStore SPI - https://github.com/apache/incubator-openwhisk/pull/2695 - 
enables log collection to be delegated to an decoupled system (instead of 
consuming cycles within the invoker JVM); Affected areas are ContainerProxy 
(log collection delegates to the SPI), RestAPIs (delegate log retrieval to the 
SPI), and ContainerFactory impls (delegate docker run --log-opt parameters to 
the SPI); still needs some cleanup (WIP). Relies on ContainerFactory, mostly 
since it organizes Container classes around the ones moved to common/scala


Building on these some upcoming additions include:
- direct LoadBalancer - allow controller to route actions directly to shared 
containers (i.e. to support better throughput with fewer containers, if the 
action supports it, via multiple loadbalancers - one for kafka+invoker routing, 
and one for direct container routing)
- mesos ContainerFactory - to support launching containers in a mesos cluster 
(run more containers with fewer, or just 1, invokers per cluster, since the 
invoker has access to the entire mesos cluster for container launching)


Let me know via comments in PR (or on dev list) if you have questions on these.

Thanks in advance for feedback!
Best
Tyson

Re: Moving out runtime images and tests

[Markus Thömmes]

Sounds all sensible to me.

To 1: Sounds like the right thing to do, if versions are common enough? Like 
major bumps could probably use different repos but I guess we can just try and 
find out what works. As a starter, the list you posted sounds sensible to me!

2. By Integration Tests I meant: Do the repos need to standup a whole OpenWhisk 
system and then run integrations tests or can we get away with just unit tests. 
I think, the best would be the latter since its the most agile but we might not 
be quite there yet. Again something we should just try and will discover on the 
way I think!

3. Yep, it's probably just build and push the images on each commit just like 
we do for master.

One more thing (hehe): Do we support independent tags of runtime images vs. the 
deployed system? Today we use the same tag that we use to deploy the system to 
pull the runtime images. There might be something left to do here like: Do we 
need a specific version for each runtime (most flexible) or can we get away 
with just using latest for each runtime (easiest to maintain). Versioning might 
be needed to be able to make breaking changes but still pin versions and to 
keep production systems stable that way (otherwise they might pull an 
incompatible image while in action). Thinking about it: We need independent 
versioning per runtime.

Am 12. September 2017 um 21:56 schrieb Rodric Rabbah :

+1 good points Tyson.

-r

On Sep 12, 2017, at 1:25 PM, Tyson Norris  wrote:

+1 in general


Tasks we'd need to do:
1. Setup a repo per runtime.

Maybe repo per runtime-type, instead of literally each runtime - specifically 
names like: openwhisk-action-nodejs, openwhisk-action-java, 
openwhisk-action-dockerskeleton, openwhisk-action-php, openwhisk-action-python, 
openwhisk-action-swift
(Since each type should share much of the same tests, I think.)

2. Move the runtime build + tests there (testwise I would rather copy and own some 
dependencies than trying to go DRY. The current setup for dependent repos needs quite 
some cleanup and is super hard to maintain for updates in the main repo). We can discuss 
if we need Integration Tests for each of the runtimes or if the "unit" tests we 
have are sufficient here.

We can DRY it *later* if wanted by releasing mvn artifacts of test base classes 
(or all of tests) from core, or something like that, but until that is 
implemented, agree that copying is better.

By “Integration Tests” do you mean to run as part of core tests? I think something 
minimal would be good like “test that at least nodejs runtime works” (e.g. health 
check), but not “test every runtime as part of core integration tests"

3. Implement a release process for the runtime images to Dockerhub.


This should be the same as the release process for core images right?

Thanks
Tyson


The runtimes update fairly rarely so I wouldn't really bother with too strict 
of a versioning there, at least not for the first shot. Process wise it does 
seem straightforwardly doable.

What do you think?

Cheers
Markus

Moving out runtime images and tests

[Markus Thömmes]

Hey all,

again we're hitting the Travis build time limit of 50 minutes and we need to 
disable more tests again to bring us in good shape there. This is not really 
acceptable and it's really painful for me to turn down testcoverage.

A thought has been lingering for some time now and with the work Rodric did by 
making runtimes configurable we should be really there to move out the runtimes 
from our main repository and leave only a default one in there if any.

Tasks we'd need to do:
1. Setup a repo per runtime.
2. Move the runtime build + tests there (testwise I would rather copy and own some 
dependencies than trying to go DRY. The current setup for dependent repos needs quite 
some cleanup and is super hard to maintain for updates in the main repo). We can discuss 
if we need Integration Tests for each of the runtimes or if the "unit" tests we 
have are sufficient here.
3. Implement a release process for the runtime images to Dockerhub.

The runtimes update fairly rarely so I wouldn't really bother with too strict 
of a versioning there, at least not for the first shot. Process wise it does 
seem straightforwardly doable.

What do you think?

Cheers
Markus

Re: Warm containers

[Markus Thömmes]

Hi,

that should still be possible without needing excessive container recreates. Do 
you see anything blocking your use-case?

Am 06. September 2017 um 08:17 schrieb mandeep gandhi 
:

Hi Markus,

Actually, I am trying to orchestrate a workflow via open-whisk which would
involve a request payload passing through a sequence of actions. So I have
a mixed bag of heterogeneous micro services I want to mimic as actions.

I had already gone through your blog earlier.



On Wed, Sep 6, 2017 at 11:27 AM, Markus Thömmes 
wrote:

Hi Mandeep,

OpenWhisk relies on container reuse (i.e. using warm containers) heavily
to reach its performance goals. Currently, containers are kept around for a
maximum of 10 minutes if they are not replaced by the need of some other
container.

You can refer to this article written by me (shameless self-advertisement)
to get some insights into how our container caching works:
https://medium.com/openwhisk/squeezing-the-milliseconds-how-to-make-
serverless-platforms-blazing-fast-aea0e9951bd0

You can refer to the relevant source code here:
- https://github.com/apache/incubator-openwhisk/blob/
master/core/invoker/src/main/scala/whisk/core/
containerpool/ContainerProxy.scala
- https://github.com/apache/incubator-openwhisk/blob/
master/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.
scala

Anything more you can think of for your use-case?

Cheers,
Markus

Am 06. September 2017 um 07:50 schrieb mandeep gandhi <
welcomemand...@gmail.com>:

Hi,

I was looking for the support of warm containers for JVM and Python based
containers. I have a use case where I want to run some scala/java micro
services and some TensorFlow containers for my workflow.

Now as these containers would take some seconds to get up, I would like to
know -

a. What is the current status for the same?
b. How can we contribute to it if the support is missing?


Thanks,
Mandeep Gandhi

Re: Warm containers

[Markus Thömmes]

Hi Mandeep,

OpenWhisk relies on container reuse (i.e. using warm containers) heavily to 
reach its performance goals. Currently, containers are kept around for a 
maximum of 10 minutes if they are not replaced by the need of some other 
container.

You can refer to this article written by me (shameless self-advertisement) to 
get some insights into how our container caching works: 
https://medium.com/openwhisk/squeezing-the-milliseconds-how-to-make-serverless-platforms-blazing-fast-aea0e9951bd0

You can refer to the relevant source code here:
- 
https://github.com/apache/incubator-openwhisk/blob/master/core/invoker/src/main/scala/whisk/core/containerpool/ContainerProxy.scala
- 
https://github.com/apache/incubator-openwhisk/blob/master/core/invoker/src/main/scala/whisk/core/containerpool/ContainerPool.scala

Anything more you can think of for your use-case?

Cheers,
Markus

Am 06. September 2017 um 07:50 schrieb mandeep gandhi 
:

Hi,

I was looking for the support of warm containers for JVM and Python based
containers. I have a use case where I want to run some scala/java micro
services and some TensorFlow containers for my workflow.

Now as these containers would take some seconds to get up, I would like to
know -

a. What is the current status for the same?
b. How can we contribute to it if the support is missing?


Thanks,
Mandeep Gandhi

Re: Today - Join us for the OpenWhisk “Tech. Interchange” (bi-weekly) Zoom Meeting

[Markus Thömmes]

Heyho,

sorry for taking so long. Here's the video of the call: 
https://www.youtube.com/watch?v=-sXiZlPnX98&t=29s

And the summary:
Attendees:
Markus Thömmes (moderator)
Michael Behrendt
Dan Lavine
Lionel Villard
Nick Mitchell
Lorna Mitchell
Carlos Santana
Vadim Raskin
Mark Deuser
Dragos Dascilate
Michael Marth
Tyson Norris
Lloyd Roseblade (Swift IBM)
Dror Gensler (Startup company related to Serverless)

What’s new:
Markus: Proposal: Weekly/Monthly digest of what has happened
Dan: Kube
A couple of blockers still
More configuration in docker images themselves —> environment variables
Don’t rely on ansible
Markus: Main repository
Deprecation of old containerpool —> new pool is default
Old codebase to be removed soon —> end confusion
Features:
Namespace specific throttles

Topics:
Nick: OpenWhisk Shell
See what it is and can do here: 
https://openwhisk.ng.bluemix.net/api/v1/web/nickm_wskng_demos/public/index.html

Lorna: Nice for demos, but not sure how it fits in a scriptable environment. 
Should we create a separate tool vs. improving our wsk CLI?
Nick: This will be open-sourced very soon.
Rodric: Whatever you do in Whisk Shell you can export a script with raw CLI 
commands, you can use this in CI then.
Nick: Goal of the shell: Take the meaningful atomic tasks and make those 
shorter and easier to use. We could make the REPL executable outside of the GUI 
to enable scripting in the Whisk Shell syntax: simple syntactically, quickly 
writable, no low-level script.
We should embody common tasks into the REPL.
Lorna: My reservations are around how useful it is for people who need to 
deploy in a scriptable way, who don’t want to learn a new shell or write 
javascript. I love it and it looks nice.
Rodric: There’s such a big tooling gap in serverless. This is just one more 
tool in that space. It could find adoption for people who want an overview of 
how things execute and develop iteratively.
Nick: We need a way to quickly extend the command set. API change relatively 
slowly. How do we add support for project level support, debuggers, schema 
inference? We need a platform for that.

Michael Marth: Performance tests
Michael: Should we move Markus’ performance repository to the apache repos and 
make them “official”?
Michael: It is great we can discuss performance issues and we need common tests 
we can everywhere and talk about the same things.
Markus: No problem, we could just move the repository and I agree: We need a 
common testbase to quickly determine performance benefits and regressions.
Markus: Where do we run them continuously? Apache Jenkins? Automatically run 
them on pull-requests. We should do it incrementally. As a quick step: Run the 
tests on a set of machines, specify them and put a stake in the ground on 
performance numbers.
Current tests are very simple, everything is warm, no cold-containers for 
instance. Can we have more sophisticated configurations?
Rodric: What are the benchmarks we should have? Raw latency, raw throughput, 
latency under different scenarios. We need to specify a set of those benchmarks 
and not just talk about infrastructure and frameworks to implement them. We saw 
a couple of devs trying to benchmark their environments. A nice thing would 
have been a standardized way to setup and run the benchmarks with good 
documentation.
We should figure out how we get infrastructure out of Apache.
Michael: I reached out to INFRA and I never got a proper reply. Shouldn’t stop 
us to move the scripts though.
Rodric: Can our mentors help here?
Michael: I asked Felix and pinged the mentors in the thread we had. No reply.
Markus: I wanted to at least a specified run on some of our hardware so people 
can compare. We should agree on a common framework though. I looked into Gatlin 
it has pros and cons.
Tyson: We looked at Locust, which also has its drawbacks.
Gatlin is a scala-based load-testing tool, a DSL around something like JMeter. 
You can code quite sophisticated scenarios with it. It can produce great load, 
but the reporting is terrible.
Locust is opposite, scriptability is more limited, reporting is much better. It 
is able to be running distributedly, while Gatling will only run on a single 
node.
Gatling is a reasonable compromise. We haven’t found an obvious choice though.
Dragos: I worked with Tsung, written in Erlang. Nice performance. Quite 
difficult to cluster. I find Locust to be my first choice because it runs 
distributed.
Markus: Nice, I’ll look into Locust as well. I found Gatling reporting good 
though, at least the report generated in the end. I care mostly about reporting 
containing everything we need, like latency changing over time.
Tyson: Gatling is better in time-based views. Locust also only gives you a 
summary aggregate.
Dragos: We can even do a screen-sharing session to teach the community.
Rodric: One outcome is, we can move the repository to Apache. Everything else 
we’ll do with pull-requests.
Tyson: It might even be useful

Today - Join us for the OpenWhisk “Tech. Interchange” (bi-weekly) Zoom Meeting

[Markus Thömmes]

Hi there,

It's biweekly meeting time again. Matt won't be able to join so we'll need to 
use another zoom meeting (thanks Michael Behrendt for helping out)

Time: Wed., August 2nd, 11am US EDT, 8am US PDT, 5pm CEST 


Join from PC, Mac, Linux, iOS or Android: https://ibm.zoom.us/j/357640980

Or iPhone one-tap (US Toll): +14086380968,,357640980# or 
+16465588656,,357640980#

Or Telephone:
Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll)
Meeting ID: 357 640 980
International numbers available: 
https://ibm.zoom.us/zoomconference?m=nMjjJbNvSpqZbEIUB0b66gqJcuvQFZDO

---

Agenda:
1. Introduction/New faces on the call?
2. What's new? Short status updates depending on who's joining the call.
3. Topics:
  3.1: Nick Mitchell - OpenWhisk Shell
  3.2: Markus Thömmes - Invoker Reactive: What's the fuzz and how does it work?
  3.3: Anything else you are interested in!
4. Confirm moderator for the next call

Note: This is not a fixed agenda. If we have consensus on items we want to 
discuss, we can adjust the agenda ad-hoc

Hope to see you there!

Deprecation of the old containerpool

[Markus Thömmes]

Hi all,

development of the reactive containerpool is finished and it's been tested thoroughly in 
our (IBM's) pipeline. It is used in the Bluemix hosted OpenWhisk deployment for a while 
now which makes me believe we should start the deprecation process for the 
"old" containerpool since it's subject to confusion what the difference between 
both are.

As a first stepping stone, I opened: 
https://github.com/apache/incubator-openwhisk/pull/2528 to enable the pool in all 
environments managed by the open repository. That will leave "custom" 
deployments alone and should keep the update path for production environments untouched 
for now. Note that our whole testcoverage will from then on be based on the new pool.

The next step will be to enable the new pool as a default, environments would 
have to opt-in to get the old implementation. I'd do that on 02.08.2017 (2nd of 
August). Two weeks later (16th of August), I'd like to completely remove the 
old implementation to further reduce confusion about the duplicated codebase.

There is no change in protocols regarding the new pool and it should be a 
drop-in replacement for the old one.

Any concerns?

Cheers,
Markus

Re: OpenWhisk on Mesos

[Markus Thömmes]

Hi Ben,

this looks great! Any feedback on the container abstractions?

I agree the ContainerPool takes a couple of assumptions as of today, I'd be 
greatly interested in reducing those and support using orchestrator APIs just 
as you're trying to do.

Great work :)

Cheers,
Markus

Von meinem iPhone gesendet

> Am 15.07.2017 um 15:50 schrieb Ben Browning :
> 
> I'm one of those that desires an alternative implementation to launching
> containers on Kubernetes, as Tyson mentioned. Instead of talking to Docker
> directly we're exploring talking to the Kubernetes API so that resource
> scheduling is still handled by the Kubernetes scheduler and we only have a
> couple of Invoker instances for high availability.
> 
> The route we have right now is an alternative Container implementation (
> https://github.com/projectodd/incubator-openwhisk/blob/3772c344dbfa421bf361631f96f00f7876eb2adf/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala)
> that we're prototyping. That implementation is not ready for production use
> by any means and has some serious performance issues we'll need to address.
> But, it works enough that we can explore the tradeoffs when talking
> directly to Docker versus talking to the Kubernetes API and letting
> Kubernetes schedule function runtimes.
> 
> This may or may not be the right approach to take on Kubernetes and Mesos.
> The load-balancing done in the controller and the container pooling done in
> the Invokers today really assume each Invoker manages an independent set of
> Docker containers on separate hardware versus Invokers managing independent
> sets of Docker containers on a shared resource, such as the Kubernetes
> cluster.
> 
> Ben
> 
> 
> On Fri, Jul 14, 2017 at 1:07 PM, Tyson Norris 
> wrote:
> 
>> Hi Tim -
>> As you mention, running OW in a mixed-workload Mesos cluster can create
>> some issues since the resource scheduling will be conflicting between Mesos
>> and OW. I think this will be true of other cluster managers as well, but
>> I’m not sure.
>> 
>> We are experimenting with how to allow OW to operate as a Mesos framework,
>> and delegate container management to Mesos. There are a number of obstacles
>> currently, such as having no way to provide an alternate mechanism for
>> launching containers. The simplest version of this would be a system where
>> a single invoker exists and acts as the Mesos framework, but really needs
>> to support multiple instances with a shared cache of data to provide high
>> availability. We are working through some ideas on how to do this, and I
>> believe there is also some similar interest from others that are using
>> Kubernetes to do something similar. We will be sharing our work on this
>> with the community asap, but there are a few steps to getting to that point.
>> 
>> Regards,
>> Tyson
>> 
>> 
>>> On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <
>> timothy.bo...@ge.com> wrote:
>>> 
>>> Duy,
>>> 
>>> Thanks for the updated PR! It contains the fixes I was looking for.
>>> 
>>> One of the concerns I have about the current situation is that the
>> config assumes that OW has carte-blanche permissions in Mesos. For example,
>> both the registrator and invoker talk directly to the Docker socket,
>> several of the services use HOST networking, and most of the services
>> reserve dedicated ports instead of using randomly assigned ones. The owners
>> of our shared Mesos cluster don’t like me doing stuff like this. :-)
>>> 
>>> Are most OW-on-Mesos users assuming they will have a dedicated Mesos
>> cluster for OW, or is the intention to make OW work with a locked-down,
>> shared cluster eventually?
>>> 
>>> Tim
>>> 
>>> 
>>> On 7/13/17, 3:49 AM, "Duy Nguyen"  wrote:
>>> 
>>>   Hi Tim,
>>> 
>>>   Great to see more interest in running OpenWhisk on Mesos.
>>> 
>>>   With the deployment structure in the PR (https://na01.safelinks.
>> protection.outlook.com/?url=https%3A%2F%2Fgithub.com%
>> 2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> 6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0
>>>   ubator-openwhisk-devtools/pull/20), we have our OW instance running
>> on
>>>   DC/OS. Basically, we wrap each service in a DC/OS package and publish
>> a
>>>   dedicated universe for it. A deployment means launching those
>> services from
>>>   the pre-defined packages and apply your apigateway config to set up
>> the
>>>   server endpoint.
>>> 
>>>   Currently one of the main limitations is with the Invoker service: it
>>>   independently manages resources for running action containers on a
>> single
>>>   node and does not communicate back to Marathon, hence Marathon is
>> unable to
>>>   make proper decision on allocating its resources. Some ideas to solve
>> this
>>>   were discussed in a previous thread [1].
>>> 
>>>   I have updated the PR which works well for us with the latest
>> OpenWhisk
>>>   image

Re: Improving support for UI driven use cases

[Markus Thömmes]

Right, I think the UI workflows are just an example of apps that are latency 
sensitive in general.

I had a discussion with Stephen Fink on the matter of detecting ourselves that 
an action is latency sensitive by using the blocking parameter or as mentioned 
the user's configuration in terms of web-action vs. non-web action. The 
conclusion there was, that we probably cannot reliably detect latency 
sensitivity without asking the user to do so. Having such an option has 
implications on other aspects of the platform: Why would one not choose that 
option?

To Rodric's points I think there are two topics to speak about and discuss:

1. The programming model: The current model encourages users to break their actions apart in 
"functions" that take payload and return payload. Having a deployment model outlined 
could as noted encourage users to use OpenWhisk as a way to rapidly deploy/undeploy their usual 
webserver based applications. The current model is nice in that it solves a lot of problems for the 
customer in terms of scalability and "crash safeness".

2. Raw throughput of our deployment model: Setting the concerns aside I think it is valid 
to explore concurrent invocations of actions on the same container. This does not 
necessarily mean that users start to deploy monolithic apps as noted above, but it 
certainly could. Keeping our JSON-in/JSON-out at least for now though, could encourage 
users to continue to think in functions. Having a toggle per action which is disabled by 
default might be a good way to start here, since many users might need to change action 
code to support that notion and for some applications it might not be valid at all. I 
think it was also already noted, that this imposes some of the "old-fashioned" 
problems on the user, like: How many concurrent requests will my action be able to 
handle? That kinda defeats the seemless-scalability point of serverless.

Cheers,
Markus

Am 02. Juli 2017 um 10:42 schrieb Rodric Rabbah :

The thoughts I shared around how to realize better packing with intrinsic 
actions are aligned with the your goals: getting more compute density with a 
smaller number of machines. This is a very worthwhile goal.

I noted earlier that packing more activations into a single container warrants a different resource manager with its own container life cycle management (e.g., it's almost at the level of: provision a container for me quickly and let me have it to run my monolithic code for as long as I want). 

Already some challenges were mentioned, wrt sharing state, resource leaks and possible data races. Perhaps defining the resource isolation model intra container - processes, threads, "node vm", ... - is helpful as you refine your proposal. This can address how one might deal with intra container noisy neighbors as well. 


Hence in terms of resource management as the platform level, I think it would 
be a mistake to treat intra container concurrency the same way as ephemeral 
activations, that are run and done. Once the architecture and scheduler 
supports a heterogenous mix of resources, then treating some actions as 
intrinsic operations becomes easier to realize; in other words complementary to 
the overall proposed direction if the architecture is done right.

To Alex's point, when you're optimizing for latency, you don't need to be 
constrained to UI applications. Maybe this is more of a practical motivation 
based on your workloads.

-r

On Jul 2, 2017, at 2:32 AM, Dascalita Dragos  wrote:

I think the opportunities for packing computation at finer granularity
will be there. In your approach you're tending, it seems, toward taking
monolithic codes and overlapping their computation. I tend to think this
will work better with another approach.

+1 to making the serverless system smarter in managing and running the code
at scale. I don't think the current state is there right now. There are
limitations which could be improved by simply allowing developers to
control which action can be invoked concurrently. We could also consider
designing the system to "learn" this intent by observing how the action is
configured by the developer: if it's an HTTP endpoint, or an event handler.

As long as today we can improve the performance by allowing concurrency in
actions, and by invoking them faster, why would we not benefit from this
now, and update the implementation later, once the system improves ? Or are
there better ways available now to match this performance that are not
captured in the proposal ?

Re: Improving support for UI driven use cases

[Markus Thömmes]

Thanks for the vry detailed writeup! Great job!

One thing I'd note: As Rodric pointed out we should break the issues apart and 
address them one by one.

For instance the proposed loadbalancer changes (Controllers knows of Containers 
downstream) is desirable for each workload and not necessary bound to the core 
of the proposal, which I'd say is the concurrency discussion. I agree that 100% 
warm container usage is crucial there, but each load will benefit from high 
container reuse.

Just not to get the discussion too broad and unfocused.

On the topic itself: I think its a great idea to further push the performance 
and, as pointed out, make operating OpenWhisk more efficient. Most issue I see 
have already been pointed out so I won't repeat them. But I'm quite sure we can 
figure those out.

Have great weekend!

Von meinem iPhone gesendet

> Am 01.07.2017 um 19:26 schrieb Tyson Norris :
> 
> RE: separate policies - I agree that it would make sense for separating 
> container pools in some way by “event driven” and “ui driven” - I don’t think 
> anything precludes that from happening, but its different than the notion of 
> “long running” and “short running”. e.g. if events flow into the system, the 
> container would be equally long running as if users are continually using the 
> system. I’m not suggesting changing the cold/warm start behavior, rather the 
> response and concurrency behavior, to be more in line with UI drive use 
> cases. In fact the “first user” experience would be exactly the same, its the 
> “99 users that arrive before the first user is complete” experience that 
> would be different. (It may also be appealing to have a notion of “prehot” 
> containers, but I think existing “prewarm” is good enough for many cases). If 
> it's useful to cap the pool usage for either of these cases, nothing prevents 
> that from happening, but I would start with a (simpler) case where there is.a 
> single pool that supports both usages - currently there is no pool that 
> reliably supports the ui case, since “bursty” traffic is immediately 
> excessively latent.
> 
> RE: concurrent requests resource usage: I would argue that you would 
> determine resource allocation in (nearly) the same way you should with 
> single-tasked containers. i.e. the only pragmatic way to estimate resource 
> usage is to measure it. In single task case, you might use curl to simulate a 
> single user. In concurrent tasks case, you might use wrk or gatling (or 
> something else) to simulate multiple users. Regardless of the tool, analyzing 
> code etc will not get close enough to accurate measurements compared to 
> empirical testing.
> 
> RE: motivation compared to having a number of hot containers - efficiency of 
> resource usage for the *OpenWhisk operator*. No one will be able to afford to 
> run open whisk if they have to run 100 containers *per action* to support a 
> burst of 100 users using any particular action. Consider a burst of 1000 or 
> 1 users, and a 1000 actions. If a single container can handle the burst 
> of 100 users, it will solve a lot of low-medium use cases efficiently, and in 
> the case of 1 users, running 100 containers will be more efficient than 
> the 1 containers you would have to run as single task.
> 
> WDYT?
> 
> Thanks for the feedback!
> Tyson
> 
> 
> On Jul 1, 2017, at 9:36 AM, Alex Glikson 
> mailto:glik...@il.ibm.com>> wrote:
> 
> Having different policies for different container pools certainly makes
> sense. Moreover, enhancing the design/implementation so that there is more
> concurrency and less bottlenecks also sounds like an excellent idea.
> However, I am unsure specifically regarding the idea of handling multiple
> requests concurrently by the same container. For example, I wonder how one
> would determine the desired resource allocation for such container?
> Wouldn't this re-introduce issues related to sizing, scaling and
> fragmentation of resources - nicely avoided with single-tasked containers?
> Also, I wonder what would be the main motivation to implement such a
> policy compared to just having a number of hot containers, ready to
> process incoming requests?
> 
> Regards,
> Alex
> 
> 
> 
> From:   Rodric Rabbah mailto:rod...@gmail.com>>
> To: dev@openwhisk.apache.org
> Cc: Dragos Dascalita Haut mailto:ddas...@adobe.com>>
> Date:   01/07/2017 06:56 PM
> Subject:Re: Improving support for UI driven use cases
> 
> 
> 
> Summarizing the wiki notes:
> 
> 1. separate control and data plane so that data plane is routed directly
> to
> the container
> 2. desire multiple concurrent function activations in the same container
> 
> On 1, I think this is inline with an outstanding desire and corresponding
> issues to take the data flow out of the system and off the control message
> critical path. As you pointed out, this has a lot of benefits - including
> one you didn't mention: streaming (web socket style) in/out 

Re: Adding OpenTracing support

[Markus Thömmes]

As Juca stated: I'd also prefer the pluggable solution if we don't have a hard 
dependence on features of one specific solution.

Welcome to the family!

Cheers,
Markus

Von meinem iPhone gesendet

> Am 14.06.2017 um 13:28 schrieb Rodric Rabbah :
> 
> Thanks Juca for starting this discussion on the dev list.
> 
> The topic of making the tracing pluggable came up both in the related issue 
> https://github.com/apache/incubator-openwhisk/issues/2192#issuecomment-300679362
>  and also in the pull request.
> 
> I continue to think a plugable solution is the preferred solution so that 
> others don't also build more one offs that will not suite everyone's purpose. 
> I though opentracing had already come up but maybe that was just a 
> off-list/Slack discussion with Markus Thoemmes.
> 
> -r

Loadbalancer Improvements

[Markus Thömmes]

Hey folks,

it's me again with the latest news on performance :).

As some of you probably now: Our current loadbalancer strategy is quite 
"simple" and doesn't take load in the system into account at all. It hops to 
the next available invoker after you've invoked an action X times (where X is a fixed 
value defined at deployment time). For many many cases that's suboptimal behavior and 
induces lots of cold-starts, even in a fairly unused system. 

To improve on this here is a proposal to take the loadbalancer state we already 
have and make something out of it.

In a nutshell, the plan is: Before you schedule to an invoker, take into 
account how much load is on the invoker you want to schedule to. If it seems 
full already (determined by outstanding active-ack responses) search for 
another invoker.
Via hashing, we define a home invoker to for every subject/action combination. That is the invoker 
with the highest probability of having a warm container for that action. If that invoker is already 
busy, choose another invoker. "Stepping" through the invokers should be stable as well, 
as in: For a given subject/action it should always try the invokers in the same order. That way, 
the probability of getting a warm container is higher than if we chose randomly, but of course it 
gets lower the more "hops" you need to make.
The step-width is determined via hashing into a series of coprime numbers to 
the amount of invokers in the system to minimize collisions and chasing.

The proposal is expected to lead to a more stable warm-container rate and lead 
to a better utilization of the system as a whole.

I already took a stab at implementing the proposal above. The pull-request can 
be found here: https://github.com/apache/incubator-openwhisk/pull/2360

As always, comments, objections, praise. All feedback is very welcome :)

Cheers,
Markus

Re: Red Hat and Apache OpenWhisk

[Markus Thömmes]

Hi Ben,

That sounds awesome :). Welcome to the family and looking forward to working 
with all of you.

Cheers,
Markus

Von meinem iPhone gesendet

> Am 31.05.2017 um 22:59 schrieb Ben Browning :
> 
> My name is Ben Browning and I lead the Serverless and
> Function-as-a-Service engineering team at Red Hat. After much
> research, evaluation, and internal discussion I'm pleased to announce
> that we've decided to refocus our own Serverless efforts from the
> Funktion project (https://funktion.fabric8.io/) to Apache OpenWhisk.
> 
> We look forward to filing issues, creating pull requests,
> participating in dev discussions, giving conference talks, and
> contributing however we can to ensure Apache OpenWhisk's continued
> success.
> 
> Thanks for all your hard work that makes my job so much easier!
> 
> Ben

Re: performance tests

[Markus Thömmes]

Hi Tyson,

awesome, we had the same train of thought :). I looked into Gatling as well, 
mainly because it's in Scala so it'd be easy for us to read and build new 
scenarios. Gatling also has a Jenkins plugin (we use Jenkins) which makes it 
very easy to integrate test results.

I also liked how you define a scenario in Gatling and it seemed to fit 
OpenWhisk quite well, especially when looking at more sophisticated use-cases 
with different load-patterns.

My initial experiments are here: 
https://github.com/markusthoemmes/gatling-example. I also have a branch 
somewhere which builds a docker container around the test. That's mainly why I 
used env variables for configuration.

Yes I think we should have those in the openwhisk-performance repo. Eventually 
Gatling/Locust (have to look at that) tests might even subsume the existing 
very basic ones.

Cheers
Markus

Am 10. Mai 2017 um 06:46 schrieb Tyson Norris :

Hi -
I spent some time experimenting with gatling and locust.io 
load tests, and wanted to find out if people here have some preference?

From my limited usage, some differences are:
- gatling has some good facilities for managing complex tests and decent 
reports with little effort, compared to locust
- locust has some distributed testing capabilities (generate load from multiple 
hosts in a coordinated fashion), but is lighter on report details and test 
scenario complexity.

Branches in my fork are here, I can create pull requests if people are 
interested in pulling this into the openwhisk-performance repo?
https://github.com/tysonnorris/openwhisk-performance/tree/gatling-tests
https://github.com/tysonnorris/openwhisk-performance/tree/locust-tests
(In each branch I created a separate readme with pertinent details of that test 
harness, open to merge with main README.md before creating a PR)

Its also an option to include both, if people are interested.

Thanks
Tyson

Re: concurrent requests on actions

[Markus Thömmes]

Hi Tyson,

Sounds like you did a lot of investigation here, thanks a lot for that :)

Seeing the numbers, 4 RPS in the "off" case seem very odd. The Travis build 
that runs the current system as is also reaches 40+ RPS. So we'd need to look 
at a mismatch here.

Other than that I'd indeed suspect a great improvement in throughput from your 
work!

Implementationwise I don't have a strong opionion but it might be worth to 
discuss the details first and land your impl. once all my staging is done (the 
open PRs). That'd ease git operation. If you want to discuss your impl. now I 
suggest you send a PR to my new-containerpool branch and share the diff here 
for discussion.

Cheers,
Markus

Von meinem iPhone gesendet

> Am 01.05.2017 um 23:16 schrieb Tyson Norris :
> 
> Hi Michael -
> Concurrent requests would only reuse a running/warm container for same-action 
> requests. So if the action has bad/rogue behavior, it will limit its own 
> throughput only, not the throughput of other actions.
> 
> This is ignoring the current implementation of the activation feed, which I 
> guess is susceptible to a flood of slow running activations. If those 
> activations are for the same action, running concurrently should be enough to 
> not starve the system for other activations (with faster actions) to be 
> processed. In case they are all different actions, OR not allowed to execute 
> concurrently, then in the name of quality-of-service, it may also be 
> desirable to reserve some resources (i.e. separate activation feeds) for 
> known-to-be-faster actions, so that fast-running actions are not penalized 
> for existing alongside the slow-running actions. This would require a more 
> complicated throughput test to demonstrate.
> 
> Thanks
> Tyson
> 
> 
> 
> 
> 
> 
> 
> On May 1, 2017, at 1:13 PM, Michael Marth 
> mailto:mma...@adobe.com>> wrote:
> 
> Hi Tyson,
> 
> 10x more throughput, i.e. Being able to run OW at 1/10 of the cost - 
> definitely worth looking into :)
> 
> Like Rodric mentioned before I figured some features might become more 
> complex to implement, like billing, log collection, etc. But given such a 
> huge advancement on throughput that would be worth it IMHO.
> One thing I wonder about, though, is resilience against rogue actions. If an 
> action is blocking (in the Node-sense, not the OW-sense), would that not 
> block Node’s event loop and thus block other actions in that container? One 
> could argue, though, that this rogue action would only block other executions 
> of itself, not affect other actions or customers. WDYT?
> 
> Michael
> 
> 
> 
> 
> On 01/05/17 17:54, "Tyson Norris" 
> mailto:tnor...@adobe.com>> wrote:
> 
> Hi All -
> I created this issue some time ago to discuss concurrent requests on actions: 
> [1] Some people mentioned discussing on the mailing list so I wanted to start 
> that discussion.
> 
> I’ve been doing some testing against this branch with Markus’s work on the 
> new container pool: [2]
> I believe there are a few open PRs in upstream related to this work, but this 
> seemed like a reasonable place to test against a variety of the reactive 
> invoker and pool changes - I’d be interested to hear if anyone disagrees.
> 
> Recently I ran some tests
> - with “throughput.sh” in [3] using concurrency of 10 (it will also be 
> interesting to test with the --rps option in loadtest...)
> - using a change that checks actions for an annotation “max-concurrent” (in 
> case there is some reason actions want to enforce current behavior of strict 
> serial invocation per container?)
> - when scheduling an actions against the pool, if there is a currently “busy” 
> container with this action, AND the annotation is present for this action, 
> AND concurrent requests < max-concurrent, the this container is used to 
> invoke the action
> 
> Below is a summary (approx 10x throughput with concurrent requests) and I 
> would like to get some feedback on:
> - what are the cases for having actions that require container isolation per 
> request? node is a good example that should NOT need this, but maybe there 
> are cases where it is more important, e.g. if there are cases where stateful 
> actions are used?
> - log collection approach: I have not attempted to resolve log collection 
> issues; I would expect that revising the log sentinel marker to include the 
> activation ID would help, and logs stored with the activation would include 
> interleaved activations in some cases (which should be expected with 
> concurrent request processing?), and require some different logic to process 
> logs after an activation completes (e.g. logs emitted at the start of an 
> activation may have already been collected as part of another activation’s 
> log collection, etc).
> - advice on creating a PR to discuss this in more detail - should I wait for 
> more of the container pooling changes to get to master? Or submit a PR to 
> Markus’s new-containerpool branch?
> 
> Thanks
> Tyson
> 
> Summary of l

Re: Performance tests for OpenWhisk

[Markus Thömmes]

Hi Michael,

yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with 4+ 
cores and 8GB memory. One VM would host the management stack while one would be 
dedicated to an Invoker only. That way we could assert single-invoker 
performance the easiest.

Thanks for helping!

Cheers,
Markus

Am 26. April 2017 um 11:36 schrieb Michael Marth :

Markus,

Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.

Let me know
Michael



On 26/04/17 07:52, "Bertrand Delacretaz"  wrote:

Hi Michael,

On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth  wrote:
...Maybe our mentors can chime in. Has this been discussed in the ASF board or 
so?...

Best would be to ask the ASF infrastructure team via
us...@infra.apache.org - briefly describe what you need to see what's
possible.

-Bertrand

Performance tests for OpenWhisk

[Markus Thömmes]

Hi all,

Since many have asked I today started an effort to bring a comprehensive 
performance test suite for OpenWhisk out in the open.

Without much talking, here's the repository: 
https://github.com/markusthoemmes/openwhisk-performance

My initial idea was to have the tests run on Travis to get comparable numbers. 
Turns out, the Travis machines are way to weak (especially in terms of CPU 
cores) to host a whole OpenWhisk system and be able to perform extraordinary 
good.

The tests I did are based on the npm loadtest module, which is pretty much 
Apache Benchmark, with a few more features and nicer output. Currently there is 
only one test which surfaces end-to-end user latency of blocking invocations 
and maximum throughput. README.md has some more information on how exactly the 
tests are set up.

I do realize that those tests are the simplest you can get, but that's where 
I'd like to start. We can get more sophisticated tests any time.

To proceed from here, I'd like to have those tests run on beefier machines, to flesh out 
the real performance. Any ideas on how to do this in an open-source fashion, e.g. 
"out in the wild"?

Cheers
Markus

Re: New ContainerPool

[Markus Thömmes]

Thanks James,

I will, I already drafted a baseline post :).

Am 03. April 2017 um 13:59 schrieb James Thomas :

This looks fantastic - great work.
You should definitely write this up as a new blog post!

On 1 April 2017 at 14:05, Markus Thömmes  wrote:

Hi out there,

Over the past couple of weeks, I started working on a new ContainerPool
(and thus eventually a new Invoker). It started as a weekend investigation
into how one would write the Invoker if one started on a green field and
turned out a valuable step forward after all.

The new ContainerPool is modeled around Akka Actors and I put an emphasis
on the testability of the code. Before diving deeper into performance work
on OpenWhisk we need to be able to quickly verify new models of scheduling
and thus I abstracted the "container providing" code away from the pool
itself. One can now easily simulate different workloads without actually
talking to docker itself. A nice side-effect of this is, that the Container
interface (it's literally a trait) is completely pluggable and can also be
filled in by "insert-your-favorite-container-solution-here".

In terms of performance I did see a significant improvement in single-user
throughput performance but haven't yet got around to make a proper write-up
of the experiment's setup and methodology, hence I'll not show hard numbers
for now. We're still missing out on a common load test suite which we can
all use to verify our changes.

So all in all features:
- Eliminated concurrency behavior through the actor model
- Eliminated busy looping to get a container
- Increased testability by drawing sharp-abstraction layers and making
sure each component is testable separately
- Increased "experimentability" with scheduling algorithms (they are
encapsulated themselves)
- Performance increases very likely due implementation of a "pause grace"
(the container is not paused for a defined amount of time and can continue
its work immediately if another request comes in at that time)
- Pluggability of the container interface

The pull-request is open at https://github.com/
openwhisk/openwhisk/pull/2021. It's passing tests already. Test
coverage is still a bit low, but Sven Lange-Last and I are working to get
it done quite soon.

It will be delivered in multiple smallish pull-requests, the first of
which is open here: https://github.com/openwhisk/openwhisk/pull/2092. At
first, the new pool is going to be behind a feature flag. Once the pool has
battle proven, we can then flip the switch and remove old code as
applicable.

Feel free to have a play with it, feedback and code-reviews are very very
welcome!

Cheers,
Markus




--
Regards,
James Thomas

New ContainerPool

[Markus Thömmes]

Hi out there,
 
Over the past couple of weeks, I started working on a new ContainerPool (and 
thus eventually a new Invoker). It started as a weekend investigation into how 
one would write the Invoker if one started on a green field and turned out a 
valuable step forward after all.
 
The new ContainerPool is modeled around Akka Actors and I put an emphasis on the testability of the 
code. Before diving deeper into performance work on OpenWhisk we need to be able to quickly verify 
new models of scheduling and thus I abstracted the "container providing" code away from 
the pool itself. One can now easily simulate different workloads without actually talking to docker 
itself. A nice side-effect of this is, that the Container interface (it's literally a trait) is 
completely pluggable and can also be filled in by 
"insert-your-favorite-container-solution-here".
 
In terms of performance I did see a significant improvement in single-user 
throughput performance but haven't yet got around to make a proper write-up of 
the experiment's setup and methodology, hence I'll not show hard numbers for 
now. We're still missing out on a common load test suite which we can all use 
to verify our changes.
 
So all in all features:
- Eliminated concurrency behavior through the actor model
- Eliminated busy looping to get a container
- Increased testability by drawing sharp-abstraction layers and making sure 
each component is testable separately
- Increased "experimentability" with scheduling algorithms (they are 
encapsulated themselves)
- Performance increases very likely due implementation of a "pause grace" (the 
container is not paused for a defined amount of time and can continue its work 
immediately if another request comes in at that time)
- Pluggability of the container interface
 
The pull-request is open at https://github.com/openwhisk/openwhisk/pull/2021. 
It's passing tests already. Test coverage is still a bit low, but Sven 
Lange-Last and I are working to get it done quite soon.
 
It will be delivered in multiple smallish pull-requests, the first of which is 
open here: https://github.com/openwhisk/openwhisk/pull/2092. At first, the new 
pool is going to be behind a feature flag. Once the pool has battle proven, we 
can then flip the switch and remove old code as applicable.
 
Feel free to have a play with it, feedback and code-reviews are very very 
welcome!
 
Cheers,
Markus

Re: factoring out "runtimes" into deployment configuration

[Markus Thömmes]

What you're referring to is basically a "kind-aware" blackbox action, where you 
get the easyness of injecting your code and the flexibility of using whatever 
you want in the image as long as the interface fits.

I generally like the idea and brought it up as well once (i custom built a 
blackbox to do something similar). Note though that the same performance 
implications as with blackbox images apply.

Von meinem iPhone gesendet

> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos :
> 
> With the "runtimeManifest" property I'm wondering if we can also expose the
> name of the container image instead of having to compute it in the code ?
> Extending the thought: what if we allow users to specify a custom container
> name when creating / updating actions, in case users want to ?
> 
> I'm thinking at the use-case when a package with a group of actions
> requires multiple libraries for each action. To simplify my example I'm
> going to refer only to JS actions:
> -   Today: when actions needs extra libs users have 2 options :
> a) create a zip
> b) browserify/"compile" JS into a single JS file
> -   What I'm thinking: allow users to create their own "runtime" image
> where they can bundle those dependent libs into. All the actions in the
> package would be smaller, they would use less space and would probably flow
> easier through the system when being invoked. A potential problem in this
> case would be whether it's secure to let users push their own images in the
> OW nodes, but at the same time allowing them to upload a ZIP is
> conceptually also a "package"/"container" that can contain anything; I
> guess the security aspect would still remain even with user-defined
> containers.
> 
> Dragos
> 
> 
> 
>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth  wrote:
>> 
>> Hi Rodric,
>> 
>> I think that’s great.
>> 
>> Only comment (more to Matt than you):
>> If the runtimes are dynamic by deployment (and of course over time) this
>> probably should be reflected in the packaging format spec [1].
>> (see also previous comment [2]:)
>> 
>> - line 252: not sure if the list of runtime names should be in the
>> normative
>> section of a spec, because (as you also write) this list will fluctuate
>> between
>> deployments and over time. OTOH there should be established well known
>> names for
>> well-known runtimes. Maybe point to a community-curated, separate markdown
>> file
>> (where new entries get a PR that can be voted upon)?
>> 
>> Cheers
>> Michael
>> 
>> [1]
>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/specification/openwhisk_v0.8.pdf
>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>> 
>> 
>> 
>> On 05/03/17 17:39, "Rodric Rabbah" > rod...@gmail.com>> wrote:
>> 
>> I've been thinking about how we can add more action runtimes more easily -
>> without changing the backend (API, controller, invoker, CLI). One way I'm
>> leaning toward and started prototyping to get a better understanding of the
>> feasibility is to strip all runtime types from the backend (and CLI) and
>> encode the information about what action runtimes the system supports
>> through configuration parameters.
>> 
>> This will make it possible to decouple deploying the core components from
>> managing the action runtime images. An example of encoding the runtime
>> information in a deployment configuration is
>> https://github.com/openwhisk/openwhisk/pull/1948.
>> 
>> In support of this general direction (even if we settle on a different
>> approach), I have increasingly flattened the "Exec" type hierarchy in the
>> backend[1-3]
>> 
>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>> 
>> A sketch of how to use the runtime manifest can be seen in this WIP commit:
>> 
>> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>> 
>> I can see how using this we can add new versions of Node.js, python:3, and
>> other handy images by curating a list of these in a separate process.
>> 
>> Feedback, as always, solicited and appreciated.
>> 
>> -r
>> 
>>