Thanks for summarizing Markus.
Yes this is confusing in context of current system, which stores in kafka, but
not to indefinitely wait, since timeout begins immediately
So, I think the problem of buffering/queueing is: when does the timeout begin?
If not everything is buffered the same, their timeout should not begin until
processing begins.
Maybe it would make sense to:
* always buffer (indefinitely) to queue for async, never for sync
* timeout for async not started till read from queue - which may be delayed
from time of trigger or http request
* this should also come with some system monitoring to indicate the queue
processing is not keeping up with some configurable max delay threshold ("I
can’t tolerate delays of > 5 minutes", etc)
* ContainerRouters can only pull from async queue when
* increasing the number of pending activations won’t exceed some
threshold (prevent excessive load of async on ContainerRouters)
* ContainerManager is not overloaded (can still create containers, or
has some configurable way to indicate the cluster is healthy enough to cope
with extra processing)
We could of course make this configurable so that operators can choose to:
* treat async/sync activations the same for sync/async (the overloaded system
fails when either ContainerManager or ContainerRouters are max capacity)
* treat async/sync with preference for:
* sync - where async is buffered for unknown period before processing,
incoming sync traffic (or lack of)
* async - where sync is sent to the queue, to be processed in order of
receipt interleaved with async traffic (similar to today, I think)
I think the impact here (aside from technical) is the timing difference if we
introduce latency in side affects based on the activation being sync vs async.
I’m also not sure prioritizing message processing between sync/async internally
in ContainerRouter is better than just have some dedicated ContainerRouters
that receive all async activations, and others that receive all sync
activations, but the end result is the same, I think.
> On Aug 19, 2018, at 4:29 AM, Markus Thömmes <markusthoem...@apache.org> wrote:
>
> Hi Tyson, Carlos,
>
> FWIW I should change that to no longer say "Kafka" but "buffer" or "message
> queue".
>
> I see two use-cases for a queue here:
> 1. What you two are alluding to: Buffering asynchronous requests because of
> a different notion of "latency sensitivity" if the system is in an overload
> scenario.
> 2. As a work-stealing type balancing layer between the ContainerRouters. If
> we assume round-robin/least-connected (essentially random) scheduling
> between ContainerRouters, we will get load discrepancies between them. To
> smoothen those out, a ContainerRouter can put the work on a queue to be
> stolen by a Router that actually has space for that work (for example:
> Router1 requests a new container, puts the work on the queue while it waits
> for that container, Router2 already has a free container and executes the
> action by stealing it from the queue). This does has the added complexity
> of breaking a streaming communication between User and Container (to
> support essentially unbounded payloads). A nasty wrinkle that might render
> this design alternative invalid! We could come up with something smarter
> here, i.e. only putting a reference to the work on the queue and the
> stealer connects to the initial owner directly which then streams the
> payload through to the stealer, rather than persisting it somewhere.
>
> It is important to note, that in this design, blocking invokes could
> potentially gain the ability to have unbounded entities, where
> trigger/non-blocking invokes might need to be subject to a bound here to be
> able to support eventual execution efficiently.
>
> Personally, I'm much more torn to the work-stealing type case. It implies a
> wholy different notion of using the queue though and doesn't have much to
> do with the way we use it today, which might be confusing. It could also
> well be the case, that work-stealing type algorithms are easier to back on
> a proper MQ vs. trying to make it work on Kafka.
>
> It might also be important to note that those two use-cases might require
> different technologies (buffering vs. queue-backend for work-stealing) and
> could well be seperated in the design as well. For instance, buffering
> triggers fires etc. does not necessarily need to be done on the execution
> layer but could instead be pushed to another layer. Having the notion of
> "async" vs "sync" in the execution layer could be benefitial for
> loadbalancing itself though. Something worth exploring imho.
>
> Sorry for the wall of text, I hope this clarifies things!
>
> Cheers,
> Markus
>
> Am Sa., 18. Aug. 2018 um 02:36 Uhr schrieb Carlos Santana <
> csantan...@gmail.com>:
>
>> triggers get responded right away (202) with an activation is and then
>> sent to the queue to be processed async same as async action invokes.
>>
>> I think we would keep same contract as today for this type of activations
>> that are eventually process different from blocking invokes including we
>> Actions were the http client hold a connection waiting for the result back.
>>
>> - Carlos Santana
>> @csantanapr
>>
>>> On Aug 17, 2018, at 6:14 PM, Tyson Norris <tnor...@adobe.com.INVALID>
>> wrote:
>>>
>>> Hi -
>>> Separate thread regarding the proposal: what is considered for routing
>> activations as overload and destined for kafka?
>>>
>>> In general, if kafka is not on the blocking activation path, why would
>> it be used at all, if the timeouts and processing expectations of blocking
>> and non-blocking are the same?
>>>
>>> One case I can imagine: triggers + non-blocking invokes, but only in the
>> case where those have some different timeout characteristics. e.g. if a
>> trigger fires an action, is there any case where the activation should be
>> buffered to kafka if it will timeout same as a blocking activation?
>>>
>>> Sorry if I’m missing something obvious.
>>>
>>> Thanks
>>> Tyson
>>>
>>>
>>
