Hi, Lari. Thanks for your proposal. Sounds good to me.
Supporting function pluggable essentially decouples the scheduling layer and the computing layer. This can not only extend the function runtime in more languages but even use this feature to implement a new non-function runtime plugin, such as the EventSouring development model. Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. At present, the scheduling logic of the function will not be related to the key of the message, I understand that it belongs to the internal feature of the runtime, and this can also be enhanced based on the current runtime? I don't quite understand what this has to do with runtime pluginization. Can you explain? In addition, the Pulsar Functions have a missing piece in how functions are > mapped to instances. It's not very efficient to even run each and every > function as a separate deployable entity. The cost of each independent JVMs > is high. It would be also better to have a model where where could be a > group of functions that are provided by one instance and always run > together. Having this option could bring down the cost and also improve the > developer experience. The framework shouldn't require the developer that > each individual function is deployed in a separate .jar file which gets run > in a separate JVM. This means that we need to enhance the function interface to that the user tells the runtime layer which topic the function subscription cares about. In addition, if we want to implement concurrent sequential processing based on keys, we can also let the function register the key of interest. These can be implemented in the current runtime, which has nothing to do with plug-in runtime, right? I wonder what are the future plans after plugging in the runtime? Do you want to develop a new runtime that implements many of the benefits you enumerate, or will you continue to enhance it on the original runtime (Java). Thanks, Baodi Shi On Jun 20, 2023 at 15:33:22, Lari Hotari <lhot...@apache.org> wrote: > Dear Pulsar Community Members, > > I would like to initiate a discussion on making the Pulsar Functions > runtime "pluggable". In doing so, we can ensure that the addition of new > runtime types becomes more straightforward. > > This use case will allow us to add support for Pulsar Functions based on > various platforms such as: > > * Pulsar Client Reactive > * Node.js / JavaScript > * WebAssembly (WASM) > * Spring Pulsar & Reactive Spring > > One of the weak points in the current Pulsar Functions runtime is the > default handling of messages individually. Individual message processing > can be slow and inefficient in cases where the main function of the > Pulsar Function (or Sink) is to do backend API calls. > > Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. > > The Reactive Pulsar Client provides an inbuilt solution for implementing > pipelining. With its ReactiveMessagePipelineBuilder, we can configure > concurrency levels with key-ordered processing support. This capability > could potentially eliminate the need to use key-shared subscriptions to > scale Pulsar processing. If a reactive Pulsar Function were primarily to > serve as a router for API calls, we could adjust the concurrency level > to hundreds or even thousands, provided the backend could handle the > load. > > With a pluggable Pulsar Functions runtime, we could introduce new > runtime types without the need for implementing each type in the > upstream project. This strategy could likely lead to new opportunities > for innovative ideas and contributions in this field. > > I am interested to know your thoughts on making the Pulsar Functions > runtime pluggable so that we can add new runtime types. > > Best Regards, > > -Lari >
