It looks like the purpose of the StreamingDispatcher was to improve performance by implementing an improved readahead mechanism. Was it actually able to accomplish this objective? If so, why do we need a separate code path for it instead of updating the existing dispatcher to use the improvement? It's not clear to me if the readahead and microbatching are mutually exclusive concepts, and microbatching tends to improve throughput in production, so it seems in theory like both optimizations would be desirable if they can be combined.
However, I haven't looked deeply into the feasibility of this, so I'd like to hear more from those with deeper understanding of these code paths. On Tue, Apr 4, 2023, 2:57 AM Yunze Xu <y...@streamnative.io.invalid> wrote: > If the flaky tests were the only concern, I think we can just disable > these tests. Whatever, this config in `ServiceConfiguration` has > existed for a long time, though when it was introduced, the PIP rule > was not clear so there is no PIP for it. > > Thanks, > Yunze > > On Tue, Apr 4, 2023 at 3:09 PM Gavin gao <gaozhang...@gmail.com> wrote: > > > > +1, I totally agree with this idea. > > > > Enrico Olivelli <eolive...@gmail.com> 于2023年4月4日周二 14:47写道： > > > > > Hello, > > > It has been a long time that we have in the Pulsar code a new > > > experimental Dispatcher implementation named StreamingDispatcher. > > > > > > https://github.com/apache/pulsar/pull/9056 > > > > > > There are many flaky tests about that feature and I believe that it > > > has never been used in Production by anyone, because it happened a few > > > times that we did some changes in the regular Dispatcher and > > > introduced bugs on the StreamingDispacther (usually manifested as > > > flaky tests) > > > > > > > > > I propose to drop the StreamingDispatcher code for Pulsar 3.0. > > > I don't think we need a PIP for this, it is an experimental code that > > > was never delivered as a production ready feature. > > > > > > If anyone is aware of users please chime in. > > > > > > If anyone wants to sponsor that feature and objects in removing this > > > dead code (that we still have to maintain) please help us in > > > completing the feature. > > > > > > On paper it is a very appealing feature, and I am disappointed in > dropping > > > it. > > > On the other hand, this is dead code that we have to maintain with zero > > > benefit > > > > > > Thoughts ? > > > > > > Enrico > > > >
