Bumping this up! Hoping this can be discussed so that I get rule out that this approach has any fatal flaws.
Regards On Fri, Jan 19, 2024 at 11:58 AM Girish Sharma <scrapmachi...@gmail.com> wrote: > Hello everyone, > > A a true cloud native platform, which supports scale up and scale down, I > feel like there is a need to be able to reduce partition count in pulsar to > truly achieve a scale down after events like sales (akin to black friday, > etc) or huge temporary publish burst due to backfill. > > I looked through the archives (upto 2021) and did not find any prior > discussion on the same topic. > > I have given this an initial thought to figure out what would it need to > support such a feature in the lowest footprint possible. I am attaching the > document explaining the need, requirements and initial high level details > [0]. What I would like is to understand if the community also finds this > feature helpful and does the approach described in the document have some > fatal flaw? Summarizing the approach here as well: > > - Introduce an ability to convert a normal topic object into a > read-only topic via admin api and an additional partitioned-topic metadata > property (just like shadow source, etc) > - Add logic to block produce but allow new consumers and dispatch call > based on this flag > - Add logic in GC to clean out read only topics when all of their > ledgers expire (TTL/retention) > > Goal is that there is no data movement involved and no impact on existing > partitions during this scale down. > > Looking forward to the discussion. > > [0] > https://docs.google.com/document/d/1sbGQSwDihQftIRsxAXg5Zm4uxKQ0kRk9HadKYRFTswI/edit?usp=sharing > > Regards > -- > Girish Sharma > -- Girish Sharma
