hi Enrico, I still think it doesn't matter
1. Offload a topic or not, configured by `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`, even though we may trigger a bunch of stuff, it's expected behavior 2. After triggered to offload a topic, the offload task will be added to an Executor, currently, we have 2 threads to execute offload with sync mode, so, even though there are thousands of topics, only 2 tasks can be executed at the same time, other tasks wait. 3. If you are worrying about system overwhelmed, there is a PR to introduce offload throttling(https://github.com/apache/pulsar/pull/22385), it is for the purpose of protect the broker, it has been pending in review for a loooong time, I'm very appreciate it that if you can help review 4. There are toooo many flags in pulsar(~500), I think we should be cautious when introducing new FLAGs Thanks, Tao Jiuming Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日13:44写道： > daojun, > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <dao...@apache.org> ha > scritto: > > > it doesn't matter, we don't need to wait until offload finished when > > loading topic, since offloading may take a long time. > > it just trigger offloading, and we don't need to handle the result. > > > > We don't have to wait for the results, but we still trigger a bunch of > stuff and if the broker has thousand of topics, there may be an herd effect > as this is a behavior change, I think it is safest to have a flag to > disable this new behavior, > otherwise upgrading an existing big cluster may lead to unexpected results, > like overwhelmed systems > > > Enrico > > > > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日02:41写道： > > > > > Overall I think it is a good idea. > > > > > > What about adding a flag to enable this behavior? > > > > > > In case there are problems with the offloader the topic won't be able > to > > > load. > > > > > > Enrico > > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <chenh...@apache.org> ha scritto: > > > > > > > Hi guys, > > > > I pushed a new proposal to improve the tiered storage offload > > > > efficiency. Please help take a look, thanks. > > > > > > > > PIP: https://github.com/apache/pulsar/pull/22650 > > > > > > > > Thanks, > > > > Hang > > > > > > > > > >
