Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <chenh...@apache.org> ha scritto:
> Hi Enrico, > I agree with Jiuming, we do not need a flag to control this > behavior. We will call `maybeOffloadInBackground` method in the > managedledger init stage. This method only checks the offload policies > to decide if we need to trigger offload. If the offload conditions are > met, use the scheduled executor to trigger offload. The whole check > process has no blocking calls and won't impact the managedledger init. > > https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489 > > I see, but when a broker loads a bundle with many topics, you trigger immediately offloading for them, even if they don't have much traffic. This is kind of a "bad surprise" when you update a cluster to the newer version and you don't expect load to spike unexpectedly. My concerns is only about the upgrade path for users coming from older versions Maybe it is not a big deal ? Enrico > > Thanks, > Hang > > 太上玄元道君 <dao...@apache.org> 于2024年5月6日周一 00:31写道： > > > > hi Enrico, > > > > I still think it doesn't matter > > > > 1. Offload a topic or not, configured by > > > `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`, > > even though we may trigger a bunch of stuff, it's expected behavior > > > > 2. After triggered to offload a topic, the offload task will be added to > an > > Executor, currently, we have 2 threads to execute offload with sync mode, > > so, even though there are thousands of topics, only 2 tasks can be > executed > > at the same time, other tasks wait. > > > > 3. If you are worrying about system overwhelmed, there is a PR to > introduce > > offload throttling(https://github.com/apache/pulsar/pull/22385), > > it is for the purpose of protect the broker, it has been pending in > review > > for a loooong time, I'm very appreciate it that if you can help review > > > > 4. There are toooo many flags in pulsar(~500), I think we should be > > cautious when introducing new FLAGs > > > > Thanks, > > Tao Jiuming > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日13:44写道： > > > > > daojun, > > > > > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <dao...@apache.org> ha > > > scritto: > > > > > > > it doesn't matter, we don't need to wait until offload finished when > > > > loading topic, since offloading may take a long time. > > > > it just trigger offloading, and we don't need to handle the result. > > > > > > > > > > We don't have to wait for the results, but we still trigger a bunch of > > > stuff and if the broker has thousand of topics, there may be an herd > effect > > > as this is a behavior change, I think it is safest to have a flag to > > > disable this new behavior, > > > otherwise upgrading an existing big cluster may lead to unexpected > results, > > > like overwhelmed systems > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日02:41写道： > > > > > > > > > Overall I think it is a good idea. > > > > > > > > > > What about adding a flag to enable this behavior? > > > > > > > > > > In case there are problems with the offloader the topic won't be > able > > > to > > > > > load. > > > > > > > > > > Enrico > > > > > > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <chenh...@apache.org> ha > scritto: > > > > > > > > > > > Hi guys, > > > > > > I pushed a new proposal to improve the tiered storage > offload > > > > > > efficiency. Please help take a look, thanks. > > > > > > > > > > > > PIP: https://github.com/apache/pulsar/pull/22650 > > > > > > > > > > > > Thanks, > > > > > > Hang > > > > > > > > > > > > > > > > > > >
