Hi guys,
If there are no objections, I will send out the vote.
Best,
Hang
Hang Chen <chenh...@apache.org> 于2024年5月6日周一 22:23写道：
>
> >I see, but when a broker loads a bundle with many topics, you trigger
> immediately offloading for them, even if they don't have much traffic.
> This is kind of a "bad surprise" when you update a cluster to the newer
> version and you don't expect load to spike unexpectedly.
> My concerns is only about the upgrade path for users coming from older
> versions
>
> I see. That makes sense. Let me add a flag to control this feature.
>
> Thanks,
> Hang
>
> Enrico Olivelli <eolive...@gmail.com> 于2024年5月6日周一 15:21写道：
> >
> > Il giorno lun 6 mag 2024 alle ore 04:15 Hang Chen <chenh...@apache.org> ha
> > scritto:
> >
> > > Hi Enrico,
> > > I agree with Jiuming, we do not need a flag to control this
> > > behavior. We will call `maybeOffloadInBackground` method in the
> > > managedledger init stage. This method only checks the offload policies
> > > to decide if we need to trigger offload. If the offload conditions are
> > > met, use the scheduled executor to trigger offload. The whole check
> > > process has no blocking calls and won't impact the managedledger init.
> > >
> > > https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
> > >
> > >
> >
> > I see, but when a broker loads a bundle with many topics, you trigger
> > immediately offloading for them, even if they don't have much traffic.
> > This is kind of a "bad surprise" when you update a cluster to the newer
> > version and you don't expect load to spike unexpectedly.
> > My concerns is only about the upgrade path for users coming from older
> > versions
> >
> > Maybe it is not a big deal ?
> >
> > Enrico
> >
> >
> > >
> > > Thanks,
> > > Hang
> > >
> > > 太上玄元道君 <dao...@apache.org> 于2024年5月6日周一 00:31写道：
> > > >
> > > > hi Enrico,
> > > >
> > > > I still think it doesn't matter
> > > >
> > > > 1. Offload a topic or not, configured by
> > > >
> > > `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> > > > even though we may trigger a bunch of stuff, it's expected behavior
> > > >
> > > > 2. After triggered to offload a topic, the offload task will be added to
> > > an
> > > > Executor, currently, we have 2 threads to execute offload with sync
> > > > mode,
> > > > so, even though there are thousands of topics, only 2 tasks can be
> > > executed
> > > > at the same time, other tasks wait.
> > > >
> > > > 3. If you are worrying about system overwhelmed, there is a PR to
> > > introduce
> > > > offload throttling(https://github.com/apache/pulsar/pull/22385),
> > > > it is for the purpose of protect the broker, it has been pending in
> > > review
> > > > for a loooong time, I'm very appreciate it that if you can help review
> > > >
> > > > 4. There are toooo many flags in pulsar(~500), I think we should be
> > > > cautious when introducing new FLAGs
> > > >
> > > > Thanks,
> > > > Tao Jiuming
> > > >
> > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日13:44写道：
> > > >
> > > > > daojun,
> > > > >
> > > > > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <dao...@apache.org> ha
> > > > > scritto:
> > > > >
> > > > > > it doesn't matter, we don't need to wait until offload finished when
> > > > > > loading topic, since offloading may take a long time.
> > > > > > it just trigger offloading, and we don't need to handle the result.
> > > > > >
> > > > >
> > > > > We don't have to wait for the results, but we still trigger a bunch of
> > > > > stuff and if the broker has thousand of topics, there may be an herd
> > > effect
> > > > > as this is a behavior change, I think it is safest to have a flag to
> > > > > disable this new behavior,
> > > > > otherwise upgrading an existing big cluster may lead to unexpected
> > > results,
> > > > > like overwhelmed systems
> > > > >
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日02:41写道：
> > > > > >
> > > > > > > Overall I think it is a good idea.
> > > > > > >
> > > > > > > What about adding a flag to enable this behavior?
> > > > > > >
> > > > > > > In case there are problems with the offloader the topic won't be
> > > able
> > > > > to
> > > > > > > load.
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <chenh...@apache.org> ha
> > > scritto:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > > I pushed a new proposal to improve the tiered storage
> > > offload
> > > > > > > > efficiency. Please help take a look, thanks.
> > > > > > > >
> > > > > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Hang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
