> we do not know anything about the availability of the owner of the topic.
If the owner broker is not available, other brokers will take over. > We could make it simpler and when a broker wants to push its data, it looks up the REST address of the "leader broker" and then pushes the data to it, I mean, without involving a "topic" Any broker may become the leader broker, in this case, the brokers need to know all the addresses of the brokers in the cluster. With the topic approach, they only need to know the topic name. Penghui On Thu, Mar 17, 2022 at 12:35 AM Enrico Olivelli <eolive...@gmail.com> wrote: > But in order to read from a topic you need a broker that is the owner of > the owner of the special "temporary topic". > > While the metadata service (ZooKeeper) is already a central point and it is > meant to be available (otherwise Pulsar doesn't work), we do not know > anything about the availability of the owner of the topic. > > Or do you mean to create a special topic that is always owned by the > "leader broker" ? > > We could make it simpler and when a broker wants to push its data, it looks > up the REST address of the "leader broker" and then pushes the data to it, > I mean, without involving a "topic". > > > Enrico > > > > Il Mer 16 Mar 2022, 12:55 PengHui Li <peng...@apache.org> ha scritto: > > > +1 > > > > The load data don't need to be persistent to the storage layer, > > Using a non-persistent topic is more efficient. > > > > Thanks, > > Penghui > > > > On Wed, Mar 16, 2022 at 2:14 PM Kai Wang <kw...@streamnative.io.invalid> > > wrote: > > > > > Hi Pulsar Community, > > > > > > Currently, Pulsar LoadManager is using Zookeeper to store the local > > broker > > > data, the LoadReportUpdaterTask will report the local load data to > > > Zookeeper, the leader broker will collect load data and store it to > > > Zookeeper. > > > > > > When we have a lot of brokers and bundles, this load datas will put > some > > > pressure on Zookeeper. > > > > > > Since the load data are not strongly consistent, we can use the > > > non-persistent topics to sync the load data. And it will reduce our > > > dependence on Zookeeper. > > > > > > If this proposal is acceptable, I will draft a PIP. > > > > > > Any suggestions are appreciated. > > > > > > Thanks, > > > Kai > > > > > >
