To clarify, historical profile that is persisted in ZK On Thu, Mar 17, 2022 at 11:54 AM Joe F <joefranc...@gmail.com> wrote:
> IIRC, there is a historical load profile for topic that feeds into > decisions by the load balancer. > > What happens during a cluster startup, with this new proposal? > > > > > On Thu, Mar 17, 2022 at 7:50 AM PengHui Li <peng...@apache.org> wrote: > >> > But which brokers will own that topic ? >> in a Pulsar cluster with a high level of isolation of tenants, we must >> ensure that: >> - at least one broker is allowed to own the topic >> - brokers dedicated to tenants do not own the topic >> With the current approach the data in on zookeeper, and this is shared >> among all the brokers >> >> We have "pulsar/system" namespace which can be used to maintain >> system topics. If users consider broker isolation, it's all transparent. >> >> Using a topic we also can shared the data among all brokers. >> Who want a data copy, only need to create a reader when starting. >> And we have introduced table view, which will make it easier to cache >> the load data, and perform the load cache update. >> >> > Another point: >> will users be allowed to produce/consume this topic ? how do we deal >> with permissions = >> >> Good point. We should avoid the user's producers/consumers, and only >> the super user can access the system topic. >> >> Thanks, >> Penghui >> >> On Thu, Mar 17, 2022 at 10:08 PM Enrico Olivelli <eolive...@gmail.com> >> wrote: >> >> > Il giorno gio 17 mar 2022 alle ore 02:42 PengHui Li >> > <peng...@apache.org> ha scritto: >> > > >> > > > we do not know >> > > anything about the availability of the owner of the topic. >> > > >> > > If the owner broker is not available, other brokers will take over. >> > > >> > > > We could make it simpler and when a broker wants to push its data, >> it >> > > looks >> > > up the REST address of the "leader broker" and then pushes the data to >> > it, >> > > I mean, without involving a "topic" >> > > >> > > Any broker may become the leader broker, in this case, the brokers >> need >> > to >> > > know all the addresses of the brokers in the cluster. With the topic >> > > approach, >> > > they only need to know the topic name. >> > >> > I thought about this a little more. >> > Using a non persistent topic makes sense. So I am closer to be >> > convinced about this move. >> > >> > But which brokers will own that topic ? >> > in a Pulsar cluster with a high level of isolation of tenants, we must >> > ensure that: >> > - at least one broker is allowed to own the topic >> > - brokers dedicated to tenants do not own the topic >> > With the current approach the data in on zookeeper, and this is shared >> > among all the brokers >> > >> > Another point: >> > will users be allowed to produce/consume this topic ? how do we deal >> > with permissions = >> > >> > >> > Enrico >> > >> > > >> > > Penghui >> > > >> > > On Thu, Mar 17, 2022 at 12:35 AM Enrico Olivelli <eolive...@gmail.com >> > >> > > wrote: >> > > >> > > > But in order to read from a topic you need a broker that is the >> owner >> > of >> > > > the owner of the special "temporary topic". >> > > > >> > > > While the metadata service (ZooKeeper) is already a central point >> and >> > it is >> > > > meant to be available (otherwise Pulsar doesn't work), we do not >> know >> > > > anything about the availability of the owner of the topic. >> > > > >> > > > Or do you mean to create a special topic that is always owned by the >> > > > "leader broker" ? >> > > > >> > > > We could make it simpler and when a broker wants to push its data, >> it >> > looks >> > > > up the REST address of the "leader broker" and then pushes the data >> to >> > it, >> > > > I mean, without involving a "topic". >> > > > >> > > > >> > > > Enrico >> > > > >> > > > >> > > > >> > > > Il Mer 16 Mar 2022, 12:55 PengHui Li <peng...@apache.org> ha >> scritto: >> > > > >> > > > > +1 >> > > > > >> > > > > The load data don't need to be persistent to the storage layer, >> > > > > Using a non-persistent topic is more efficient. >> > > > > >> > > > > Thanks, >> > > > > Penghui >> > > > > >> > > > > On Wed, Mar 16, 2022 at 2:14 PM Kai Wang >> > <kw...@streamnative.io.invalid> >> > > > > wrote: >> > > > > >> > > > > > Hi Pulsar Community, >> > > > > > >> > > > > > Currently, Pulsar LoadManager is using Zookeeper to store the >> local >> > > > > broker >> > > > > > data, the LoadReportUpdaterTask will report the local load data >> to >> > > > > > Zookeeper, the leader broker will collect load data and store >> it to >> > > > > > Zookeeper. >> > > > > > >> > > > > > When we have a lot of brokers and bundles, this load datas will >> put >> > > > some >> > > > > > pressure on Zookeeper. >> > > > > > >> > > > > > Since the load data are not strongly consistent, we can use the >> > > > > > non-persistent topics to sync the load data. And it will reduce >> our >> > > > > > dependence on Zookeeper. >> > > > > > >> > > > > > If this proposal is acceptable, I will draft a PIP. >> > > > > > >> > > > > > Any suggestions are appreciated. >> > > > > > >> > > > > > Thanks, >> > > > > > Kai >> > > > > > >> > > > > >> > > > >> > >> >
