Hi, I would like to start VOTE on PIP-136: https://github.com/apache/pulsar/issues/13728
Thanks, Rajan On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dhabalia...@gmail.com> wrote: > > >> How do we designate the host broker? Is it manual? How does it work > when the host broker is removed from the cluster? > No, it will not be manual but as I explained earlier a broker which has a > failover consumer to consume remote events will be the publisher for > metadata update. If that broker is removed then a new failover > consumer/broker will be selected for the same. > > >> I look forward to seeing more about this design for conflict resolution. > Sure, I have updated PIP to handle such race condition: > https://github.com/apache/pulsar/issues/13728 > > > >> (1) scenarios where the Pulsar cluster operators and tenant admins are > different entities and tenants can be malicious, or more probably, write > bad code that will produce malicious outcomes. > I agree, Pulsar should have provision to prevent such scenarios where > changes from one tenant in a cluster can impact other clusters. This PIP > considers the tenant/admin will be the same at both the ends but that can > not be true in all cases. We can add an enhancement later or we can create > a separate PIP to start discussion with the possible solutions. > > Thanks, > Rajan > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <joefranc...@gmail.com> wrote: > >> >On my first reading, it wasn't clear if there was only one topic >> required for this feature. I now see that the topic is not tied to a >> specific tenant or namespace. As such, we can avoid complicated >> authorization questions by putting the required event topic(s) into a >> "system" tenant and namespace >> >> We should consider complicated questions. We can say why we chose not to >> address it, or why it does not apply. for a particular situation >> >> Many namespace policies are administered by tenants. As such any tenant >> can load this topic. Is it possible for one abusive tenant to make your >> system topic dysfunctional? >> >> Pulsar committers should think about >> (1) scenarios where the Pulsar cluster operators and tenant admins are >> different entities and tenants can be malicious, or more probably, write >> bad code that will produce malicious outcomes. >> (2) whether the changes introduce additional SPOFs into the cluster. >> >> I don't think this PIP has those issues, but as a matter of practice, I >> would like to see backend/system PIPs consider these questions and >> explicitly state the conclusions with rationale >> >> Joe >> >> >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mmarsh...@apache.org> >> wrote: >> >> > Thanks for your responses. >> > >> > > I don't see a need of protobuf for this particular usecase >> > >> > If no one else feels strongly on this point, I am good with using a >> POJO. >> > >> > > It doesn't matter if it's system-topic or not because it's >> > > configurable and the admin of the system can decide and configure it >> > > according to the required persistent policy. >> > >> > On my first reading, it wasn't clear if there was only one topic >> > required for this feature. I now see that the topic is not tied to a >> > specific tenant or namespace. As such, we can avoid complicated >> > authorization questions by putting the required event topic(s) into a >> > "system" tenant and namespace, by default. The `pulsar/system` tenant >> > and namespace seem appropriate to me. >> > >> > > I would keep the system topic >> > > separate because this topic serves a specific purpose with specific >> > schema, >> > > replication policy and retention policy. >> > >> > I think we need a more formal definition for system topics. This topic >> > is exactly the kind of topic I would call a system topic: its intended >> > producers and consumers are Pulsar components. However, because >> > this feature can live on a topic in a system namespace, we can avoid >> > the classification discussion for this PIP. >> > >> > > Source region will have a broker which will create a failover >> consumer on >> > > that topic and a broker with an active consumer will watch the >> metadata >> > > changes and publish the changes to the event topic. >> > >> > How do we designate the host broker? Is it manual? How does it work >> > when the host broker is removed from the cluster? >> > >> > If we collocate the active consumer with the broker hosting the event >> > topic, can we skip creating the failover consumer? >> > >> > > PIP briefly talks about it but I will update the PIP with more >> > > explanation. >> > >> > I look forward to seeing more about this design for conflict resolution. >> > >> > Thanks, >> > Michael >> > >> > >> > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dhabalia...@gmail.com> >> > wrote: >> > > >> > > Please find my response inline. >> > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall < >> mmarsh...@apache.org> >> > > wrote: >> > > >> > > > I think this is a very appropriate direction to take Pulsar's >> > > > geo-replication. Your proposal is essentially to make the >> > > > inter-cluster configuration event driven. This increases fault >> > > > tolerance and better decouples clusters. >> > > > >> > > > Thank you for your detailed proposal. After reading through it, I >> have >> > > > some questions :) >> > > > >> > > > 1. What do you think about using protobuf to define the event >> > > > protocol? I know we already have a topic policy event stream >> > > > defined with Java POJOs, but since this feature is specifically >> > > > designed for egressing cloud providers, ensuring compact data >> transfer >> > > > would keep egress costs down. Additionally, protobuf can help make >> it >> > > > clear that the schema is strict, should evolve thoughtfully, and >> > > > should be designed to work between clusters of different versions. >> > > > >> > > >> > > >>> I don't see a need of protobuf for this particular usecase >> because >> > of >> > > two reasons: >> > > >> a. policy changes don't generate huge traffic which could be 1 >> rps >> > b. >> > > and it doesn't need performance optimization. >> > > >> It should be similar as storing policy in text instead protobuf >> > which >> > > doesn't impact footprint size or performance due to limited number of >> > > >> update operations and relatively less complexity. I agree that >> > protobuf >> > > could be another option but in this case it's not needed. Also, POJO >> > > >> can also support schema and versioning. >> > > >> > > >> > > >> > > > >> > > > 2. In your view, which tenant/namespace will host >> > > > `metadataSyncEventTopic`? Will there be several of these topics or >> is >> > > > it just hosted in a system tenant/namespace? This question gets back >> > > > to my questions about system topics on this mailing list last week >> > [0]. I >> > > > view this topic as a system topic, so we'd need to make sure that it >> > > > has the right authorization rules and that it won't be affected by >> > calls >> > > > like "clearNamespaceBacklog". >> > > >> > > >> > > >> It doesn't matter if it's system-topic or not because it's >> > > configurable and the admin of the system can decide and configure it >> > > according to the required persistent policy. I would keep the system >> > topic >> > > separate because this topic serves a specific purpose with specific >> > schema, >> > > replication policy and retention policy. >> > > >> > > >> > > >> > > > >> > > > 3. Which broker will host the metadata update publisher? I assume we >> > > > want the producer to be collocated with the bundle that hosts the >> > > > event topic. How will this be coordinated? >> > > > >> > > >> It's already explained into PIP in section: "Event publisher and >> > handler" >> > > >> Every isolated cluster deployed on a separate cloud platform will >> > have a >> > > source region and part of replicated clusters for the event topic. The >> > > Source region will have a broker which will create a failover >> consumer on >> > > that topic and a broker with an active consumer will watch the >> metadata >> > > changes and publish the changes to the event topic. >> > > >> > > >> > > >> > > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the topic >> level >> > > > policies already have this feature? If so, is there a way to >> integrate >> > > > this feature with the existing topic policy feature? >> > > > >> > > >> Yes, ResourceType can be extensible to a topic as well. >> > > >> > > >> > > >> > > > >> > > > 5. By decentralizing the metadata store, it looks like there is a >> > > > chance for conflicts due to concurrent updates. How do we handle >> those >> > > > conflicts? >> > > > >> > > >> PIP briefly talks about it but I will update the PIP with more >> > > explanation. MetadataChangeEvent contains source-cluster and updated >> > time. >> > > Also, resources Tenant/Namespace will also contain lastUpdatedTime >> which >> > > will help to destination clusters to handle stale/duplicate events and >> > race >> > > conditions. Also, snapshot-sync an additional task helps all clusters >> to >> > be >> > > synced with each other eventually. >> > > >> > > >> > > >> > > > I'll also note that I previously proposed a system event topic here >> > > > [1] and it was proposed again here [2]. Those features were for >> > > > different use cases, but ultimately looked very similar. In my >> view, a >> > > > stream of system events is a very natural feature to expect in a >> > > > streaming technology. I wonder if there is a way to generalize this >> > > > feature to fulfill local cluster consumers and geo-replication >> > > > consumers. Even if this PIP only implements the geo-replication >> > > > portion of the feature, it'd be good to design it in an extensible >> > fashion. >> > > > >> > > >> I think answer (2) addresses this concern as well. >> > > >> > > >> > > >> > > > Thanks, >> > > > Michael >> > > > >> > > > [0] >> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17 >> > > > [1] >> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871 >> > > > [2] >> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx >> > > > >> > > > >> > > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia < >> rdhaba...@apache.org> >> > > > wrote: >> > > > > >> > > > > Hi, >> > > > > >> > > > > I would like to start a discussion about PIP-136: Sync Pulsar >> > policies >> > > > > across multiple clouds. >> > > > > >> > > > > PIP documentation: https://github.com/apache/pulsar/issues/13728 >> > > > > >> > > > > *Motivation* >> > > > > Apache Pulsar is a cloud-native, distributed messaging framework >> > which >> > > > > natively provides geo-replication. Many organizations deploy >> pulsar >> > > > > instances on-prem and on multiple different cloud providers and at >> > the >> > > > same >> > > > > time they would like to enable replication between multiple >> clusters >> > > > > deployed in different cloud providers. Pulsar already provides >> > various >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to >> > > > fulfill >> > > > > security requirements when brokers are deployed on different >> security >> > > > zones >> > > > > connected with each other. However, sometimes it's not possible to >> > share >> > > > > metadata-store (global zookeeper) between pulsar clusters >> deployed on >> > > > > separate cloud provider platforms, and synchronizing configuration >> > > > metadata >> > > > > (policies) can be a critical path to share tenant/namespace/topic >> > > > policies >> > > > > between clusters and administrate pulsar policies uniformly across >> > all >> > > > > clusters. Therefore, we need a mechanism to sync configuration >> > metadata >> > > > > between clusters deployed on the different cloud platforms. >> > > > > >> > > > > *Sync Pulsar policies across multiple clouds* >> > > > > https://github.com/apache/pulsar/issues/13728 >> > > > > Prototype git-hub-link >> > > > > < >> > > > >> > >> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6 >> > > > > >> > > > > Thanks, >> > > > > Rajan >> > > > >> > >> >
