rdhabalia opened a new issue #13728: URL: https://github.com/apache/pulsar/issues/13728
# Sync Pulsar policies across multiple clouds ## Motivation Apache Pulsar is a cloud-native, distributed messaging framework which natively provides geo-replication. Many organizations deploy pulsar instances on-prem and on multiple different cloud providers and at the same time they would like to enable replication between multiple clusters deployed in different cloud providers. Pulsar already provides various proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to fulfill security requirements when brokers are deployed on different security zones connected with each other.  However, sometimes it's not possible to share metadata-store (global zookeeper) between pulsar clusters deployed on separate cloud provider platforms, and synchronizing configuration metadata (policies) can be a critical path to share tenant/namespace/topic policies between clusters and administrate pulsar policies uniformly across all clusters. Therefore, we need a mechanism to sync configuration metadata between clusters deployed on the different cloud platforms. ## Goal ### Replicated metadata event topic All regions in a cluster that share the same metadata-store (eg: global zookeeper which persists policies) are already in sync but they are not in sync with regions which are in different clusters and not sharing the same metadata-store. We want to sync clusters that are not sharing the same config metadata store and in order to synchronize metadata store, we can pick one region from each cluster and set up a replicated topic across those regions where they can exchange change of metadata event and try to sync cluster with the change of events occurred at different clusters. This PIP will introduce metadata event-topic which will be replicated between one region in every cloud platform or clusters that are not in sync and don’t share the same metadata store. The one region in every cluster/cloud-platform which will be responsible to generate local change events is known as a source region. Each source-region in a cluster will generate a local change event to this replicated topic and this event eventually will be replicated to all other clusters present into other cloud platforms and those clusters can update their metadata stores with the new change of event. In some cases, the source cluster might miss generating change events, or a new cluster is added which needs to be synchronized with metadata stored into another existing cluster. This solution also allows triggering snapshot synchronization to sync metadata-store into the new cluster. Below data structure shows the payload of change events published into the event topic. Each event contains metadata of individual namespace/tenant/topic along with source cluster name and updated time of the event. Source-cluster name and updateTime helps destination clusters to handle stale or duplicate events. **Event** ``` public class MetadataChangeEvent { private EventType type; private ResourceType resource; private String resourceName; private byte[] data; private String sourceCluster; private long updateTime; public enum EventType { Created, Modified, Deleted; } public enum ResourceType { Tenants, Namespaces; } } ``` ## Implementation ### Event publisher and handler Every isolated cluster deployed on a separate cloud platform will have a source region and part of replicated clusters for the event topic. The Source region will have a broker which will create a failover consumer on that topic and a broker with an active consumer will watch the metadata changes and publish the changes to the event topic. The active consumer of the topic consumes the event and updates the metadata to its metadata store cluster. It also allows snapshot synchronization to sync metadata-store into the new cluster by scheduling a job that publishes snapshot of each policy periodically.  ### Broker changes **Configuration** ``` # topic name to share policy changes private String metadataSyncEventTopic; # frequency of generating snapshot for stored policy to the event topic private long metadataSyncSnapshotDurationSecond; ``` **Tenant/Namespace metadata** Tenant and Namespace metadata will have below optional fields if the sync metadata feature is enabled. These fields are used to handle duplicate and stale events. ``` String lastUpdatedBy; // source cluster name long lastUpdatedTime;// when source cluster has updated the metadata. ``` ### Event topic consumer and publisher User can enable this feature by configuring metadataSyncEventTopic into broker and broker initializes MetadataPolicySyncer component which creates failover consumer to listen and handle metadata’s change events. It also sets the watch on metadata changes and publishes those changes to event topic so, other cluster’s source region can consume those events and sync their local metadata store. ### Prototype [git-hub-link](https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
