codelipenghui commented on code in PR #24368: URL: https://github.com/apache/pulsar/pull/24368#discussion_r2121371920
########## pip/pip-422.md: ########## @@ -0,0 +1,103 @@ +# PIP-422: Support global topic-level policy: replicated clusters and new API to delete topic-level policies + +# Background knowledge + +Users have two choices of config metadata store when using the feature Geo-Replication: shared or not. +All clusters share data that are stored in global config metadata store if users choose to use a shared metadata store. +By the way, the share config metadata store often is build crossing multi zone/region to confirm fault tolerance. + +Since the data that are stored in the shared metadata store contains the following pulsar resources: tenants, namespaces, partitioned topic, +the Admin API can not be used to delete these pulsar resources for a single cluster. +Pulsar has a feature that used to delete namespace level resources for a specify cluster when using a shared metadata store, it works as follows: +- The original namespace-level policy that named `replicated clusters` contains two clusters `c1` and `c2`. +- The topics under the cluster `c1` will be deleted automatically when users remove `c1` from the namespace-level policy. + +The feature above very useful at the following scenarios: +- Transfer a namespace from a old cluster to a new cluster: at last, the namespace should be deleted from the old cluster. +- Delete topics: since Pulsar does not allow deleting topics when enabled Geo-Replication, users should disable replication first. + +# Motivation + +When using shared metadata store and enabled Geo-Replication, there is no mechanism for the following scenarios: +- Transfer a partitioned topic from a old cluster to a new cluster. +- Delete a partitioned topic form a specify cluster. Review Comment: ```suggestion - Delete a partitioned topic from a specify cluster. ``` ########## pip/pip-422.md: ########## @@ -0,0 +1,103 @@ +# PIP-422: Support global topic-level policy: replicated clusters and new API to delete topic-level policies + +# Background knowledge + +Users have two choices of config metadata store when using the feature Geo-Replication: shared or not. +All clusters share data that are stored in global config metadata store if users choose to use a shared metadata store. +By the way, the share config metadata store often is build crossing multi zone/region to confirm fault tolerance. + +Since the data that are stored in the shared metadata store contains the following pulsar resources: tenants, namespaces, partitioned topic, +the Admin API can not be used to delete these pulsar resources for a single cluster. +Pulsar has a feature that used to delete namespace level resources for a specify cluster when using a shared metadata store, it works as follows: +- The original namespace-level policy that named `replicated clusters` contains two clusters `c1` and `c2`. +- The topics under the cluster `c1` will be deleted automatically when users remove `c1` from the namespace-level policy. + +The feature above very useful at the following scenarios: +- Transfer a namespace from a old cluster to a new cluster: at last, the namespace should be deleted from the old cluster. +- Delete topics: since Pulsar does not allow deleting topics when enabled Geo-Replication, users should disable replication first. + +# Motivation + +When using shared metadata store and enabled Geo-Replication, there is no mechanism for the following scenarios: +- Transfer a partitioned topic from a old cluster to a new cluster. +- Delete a partitioned topic form a specify cluster. + +Passed solutions: +1. Disable namespace-level Geo-Replication then delete the partitioned topic: this will trigger a deletion for other topics, which is not expected. +2. Disable topic-level Geo-Replication under each cluster, and delete the partitioned topic by Admin API: it will delete the shared partitioned topic metadata, which was also deleted from another cluster. +3. Disable topic-level Geo-Replication under each cluster, and delete the sub-topics manually by Admin API: this is the best solution without the current PIP, the defect is that the schema data and topic-level policies are still there. + +# Goals & Designs + +Add two new APIs +- `pulsar-admin topicPolicies -c <clusters> --g <topic>` + - Pulsar only support setting a local topic-level `replicated clusters`, does not support global policies so far. +- `pulsar-admin topicPolicies delete <topic>` + - To delete topic level policies, including global and local policies, it can be called even if the topic has been deleted. + +The new APIs will be used for the following scenarios: + +A. Delete a partitioned topic under all clusters, if enabled Geo-Replication with a shared config metadata store. +- Disable global topic level `replicated clusters`, in other words, set `replicated clusters` with a single list. + - The clusters are not contained in the `replicated clusters` will delete the sub-topics automatically, such as the namespace-level implementation. + - The schemas and local topic policies of the topic will be deleted after the latest sub-topic is deleted. +- Delete partitioned topic since we have disabled Geo-Replication. +- Manually delete global topic-level policies by the new API `pulsar-admin topicPolicies delete <topic>` for other clusters that are not contained in `replicated clusters`. + +B. Delete a partitioned topic under a specify cluster. +- Support setting up a global topic-level `replicated clusters` policy , which does not contain the cluster that you want to remove form. + - Broker will delete the sub-topics automatically. + - The schemas and local topic policies of the topic will be deleted after the latest sub-topic is deleted. +- Note: + - The global topic-level policy will not be removed, to avoid the namespace-level `replicated cluster` being taken affect. + - Once you want to delete the topic under all clusters, please use the solution A to remove the global topic-level policy left. + +**Question**: why not to add a mechanism that brokers clear global topic-level policies that the related topic was deleted regularly, which avoids to call `pulsar-admin topicPolicies delete <topic>`? + +**Answer**: When users quickly delete and create topics, it will result in the previous policies not being deleted by the scheduled cleaning mechanism, thereby causing confusion. + +### Public API + +**new Pulsar Admin API** +- `pulsar-admin topicPolicies --clusters <clusters> --global <topic>` Review Comment: We already have `bin/pulsar-admin topics set-replication-cluster`, it seems we can only add an option instead of a new command. ########## pip/pip-422.md: ########## @@ -0,0 +1,103 @@ +# PIP-422: Support global topic-level policy: replicated clusters and new API to delete topic-level policies + +# Background knowledge + +Users have two choices of config metadata store when using the feature Geo-Replication: shared or not. +All clusters share data that are stored in global config metadata store if users choose to use a shared metadata store. +By the way, the share config metadata store often is build crossing multi zone/region to confirm fault tolerance. + +Since the data that are stored in the shared metadata store contains the following pulsar resources: tenants, namespaces, partitioned topic, +the Admin API can not be used to delete these pulsar resources for a single cluster. +Pulsar has a feature that used to delete namespace level resources for a specify cluster when using a shared metadata store, it works as follows: +- The original namespace-level policy that named `replicated clusters` contains two clusters `c1` and `c2`. +- The topics under the cluster `c1` will be deleted automatically when users remove `c1` from the namespace-level policy. + +The feature above very useful at the following scenarios: +- Transfer a namespace from a old cluster to a new cluster: at last, the namespace should be deleted from the old cluster. +- Delete topics: since Pulsar does not allow deleting topics when enabled Geo-Replication, users should disable replication first. + +# Motivation + +When using shared metadata store and enabled Geo-Replication, there is no mechanism for the following scenarios: +- Transfer a partitioned topic from a old cluster to a new cluster. +- Delete a partitioned topic form a specify cluster. + +Passed solutions: +1. Disable namespace-level Geo-Replication then delete the partitioned topic: this will trigger a deletion for other topics, which is not expected. +2. Disable topic-level Geo-Replication under each cluster, and delete the partitioned topic by Admin API: it will delete the shared partitioned topic metadata, which was also deleted from another cluster. +3. Disable topic-level Geo-Replication under each cluster, and delete the sub-topics manually by Admin API: this is the best solution without the current PIP, the defect is that the schema data and topic-level policies are still there. Review Comment: Is it better to move to `# Alternatives`? ########## pip/pip-422.md: ########## @@ -0,0 +1,103 @@ +# PIP-422: Support global topic-level policy: replicated clusters and new API to delete topic-level policies + +# Background knowledge + +Users have two choices of config metadata store when using the feature Geo-Replication: shared or not. +All clusters share data that are stored in global config metadata store if users choose to use a shared metadata store. +By the way, the share config metadata store often is build crossing multi zone/region to confirm fault tolerance. Review Comment: ```suggestion By the way, the share config metadata store often is build crossing multi zone/region to ensure fault tolerance. ``` ########## pip/pip-422.md: ########## @@ -0,0 +1,103 @@ +# PIP-422: Support global topic-level policy: replicated clusters and new API to delete topic-level policies + +# Background knowledge + +Users have two choices of config metadata store when using the feature Geo-Replication: shared or not. +All clusters share data that are stored in global config metadata store if users choose to use a shared metadata store. +By the way, the share config metadata store often is build crossing multi zone/region to confirm fault tolerance. + +Since the data that are stored in the shared metadata store contains the following pulsar resources: tenants, namespaces, partitioned topic, +the Admin API can not be used to delete these pulsar resources for a single cluster. +Pulsar has a feature that used to delete namespace level resources for a specify cluster when using a shared metadata store, it works as follows: +- The original namespace-level policy that named `replicated clusters` contains two clusters `c1` and `c2`. +- The topics under the cluster `c1` will be deleted automatically when users remove `c1` from the namespace-level policy. + +The feature above very useful at the following scenarios: +- Transfer a namespace from a old cluster to a new cluster: at last, the namespace should be deleted from the old cluster. +- Delete topics: since Pulsar does not allow deleting topics when enabled Geo-Replication, users should disable replication first. + +# Motivation + +When using shared metadata store and enabled Geo-Replication, there is no mechanism for the following scenarios: +- Transfer a partitioned topic from a old cluster to a new cluster. +- Delete a partitioned topic form a specify cluster. + +Passed solutions: +1. Disable namespace-level Geo-Replication then delete the partitioned topic: this will trigger a deletion for other topics, which is not expected. +2. Disable topic-level Geo-Replication under each cluster, and delete the partitioned topic by Admin API: it will delete the shared partitioned topic metadata, which was also deleted from another cluster. +3. Disable topic-level Geo-Replication under each cluster, and delete the sub-topics manually by Admin API: this is the best solution without the current PIP, the defect is that the schema data and topic-level policies are still there. + +# Goals & Designs + +Add two new APIs +- `pulsar-admin topicPolicies -c <clusters> --g <topic>` + - Pulsar only support setting a local topic-level `replicated clusters`, does not support global policies so far. +- `pulsar-admin topicPolicies delete <topic>` + - To delete topic level policies, including global and local policies, it can be called even if the topic has been deleted. + +The new APIs will be used for the following scenarios: + +A. Delete a partitioned topic under all clusters, if enabled Geo-Replication with a shared config metadata store. +- Disable global topic level `replicated clusters`, in other words, set `replicated clusters` with a single list. + - The clusters are not contained in the `replicated clusters` will delete the sub-topics automatically, such as the namespace-level implementation. Review Comment: If there is any active producers or consumers connected the cluster which been removed from the replicated cluster list, the topic will be create again if topic auto-creation is enabled? Or it will rely on the global topic policy which defined the replication_cluster. All the producers and consumers will be rejected if the connected cluster is not listed in replicated_clusters? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
