codelipenghui commented on code in PR #21648:
URL: https://github.com/apache/pulsar/pull/21648#discussion_r1436842208
##########
pip/pip-321.md:
##########
@@ -0,0 +1,285 @@
+# PIP-321: Split the responsibilities of namespace replication-clusters
+
+# Background knowledge
+
+Pulsar's geo-replication mechanism is typically used for disaster recovery,
enabling the replication of persistently stored message data across multiple
data centers. For instance, your application publishes data in one region, and
you would like to process it for consumption in other regions. With Pulsar's
geo-replication mechanism, messages can be produced and consumed in different
geo-replicated regions. See the introduction of geo-replication to get more
information.[1]
+
+A client can set allowed clusters for a tenant. The allowed-cluster for the
tenant is a cluster that the tenant can access.
+
+A client can set replication clusters for a namespace, and the pulsar broker
internally manages replication to all the replication clusters. And the
replication clusters for a namespace must be a subgroup of the tenant's allowed
clusters.
+
+A client can not set allowed clusters at the namespace level, although PIP-8
[2] essentially achieves a similar functionality. It does not formally propose
this configuration, but the implementation uses replication-clusters as
allowed-clusters for a namespace. It introduces peer cluster for global
namespace redirection and fails `PartitionedMetadata-Lookup` request if global
namespace's replication-clusters doesn't contain current/peer-clusters. See
more information about this in PIP-8. [2]
+
+A namespace has multiple topics. Once a namespace is configured with
replication clusters, all the topics under this namespace will enable
replication in these clusters.
+
+Namespace Policy is a configuration in the namespace level, that is stored in
the metadata store, e.g. zookeeper, and this configuration can not be accessed
across multiple clusters with different metadata stores.
+
+Replication clusters can be configured at the message level. Pulsar support
setting replication clusters when send messages.
+```java
+producer.newMessage().replicationClusters(List.of("cluster1",
"cluster2")).send();
+```
+
+[1] https://pulsar.apache.org/docs/3.1.x/concepts-replication
+[2] https://github.com/apache/pulsar/pull/903
+[3]
https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
+
+# Motivation
+
+Geo-replication at the topic level and message level can't work as expected
when geo-replication is disabled at the namespace level and the clusters use a
shared configuration store.
+Let's see an example:
+
+**Example For Topic Level:**
+
+- Environment:
+ - cluster1 and cluster2 in different regions sharing the same configuration
store.
+
+- Replication clusters configuration:
+ - Set namespace `ns` replication clusters : cluster1 (local cluster)
+ - Set topic `ns/topic1` replication clusters : cluster1, cluster2.
+
+- Expected:
+ - Topic `ns/topic1` can replicate between cluster1 and cluster2.
+
+- Actual:
+ - Topic cannot be created at cluster2.
+```
+PRECONDITION_FAILED: Namespace missing local cluster name in clusters list:
local_cluster=cluster2 ns=ns clusters=[cluster1]
+```
+
+**Example For Message Level**
+
+- Environment:
+ - cluster1 and cluster2 in different regions sharing the same Zookeeper
cluster.
+
+- Replication clusters configuration:
+ - Set namespace `ns` replication clusters : cluster1 (local cluster)
+ - Set replication clusters when send message1: cluster1, cluster2.
+
+- Expected:
+ - Message1 can replicate between cluster1 and cluster2.
+
+- Actual:
+ - Topic cannot be created at cluster2, and so the message1 can not be
replicated to cluster2.
+```
+PRECONDITION_FAILED: Namespace missing local cluster name in clusters list:
local_cluster=cluster2 ns=ns clusters=[cluster1]
+```
+
+The root cause of these issues is that topics cannot access clusters that are
not included in the replication-clusters of the namespace policy.
+If you set both clusters to the namespace's replication clusters. All the
topics under this namespace will start to replicate data between clusters
unless they set replication clusters to one cluster to the topic level for all
topics. It's super hard for Pulsar maintainers and impossible to control the
newly created topics (create a topic first and then set topic policies). The
replication clusters and allowed clusters are different. Employing one
configuration for two purposes is insoluble.
+But in the current implementation, the replication-clusters and
allowed-Clusters are all configured by specifying the replication clusters.
+This will make the topic unable to have its replication policies.
+
+To support geo-replication policies at the topic level and the message level,
we must make the cluster configuration at the namespace level more clearly.
+Introduce `allowed-clusters` at the namespace level and make
`replication-clusters` only the default replication clusters for the topics
under the namespace.
+
+# Goals
+
+## In Scope
+
+The namespace will have a clearer configuration for clusters. Users can use
`replication-clusters` and `allowed-clusters` to specify the clusters that the
data of the namespace will replicate and the clusters that can load the topics
under the namespace; it's similar to the tenant's `allowed-clusters.`
+
+## Out of Scope
+
+This proposal can be used to solve the problem of topic-level and message
level geo-replication can not work as expected. It is the initial motivation
for this proposal, but this proposal does not involve modifications to
geo-replication.
+
+Out of this proposal, there are others actions needed to perform.
+1. Limit the replication configuration at the namespace level, topic level and
message level.
+ - The `replication_clusters` at the namespace level, topic level and
message level should be the subgroup of `allowed_clusters` at the namespace
level.
+ - Otherwise, `400 Bad Request` will be returned when specify the
`replication_clusters` at the namespace level or topic level.
+ - Fail send request with a `NotAllowedException` exception when the
`replication_clusters` of the message is not the subgroup of the
`allowed_clusters` at the namespace level.
+2. Implement `allowed_clusters` at the topic level, this should need another
proposal.
+ - If `allowed_clusters` is implemented in the topic policy, the
`replication_clusters` at the topic level and message level should be the
subgroup of the `allowed_clusters` of the topic level.
+ - Otherwise, `400 Bad Request` will be returned when specify the
`replication_clusters`at the namespace level.
+ - Fail send request with a `NotAllowedException` exception when the
`replication_clusters` of the message is not the subgroup of the
`allowed_clusters` at the topic level.
+
+Fail request of sending message when the message set the configuration of the
`replication_clusters` at the message level.
+
+# High Level Design
+A new namespace policy option `allowed_clusters` will be added. The
`allowed_clusters` policy will specify the clusters where topics under this
namespace can be created or loaded. The `replication_clusters` indicates the
clusters that are used to create a full mesh replication for all topics under
this namespace.
+
+When a namespace has the policy with `allowed_clusters` and
`replication_clusters`, the topics under this namespace will replicate data to
all `replication_clusters` by default. Additionally, the topic can have a
flexible replication clusters configuration, which should be a subset of the
`allowed_clusters` of the namespace.
+
+If `allowed_clusters` is not set, `replication_clusters` will be used as the
default value for `allowed_clusters`.
+
+If neither `allowed_clusters` nor `replication_clusters` are set, topics under
this namespace will only be able to publish/subscribe at the local cluster. The
local cluster will be added in the `allowed_clusters` automatically when
creating namespace.
+
+Message-level replication is similar to topic-level replication. The
replication clusters of a message should be the subset of the
`allowed_clusters`, and are the `replication_clusters` configured at the topic
level or namespace level by default.
+
+# Detailed Design
+
+## Public-facing Changes
+
+### Public API
+
+#### `setNamespaceAllowedClusters` Endpoint
+
+This new endpoint allows setting the list of allowed clusters for a specific
namespace.
+
+**Method:**
+```
+POST
+```
+
+**Path:**
+```
+/namespaces/{tenant}/{namespace}/allowedClusters
+```
+
+**HTTP Body Parameters:**
+
+- `clusterIds`: A list of cluster IDs.
+
+**Response Codes:**
+
+- `400 Bad Request`: The list of allowed clusters should include all
replication clusters.
+- `403 Forbidden`: The requester does not have admin permissions.
+- `404 Not Found`: The specified tenant, cluster, or namespace does not exist.
+- `409 Conflict`: A peer-cluster cannot be part of an allowed-cluster.
+- `412 Precondition Failed`: The namespace is not global or the provided
cluster IDs are invalid.
+
+**Explanation for 409 Conflict:** This follows the behavior of namespace
replication clusters. As per PIP-8, a peer-cluster cannot be part of a
replication-cluster. Similarly, for allowed-clusters, users could enable
replication at the topic level, hence a peer-cluster cannot be part of
allowed-clusters as well.
+
+#### `getNamespaceAllowedClusters` Endpoint
+
+This new endpoint allows retrieving the list of allowed clusters for a
specific namespace.
+
+**Method:**
+```
+GET
+```
+
+**Path:**
+```
+/namespaces/{tenant}/{namespace}/allowedClusters
+```
+
+**Response Codes:**
+
+- `403 Forbidden`: The requester does not have admin permissions.
+- `404 Not Found`: The specified tenant, cluster, or namespace does not exist.
+- `412 Precondition Failed`: The namespace is not global.
+
+**Example Response:**
+```
+[
+ "cluster1",
+ "cluster2",
+ "cluster3"
+]
+```
+
+### Binary protocol
+
+### Configuration
+
+### CLI
+
+#### `setNamespaceAllowedClusters` Command
+
+This new command allows you to set the list of allowed clusters for a specific
namespace.
+
+**Usage:**
+
+```
+$ pulsar admin namespaces set-allowed-clusters --clusters <cluster-ids>
<tenant>/<namespace>
+```
+
+**Options:**
+ * --clusters, -c
+ - A comma-separated list of cluster IDs.
+
+**Response Codes:**
+
+- `400 Bad Request`: The allowed clusters should contain all replication
clusters.
+- `403 Forbidden`: You do not have admin permission.
+- `404 Not Found`: The tenant, cluster, or namespace does not exist.
+- `409 Conflict`: A peer-cluster cannot be part of an allowed-cluster.
+- `412 Precondition Failed`: The namespace is not global or the cluster IDs
are invalid.
+
+**Explanation for 409 Conflict:** This follows the behavior of namespace
replication clusters. In PIP-8, it introduced the concept of a peer-cluster,
which cannot be part of a replication-cluster. For the allowed-clusters, users
could enable replication at the topic level, so the peer-cluster cannot be part
of the allowed-clusters too.
+
+#### `getNamespaceAllowedClusters` Command
+
+This new command allows you to retrieve the list of allowed clusters for a
specific namespace.
+
+**Usage:**
+
+```
+$ pulsar admin namespaces get-allowed-clusters <tenant>/<namespace>
+```
+
+**Response Codes:**
+
+- `403 Forbidden`: You do not have admin permission.
+- `404 Not Found`: The tenant, cluster, or namespace does not exist.
+- `412 Precondition Failed`: The namespace is not global.
+
+**Example Response:**
+
+```
+"cluster1"
+"cluster2"
+"cluster3"
+```
+
+#### `CreateNamespace` Command
+
+Add a new option to this command to set allowed clusters when create a new
namespace.
+
+**Usage:**
+
+```
+$ pulsar admin namespaces create [options] tenant/namespace
+```
+
+**Options:**
+ * --bundles, -b
+ - number of bundles to activate. Default: 0
+ * --clusters, -c
+ - List of replication clusters this namespace will be assigned.
(Modified*)
+ * --allowed-clusters, -a (New*)
Review Comment:
```suggestion
--allowed-clusters, -a (New*)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
