nodece commented on code in PR #23770: URL: https://github.com/apache/pulsar/pull/23770#discussion_r2307837323
########## pip/pip-398.md: ########## @@ -0,0 +1,253 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(false/*true*/) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +The PIP aims to provide management of subscription replication at the broker, namespace and topic levels using the +Pulsar Admin CLI and API. + +# High Level Design + +Introduces the `replicateSubscriptionState` configuration to enabling subscription replication on the broker, namespace +and topic levels, when enabled, all consumers under the broker/namespace/topic will automatically replicate their +subscription states to remote clusters. + +The priority for the subscription replication configuration is as follows: + +- consumer level > topic level > namespace level > broker level. +- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic, namespace, and broker levels are + ignored. +- If set at the topic level, the namespace-level configuration is ignored. +- If set at the namespace level, the broker-level configuration is ignored. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = true, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, Boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, Boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + } Review Comment: > @nodece With a single `Boolean` value, how would you distinguish between the 2 different cases (1. and 3. above) where subscription replication is enabled? > override any setting by the consumer (or lower level policy) and enable subscription replication for all subscriptions Subscription replication is enabled if: - The consumer sets true - The first non-null policy found in the following order is true: topic → namespace → broker. Once a policy is found (even if false), lower levels are ignored. > no setting at all (higher level setting would take priority). If all policies (broker, ns, topic) are "empty", the consumer setting would get used directly. Correct. I understand the concern about using `null` for removing the setting. To make the semantics clearer, I'll add an explicit `removeReplicateSubscriptionState()` API instead of relying on `null`. So no need to worry about `Boolean` vs `boolean`, the new API will handle the unset case explicitly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
