coderzc commented on code in PR #25134:
URL: https://github.com/apache/pulsar/pull/25134#discussion_r2681636944
##########
pip/pip-452.md:
##########
@@ -0,0 +1,212 @@
+# PIP-452: Customizable Topic Listing in GetTopicsOfNamespace with properties
+
+# Motivation
+Currently, the CommandGetTopicsOfNamespace logic in the Pulsar Broker is
hard-coded to scan the metadata store (ZooKeeper) for all children nodes under
a namespace.
+
+This implementation limits the flexibility required for complex multi-tenant
scenarios:
+
+No Client Context: The broker cannot distinguish who is asking for the topics
or why. It cannot filter topics based on client properties (This properties may
be related to topic properties).
+
+Inefficient Filtering: For namespaces with millions of topics, the broker must
fetch the full list into memory before applying the topics_pattern regex. There
is no way to "push down" the filtering to the data source (e.g., a database
with an index).
+
+To address these issues, I propose making the topic listing logic pluggable
and extending the protocol to accept client properties.
+
+# Goals
+Protocol: Add a properties field to `CommandGetTopicsOfNamespace` to carry
client-side context.
+
+Broker: Introduce a `NamespaceTopicListingStrategy` SPI to customize how
topics are retrieved.
+
+Client: Update the Java Client to forward Consumer properties to the lookup
service when using Regex subscriptions.
+
+# High Level Design
+We will modify the Pulsar Protocol to carry a properties map. On the Broker
side, we will abstract the topic retrieval logic into a
NamespaceTopicListingStrategy.
+The default implementation will preserve the existing behavior (fetching from
ZooKeeper). The Broker's connection handler will simply delegate the request to
this strategy.
+
+# Detailed Design
+1. Protocol Changes
+We update `PulsarApi.proto` to include the properties field.
+
+PulsarApi.proto
+```protobuf
+message CommandGetTopicsOfNamespace {
+ required uint64 request_id = 1;
+ required string namespace = 2;
+ optional Mode mode = 3 [default = PERSISTENT];
+
+ // Existing fields for filtering and hash optimization
+ optional string topics_pattern = 4;
+ optional string topics_hash = 5;
+
+ // New field: Context properties from the client
+ repeated KeyValue properties = 6;
+}
+```
+2. Broker Changes
+New Interface: `NamespaceTopicListingStrategy`
+We introduce a strategy interface in the broker.
+
+```java
+package org.apache.pulsar.broker.namespace;
+
+import java.util.List;
+
+public class NamespaceTopicListingResult {
+ private final List<String> topics;
+
+ // Indicates if the plugin has already applied the regex filtering
provided in the request.
+ // If true, the Broker will SKIP the subsequent filtering step.
+ private final boolean filtered;
+
+ public NamespaceTopicListingResult(List<String> topics, boolean filtered) {
+ this.topics = topics;
+ this.filtered = filtered;
+ }
+
+ public List<String> getTopics() {
+ return topics;
+ }
+
+ public boolean isFiltered() {
+ return filtered;
+ }
+}
+```
+
+```Java
+
+package org.apache.pulsar.broker.namespace;
+
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.api.proto.CommandGetTopicsOfNamespace.Mode;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+public interface NamespaceTopicListingStrategy {
Review Comment:
Applied the suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
