leizhiyuan opened a new issue, #10559:
URL: https://github.com/apache/rocketmq/issues/10559
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement
rather than a bug/feature.
### Summary
Support wildcard pattern matching for LiteTopic consumer subscriptions,
enabling hierarchical topic organization and fine-grained subscriber-side
filtering.
### Motivation
PR #10204 introduced Lite wildcard groups, but the current implementation
has several limitations:
1. **No consumer-side pattern filtering** — A wildcard consumer group
receives all LiteTopics under the parent topic. Consumers cannot express which
subset they want; they always get everything.
2. **Full LMQ scan on every dispatch cycle** —
`doFullDispatchForWildcardGroup` calls `forEachLiteTopic()` which scans the
entire ConsumeQueue table (O(N) over all LMQs), not just the topics under the
bound parent topic.
3. **No hierarchical topic organization** — LiteTopic names are flat strings
with no way to express logical grouping (e.g., "all payment-related subtopics").
This makes it difficult to build systems where different consumers within
the same group handle different subsets of events.
### Describe the Solution You'd Like
**Wildcard pattern matching** for LiteTopic consumer subscriptions, inspired
by NATS subject matching.
#### Key Design Decisions
**Separator: `__` (double underscore)**
- `.` is already used in RocketMQ topic names (e.g., `%LMQ%$parent$child`)
and in ConsumeQueue file paths
- `__` is visually distinctive, rarely appears in topic names, and avoids
conflicts
**Pattern Tokens**
| Token | Meaning | Position | NATS Equivalent |
|-------|---------|----------|-----------------|
| `*` | Matches exactly one `__`-delimited segment | Anywhere | `*` |
| `**` | Matches one or more segments | Must be at the end | `>` |
**Examples** (parent topic = `order_events`)
| Pattern | Matches | Does Not Match |
|---------|---------|----------------|
| `pay__refund` | `pay__refund` | `pay__refund__notify`, `notify__refund` |
| `pay__*` | `pay__refund`, `pay__success` | `pay__refund__notify` |
| `*__refund` | `pay__refund`, `notify__refund` | `pay__refund__notify` |
| `pay__*__notify` | `pay__refund__notify`, `pay__success__notify` |
`pay__refund` |
| `**` | `pay`, `pay__refund`, `pay__refund__notify` | (matches all) |
| `pay__**` | `pay__refund`, `pay__refund__notify` | `notify__refund` |
#### Implementation Plan
**Core matching engine:** `LitePatternMatcher` in `common` module, providing
three static methods:
- `matches(pattern, liteTopic)` — recursive segment-by-segment matching, no
regex
- `expand(pattern, candidates)` — expand a pattern against a set of
candidate topics
- `validate(pattern)` — validate pattern syntax
**Two-phase subscription expansion:**
1. **Eager expansion** at `COMPLETE_ADD` time — expand all patterns against
existing LiteTopics under the parent topic, register matched topics to
`liteTopic2Group`
2. **Lazy re-expansion** when new LiteTopics are created via `onRegister()`
— check if the new topic matches any client's patterns, add to subscription if
so
**Dispatch optimization:** Replace full LMQ scan (`forEachLiteTopic`, O(N))
with targeted iteration via `collectByParentTopic` (O(M), where M = topics
under the parent topic).
**Multi-pattern support (v1):** `liteTopicSet` is a `Set<String>`, so
multiple patterns can be sent in a single `COMPLETE_ADD` request. The union of
all matched topics becomes the client's effective subscription.
**Protocol compatibility:** Reuse `LiteSubscriptionDTO.liteTopicSet` to
carry pattern strings for wildcard groups. No wire protocol changes needed —
the broker distinguishes via `isWildcardGroup()`.
#### Consumer API Example
```java
DefaultLitePullConsumer consumer = new
DefaultLitePullConsumer("order_consumer_group");
consumer.setLiteBindTopic("order_events");
consumer.subscribeLite("pay__*"); // single-level payment events
consumer.subscribeLite("notify__**"); // all notification events
(multi-level)
```
#### Key Changes
| Component | Change |
|-----------|--------|
| `LitePatternMatcher` (new) | Core matching engine in `common` |
| `LiteSubscription` | + `wildcardPatterns: Set<String>` |
| `LiteSubscriptionRegistryImpl` | Multi-pattern expansion in
`addCompleteSubscription` |
| `LiteEventDispatcher` | Lazy pattern re-expansion in `onRegister`;
dispatch optimization |
### Describe Alternatives You've Considered
| Alternative | Why Rejected |
|-------------|-------------|
| Store compiled Trie per wildcard group | Over-engineering for v1;
`String.split` + recursive matching is fast enough for < 10K topics |
| Support `**` in the middle of a pattern (e.g., `order__**__refund`) | NATS
does not support this; greedy vs. non-greedy makes matching ambiguous; deferred
to v2 |
| Use `.` as separator | Conflicts with existing topic naming conventions in
RocketMQ |
| Per-message pattern matching (no eager expansion) | Would require checking
patterns on every `doDispatch` call; eager expansion is more efficient |
| Add a dedicated `pattern` field to `LiteSubscriptionDTO` | Unnecessary;
reusing `liteTopicSet` is simpler and avoids wire protocol changes |
### Additional Context
- Inspired by NATS subject-based addressing and MQTT topic filters (with `+`
/ `#`), but using `__` separator to avoid conflicts with RocketMQ's existing
naming conventions
- Aligned with FlowMQ's "unified topic routing" philosophy — one topic
namespace, multiple protocol bindings, unified wildcard matching
- Performance target: `matches()` < 5μs per call, `expand()` < 10ms for
1,000 candidates
- Full backward compatibility: wildcard groups without patterns continue to
receive all LiteTopics under the parent topic (existing behavior)
- Registers via `LiteSubscriptionCtlRequestBody` without any wire protocol
changes
- Reuses the existing `collectByParentTopic` API for dispatch optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
