Hi, Yubiao
Is it better to create the same number of retry/dlq topic partitions as the
original topic?
I mean, if the original topic has 16 partitions, we also create 16(or some
other numbers, can be configurable) partitions for the retry/dlq topic.
If the topic/partition is a retry/dlq topic, it's name matches the pattern:
`{topic}-partition-{n}-{subscription}-DLQ`, right?
Can we just parse the name and then get the origin topic name, and create
retry/dlq topic with the same partitions?
Say, if client wants to create
`persistent://public/default/some_topic-partition-1-test_sub-DLQ`,
we parse the topic name, then we get the origin topic name
`persistent://public/default/some_topic` and the subscription name `test_sub`.
After this, we look up the number of partitions of the origin topic, for
instance, 16 partitions, in the last, create a partitioned topic
`admin.topics.createPartitionedTopic("persistent://public/default/some_topic-test_sub-DLQ",
16)`.
And we can also increase the retry/dlq topic partitions if the origin topic
partitions increased.
WDYT, is it feasible?
Thanks,
Tao Jiuming
On 2024/05/13 18:56:01 Yubiao Feng wrote:
> Hi community
>
> I want to change the behavior of DLQ/Retry Topic auto creation.
>
> ### Motivation
> After you set `defaultNumPartitions` to `16`, you will get `16` partitions
> per topic, and `16*16` DLQ partitions per subscription, you will get a huge
> number of DLQs if you have more than one subscription under one topic. For
> example:
> - create topic `t1` with `16` partitions
> - you will get `t1-partition-0`, `t1-partition-1`...`tp-partition-15`.
> - after you enable DLQ, you will get the following `16*16` DLQs per
> subscription
> - `t1-partition-0-{subscription}-DLQ-partition-0`
> - `t1-partition-0-{subscription}-DLQ-partition-1`
> - ...
> - `t1-partition-15-{subscription}-DLQ-partition-15`
> - `t1-partition-1-{subscription}-DLQ-partition-0`
> - `t1-partition-1-{subscription}-DLQ-partition-1`
> - ...
> - ...
> - ...
> - `t1-partition-15-{subscription}-partition-15`
>
> ### Goal
> - Do not create partitioned DLQs/Retry topics automatically.
> - Users can also create partitioned DLQ manually if they need it.
>
> I have drafted a PR, please take a look
> - https://github.com/apache/pulsar/pull/22705
>
> Thanks
> Yubiao Feng
>
