Hi Yubiao, Appreciate your effort in initiating this PIP. I believe these changes will address the existing issues and make DLQ and Retry Topic handling more efficient and straightforward.
The goals you outlined are clear and, upon implementation, will improve the overall functionality of Pulsar. The proposed API changes also seem suitable for achieving the desired outcomes. Looking forward to the progress on this PIP. Best regards, Xiangying On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng <yubiao.f...@streamnative.io.invalid> wrote: > Hi community > > I started a PIP about "Just auto-create no-partitioned DLQ And Prevent > auto-create a DLQ for a DLQ". > > PIP link: https://github.com/apache/pulsar/issues/20033 > > ### Motivation > > #### Just auto-create no-partitioned DLQ/Retry Topic > If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a > topic when the client loads it; After setting config > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar > will auto-create a partitioned topic(which have two partitions) when the > client loads it. > > After the above, if using the feature [Retry Topic]( > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > ) > and [DLQ]( > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > ) > enable topic auto-creation, we will get a partitioned DLQ and a partitioned > Retry Topic like this: > - `{primary_topic_name}-{sub_name}-DLQ` > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > - `{primary_topic_name}-{sub_name}-RETRY` > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > ---- > > I feel that almost all users will not use the multi-partitioned DLQ or > multi-partitioned Retry topic because there is a bug that causes the above > behavior to be incorrect, but we have yet to receive any issues about it. > This bug causes the above behavior to look like this: When the partitioned > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will create a > partitioned topic meta which has two partitioned but only create a topic > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug > description. > > So I want to change the behavior to Just auto-create no-partitioned > DLQ/Retry Topic. > > ---- > > #### Prevent auto-create the DLQ for a DLQ > Please look at this [Discussion]( > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the > detail. > > ---- > > ### Goal > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other words: > prevent auto-create partitioned DLQ) > - DLQ/Retry topic should not create for a DLQ/Retry Topic > - roles: > - DLQ will not auto-create for a DLQ > - Retry Topic will not auto-create for a Retry Topic > - DLQ will not auto-create for a Retry Topic > - Retry Topic will not auto-create for a DLQ > - client changes: Clients will not create a DLQ for a DLQ > - broker changes: rejected the request which wants to auto-create a DLQ > for a DLQ > > ---- > > ### API Changes > > #### CommandSubscribe.java > ```java > /** > * This is an enumeration value with tree options: "standard", "dead > letter", "retry letter". > */ > private String topicPurpose; > ``` > > #### Properties of Topic > ```properties > "purposeOfAutoCreatedTopic": value with tree options: "standard", "dead > letter", "retry letter" > ``` > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > Because there is a scenario like this: auto-create a topic, use it as a DLQ > after a few days, and not use it as a DLQ after a few days, this Topic will > be allowed to have DLQ/Retry Topic. We only mark the topics created for > DLQ/Retry purposes. > > > Thanks > Yubiao Feng >
