Yubiao, I understand your problem: DLQ and retry topic don't play well with automatic topic creation.
We are introducing wire protocol changes and also starting to add topic properties for every new topic (they come with a cost on metadata). I am not sure that adding the purposeOfAutoCreatedTopic property adds much value and I suggest to drop it. Logging on the broker that you created the topic is enough. I don't think that any application or even the broker would use that piece of information. Regarding topicPurpose I think that having this "list" of possibilities will become hard to maintain un the future. I think that it is better to add flags like: - allowAutoTopiCreation: default "true", if "false" the broker won't create the topic in any case - autoTopicCreationMode: undefined/partitioned/non-partitioned One problem with partitioned topic and DLQ is that you probably want the same number of partitions on the side topic as the original topic, so probably you have to carry that parameter (number of partitions). IIRC the Subscribe command is sent directly per each partition, so I don't know how these things may work well together. Enrico Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng <xiangy...@apache.org> ha scritto: > > Hi Yubiao, > > Appreciate your effort in initiating this PIP. I believe these changes will > address the existing issues and make DLQ and Retry Topic handling more > efficient and straightforward. > > The goals you outlined are clear and, upon implementation, will improve the > overall functionality of Pulsar. The proposed API changes also seem > suitable for achieving the desired outcomes. > > Looking forward to the progress on this PIP. > > Best regards, > Xiangying > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng > <yubiao.f...@streamnative.io.invalid> wrote: > > > Hi community > > > > I started a PIP about "Just auto-create no-partitioned DLQ And Prevent > > auto-create a DLQ for a DLQ". > > > > PIP link: https://github.com/apache/pulsar/issues/20033 > > > > ### Motivation > > > > #### Just auto-create no-partitioned DLQ/Retry Topic > > If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a > > topic when the client loads it; After setting config > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar > > will auto-create a partitioned topic(which have two partitions) when the > > client loads it. > > > > After the above, if using the feature [Retry Topic]( > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > > ) > > and [DLQ]( > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > > ) > > enable topic auto-creation, we will get a partitioned DLQ and a partitioned > > Retry Topic like this: > > - `{primary_topic_name}-{sub_name}-DLQ` > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > > - `{primary_topic_name}-{sub_name}-RETRY` > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > > > ---- > > > > I feel that almost all users will not use the multi-partitioned DLQ or > > multi-partitioned Retry topic because there is a bug that causes the above > > behavior to be incorrect, but we have yet to receive any issues about it. > > This bug causes the above behavior to look like this: When the partitioned > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will create a > > partitioned topic meta which has two partitioned but only create a topic > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug > > description. > > > > So I want to change the behavior to Just auto-create no-partitioned > > DLQ/Retry Topic. > > > > ---- > > > > #### Prevent auto-create the DLQ for a DLQ > > Please look at this [Discussion]( > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the > > detail. > > > > ---- > > > > ### Goal > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other words: > > prevent auto-create partitioned DLQ) > > - DLQ/Retry topic should not create for a DLQ/Retry Topic > > - roles: > > - DLQ will not auto-create for a DLQ > > - Retry Topic will not auto-create for a Retry Topic > > - DLQ will not auto-create for a Retry Topic > > - Retry Topic will not auto-create for a DLQ > > - client changes: Clients will not create a DLQ for a DLQ > > - broker changes: rejected the request which wants to auto-create a DLQ > > for a DLQ > > > > ---- > > > > ### API Changes > > > > #### CommandSubscribe.java > > ```java > > /** > > * This is an enumeration value with tree options: "standard", "dead > > letter", "retry letter". > > */ > > private String topicPurpose; > > ``` > > > > #### Properties of Topic > > ```properties > > "purposeOfAutoCreatedTopic": value with tree options: "standard", "dead > > letter", "retry letter" > > ``` > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > > Because there is a scenario like this: auto-create a topic, use it as a DLQ > > after a few days, and not use it as a DLQ after a few days, this Topic will > > be allowed to have DLQ/Retry Topic. We only mark the topics created for > > DLQ/Retry purposes. > > > > > > Thanks > > Yubiao Feng > >
