Hi @Enrico > I think that it is better to add flags like: > - allowAutoTopiCreation: default "true", if "false" the broker > won't create the topic in any case > autoTopicCreationMode: undefined/partitioned/non-partitioned
I agree with you, but now the problem is that we still need an identifier to say that it is a DLQ. Do you have some suggestions? Thanks Yubiao Feng On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eolive...@gmail.com> wrote: > Yubiao, > I understand your problem: DLQ and retry topic don't play well with > automatic topic creation. > > We are introducing wire protocol changes and also starting to add > topic properties for every new topic (they come with a cost on > metadata). > > I am not sure that adding the purposeOfAutoCreatedTopic property adds > much value and I suggest to drop it. > Logging on the broker that you created the topic is enough. I don't > think that any application or even the broker would use that piece of > information. > > > Regarding topicPurpose I think that having this "list" of > possibilities will become hard to maintain un the future. > > I think that it is better to add flags like: > - allowAutoTopiCreation: default "true", if "false" the broker won't > create the topic in any case > - autoTopicCreationMode: undefined/partitioned/non-partitioned > > One problem with partitioned topic and DLQ is that you probably want > the same number of partitions on the side topic as the original topic, > so probably you have to carry that parameter (number of partitions). > > IIRC the Subscribe command is sent directly per each partition, so I > don't know how these things may work well together. > > Enrico > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng > <xiangy...@apache.org> ha scritto: > > > > Hi Yubiao, > > > > Appreciate your effort in initiating this PIP. I believe these changes > will > > address the existing issues and make DLQ and Retry Topic handling more > > efficient and straightforward. > > > > The goals you outlined are clear and, upon implementation, will improve > the > > overall functionality of Pulsar. The proposed API changes also seem > > suitable for achieving the desired outcomes. > > > > Looking forward to the progress on this PIP. > > > > Best regards, > > Xiangying > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > Hi community > > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And Prevent > > > auto-create a DLQ for a DLQ". > > > > > > PIP link: https://github.com/apache/pulsar/issues/20033 > > > > > > ### Motivation > > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic > > > If enabled the config `allowAutoTopicCreation,` Pulsar will > auto-create a > > > topic when the client loads it; After setting config > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, > Pulsar > > > will auto-create a partitioned topic(which have two partitions) when > the > > > client loads it. > > > > > > After the above, if using the feature [Retry Topic]( > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > > > ) > > > and [DLQ]( > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > > > ) > > > enable topic auto-creation, we will get a partitioned DLQ and a > partitioned > > > Retry Topic like this: > > > - `{primary_topic_name}-{sub_name}-DLQ` > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > > > - `{primary_topic_name}-{sub_name}-RETRY` > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > > > > > ---- > > > > > > I feel that almost all users will not use the multi-partitioned DLQ or > > > multi-partitioned Retry topic because there is a bug that causes the > above > > > behavior to be incorrect, but we have yet to receive any issues about > it. > > > This bug causes the above behavior to look like this: When the > partitioned > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will > create a > > > partitioned topic meta which has two partitioned but only create a > topic > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug > > > description. > > > > > > So I want to change the behavior to Just auto-create no-partitioned > > > DLQ/Retry Topic. > > > > > > ---- > > > > > > #### Prevent auto-create the DLQ for a DLQ > > > Please look at this [Discussion]( > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for > the > > > detail. > > > > > > ---- > > > > > > ### Goal > > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other words: > > > prevent auto-create partitioned DLQ) > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic > > > - roles: > > > - DLQ will not auto-create for a DLQ > > > - Retry Topic will not auto-create for a Retry Topic > > > - DLQ will not auto-create for a Retry Topic > > > - Retry Topic will not auto-create for a DLQ > > > - client changes: Clients will not create a DLQ for a DLQ > > > - broker changes: rejected the request which wants to auto-create a > DLQ > > > for a DLQ > > > > > > ---- > > > > > > ### API Changes > > > > > > #### CommandSubscribe.java > > > ```java > > > /** > > > * This is an enumeration value with tree options: "standard", "dead > > > letter", "retry letter". > > > */ > > > private String topicPurpose; > > > ``` > > > > > > #### Properties of Topic > > > ```properties > > > "purposeOfAutoCreatedTopic": value with tree options: "standard", "dead > > > letter", "retry letter" > > > ``` > > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > > > Because there is a scenario like this: auto-create a topic, use it as > a DLQ > > > after a few days, and not use it as a DLQ after a few days, this Topic > will > > > be allowed to have DLQ/Retry Topic. We only mark the topics created for > > > DLQ/Retry purposes. > > > > > > > > > Thanks > > > Yubiao Feng > > > >
