We've had several bugs related to this topic of auto created partitioned topics recently. These three PRs come to mind:
* https://github.com/apache/pulsar/pull/20370 * https://github.com/apache/pulsar/pull/20392 * https://github.com/apache/pulsar/pull/20397 Those PRs rely on server side inference based on the topic name to know that a topic should not be a partitioned topic. In general, these problems hint to me that we need better definitions for "system" and "special" topics that break the rules of auto created topics in well defined ways. I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead. > I agree with you, but now the problem is that we still need > an identifier to say that it is a DLQ. Do you have some > suggestions? The identifier is the name. The topics have `-DLQ` and `-RETRY` as suffixes, which makes them "special" (which reminds me of this thread [0]). We could choose to always make these non partitioned or to make them configurable in some other way. Or, we could consider giving producers and consumers the option to configure a `partitionCountHint`. This could be used by the broker during auto creation to create a partitioned topic with x partitions when greater than 0 and to create a non-partitioned topic when 0. This moves the logic out of inference on topic names and into the realm of client configuration. However, it could create hard to debug scenarios. Thanks, Michael [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w On Wed, May 31, 2023 at 7:42 AM Yubiao Feng <yubiao.f...@streamnative.io.invalid> wrote: > > Hi @Enrico > > > I think that it is better to add flags like: > > - allowAutoTopiCreation: default "true", if "false" the broker > > won't create the topic in any case > > autoTopicCreationMode: undefined/partitioned/non-partitioned > > I agree with you, but now the problem is that we still need > an identifier to say that it is a DLQ. Do you have some > suggestions? > > Thanks > > Yubiao Feng > > > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eolive...@gmail.com> wrote: > > > Yubiao, > > I understand your problem: DLQ and retry topic don't play well with > > automatic topic creation. > > > > We are introducing wire protocol changes and also starting to add > > topic properties for every new topic (they come with a cost on > > metadata). > > > > I am not sure that adding the purposeOfAutoCreatedTopic property adds > > much value and I suggest to drop it. > > Logging on the broker that you created the topic is enough. I don't > > think that any application or even the broker would use that piece of > > information. > > > > > > Regarding topicPurpose I think that having this "list" of > > possibilities will become hard to maintain un the future. > > > > I think that it is better to add flags like: > > - allowAutoTopiCreation: default "true", if "false" the broker won't > > create the topic in any case > > - autoTopicCreationMode: undefined/partitioned/non-partitioned > > > > One problem with partitioned topic and DLQ is that you probably want > > the same number of partitions on the side topic as the original topic, > > so probably you have to carry that parameter (number of partitions). > > > > IIRC the Subscribe command is sent directly per each partition, so I > > don't know how these things may work well together. > > > > Enrico > > > > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng > > <xiangy...@apache.org> ha scritto: > > > > > > Hi Yubiao, > > > > > > Appreciate your effort in initiating this PIP. I believe these changes > > will > > > address the existing issues and make DLQ and Retry Topic handling more > > > efficient and straightforward. > > > > > > The goals you outlined are clear and, upon implementation, will improve > > the > > > overall functionality of Pulsar. The proposed API changes also seem > > > suitable for achieving the desired outcomes. > > > > > > Looking forward to the progress on this PIP. > > > > > > Best regards, > > > Xiangying > > > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng > > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > > > Hi community > > > > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And Prevent > > > > auto-create a DLQ for a DLQ". > > > > > > > > PIP link: https://github.com/apache/pulsar/issues/20033 > > > > > > > > ### Motivation > > > > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will > > auto-create a > > > > topic when the client loads it; After setting config > > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, > > Pulsar > > > > will auto-create a partitioned topic(which have two partitions) when > > the > > > > client loads it. > > > > > > > > After the above, if using the feature [Retry Topic]( > > > > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > > > > ) > > > > and [DLQ]( > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > > > > ) > > > > enable topic auto-creation, we will get a partitioned DLQ and a > > partitioned > > > > Retry Topic like this: > > > > - `{primary_topic_name}-{sub_name}-DLQ` > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > > > > - `{primary_topic_name}-{sub_name}-RETRY` > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > > > > > > > ---- > > > > > > > > I feel that almost all users will not use the multi-partitioned DLQ or > > > > multi-partitioned Retry topic because there is a bug that causes the > > above > > > > behavior to be incorrect, but we have yet to receive any issues about > > it. > > > > This bug causes the above behavior to look like this: When the > > partitioned > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will > > create a > > > > partitioned topic meta which has two partitioned but only create a > > topic > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug > > > > description. > > > > > > > > So I want to change the behavior to Just auto-create no-partitioned > > > > DLQ/Retry Topic. > > > > > > > > ---- > > > > > > > > #### Prevent auto-create the DLQ for a DLQ > > > > Please look at this [Discussion]( > > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for > > the > > > > detail. > > > > > > > > ---- > > > > > > > > ### Goal > > > > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other words: > > > > prevent auto-create partitioned DLQ) > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic > > > > - roles: > > > > - DLQ will not auto-create for a DLQ > > > > - Retry Topic will not auto-create for a Retry Topic > > > > - DLQ will not auto-create for a Retry Topic > > > > - Retry Topic will not auto-create for a DLQ > > > > - client changes: Clients will not create a DLQ for a DLQ > > > > - broker changes: rejected the request which wants to auto-create a > > DLQ > > > > for a DLQ > > > > > > > > ---- > > > > > > > > ### API Changes > > > > > > > > #### CommandSubscribe.java > > > > ```java > > > > /** > > > > * This is an enumeration value with tree options: "standard", "dead > > > > letter", "retry letter". > > > > */ > > > > private String topicPurpose; > > > > ``` > > > > > > > > #### Properties of Topic > > > > ```properties > > > > "purposeOfAutoCreatedTopic": value with tree options: "standard", "dead > > > > letter", "retry letter" > > > > ``` > > > > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > > > > Because there is a scenario like this: auto-create a topic, use it as > > a DLQ > > > > after a few days, and not use it as a DLQ after a few days, this Topic > > will > > > > be allowed to have DLQ/Retry Topic. We only mark the topics created for > > > > DLQ/Retry purposes. > > > > > > > > > > > > Thanks > > > > Yubiao Feng > > > > > >
