Hi Michael, Enrico > The identifier is the name. The topics have `-DLQ` and `-RETRY` > as suffixes, which makes them "special" (which reminds me of this > thread [0]). We could choose to always make these non partitioned > or to make them configurable in some other way.
I changed the design like this: ### Properties of Topic ```properties "wasAutoCreated": boolean ``` - If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned topics will be automatically created - If the property `wasAutoCreated` of the topic is true and the topic name ends with "-RETRY" or "-DLQ", retry topics and dead letter queues will no longer be created for this topic. How are you feeling? Thanks Yubiao Feng On Thu, Jun 1, 2023 at 12:48 PM Michael Marshall <mmarsh...@apache.org> wrote: > We've had several bugs related to this topic of auto created > partitioned topics recently. These three PRs come to mind: > > * https://github.com/apache/pulsar/pull/20370 > * https://github.com/apache/pulsar/pull/20392 > * https://github.com/apache/pulsar/pull/20397 > > Those PRs rely on server side inference based on the topic name to > know that a topic should not be a partitioned topic. > > In general, these problems hint to me that we need better definitions > for "system" and "special" topics that break the rules of auto created > topics in well defined ways. > > I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead. > > > I agree with you, but now the problem is that we still need > > an identifier to say that it is a DLQ. Do you have some > > suggestions? > > The identifier is the name. The topics have `-DLQ` and `-RETRY` as > suffixes, which makes them "special" (which reminds me of this thread > [0]). We could choose to always make these non partitioned or to make > them configurable in some other way. > > Or, we could consider giving producers and consumers the option to > configure a `partitionCountHint`. This could be used by the broker > during auto creation to create a partitioned topic with x partitions > when greater than 0 and to create a non-partitioned topic when 0. This > moves the logic out of inference on topic names and into the realm of > client configuration. However, it could create hard to debug > scenarios. > > Thanks, > Michael > > [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w > > On Wed, May 31, 2023 at 7:42 AM Yubiao Feng > <yubiao.f...@streamnative.io.invalid> wrote: > > > > Hi @Enrico > > > > > I think that it is better to add flags like: > > > - allowAutoTopiCreation: default "true", if "false" the broker > > > won't create the topic in any case > > > autoTopicCreationMode: undefined/partitioned/non-partitioned > > > > I agree with you, but now the problem is that we still need > > an identifier to say that it is a DLQ. Do you have some > > suggestions? > > > > Thanks > > > > Yubiao Feng > > > > > > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eolive...@gmail.com> > wrote: > > > > > Yubiao, > > > I understand your problem: DLQ and retry topic don't play well with > > > automatic topic creation. > > > > > > We are introducing wire protocol changes and also starting to add > > > topic properties for every new topic (they come with a cost on > > > metadata). > > > > > > I am not sure that adding the purposeOfAutoCreatedTopic property adds > > > much value and I suggest to drop it. > > > Logging on the broker that you created the topic is enough. I don't > > > think that any application or even the broker would use that piece of > > > information. > > > > > > > > > Regarding topicPurpose I think that having this "list" of > > > possibilities will become hard to maintain un the future. > > > > > > I think that it is better to add flags like: > > > - allowAutoTopiCreation: default "true", if "false" the broker won't > > > create the topic in any case > > > - autoTopicCreationMode: undefined/partitioned/non-partitioned > > > > > > One problem with partitioned topic and DLQ is that you probably want > > > the same number of partitions on the side topic as the original topic, > > > so probably you have to carry that parameter (number of partitions). > > > > > > IIRC the Subscribe command is sent directly per each partition, so I > > > don't know how these things may work well together. > > > > > > Enrico > > > > > > > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng > > > <xiangy...@apache.org> ha scritto: > > > > > > > > Hi Yubiao, > > > > > > > > Appreciate your effort in initiating this PIP. I believe these > changes > > > will > > > > address the existing issues and make DLQ and Retry Topic handling > more > > > > efficient and straightforward. > > > > > > > > The goals you outlined are clear and, upon implementation, will > improve > > > the > > > > overall functionality of Pulsar. The proposed API changes also seem > > > > suitable for achieving the desired outcomes. > > > > > > > > Looking forward to the progress on this PIP. > > > > > > > > Best regards, > > > > Xiangying > > > > > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng > > > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > > > > > Hi community > > > > > > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And > Prevent > > > > > auto-create a DLQ for a DLQ". > > > > > > > > > > PIP link: https://github.com/apache/pulsar/issues/20033 > > > > > > > > > > ### Motivation > > > > > > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic > > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will > > > auto-create a > > > > > topic when the client loads it; After setting config > > > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, > > > Pulsar > > > > > will auto-create a partitioned topic(which have two partitions) > when > > > the > > > > > client loads it. > > > > > > > > > > After the above, if using the feature [Retry Topic]( > > > > > > > > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > > > > > ) > > > > > and [DLQ]( > > > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > > > > > ) > > > > > enable topic auto-creation, we will get a partitioned DLQ and a > > > partitioned > > > > > Retry Topic like this: > > > > > - `{primary_topic_name}-{sub_name}-DLQ` > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > > > > > - `{primary_topic_name}-{sub_name}-RETRY` > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > > > > > > > > > ---- > > > > > > > > > > I feel that almost all users will not use the multi-partitioned > DLQ or > > > > > multi-partitioned Retry topic because there is a bug that causes > the > > > above > > > > > behavior to be incorrect, but we have yet to receive any issues > about > > > it. > > > > > This bug causes the above behavior to look like this: When the > > > partitioned > > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will > > > create a > > > > > partitioned topic meta which has two partitioned but only create a > > > topic > > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic > named > > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at > this > > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed > bug > > > > > description. > > > > > > > > > > So I want to change the behavior to Just auto-create no-partitioned > > > > > DLQ/Retry Topic. > > > > > > > > > > ---- > > > > > > > > > > #### Prevent auto-create the DLQ for a DLQ > > > > > Please look at this [Discussion]( > > > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) > for > > > the > > > > > detail. > > > > > > > > > > ---- > > > > > > > > > > ### Goal > > > > > > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other > words: > > > > > prevent auto-create partitioned DLQ) > > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic > > > > > - roles: > > > > > - DLQ will not auto-create for a DLQ > > > > > - Retry Topic will not auto-create for a Retry Topic > > > > > - DLQ will not auto-create for a Retry Topic > > > > > - Retry Topic will not auto-create for a DLQ > > > > > - client changes: Clients will not create a DLQ for a DLQ > > > > > - broker changes: rejected the request which wants to > auto-create a > > > DLQ > > > > > for a DLQ > > > > > > > > > > ---- > > > > > > > > > > ### API Changes > > > > > > > > > > #### CommandSubscribe.java > > > > > ```java > > > > > /** > > > > > * This is an enumeration value with tree options: "standard", > "dead > > > > > letter", "retry letter". > > > > > */ > > > > > private String topicPurpose; > > > > > ``` > > > > > > > > > > #### Properties of Topic > > > > > ```properties > > > > > "purposeOfAutoCreatedTopic": value with tree options: "standard", > "dead > > > > > letter", "retry letter" > > > > > ``` > > > > > > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > > > > > Because there is a scenario like this: auto-create a topic, use it > as > > > a DLQ > > > > > after a few days, and not use it as a DLQ after a few days, this > Topic > > > will > > > > > be allowed to have DLQ/Retry Topic. We only mark the topics > created for > > > > > DLQ/Retry purposes. > > > > > > > > > > > > > > > Thanks > > > > > Yubiao Feng > > > > > > > > >
