Before we get further into the implementation, I'd like to discuss whether the current behavior is the expected behavior, as this is the key motivation for this feature.
I think the DLQ's current behavior is the expected behavior because the DLQ is only a topic and topics lose messages unless they have a subscription or a retention policy. I admit that it is not necessarily a nice default behavior to potentially lose messages, but this is the design for all topics. Based on the current design, an admin can create a retention policy for the topic or namespace. Then, consumers of the topic have the duration of the retention policy to discover the topic and create a subscription before messages are lost. Is there a reason this solution doesn't work for the DLQ topic? Perhaps the disconnect here is that users of the DLQ feature do not view the DLQ as only a Pulsar topic. I look forward to your thoughts. As an aside, I wonder if topic discoverability is part of the problem here. It would be extremely valuable to get notifications any time a topic is created. That would allow users to move away from polling for current topic names towards a more reactive design. Thanks, Michael On Tue, Dec 28, 2021 at 7:59 PM Zike Yang <zky...@streamnative.io.invalid> wrote: > > > Oh, that's a very interesting point. I think it'd be easy to add that > > as "internal" feature, though I'm a bit puzzled on how to add that to > > the producer API > > I think we can add a field `String initialSubscriptionName` to the > Producer Configuration. And add a new field `optional string > initial_subscription_name` to the `CommnadProducer`. > When the Broker handles the CommandProducer, if it checks that the > initialSubscriptionName is not empty or null, it will use > initialSubscriptionName to create a subscription on that topic. When > creating the deadLetterProducer or retryLetterProducer, we can specify > and create the initial subscription directly through the Producer. > What do you think? > > On Thu, Dec 23, 2021 at 7:42 AM Matteo Merli <matteo.me...@gmail.com> wrote: > > > > > What if we extended the `CommandProducer` command to add a > > > `create_subscription` field? Then, any time a topic is auto > > > created and this field is true, the broker would auto create a > > > subscription. There are some details to work out, but I think this > > > feature would fulfill the needs of this PIP and would also be broadly > > > useful for many client applications that dynamically create topics. > > > > Oh, that's a very interesting point. I think it'd be easy to add that > > as "internal" feature, though I'm a bit puzzled on how to add that to > > the producer API > > > > -- > Zike Yang
