I have no objection to adding new fields,
We should have no other disagreements in other respects by now,
Zike, could you please help start the new VOTE email thread for this
proposal?
Regards,
Penghui
On Tue, Jan 25, 2022 at 2:46 PM Michael Marshall
wrote:
> > Yes, but it doesn't seem like a
> Yes, but it doesn't seem like a big deal.
I mentioned it to identify a subtle implication of the proposed
design. I think we should be intentional about what we expose
to end users.
By putting this field in the protobuf, we will have more control
over how users interact with this feature.
>
> This is true of the DLQ producer, but by using the metadata map, won't
we be exposing this feature to end users implicitly because they could
add the field to the metadata map for other producers? (I am okay with
giving users this functionality, but I know that Matteo said in a
community meeting
> users will not touch the metadata of the producer of a DLQ
This is true of the DLQ producer, but by using the metadata map, won't
we be exposing this feature to end users implicitly because they could
add the field to the metadata map for other producers? (I am okay with
giving users this
+1 for adding init_subscription filed to the metadata of CommandProducer.
On Thu, Jan 20, 2022 at 2:52 PM PengHui Li wrote:
>
> > What is the justification for avoiding the new protobuf field? If we
> add a structured field to a map of , we are still
> modifying the protocol, even if we aren't
> What is the justification for avoiding the new protobuf field? If we
add a structured field to a map of , we are still
modifying the protocol, even if we aren't modifying the protobuf.
Not all cases need a new field named init_subscription when creating
producer,
and users will not touch the
> I think that, good or bad, the impression that users have that the DLQ
> is not a "normal" topic
Thanks for your perspective, Matteo. I still prefer my alternative
design that bypasses subscription creation, but it seems there
is insufficient interest in it, so we should move forward
discussing
+1 for adding the DLQ_init_sub to producer metadata so that we don't
need to introduce a new field in CommandProducer, and the new field
looks a little confusing
Thanks,
Penghui
On Mon, Jan 17, 2022 at 10:19 PM Hang Chen wrote:
> Thanks for creating this proposal Zike Yang. I have two ideas
Thanks for creating this proposal Zike Yang. I have two ideas about it.
1. Instead of modifying the current protocol, we can use producer
metadata to carry the init subscription
2. Please add auth for subscription creation when create topic by
producer, otherwise, it will be easily attacked.
> If we want to hold that the DLQ is not a normal topic, then I can see
> why we would have a DLQ specific feature here.
I think that, good or bad, the impression that users have that the DLQ
is not a "normal" topic comes from 2 factors:
1. The experience with traditional messaging systems JMS
> It looks like a feature that supports retaining data while no subscriptions.
Yes, that is my proposed feature. How we handle messages on a topic
with an empty set of subscriptions is a design decision.
Note that when there are no subscriptions for a topic, the following two
statements are both
> I think we should consider adding a new policy for Pulsar topics: a
namespace (or topic) policy that makes it possible to retain messages
indefinitely when a topic has no subscriptions.
It looks like a feature that supports retaining data while no subscriptions.
With infinite data retention,
Thanks for your response, Penghui.
I support simplifying message loss prevention for DLQ topics. However,
it's not clear to me why we should only simplify it for DLQ topics.
As a Pulsar user, I encountered many of the challenges you mention
when producing to auto created topics. In my
Thanks for the great comments, Michael.
Let me try to clarify some context about the issue that users encountered
and the improvement that the proposal wants to Introduce.
> Before we get further into the implementation, I'd like to discuss
whether the current behavior is the expected behavior,
Before we get further into the implementation, I'd like to discuss
whether the current behavior is the expected behavior, as this is
the key motivation for this feature.
I think the DLQ's current behavior is the expected behavior because
the DLQ is only a topic and topics lose messages unless
> Oh, that's a very interesting point. I think it'd be easy to add that
> as "internal" feature, though I'm a bit puzzled on how to add that to
> the producer API
I think we can add a field `String initialSubscriptionName` to the
Producer Configuration. And add a new field `optional string
> What if we extended the `CommandProducer` command to add a
> `create_subscription` field? Then, any time a topic is auto
> created and this field is true, the broker would auto create a
> subscription. There are some details to work out, but I think this
> feature would fulfill the needs of this
> I think there is an issue with the current behavior. When the init
> subscription
> is not created before creating deadLetterProducer, it may result in the
> loss of messages.
> And this is not the expected behavior.
I'm not sure I agree with this assertion. The default in Pulsar is
that a
I think we can create a consumer with the DLQ init subscription and then
close the consumer?
This will make the implementation easier.
Penghui
On Wed, Dec 22, 2021 at 4:49 PM Zike Yang
wrote:
> > We can avoid confusion by only attempting to create the subscription
> > when
> We can avoid confusion by only attempting to create the subscription
> when `initSubscriptionName` is not `null`. As a result, this PIP
> wouldn't change the behavior for current implementations.
I think there is an issue with the current behavior. When the init
subscription
is not created
> IMO, we won't create the init subscription if `allowAutoSubscriptionCreation`
> is false. Otherwise, it will confuse the user. Users can explicitly create
> that
> subscription to handle this case.
That is a good point, but I don't think we should attempt to create
the subscription implicitly
Thanks for your review Michael.
On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall wrote:
>
> Thanks for creating this PIP Zike Yang. This sounds like an important
> improvement. I have a couple thoughts.
>
> 1. Does the `retryLetterTopic` have the same problem with immediate
> message expiration?
Thanks for creating this PIP Zike Yang. This sounds like an important
improvement. I have a couple thoughts.
1. Does the `retryLetterTopic` have the same problem with immediate
message expiration? Based on a quick glance, I don't see anything
creating a subscription for the retry topic.
2. Your
https://github.com/apache/pulsar/issues/13408
Pasted below for quoting convenience.
——
## Motivation
If we enable the DLQ when consuming messages. For some messages that can't be
processed successfully, the messages will be moved to the DLQ, but if we do not
specify the data retention for
24 matches
Mail list logo
