>From my knowledge, when a client does negative ack, after a certain amount
of time, it sends the following command to the broker containing the
negatively acked messages ids:
message CommandRedeliverUnacknowledgedMessages {
required uint64 consumer_id = 1;
repeated MessageIdData message_ids = 2;
optional uint64 consumer_epoch = 3;
}
On Tue, Jan 17, 2023 at 5:24 AM Nitin Goyal <nitin.goyal....@gmail.com>
wrote:
> Hello all,
>
> I have a few questions.
>
> 1. When a message is NACKed does it still send the commands to the broker?
> 2. Do we have any plan to replace the ReconsumeLater function with NACK
> Backoff interface in future?
>
>
> On Tue, Jan 10, 2023 at 7:17 PM Nitin Goyal <nitin.goyal....@gmail.com>
> wrote:
>
> > Hello Team,
> >
> > Thanks for pointing these things out.
> >
> > First thing is to consider that this addon feature is for the message
> > which needs to go to DLQ after a retryLater reaches the maximum limit of
> > consumer level retries. It allows an additional maximum limit if a
> message
> > has one at the time of producing.
> >
> > There are two types of retry in current architecture which consumers can
> > go with.
> >
> > 1. Process later with an NACK backoff policy: In this case if someone
> > wants to retry and wants to set a maximum retry at message processor
> level.
> > They can get the number of times the message has been retried so far
> > by calling the `reconsumeTimes` function. they will match the number with
> > their custom retry. Now there are two things that will happen. Either
> they
> > will call ACK that particular message or they have to call the
> > `reconsumeLater` function based on system requirement. If they call ACK
> > then the message is marked as delete in the ledger. And other cases would
> > be explained in point 2.
> >
> > 2. By calling reconsumeLater: In some real cases if someone is calling
> > reconsumeLater with a delay like they want to process a message again but
> > after a particular time. so the new message gets created in the consumer
> > RETRY topic which was again processed by the consumer (they may call the
> > reconsumeLater again if the system requires based on use cases.). Now if
> > the message reaches its retry limit which was configured at consumer
> level
> > they will be sent to the DLQ message.
> >
> > So my proposal will not make any changes to 1st scenario which is to use
> > NACK backoff for retrying the message and process. Consumers can use the
> > same mechanism as they are doing now.
> >
> > It adds an additional feature to pulsar in which if a producer knows how
> > many times an event should be processed a max apart from the consumer max
> > retry and send to DLQ if reaches its limit. Then the above change will
> > check if the limit is reached for reconsume later then it will send to
> DLQ
> > even if the consumer has higher reconsme later retry limit.
> >
> > if producers do not pass the max retry limit it will obey the consumer
> > retry limit.
> >
> > Also in this PIP we are not removing the consumer own retry limit it
> would
> > be there. but get overridden by message limit if passed by the producer.
> if
> > a consumer wants to re-override that limit ( temporarily change in the
> > retry limit) which also can be achieved by sending a new max retry as
> > custom property to the `reconsumeLater` function.
> >
> > So let me rephrase the subject for this pip is to `*MAX RECONSUME TIMES
> > per Message*`.
> >
> > Michael's point about adding relation between producer and consumer?
> >
> > Yeah, It added a relation b/w them but it would be loose coupling.
> Because
> > producers can send the max retry limit in a message which can be
> overridden
> > by the consumer if needed. so consumers do not have to obey the limit if
> > needed.
> >
> > Why I prefer message properties over `MessageMetadata` is because if a
> > consumer wants to override the max limit (temporarily or permanent) then
> it
> > would be difficult to update if required.
> >
> > I hope the above content answers most of the doubts about this PIP.
> >
> > On Tue, Jan 10, 2023 at 5:59 PM r...@apache.org <ranxiaolong...@gmail.com
> >
> > wrote:
> >
> >> Thanks for submitting this PIP, Nitin Goyal.
> >>
> >> Seeing that you have submitted a related pull request in the Go SDK
> >> community, I am sorry that I made the request changes first.
> >>
> >> For the retry processing of a single message, the methods currently
> >> provided are:
> >>
> >> -ReconsumeLater
> >> -Nack
> >>
> >> In actual usage scenarios, it may be a more elegant way to retry if we
> >> verify Nack multiple times. The current implementation of ReconsumeLater
> >> relies on delayed messages. This is not an elegant way, and there is no
> >> way
> >> to record the number of retries.
> >>
> >> In order to support backoff retries, we introduced the NackBackoff
> >> strategy, which itself is an interface and exposed to users. Based on
> this
> >> interface, we can do more customized things. If the NackBackoff
> interface
> >> can't meet our current needs, I prefer that we implement more parameters
> >> in
> >> the NackBackoff strategy instead of continuing to add new functional
> logic
> >> in ReconsumeLater.
> >>
> >> --
> >> Thanks
> >> Xiaolong Ran
> >>
> >> Enrico Olivelli <eolive...@gmail.com> 于2023年1月10日周二 16:23写道：
> >>
> >> > I think that Michael's point is very important.
> >> >
> >> > Producers and Consumers are decoupled and this PIP would introduce
> >> > a new concept.
> >> > Also the same Message can be consumed using multiple subscriptions
> >> > (typically Applications)
> >> > and all the applications will process the message in a different way
> >> > (by definition).
> >> >
> >> > Isn't it possible to implement what you want with an Interceptor or
> >> > with some custom handling for the DLQ ?
> >> > The DLQ (currently) in Pulsar is mostly a client-side feature.
> >> > The producer can tag the messages with a message property and the
> >> > client can decide what to do
> >> > and how many times to retry the message or give up.
> >> >
> >> > Enrico
> >> >
> >> > Il giorno mar 10 gen 2023 alle ore 01:00 Michael Marshall
> >> > <mmarsh...@apache.org> ha scritto:
> >> > >
> >> > > Thanks for submitting this PIP, Nitin Goyal.
> >> > >
> >> > > At the heart of this PIP is an assumption about the relationship
> >> > > between producers and consumers. The PIP assumes a producer knows
> how
> >> > > many times a consumer should attempt to consume a message before
> >> > > giving up and sending it to the DLQ. Does anyone have strong
> opinions
> >> > > on the boundaries between producers and consumers in relation to
> this
> >> > > PIP?
> >> > >
> >> > > This PIP expands the relationship between producer and consumer by
> >> > > letting the producer tell the consumer's pulsar client when to send
> a
> >> > > message to the DLQ, and as such, we should be very intentional about
> >> > > accepting this PIP.
> >> > >
> >> > > Because a user can easily implement this feature on their own and
> >> > > because it tightly couples producers and consumers, I think we
> should
> >> > > not move forward with this PIP. I am open to discussion, though.
> >> > >
> >> > > > for instance in case of major system failure you don't want to
> lose
> >> > > > all your messages.
> >> > >
> >> > > For what it's worth, the retry letter topic feature, which this PIP
> >> > > relies on, sends all messages to the DLQ, so this feature does not
> >> > > introduce conditions for message loss.
> >> > >
> >> > > As an aside, if we move forward with this feature, we need to make
> >> > > sure that the protocol documentation is updated and we should
> consider
> >> > > putting this field in the `MessageMetadata` protobuf object, not in
> a
> >> > > properties map.
> >> > >
> >> > > Thanks,
> >> > > Michael
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Jan 9, 2023 at 9:47 AM Nitin Goyal <
> nitin.goyal....@gmail.com
> >> >
> >> > wrote:
> >> > > >
> >> > > > Hello Enrico,
> >> > > >
> >> > > > For your concern about temporarily increasing of retry. It can be
> >> > achieved
> >> > > > using overriding msg property while calling reconsuming later with
> >> > custom
> >> > > > properties..
> >> > > >
> >> > > > About msg immutable as per current design if consumer call
> >> > reconsumeLater
> >> > > > function it creates a new msg in the system adding few properties
> >> to it
> >> > > > like how many times it get consumed. That also allow other custom
> >> > > > properties to be added in newly generated msg.. so if msg needs
> >> > temporary
> >> > > > high retry or change in retry count on msg they can override it
> >> using
> >> > > > custom properties…
> >> > > >
> >> > > > Thanks
> >> > > > Nitin Goyal
> >> > > >
> >> > > >
> >> > > > On Mon, 9 Jan 2023 at 1:11 PM, Enrico Olivelli <
> eolive...@gmail.com
> >> >
> >> > wrote:
> >> > > >
> >> > > > > I don't think that this is a good idea.
> >> > > > >
> >> > > > > Because IIUC we want to add a property per message that sets the
> >> > > > > maximum time of retries.
> >> > > > >
> >> > > > > Unfortunately in a real system sometimes you have to change the
> >> > number
> >> > > > > of retries temporarily,
> >> > > > > for instance in case of major system failure you don't want to
> >> lose
> >> > > > > all your messages.
> >> > > > >
> >> > > > > If we allow you to set a property on a message then you won't be
> >> able
> >> > > > > to change it because the message is immutable.
> >> > > > >
> >> > > > > TTL (time to live) is a similar concept but it is related to the
> >> > > > > concept of "physical time", if you have message that represents
> >> > > > > a task to be executed within a given deadline then it makes
> sense
> >> to
> >> > > > > state it in the message metadata.
> >> > > > >
> >> > > > > But the "number of retries" depends on how you deal with the
> >> retries:
> >> > > > > - how much time do you wait ?
> >> > > > > - how often do you have a temporary failure to retry ?
> >> > > > >
> >> > > > > It would make more sense to have a QOS (quality of service)
> >> attribute
> >> > > > > on the message, like "important/"non-important"/"foo"/"bar"
> >> > > > > and have a way for the brokers and the clients to handle that. I
> >> am
> >> > > > > pretty sure that with interceptors you can already do something.
> >> > > > >
> >> > > > >
> >> > > > > I am against hard coding the behaviour described in the PIP
> (and I
> >> > > > > voted -1 in the VOTE thread)
> >> > > > >
> >> > > > > Enrico
> >> > > > >
> >> > > > > Il giorno ven 6 gen 2023 alle ore 09:11 Zike Yang <
> >> z...@apache.org>
> >> > ha
> >> > > > > scritto:
> >> > > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > This looks good to me.
> >> > > > > > +1
> >> > > > > >
> >> > > > > > I was thinking if we could add a new API for `reconsumeLater`
> to
> >> > let
> >> > > > > > users set the max retry times easily. But I saw that there are
> >> too
> >> > > > > > many reconsumeLater API and this will make the consumer more
> >> > complex.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Zike Yang
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Jan 5, 2023 at 3:58 PM Zixuan Liu <node...@gmail.com>
> >> > wrote:
> >> > > > > > >
> >> > > > > > > +1
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Zixuan
> >> > > > > > >
> >> > > > > > > Nitin Goyal <nitin.goyal....@gmail.com> 于2023年1月5日周四
> 13:50写道：
> >> > > > > > >
> >> > > > > > > > Hi all,
> >> > > > > > > >
> >> > > > > > > > I made a PIP to discuss:
> >> > > > > https://github.com/apache/pulsar/issues/19136
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Nitin Goyal
> >> > > > > > > >
> >> > > > >
> >> > > > --
> >> > > > Regards
> >> > > > Nitin Goyal
> >> >
> >>
> >
> >
> > --
> > Regards
> > Nitin Goyal
> >
>
>
> --
> Regards
> Nitin Goyal
>
