We do have a backlog quota and messageTTL
Our namespace is configured as follows:
Backlog quota:
*"message_age BacklogQuotaImpl(limit=-1, limitSize=-1, limitTime=180000,
policy=consumer_backlog_eviction)"*
Retention:
*{ "retentionTimeInMinutes" : 0, "retentionSizeInMB" : 0}*
Message TTL:
*172800*
All topics are partitioned.
*{
"partitions" : 3
}*
On Thu, Mar 10, 2022 at 11:24 PM PengHui Li <[email protected]> wrote:
> Have you changed the backlog quota policy or enabled the message TTL?
> Pulsar will not remove any cursors or skip messages by default.
>
> Penghui
> On Mar 11, 2022, 1:25 PM +0800, Tecno Brain <[email protected]>,
> wrote:
>
> Hello,
> I have an application using Pulsar just as JMS (we get single messages,
> acknowledge them when we are done processing it)
> The entire system, composed of several types of apps, uses about 40
> different topics.
>
> Yesterday, an application that subscribes to about 20 queues, suddenly
> was inundated with thousands of messages from two of the queues but I could
> not track those messages to an application producing them. We found that
> the messages were *duplicates*.
> So it seems that the cursor to these two topics was lost, and messages
> from 3 hours earlier were consumed again. I found the following paragraph :
>
> "Each subscription stores a cursor. The cursor is the current offset in
> the log. Subscriptions store their cursor in BookKeeper in ledgers. This
> makes cursor tracking scalable just like topics."
> (
> https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
> )
>
> My guess is that the cursor was lost.
> How could I verify that this was the case? I can't find anything relevant
> in the logs.
>
> The only message I found occurring around the same time is
>
> New ensemble:
> [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181] is not
> adhering to Placement Policy
>
> Any pointers are appreciated.
>
>
>
>
