Kannar, Why not find the stop position first, then read the message until a given position? Does the stop position change dynamically? You only know it once you meet it?
Yong On Wed, 8 Mar 2023 at 21:37, Alexandre DUVAL <alexandre.du...@clever-cloud.com.invalid> wrote: > Hi Michael, > > This would work but it will need something to store every messages read > to reverse them before answer which can be heavy in RAM usages. The key > point of the future is to read message by message from a MessageId to > past with stop read possible conditions. > > Best, > > Kannar > > On 3/7/23 22:10, Michael Marshall wrote: > >> The goal is to start from a known MessageId and read the N message > >> before this MessageId. > > Have you looked at the seek implementation to see if it would be > > feasible to extend the implementation and add a method to "seekBefore" > > a message id in the way you described? I haven't considered all of the > > implications, but if the main goal is to move the cursor, I think the > > solution should be about moving the cursor, not about reading a topic > > in reverse. > > > > Thanks, > > Michael > > > > On Tue, Mar 7, 2023 at 6:50 AM Alexandre DUVAL <kanna...@gmail.com> > wrote: > >> Hi Yong, > >> > >> The goal is to start from a known MessageId and read the N message > >> before this MessageId. > >> > >> Best, > >> > >> Kannar > >> > >> > >> On 3/7/23 01:53, Yong Zhang wrote: > >>> Hi Kannar, > >>> > >>> Just interested in what exactly your case. > >>> > >>> Why do you need to read messages in a reversed order? What is your > case? > >>> > >>> Best, > >>> Yong > >>> > >>> On Mon, 6 Mar 2023 at 23:37, Alexandre DUVAL<kanna...@gmail.com> > wrote: > >>> > >>>> Hi, > >>>> > >>>> I'm wondering if it is possible to introduce a new feature on Pulsar > >>>> which will enable users to read topic from a defined MessageId to > >>>> previous messages until the begin of the topic. > >>>> > >>>> I tried to use Pulsar SQL but it requires so much RAM even for little > >>>> queries (due to Presto design). > >>>> > >>>> Currently, every read in Pulsar are expected to be going forward. So > it > >>>> might be a bit tricky to prevent every weird behavior by introducing > the > >>>> feature. > >>>> > >>>> I'm currently tried to make an MVP/POC by introducting a readReverse > >>>> field in the CommandSubscribe that is used by ReaderAPI and currently > >>>> looking for to create a getFirstMessageId() on ManagedLedger > >>>> (https://github.com/CleverCloud/pulsar/pull/3). I also removed > >>>> startPosition < endPosition sanity checks in BookKeeper locally > >>>> (https://github.com/CleverCloud/bookkeeper/pull/2). > >>>> > >>>> We definitely prefer a readPrevious(), hasPreviousMessageAvailable() > in > >>>> the ReaderAPI. > >>>> > >>>> I'm not familiar with these internals such as NonDurableCursor, > >>>> RangeEntryCache, ManagedCursor so it's a bit tricky. > >>>> > >>>> So I wondering someone to help/guide me or even directly handle the > >>>> subject (or the discuss). > >>>> > >>>> Regards, > >>>> > >>>> Kannar > >>>> > >>>> > >>>> > >>>> >
