Hi Kannar, +1 to find the position first and then read like normal as mentioned by Yong and Michael.
Another problem of reading reverse is that it would break all the read ahead techniques in the storage and result in very poor performance. > This would work but it will need something to store every messages read > to reverse them before answer which can be heavy in RAM usages. Finding the position doesn't require reading all the messages body. Just use the ledger metadata info and maybe some message heads in the last ledger would be enough. Thanks, Haiting On Thu, Mar 9, 2023 at 11:09 AM Zike Yang <z...@apache.org> wrote: > > > Have you looked at the seek implementation to see if it would be > feasible to extend the implementation and add a method to "seekBefore" > a message id in the way you described? > > I think it's not very feasible for this case. Seeking before can lead > to consumer reconnection, which can cause significant performance > issues and overhead. > > > Zike Yang > > On Thu, Mar 9, 2023 at 10:22 AM Yong Zhang <zhangyong1025...@gmail.com> wrote: > > > > Kannar, > > > > Why not find the stop position first, then read the message > > until a given position? > > Does the stop position change dynamically? You only know > > it once you meet it? > > > > Yong > > > > > > > > On Wed, 8 Mar 2023 at 21:37, Alexandre DUVAL > > <alexandre.du...@clever-cloud.com.invalid> wrote: > > > > > Hi Michael, > > > > > > This would work but it will need something to store every messages read > > > to reverse them before answer which can be heavy in RAM usages. The key > > > point of the future is to read message by message from a MessageId to > > > past with stop read possible conditions. > > > > > > Best, > > > > > > Kannar > > > > > > On 3/7/23 22:10, Michael Marshall wrote: > > > >> The goal is to start from a known MessageId and read the N message > > > >> before this MessageId. > > > > Have you looked at the seek implementation to see if it would be > > > > feasible to extend the implementation and add a method to "seekBefore" > > > > a message id in the way you described? I haven't considered all of the > > > > implications, but if the main goal is to move the cursor, I think the > > > > solution should be about moving the cursor, not about reading a topic > > > > in reverse. > > > > > > > > Thanks, > > > > Michael > > > > > > > > On Tue, Mar 7, 2023 at 6:50 AM Alexandre DUVAL <kanna...@gmail.com> > > > wrote: > > > >> Hi Yong, > > > >> > > > >> The goal is to start from a known MessageId and read the N message > > > >> before this MessageId. > > > >> > > > >> Best, > > > >> > > > >> Kannar > > > >> > > > >> > > > >> On 3/7/23 01:53, Yong Zhang wrote: > > > >>> Hi Kannar, > > > >>> > > > >>> Just interested in what exactly your case. > > > >>> > > > >>> Why do you need to read messages in a reversed order? What is your > > > case? > > > >>> > > > >>> Best, > > > >>> Yong > > > >>> > > > >>> On Mon, 6 Mar 2023 at 23:37, Alexandre DUVAL<kanna...@gmail.com> > > > wrote: > > > >>> > > > >>>> Hi, > > > >>>> > > > >>>> I'm wondering if it is possible to introduce a new feature on Pulsar > > > >>>> which will enable users to read topic from a defined MessageId to > > > >>>> previous messages until the begin of the topic. > > > >>>> > > > >>>> I tried to use Pulsar SQL but it requires so much RAM even for little > > > >>>> queries (due to Presto design). > > > >>>> > > > >>>> Currently, every read in Pulsar are expected to be going forward. So > > > it > > > >>>> might be a bit tricky to prevent every weird behavior by introducing > > > the > > > >>>> feature. > > > >>>> > > > >>>> I'm currently tried to make an MVP/POC by introducting a readReverse > > > >>>> field in the CommandSubscribe that is used by ReaderAPI and currently > > > >>>> looking for to create a getFirstMessageId() on ManagedLedger > > > >>>> (https://github.com/CleverCloud/pulsar/pull/3). I also removed > > > >>>> startPosition < endPosition sanity checks in BookKeeper locally > > > >>>> (https://github.com/CleverCloud/bookkeeper/pull/2). > > > >>>> > > > >>>> We definitely prefer a readPrevious(), hasPreviousMessageAvailable() > > > in > > > >>>> the ReaderAPI. > > > >>>> > > > >>>> I'm not familiar with these internals such as NonDurableCursor, > > > >>>> RangeEntryCache, ManagedCursor so it's a bit tricky. > > > >>>> > > > >>>> So I wondering someone to help/guide me or even directly handle the > > > >>>> subject (or the discuss). > > > >>>> > > > >>>> Regards, > > > >>>> > > > >>>> Kannar > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > >
