Apparently, there is an assumption in this computation that the bookie sees entries in sequence. When ES>WQ bookie won't see them in sequence. I guess I need to understand what you are trying to do with in-memory cursor at the bookie. In our protocol, only the client(writer) will have the complete view and bookie view is persisted in the index files. If you want to know what is persisted in the journal, then you may need to maintain some in-memory value but that is just one value isn't it? It could be I don't fully understand what is the intended change here, maybe let's have a quick call or discuss at next Thursday meeting?
On Sat, Apr 21, 2018 at 12:37 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > Il sab 21 apr 2018, 09:11 Venkateswara Rao Jujjuri <jujj...@gmail.com> ha > scritto: > > > Sorry, maxEntryId can't be used for LAC. Let me review the patch and > > respond tomorrow. :) > > I was mostly thinking of sync-on-close which is my use case. Intermittent > > syncs will need a more robust way to define the LAC. > > > > > Thank you JV, > I think that about LAC updates we are already on our way as discussed on > BP14 and there is already my patch about client side LAC management (not > mergeable untill we merge server side changes). > > My problem is about reconstructing the knowledge of which entries are > stored on the bookie after a restart. > In my current impl I am keeping an in memory 'cursor' which tracks which > entries are on bookie (fsync'd to journal), but after a restart this cursor > is to be recreated. > We need this cursor because we can have temporary holes in the sequence of > entries, think about WriteAdvHandle. > LAC does not advance in presence of holes, and in particular it does not > advance without fsync. > > Even without ensemble changes a bookie can restart without breaking a > writer, for instance in the case of no writes during the restart. > > Enrico > > > > On Sat, Apr 21, 2018 at 12:00 AM, Venkateswara Rao Jujjuri < > > jujj...@gmail.com> wrote: > > > > > @Enrico I thought I had a pretty simple view on this. > > > 1. No ensemble changes allowed on a ledger in this mode. > > > 2. A sync is sent to all bookies on the ensemble and it is required > that > > > the sync must be successful to proceed forward. > > > 3. Sync and Write are serialized. or sync can return maxEntryId. > > > 4. If we get a successful sync from all bookies, then the ' smallest > > > maxEntryId ' returned will be the LAC. > > > 5. An implicit sync is sent on close. > > > > > > IMO, no ensemble change will make the whole thing simple and avoids > > scans > > > from client side. > > > > > > > > > On Wed, Apr 18, 2018 at 2:02 AM, Enrico Olivelli <eolive...@gmail.com> > > > wrote: > > > > > >> Any other suggestion? JV, Sijie? > > >> My current idea for bp14 is to have a client driven scan. > > >> > > >> > > >> > > >> Enrico > > >> > > >> Il ven 13 apr 2018, 10:22 Enrico Olivelli <eolive...@gmail.com> ha > > >> scritto: > > >> > > >> > > > >> > > > >> > Il gio 12 apr 2018, 19:59 Sam Just <sj...@salesforce.com> ha > scritto: > > >> > > > >> >> IIRC, InterleavedLedgerStorage has for each ledger an index file > > >> >> mapping the entries to entry logger offsets, you could probably > scan > > >> >> that directly (particularly if you included a lower ?bound -- > > probably > > >> >> the client's current idea of the LAC). > > >> >> -Sam > > >> >> > > >> > > > >> > Thank you Sam, > > >> > I think that the major point is about introducing a standard > interface > > >> to > > >> > be implemented by LedgerStorage, now we already have 3 > > implementations. > > >> > > > >> > Options are: > > >> > 1) let the client drive the scan > > >> > 2) add a scan() method to LedgerStorage > > >> > 3) add a getMaxEntry() to LedgerStorage > > >> > > > >> > For bp14 I just need 1) or 3) but 1) is also a protocol issue and so > > it > > >> is > > >> > more expensive for long term maintenance > > >> > > > >> > Enrico > > >> > > > >> >> > > >> >> On Thu, Apr 12, 2018 at 12:31 AM, Enrico Olivelli < > > eolive...@gmail.com > > >> > > > >> >> wrote: > > >> >> > Hi BookKeepers, > > >> >> > during implementation of BP-14 I am facing a problem so I am > asking > > >> for > > >> >> > suggestions. > > >> >> > > > >> >> > My need is to be able to know the list of all entries stored on a > > >> >> > LedgerStorage given a ledgerId. > > >> >> > > > >> >> > Scanning from 0 to LedgerStorage#getLastAddConfirmed() does not > > >> seem to > > >> >> > work because we have to deal with WriteAdvHandle, so there can be > > >> >> temporary > > >> >> > "gaps" in the sequence of entries. > > >> >> > > > >> >> > I can have a writer which writes entries 0,1,5,6,7. Its LAC will > be > > >> at > > >> >> most > > >> >> > 1 as entries 2,3,4 are not written yet. > > >> >> > I need on the bookie to able to know that entries 0, 1, 5, 6, 7 > are > > >> >> stored > > >> >> > on LedgerStorage. > > >> >> > > > >> >> > I cannot issue a scan from 0 to Long.MAX_VALUE, my current > > 'solution' > > >> >> it to > > >> >> > make the client (writer) send the 'maximum entry id' and perform > a > > >> scan > > >> >> > from 0 to maxEntryId. > > >> >> > In the example the writer will send a forceLedger RPC with > > >> maxEntryId = > > >> >> 7. > > >> >> > > > >> >> > This is need only for recoveries are bookie restart because I > have > > to > > >> >> > reconstruct the knowledge about which entries have been persisted > > >> >> durably > > >> >> > on the Bookie. > > >> >> > > > >> >> > I am not very expert about LedgerStorage implementations, and I > > don't > > >> >> know > > >> >> > if it would be feasible to have such 'scan all entries' method. > > >> >> > > > >> >> > This is the code I am talking about > > >> >> > > > >> >> https://github.com/apache/bookkeeper/pull/1317/files#diff-3b > > >> 81b1c90d1f51017627b3c032676168R1210 > > >> >> > > > >> >> > Any help is really appreciated > > >> >> > Enrico > > >> >> > > >> > -- > > >> > > > >> > > > >> > -- Enrico Olivelli > > >> > > > >> -- > > >> > > >> > > >> -- Enrico Olivelli > > >> > > > > > > > > > > > > -- > > > Jvrao > > > --- > > > First they ignore you, then they laugh at you, then they fight you, > then > > > you win. - Mahatma Gandhi > > > > > > > > > > > > > > > -- > > Jvrao > > --- > > First they ignore you, then they laugh at you, then they fight you, then > > you win. - Mahatma Gandhi > > > -- > > > -- Enrico Olivelli > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
