Matteo we are in the process of enhancing bk shell in many ways and we are well into getting patches ready. I will open jira tickets but here is what we are working on:
- Add flush command to bookie shell - Make bookie shell usable on dev box including using local bookie; so basically we must be able to run bookie specific commands even while using local bookie. - Multiple formats for ExtentId in bk shell; input/output to take log/hex/uuid formats - Spit out list of disk files, including index,journal,and ledger files - enhance readlog command just to take a particular entryId and print that. Are you guys working on anything else either in bk shell or in general diagnostic tools? On Thu, Mar 10, 2016 at 10:59 PM, Arun M. Krishnakumar < [email protected]> wrote: > Thanks Matteo. I will raise the other ticket in a bit. > > Thanks, > Arun > > On Thu, Mar 10, 2016 at 10:57 PM, Matteo Merli <[email protected]> > wrote: > > > FYI: we submitted the PR for adding the tool in "bookkeeper shell" to > dump > > ledger entries. > > > > https://issues.apache.org/jira/browse/BOOKKEEPER-894 > > > > On Thu, Mar 3, 2016 at 10:19 PM Arun M. Krishnakumar <[email protected] > > > > wrote: > > > > > Thanks Sijie for the +1. We will raise a bug on the admin portal and > > > discuss possible methods of approach there. > > > > > > Thanks, > > > Arun > > > > > > On Tue, Mar 1, 2016 at 10:18 AM, Sijie Guo <[email protected]> wrote: > > > > > > > Yup. I think it is in general a good idea to have such admin portal > > that > > > > could query or control the internal state. > > > > > > > > +1 for it. > > > > > > > > - Sijie > > > > > > > > On Mon, Feb 29, 2016 at 10:34 AM, Arun M. Krishnakumar < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Hi Sijie, > > > > > > > > > > Yes, that's correct. The work-item is about building an admin > command > > > to > > > > > force the flush and I was looking for any option or workaround to > do > > > the > > > > > job. > > > > > > > > > > As we understand it, there doesn't seem to be a nice way to > > communicate > > > > > with the bookies for such OOB (out of band) requests. So one of the > > > > options > > > > > is to have a separate communication channel from the shell to the > > > bookie. > > > > > This would enable other interesting commands such as querying the > > exact > > > > > states of various caches, getting some statistical numbers etc. > > > > > > > > > > Do you have any other requirements that would fall into this > general > > > > > operational category ? If so, and if the approach isn't too bad, we > > > could > > > > > work on a clean channel and protocol that could be used for such > > > > > communications. > > > > > > > > > > Thanks, > > > > > Arun > > > > > > > > > > On Mon, Feb 29, 2016 at 9:45 AM, Sijie Guo <[email protected]> > wrote: > > > > > > > > > > > We don't have the mechanism to trigger/force a flush by the > command > > > > right > > > > > > now. > > > > > > > > > > > > Arun: > > > > > > > > > > > > As my understanding on your question, you want to admin command > > that > > > > you > > > > > > could issue to trigger the flush, right? > > > > > > > > > > > > - Sijie > > > > > > > > > > > > On Fri, Feb 26, 2016 at 10:33 PM, Venkateswara Rao Jujjuri < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Also do we have a way to flush to ledger on disk? Some of the > > > bookie > > > > > > shell > > > > > > > commands will fail if there is no disk entry for journals. > > > > > > > > > > > > > > Thanks, > > > > > > > JV > > > > > > > > > > > > > > On Sat, Feb 27, 2016 at 5:29 AM, Arun M. Krishnakumar < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > (Please ignore the first part of the question since that can > be > > > > done > > > > > > by a > > > > > > > > shell command. I am interested in the second part.) > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Arun > > > > > > > > > > > > > > > > On Fri, Feb 26, 2016 at 1:05 PM, Arun M. Krishnakumar < > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Matteo, > > > > > > > > > > > > > > > > > > Thanks for the quick reply. > > > > > > > > > > > > > > > > > > The use case is mainly for testing and debugging during > > > > > development. > > > > > > > > > > > > > > > > > > Is there a way to check the entry in the journal device > (say > > a > > > > > > running > > > > > > > > > count of pending entries yet to be written out to the entry > > > log) > > > > ? > > > > > > > > > Ideally I was thinking of querying the entry log itself. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Arun > > > > > > > > > > > > > > > > > > > On Feb 26, 2016, at 12:34 PM, Matteo Merli < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Arun, > > > > > > > > > > > > > > > > > > > > what is your use case for waiting until the entries are > > > > "flushed" > > > > > > > into > > > > > > > > > the > > > > > > > > > > entry logs? > > > > > > > > > > > > > > > > > > > > Once the ledger.addEntry() operation completes, the > entries > > > are > > > > > > > > > guaranteed > > > > > > > > > > to have been written and fsynced on the journal device, > > from > > > > > where > > > > > > > they > > > > > > > > > > will be recovered in case the bookie crashes before > > flushing > > > to > > > > > the > > > > > > > > entry > > > > > > > > > > log. > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > On Fri, Feb 26, 2016 at 12:13 PM Arun M. Krishnakumar < > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Hi, > > > > > > > > > >> > > > > > > > > > >> (I am new to bookkeeper, so please forgive my ignorance > if > > > > there > > > > > > is > > > > > > > > > >> something fundamental I'm missing). > > > > > > > > > >> > > > > > > > > > >> When working with bookkeeper and making entries we would > > > like > > > > to > > > > > > be > > > > > > > > > able to > > > > > > > > > >> see entries flushed to disk and then examine them using > > > > existing > > > > > > > shell > > > > > > > > > >> commands (for debugging and test purposes). > > > > > > > > > >> > > > > > > > > > >> We typically use the "localbookie" command-line option > and > > > > need > > > > > to > > > > > > > > have > > > > > > > > > a > > > > > > > > > >> way to know if an entry written has made it correctly to > > > > > > bookkeeper > > > > > > > > and > > > > > > > > > the > > > > > > > > > >> disk. > > > > > > > > > >> > > > > > > > > > >> Currently the ways we are planning to do this are: > > > > > > > > > >> 1. To modify the configuration to set very low > thresholds > > > for > > > > > > > > flushing. > > > > > > > > > >> 2. Create a listener thread on the localbookie which > will > > > > listen > > > > > > to > > > > > > > > > flush > > > > > > > > > >> commands and forcibly do a flush. (I know this is a > hack, > > > but > > > > > > just > > > > > > > > > putting > > > > > > > > > >> it out there) > > > > > > > > > >> > > > > > > > > > >> Is there a cleaner way to do this ? I think people may > > have > > > > > wanted > > > > > > > > some > > > > > > > > > >> similar functionality in the past. > > > > > > > > > >> > > > > > > > > > >> Thanks, > > > > > > > > > >> Arun > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Jvrao > > > > > > > --- > > > > > > > First they ignore you, then they laugh at you, then they fight > > you, > > > > > then > > > > > > > you win. - Mahatma Gandhi > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
