@Enrico Let me understand the issue: Bookies are up and running but ZK doesn't show bookies on the list.
Do you see if session expired or not? or bookies are hung in some way? We have seen multiple situations in this area: 1. Bookie process is around, but zk session lost. Alost all the time we ran into this when bookie is hung in some sense. 2. Bookies are down, but ZK still shows the Bookie in RW list. We suspect this is a ZK bug which got fixed in later releases. 3. Bookies are hug at disk, but was able to keep up ZK session. Does your case fit into any of these situations? Or do you believe that the Bookie is healthy and up but lost ZK session? If so how did you validate that your Bookie is healthy.? JV On Fri, Jun 1, 2018 at 12:11 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > Il ven 1 giu 2018, 08:49 Sijie Guo <guosi...@gmail.com> ha scritto: > > > I don't think there is any zk changes between 4.6.2 and 4.7.0. Are you > sure > > the upgrade fixes the problem? > > > > I have checked several times and it seems to me that every zk fix in 4.7.0 > has been cherry picked to 4.6.2. > It is only a fact that with the upgrade the issue does not appear. Maybe it > is too early to say that it is working. > > I will send news > > Enrico > > > > > - Sijie > > > > On Thu, May 31, 2018 at 11:30 PM, Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > Seems that al the sites which are reporting this kind of problems are > > ONLY > > > on 4.6.2. > > > > > > After an upgrade to 4.7.0 apparently the problem disappears. > > > > > > I will send news next week > > > > > > Enrico > > > > > > Il dom 20 mag 2018, 19:18 Enrico Olivelli <eolive...@gmail.com> ha > > > scritto: > > > > > > > My guess is that is about using zk ACLs > > > > I have no evidence > > > > > > > > Enrico > > > > > > > > > > > > Il mar 15 mag 2018, 14:09 Enrico Olivelli <eolive...@gmail.com> ha > > > > scritto: > > > > > > > >> Il giorno mar 15 mag 2018 alle ore 14:04 Sijie Guo < > > guosi...@gmail.com> > > > >> ha scritto: > > > >> > > > >>> On Tue, May 15, 2018 at 4:45 AM, Enrico Olivelli < > > eolive...@gmail.com> > > > >>> wrote: > > > >>> > > > >>> > Hi, > > > >>> > it is quite some time that we are seeing Bookies in staging > > > >>> environments > > > >>> > which disappear from ZK but appartently are still up and running. > > > >>> > > > > >>> > I have not dug deeply into this problem, but at first glance it > > > should > > > >>> be > > > >>> > related to ZK session expiration, those machines are heavily > loaded > > > >>> > sometimes and it is not surprising that ZK session expires. > > > >>> > > > > >>> > > > >>> There should be already a logic on re-registration after session > > > expired, > > > >>> no? > > > >>> > > > >> > > > >> Yes. The fact is that in the month we are seeing this strange > > behaviour. > > > >> I don't know if it could be a regression on 4.7. > > > >> I have no reports from production sites, but in production we have > > > >> dedicated machines for bookies. > > > >> > > > >> > > > >>> > > > >>> ZooKeeper stats should always show whether a bookie is able to > > connect > > > to > > > >>> zookeeper. That would probably tell you what happens. > > > >>> > > > >> > > > >> I will check, thank you for your suggestion. > > > >> > > > >> Enrico > > > >> > > > >> > > > >> > > > >>> > > > >>> > > > >>> > > > > >>> > Apart from searching for a bug, I wonder if would it be useful an > > > >>> automatic > > > >>> > self check of the bookie, something like a periodic check which > > asks > > > >>> to the > > > >>> > Registration Manager if the bookie is listed in the expected > bookie > > > >>> list > > > >>> > (readonly/available....) > > > >>> > > > > >>> > This will be useful even if we are not using ZK as well, now that > > we > > > >>> have > > > >>> > this great abstraction of ZK > > > >>> > > > > >>> > Thoughts ? > > > >>> > > > > >>> > Enrico > > > >>> > > > > >>> > > > >> -- > > > > > > > > > > > > -- Enrico Olivelli > > > > > > > -- > > > > > > > > > -- Enrico Olivelli > > > > > > -- > > > -- Enrico Olivelli > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
