[ https://issues.apache.org/jira/browse/ACCUMULO-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-954: ---------------------------------- Priority: Critical (was: Minor) > ZooLock watcher can stop watching > --------------------------------- > > Key: ACCUMULO-954 > URL: https://issues.apache.org/jira/browse/ACCUMULO-954 > Project: Accumulo > Issue Type: Bug > Components: tserver > Affects Versions: 1.4.2 > Reporter: Adam Fuchs > Assignee: Keith Turner > Priority: Critical > Fix For: 1.5.0, 1.4.3 > > > Basically, this will result in tablet servers failing to recognize when they > lose their locks. I think the worst that can happen with this is a tablet > server can fail to die after it loses its lock, which could bog down clients > and create a bunch of noise in the cluster. I believe there could also be > useless files generated that wouldn't get garbage collected. !METADATA table > write protections and logger write protections should prevent any permanent > damage or data loss. We have seen this result in warnings and errors that > look like multiple hosting of tablets. > {code} > 2013-01-09 19:59:27,742 [tabletserver.TabletServer] INFO : port = 9997 > 2013-01-09 19:59:27,926 [zookeeper.ZooLock] DEBUG: event > /accumulo/655f93d8-20fc-451f-a457-458b5717a11e/tservers/172.16.2.25:9997 > NodeDeleted SyncConnected > 2013-01-09 19:59:27,931 [tabletserver.TabletServer] INFO : Waiting for tablet > server lock > 2013-01-09 19:59:32,943 [tabletserver.TabletServer] DEBUG: Obtained tablet > server lock > /accumulo/655f93d8-20fc-451f-a457-458b5717a11e/tservers/172.16.2.25:9997/zlock-0000000000 > 2013-01-09 19:59:36,703 [tabletserver.TabletServer] DEBUG: Got loadTablet > message from user: !SYSTEM > {code} > Here's what happened: > 1. Tablet server fails to get lock, triggering the watcher on the parent node. > 2. Watcher doesn't get reset, and doesn't take any action. > 3. Loop in TabletServer:~2659 retries, but uses the same ZooLock object. > 4. TabletServer loses its lock, but receives a connection loss message before > the NodeDeleted message. > 5. TabletServer continues to try to do work instead of killing itself. > We could probably patch this for 1.4 by creating the ZooLock within the > announceExistence loop, instead of reusing the one. Eventually, we ought to > have an else branch in both of the Watchers that either reset the watch > (resilient against zookeeper connection hiccups) or just kill the server to > be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira