[
https://issues.apache.org/jira/browse/ACCUMULO-327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith Turner resolved ACCUMULO-327.
-----------------------------------
Resolution: Fixed
> master lost all tablet servers
> ------------------------------
>
> Key: ACCUMULO-327
> URL: https://issues.apache.org/jira/browse/ACCUMULO-327
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Environment: running the random walk test on a small cluster
> Reporter: Eric Newton
> Assignee: Keith Turner
> Fix For: 1.4.0
>
>
> Master would occasionally take a long time to collect status information from
> a tablet server. The connection would timeout after the default 120 second
> RPC time. This probably left the connection in a bad state because I am
> seeing
> {noformat}
> org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82
> but got 0
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:445)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_halt(TabletClientService.java:893)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.halt(TabletClientService.java:876)
> {noformat}
> If the master is unable to collect statistics on the tablet server, it
> attempts to halt it (as above) and then it removes its lock in zookeeper.
> Eventually, under the pressure of random walk operations, the master killed
> every tablet server.
> Guess: a lock in the tablet server is delaying status reporting.
> I wrote a script to process the master logs. It saves each line that refers
> to the IP address of a tablet server. When it sees the zookeeper lock has
> been deleted, it prints the last N lines that refer to that tablet server.
> In 7 out of the 10 cases, a split timed out prior or during the status
> request failures.
> In 5 cases, the tablet server was hosting the root tablet (a necessary
> condition when the last server died).
> In 5 cases, the table_table info tablet was being hosted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
