Jeff, This is the bug I ran into, https://issues.apache.org/jira/browse/ACCUMULO-4143. Looks like it was around migrations instead of splitting.
Mike On Wed, Oct 12, 2016 at 8:27 PM, Michael Wall <[email protected]> wrote: > Jeff, > > There was a race condition around splitting I think, but I can't find the > issue right now. I caught a stacktrace at one point doing just as Ed > suggests, then setting 'debug on' in the shell try the droptable again. > After I found the issue, I turned off all ingest and waited for the master > stop balancing and tables to stop splitting. My droptable was successfully > issued after that (doing Ed's trick once again). > > I can look for that issue tomorrow or if you catch a stacktrace, post it > here. > > Mike > > On Wed, Oct 12, 2016 at 7:57 PM, Ed Coleman <[email protected]> wrote: > >> This is a brute force approach and should only done as a last resort. >> Sometimes bouncing the master(s) helps – make sure all of stopped, not just >> failed over to another instance., but it sounds like you have tried that. >> >> >> >> You may be able to force it by manually editing zookeeper nodes using >> zkCli.sh You are editing zookeeper so make sure you have the correct table >> id and path(s) >> >> >> >> Add permission to write in zookeeper using: >> >> >> >> zkCli> addauth digest accumulo:YOUR_SECRET >> >> >> >> Under /accumulo/INSTANCE_ID/tables/TABLE_ID/state (going from memory, >> could be fate) you can use examine the state >> >> >> >> zkCli> ls /accumulo/INSTANCE_ID/tables/TABLE_ID/ >> >> >> >> there should be mutlple zk nodes, one of which is state (or maybe fate?) >> >> >> >> zkCli> get /accumulo/INSTANCE_ID/tables/TABLE_ID/state >> >> >> >> – it should say DELETING. >> >> >> >> zkCli> set /accumulo/INSTANCE_ID/tables/TABLE_ID/state ONLINE >> >> >> >> This will force the table back online. After all tablets are accounted >> for in the monitor you should be able to re-issue the delete command via >> the accumulo shell. >> >> >> >> Ed Coleman >> >> >> >> *From:* Jeff Kubina [mailto:[email protected]] >> *Sent:* Wednesday, October 12, 2016 7:38 PM >> *To:* [email protected] >> *Subject:* Re: How can I remove or fix a hanging droptable in Accumulo? >> >> >> >> Lots of bulk ingest assignments. It's been hung for days and even >> returned after a reboot of the master and tservers. >> >> >> -- >> >> Jeff Kubina >> 410-988-4436 >> >> >> >> >> >> On Wed, Oct 12, 2016 at 6:28 PM, Michael Wall <[email protected]> wrote: >> >> Jeff, >> >> What is the master doing while the droptable is hung? >> >> Mike >> >> >> >> On Wed, Oct 12, 2016 at 6:14 PM, Jeff Kubina <[email protected]> >> wrote: >> >> How can I remove or fix a hanging droptable in Accumulo? >> >> >> >> >> > >
