The is a utility - SetGoalState that can be run from the command line accumulo SetGoalState NORMAL
(or SAFE_MODE, CLEAN_STOP) It sets a value in ZK at /accumulo/instance-id/managers/goal_state Ed Coleman ________________________________ From: Ligade, Shailesh [USA] <ligade_shail...@bah.com> Sent: Wednesday, February 9, 2022 1:54 PM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: Re: accumulo 1.10.0 masters won't start Well, i just went ahead and deleted fate in zookeeper and restarted the master..it was doing better but then i am getting different error ERROR: Problem getting real goal state from zookeeper: java.lang.IllegalArgumentException: No enum constant org.apache.accumulo.core.master.thrift.MasterGoalState I hope i didn't delete goal_state accidently ...;-( currently ls on goal_state is [], is there a way to add some value there? -S ________________________________ From: dev1 <d...@etcoleman.com> Sent: Wednesday, February 9, 2022 1:32 PM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: [External] Re: accumulo 1.10.0 masters won't start Did you try setting the increased size in the zkCli.sh command (or wherever it gets it environment from?) The ZK docs indicate that it needs to be set to the same size on all servers and clients. You should be able to use zkCli.sh to at least see what's going on - if that does not work, then it seems unlikely that the master would either. Can you: * list the nodes under /accumulo/[instance id]/fate? * use the stat command on each of the nodes - the size is one of the fields. * list nodes under any of the /accumulo/[instance_id/fate/tx-##### * there should be a node named debug - doing a get on that should show the op name. Ed Coleman ________________________________ From: Ligade, Shailesh [USA] <ligade_shail...@bah.com> Sent: Wednesday, February 9, 2022 12:54 PM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: Re: accumulo 1.10.0 masters won't start Thanks I added -Djute.maxbuffer=30000000 In conf/java.env and restart all zookeepers but still getting the same error.. documentation is kind of fuzzy on setting this property as it states in hex (default 0xffff) so not 100% sure if 30000000 is ok, but atleast I could see zookeeper was up -S ________________________________ From: dev1 <d...@etcoleman.com> Sent: Wednesday, February 9, 2022 12:26 PM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: [External] Re: accumulo 1.10.0 masters won't start Does the monitor or any of the logs show errors that relate to exceeding the ZooKeeper jute buffer size? Is so, have you tried increasing the ZooKeeper jute.maxbuffer limit(https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#Unsafe+Options<https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html*Unsafe*Options__;Iys!!May37g!dTGCMHPLPDBXwSqtLa5cIPHiTIQF7IjLCVyvGxfi1sgPbrsOI8RCEsuZ9u-jJtayEg$>)? Ed Coleman ________________________________ From: Ligade, Shailesh [USA] <ligade_shail...@bah.com> Sent: Wednesday, February 9, 2022 11:49 AM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: accumulo 1.10.0 masters won't start Hello, My both masters are stuck error on zookeeper: IOException: Packet len 2791093 is out of range! KeeperErrorCode = ConnectionLoss for /accumulo/<instance_id>/fate if use zkCli to see what is under fate, i get IOException Packet len 2791161 is out of range Unable to read additional data from server sessionid xxxx, likely server has closed socket hdfs fsck is all good How can I clear this fate? master process is up and I can get into accumulo shell, but there are no fate (fate print returns empty) Any idea how to bring the master up? Thanks S
