Hello, Josh
I heard my project menber that did a `hdfs dfs -chmod -R 644 /accumulo`, but no one did it... Thank you for reply, Takashi 2016-12-06 5:50 GMT+09:00 Josh Elser <[email protected]>: > 2016-12-02 19:30:16,170 [tserver.TabletServer] WARN : exception trying to > assign tablet +r<< hdfs://10.24.83.112:8020/accumulo/tables/+r/root_tablet > java.lang.RuntimeException: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Permission denied: > user=accumulo, access=EXECUTE, > inode="/accumulo/recovery/603194f3-dd41-44ed-8ad6-90d408149952/failed/data":accumulo:accumulo:-rw-r--r-- > > at org.apache.accumulo.tserver.log.MultiReader.<init>(MultiReader.java:113) > at > org.apache.accumulo.tserver.log.SortedLogRecovery.recover(SortedLogRecovery.java:105) > at > org.apache.accumulo.tserver.log.TabletServerLogger.recover(TabletServerLogger.java:483) > > > It looks like Accumulo doesn't have permission to list the contents of this > directory as the directory lacks the execute bit. Did someone do a `hdfs dfs > -chmod -R 644 /accumulo`? I don't know how Accumulo would have a directory > which is chmod 644 instead of 755... > > > Takashi Sasaki wrote: >> >> Oops, I didn't know stripping attachments. >> >> I'm hosting the file on my google drive. >> >> tserver.log >> https://drive.google.com/open?id=0B0ffj_ngVZxuaHJQYUtDY3doYm8 >> >> master.log >> https://drive.google.com/open?id=0B0ffj_ngVZxuMEk4MHJVQzVWZXc >> >> >> Thank you for advice, >> >> Takashi >> >> 2016-12-05 22:00 GMT+09:00 Josh Elser<[email protected]>: >>> >>> Apache mailing lists strip attachments. Please host the files somewhere >>> and >>> provide a link to them. >>> >>> On Dec 4, 2016 20:54, "Takashi Sasaki"<[email protected]> wrote: >>>> >>>> Hello, >>>> >>>> I'm sorry to take a few wrong infomation at first post. >>>> >>>> I asked the project members again about the problem. >>>> Master server did not throw AccessControlException. >>>> >>>> Actually, TabletServer threw AccessControlException. >>>> And, the stack trace was missing words, wrong path. >>>> >>>> Correct full stack trace was line 52 in attached file "tserver.log". >>>> And I attache "master.log" for your reference. >>>> >>>> Unfortunately, I could not still get debug log. >>>> >>>> Thank you for your support, >>>> Takashi >>>> >>>> >>>> 2016-12-04 18:33 GMT+09:00 Takashi Sasaki<[email protected]>: >>>>> >>>>> Hello, Christopher >>>>> >>>>>> The stack trace doesn't include anything from Accumulo, so it's not >>>>>> clear where in the Accumulo code this occurred. Do you have the full >>>>>> stack >>>>>> trace? >>>>> >>>>> Yes, I understand the stack trace isn't including from Accumulo. >>>>> I don't have full stack trace now, but I will try to find it. >>>>> >>>>> In additon, I use Accumulo on AWS EMR cluster for Enterprise >>>>> Production System, so log level isn't debug, becase of disk capacity >>>>> problem. >>>>> I will try to reproduce with debug log level. >>>>> >>>>> Thank you for your reply, >>>>> Takashi >>>>> >>>>> 2016-12-04 18:00 GMT+09:00 Christopher<[email protected]>: >>>>>> >>>>>> The stack trace doesn't include anything from Accumulo, so it's not >>>>>> clear >>>>>> where in the Accumulo code this occurred. Do you have the full stack >>>>>> trace? >>>>>> >>>>>> In particular, it's not clear to me that there should be a directory >>>>>> called >>>>>> failed/da at that location, nor is it clear why Accumulo would be >>>>>> trying to >>>>>> check for the execute permission on it, unless it's trying to recurse >>>>>> into a >>>>>> directory. There is one part of the code where, if the directory >>>>>> exists >>>>>> when >>>>>> log recovery begins, it may try to do a recursive delete, but I can't >>>>>> see >>>>>> how this location would have been created by Accumulo. If that is the >>>>>> case, >>>>>> then it should be safe to manually delete this directory and its >>>>>> contents. >>>>>> The failed marker should be a regular file, though, and should not be >>>>>> a >>>>>> directory with another directory called "da" in it. So, I can't see >>>>>> how >>>>>> this >>>>>> was even created, unless by an older version or another program. >>>>>> >>>>>> The only way I can see this occurring is if you recently did an >>>>>> upgrade, >>>>>> while Accumulo had not yet finished outstanding log recoveries from a >>>>>> previous shutdown, AND the previous version did something different >>>>>> than >>>>>> 1.7.2. If that was the case, then perhaps the older version could have >>>>>> created this problematic directory. It seems unlikely, though... >>>>>> because >>>>>> directories are usually not created without the execute bit... and the >>>>>> error >>>>>> message looks like a directory missing that bit. >>>>>> >>>>>> It's hard to know more without seeing the full stack trace with the >>>>>> relevant >>>>>> accumulo methods included. It might also help to see the master debug >>>>>> logs >>>>>> leading up to the error. >>>>>> >>>>>> On Sun, Dec 4, 2016 at 2:35 AM Takashi Sasaki<[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> I use Accumulo-1.7.2 with Haddop2.7.2 and ZooKeeper 3.4.8 >>>>>>> >>>>>>> Master server suddenly throw AccessControlException. >>>>>>> >>>>>>> java.io.IOException: >>>>>>> org.apache.hadoop.security.AccessControlException: Permission denied: >>>>>>> user=accumulo, access=EXECUTE, >>>>>>> >>>>>>> >>>>>>> >>>>>>> inode="/accumulo/recovery/603194f3-dd41-44ed-8ad6-90d408149952/failed/da":accumulo:accumulo:-rw-r--r-- >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720) >>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSDirSt >>>>>>> at AndListingOp.getFileInfo(FSDirSt >>>>>>> at AndListingOp.java:108) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3855) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTransl >>>>>>> at orPB.getFileInfo(ClientNamenodeProtocolServerSideTransl >>>>>>> at orPB.java:843) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) >>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >>>>>>> at java.security.AccessController.doPrivileged(N >>>>>>> at ive Method) >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>> at org.apache.hadoop.security.UserGroupInform >>>>>>> at ion.doAs(UserGroupInform >>>>>>> at ion.java:1657) >>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >>>>>>> >>>>>>> >>>>>>> How can I solve this Exception? >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Takashi.
