[
https://issues.apache.org/jira/browse/HBASE-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437542#comment-16437542
]
Sean Busbey commented on HBASE-20404:
-------------------------------------
{quote}Patch looks good. Do we have to print the exception at all, even at
DEBUG? Its a big dirty one. The prefatory text helps but It shows up often
enough. Thanks.
{quote}
We end up in that block if we hit literally any kind of IOException. I think we
need to print it, because the problem could be e.g. HDFS in safe mode or some
other thing that an operator could do something about.
Are we seeing the specific IOException of "you're trying to delete a directory
before the contents are deleted" a bunch of times because of a bad sort order?
i.e. does a directory sort ahead of the stuff in the directory because of the
"prioritize by size" check?
> Ugly cleanerchore complaint that dir is not empty
> -------------------------------------------------
>
> Key: HBASE-20404
> URL: https://issues.apache.org/jira/browse/HBASE-20404
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.0.0
> Reporter: stack
> Assignee: Sean Busbey
> Priority: Major
> Attachments: HBASE-20404.0.patch, HBASE-20404.1.patch
>
>
> I see these big dirty exceptions in my master log during a long-run.... Lets
> clean them up (Are they exceptions I as an operator can actually do something
> about? Are they 'problems'? Should they be LOG.warn?)
> {code}
> 2018-04-12 16:02:09,911 WARN [ForkJoinPool-1-worker-15]
> cleaner.CleanerChore: Could not delete dir under
> hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta;
> {}
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
>
> `/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta
> is non empty': Directory is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:115)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:2848)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:1048)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy26.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:568)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> at com.sun.proxy.$Proxy27.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> ...
> {code}
> Looks like log format is off too...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
