[ 
https://issues.apache.org/jira/browse/HDFS-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952979#comment-17952979
 ] 

Carl Vuosalo commented on HDFS-17791:
-------------------------------------

Example of error message produced by the {{libhdfs}} client when trying to open 
a non-existent file. Part of the Java stack trace actually appears twice. A 
message like this appears in our logs for every "file not found" error.


{code:java}
File we will access: 
/store/mc/RunIIISummer24PrePremix/Neutrino_E-10_gun/PREMIX/Premixlib2024_140X_mcRun3_2024_realistic_v26-v1/2520137/a65fd95a-1302-4e95-98a0-e8d4ad856aab.root
hdfsOpenFile(/store/mc/RunIIISummer24PrePremix/Neutrino_E-10_gun/PREMIX/Premixlib2024_140X_mcRun3_2024_realistic_v26-v1/2520137/a65fd95a-1302-4e95-98a0-e8d4ad856aab.root):
 
FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
 error:
RemoteException: File does not exist: 
/store/mc/RunIIISummer24PrePremix/Neutrino_E-10_gun/PREMIX/Premixlib2024_140X_mcRun3_2024_realistic_v26-v1/2520137/a65fd95a-1302-4e95-98a0-e8d4ad856aab.root
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
  at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)
  at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
  at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770)
  at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:460)
  at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
java.io.FileNotFoundException: File does not exist: 
/store/mc/RunIIISummer24PrePremix/Neutrino_E-10_gun/PREMIX/Premixlib2024_140X_mcRun3_2024_realistic_v26-v1/2520137/a65fd95a-1302-4e95-98a0-e8d4ad856aab.root
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
  at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)
  at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
  at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770)
  at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:460)
  at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
  at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
  at 
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at 
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
  at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
  at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:902)
  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:889)
  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:878)
  at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1046)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:343)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:339)
  at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:356)
Caused by: 
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does 
not exist: 
/store/mc/RunIIISummer24PrePremix/Neutrino_E-10_gun/PREMIX/Premixlib2024_140X_mcRun3_2024_realistic_v26-v1/2520137/a65fd95a-1302-4e95-98a0-e8d4ad856aab.root
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)
  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
  at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)
  at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
  at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770)
  at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:460)
  at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)

  at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1567)
  at org.apache.hadoop.ipc.Client.call(Client.java:1513)
  at org.apache.hadoop.ipc.Client.call(Client.java:1410)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
  at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
  at com.sun.proxy.$Proxy12.getBlockLocations(Unknown Source)
  at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:334)
  at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433)
  at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
  at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
  at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
  at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
  at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:900)
  ... 7 more
{code}


> libhdfs client error message for "file not found" includes lengthy, unneeded 
> stack trace
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-17791
>                 URL: https://issues.apache.org/jira/browse/HDFS-17791
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: libhdfs
>    Affects Versions: 3.3.6
>            Reporter: Carl Vuosalo
>            Priority: Minor
>
> The {{libhdfs}} client outputs a full Java stack trace in its error message 
> for a "file not found" error when trying to open a non-existent file. Our 
> HDFS application processes numerous queries checking for the presence of 
> files, so "file not found" errors are frequent. Each such error generates a 
> very lengthy and useless Java stack trace. These traces have no diagnostic 
> value, since the cause of the error is that the file doesn't exist.
> A fix for this problem is proposed by pull request 
> [#7696|https://github.com/apache/hadoop/pull/7696],



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to