[ https://issues.apache.org/jira/browse/HDFS-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637833#comment-13637833 ]
Yu Li commented on HDFS-4720: ----------------------------- Hello [~jerryhe], The message and stack I observed(as shown below) is quite similar but not exactly the same with yours. I'm not sure whether we're trying the same version of hadoop, the one I used is hadoop-1.1.1 {panel} hadoop distcp /tmp/jruby-complete-1.6.5.1.jar webhdfs://9.125.91.42:14000/tmp/test/ 13/04/22 04:11:24 INFO tools.DistCp: srcPaths=[/tmp/jruby-complete-1.6.5.1.jar] 13/04/22 04:11:24 INFO tools.DistCp: destPath=webhdfs://9.125.91.42:14000/tmp/test 13/04/22 04:11:25 INFO tools.DistCp: sourcePathsCount=1 13/04/22 04:11:25 INFO tools.DistCp: filesToCopyCount=1 13/04/22 04:11:25 INFO tools.DistCp: bytesToCopyCount=12.7m 13/04/22 04:11:25 WARN web.WebHdfsFileSystem: Original exception is {color: red}org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser = null, path = /tmp/test/_distcp_logs_e0nhl6{color} at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:549) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:570) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:581) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768) at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:120) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912) at java.security.AccessController.doPrivileged(AccessController.java:310) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323) at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667) at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) 13/04/22 04:11:25 INFO mapred.JobClient: Running job: job_201304212037_0005 13/04/22 04:11:26 INFO mapred.JobClient: map 0% reduce 0% 13/04/22 04:11:36 INFO mapred.JobClient: map 100% reduce 0% 13/04/22 04:11:36 INFO mapred.JobClient: Job complete: job_201304212037_0005 13/04/22 04:11:36 INFO mapred.JobClient: Counters: 21 13/04/22 04:11:36 INFO mapred.JobClient: Job Counters 13/04/22 04:11:36 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=8876 13/04/22 04:11:36 INFO mapred.JobClient: Launched map tasks=1 13/04/22 04:11:36 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/04/22 04:11:36 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/04/22 04:11:36 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 13/04/22 04:11:36 INFO mapred.JobClient: distcp 13/04/22 04:11:36 INFO mapred.JobClient: Bytes expected=13327243 13/04/22 04:11:36 INFO mapred.JobClient: Files copied=1 13/04/22 04:11:36 INFO mapred.JobClient: Bytes copied=13327243 13/04/22 04:11:36 INFO mapred.JobClient: FileSystemCounters 13/04/22 04:11:36 INFO mapred.JobClient: FILE_BYTES_WRITTEN=21271 13/04/22 04:11:36 INFO mapred.JobClient: WEBHDFS_BYTES_WRITTEN=13327243 13/04/22 04:11:36 INFO mapred.JobClient: File Output Format Counters 13/04/22 04:11:36 INFO mapred.JobClient: Bytes Written=0 13/04/22 04:11:36 INFO mapred.JobClient: Map-Reduce Framework 13/04/22 04:11:36 INFO mapred.JobClient: Virtual memory (bytes) snapshot=895299584 13/04/22 04:11:36 INFO mapred.JobClient: Map input bytes=128 13/04/22 04:11:36 INFO mapred.JobClient: Physical memory (bytes) snapshot=69713920 13/04/22 04:11:36 INFO mapred.JobClient: Map output records=0 13/04/22 04:11:36 INFO mapred.JobClient: CPU time spent (ms)=530 13/04/22 04:11:36 INFO mapred.JobClient: Map input records=1 13/04/22 04:11:36 INFO mapred.JobClient: Total committed heap usage (bytes)=8459264 13/04/22 04:11:36 INFO mapred.JobClient: Spilled Records=0 13/04/22 04:11:36 INFO mapred.JobClient: SPLIT_RAW_BYTES=111 13/04/22 04:11:36 INFO mapred.JobClient: File Input Format Counters 13/04/22 04:11:36 INFO mapred.JobClient: Bytes Read=0 {panel} > Misleading warning message in WebhdfsFileSystem when trying to check whether > path exist using webhdfs url > --------------------------------------------------------------------------------------------------------- > > Key: HDFS-4720 > URL: https://issues.apache.org/jira/browse/HDFS-4720 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs > Affects Versions: 1.1.1, 1.1.2 > Reporter: Yu Li > Assignee: Yu Li > Priority: Minor > Attachments: HDFS-4720-trunk.patch > > > When we trying to check whether the target path exists in HDFS through > webhdfs, if the given path to check doesn't exist, we will always observe > warning message like: > =========================================================== > 13/04/21 04:38:01 WARN web.WebHdfsFileSystem: Original exception is > org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser = null, path > = /testWebhdfs > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:552) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:573) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:584) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768) > =========================================================== > while actually FileNotFoundException should be expected when the operation is > GETFILESTATUS and target path doesn't exist. The fact that RemoteException > didn't include the real exception class(FileNotFoundException) in its > toString method even make the message more misleading, since from the message > user won't know what the warning is about -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira