[
https://issues.apache.org/jira/browse/HDFS-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637833#comment-13637833
]
Yu Li commented on HDFS-4720:
-----------------------------
Hello [~jerryhe],
The message and stack I observed(as shown below) is quite similar but not
exactly the same with yours. I'm not sure whether we're trying the same version
of hadoop, the one I used is hadoop-1.1.1
{panel}
hadoop distcp /tmp/jruby-complete-1.6.5.1.jar
webhdfs://9.125.91.42:14000/tmp/test/
13/04/22 04:11:24 INFO tools.DistCp: srcPaths=[/tmp/jruby-complete-1.6.5.1.jar]
13/04/22 04:11:24 INFO tools.DistCp:
destPath=webhdfs://9.125.91.42:14000/tmp/test
13/04/22 04:11:25 INFO tools.DistCp: sourcePathsCount=1
13/04/22 04:11:25 INFO tools.DistCp: filesToCopyCount=1
13/04/22 04:11:25 INFO tools.DistCp: bytesToCopyCount=12.7m
13/04/22 04:11:25 WARN web.WebHdfsFileSystem: Original exception is
{color: red}org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser =
null, path = /tmp/test/_distcp_logs_e0nhl6{color}
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:549)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:570)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:581)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
at
org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:120)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at
java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
13/04/22 04:11:25 INFO mapred.JobClient: Running job: job_201304212037_0005
13/04/22 04:11:26 INFO mapred.JobClient: map 0% reduce 0%
13/04/22 04:11:36 INFO mapred.JobClient: map 100% reduce 0%
13/04/22 04:11:36 INFO mapred.JobClient: Job complete: job_201304212037_0005
13/04/22 04:11:36 INFO mapred.JobClient: Counters: 21
13/04/22 04:11:36 INFO mapred.JobClient: Job Counters
13/04/22 04:11:36 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=8876
13/04/22 04:11:36 INFO mapred.JobClient: Launched map tasks=1
13/04/22 04:11:36 INFO mapred.JobClient: Total time spent by all reduces
waiting after reserving slots (ms)=0
13/04/22 04:11:36 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
13/04/22 04:11:36 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/04/22 04:11:36 INFO mapred.JobClient: distcp
13/04/22 04:11:36 INFO mapred.JobClient: Bytes expected=13327243
13/04/22 04:11:36 INFO mapred.JobClient: Files copied=1
13/04/22 04:11:36 INFO mapred.JobClient: Bytes copied=13327243
13/04/22 04:11:36 INFO mapred.JobClient: FileSystemCounters
13/04/22 04:11:36 INFO mapred.JobClient: FILE_BYTES_WRITTEN=21271
13/04/22 04:11:36 INFO mapred.JobClient: WEBHDFS_BYTES_WRITTEN=13327243
13/04/22 04:11:36 INFO mapred.JobClient: File Output Format Counters
13/04/22 04:11:36 INFO mapred.JobClient: Bytes Written=0
13/04/22 04:11:36 INFO mapred.JobClient: Map-Reduce Framework
13/04/22 04:11:36 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=895299584
13/04/22 04:11:36 INFO mapred.JobClient: Map input bytes=128
13/04/22 04:11:36 INFO mapred.JobClient: Physical memory (bytes)
snapshot=69713920
13/04/22 04:11:36 INFO mapred.JobClient: Map output records=0
13/04/22 04:11:36 INFO mapred.JobClient: CPU time spent (ms)=530
13/04/22 04:11:36 INFO mapred.JobClient: Map input records=1
13/04/22 04:11:36 INFO mapred.JobClient: Total committed heap usage
(bytes)=8459264
13/04/22 04:11:36 INFO mapred.JobClient: Spilled Records=0
13/04/22 04:11:36 INFO mapred.JobClient: SPLIT_RAW_BYTES=111
13/04/22 04:11:36 INFO mapred.JobClient: File Input Format Counters
13/04/22 04:11:36 INFO mapred.JobClient: Bytes Read=0
{panel}
> Misleading warning message in WebhdfsFileSystem when trying to check whether
> path exist using webhdfs url
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-4720
> URL: https://issues.apache.org/jira/browse/HDFS-4720
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 1.1.1, 1.1.2
> Reporter: Yu Li
> Assignee: Yu Li
> Priority: Minor
> Attachments: HDFS-4720-trunk.patch
>
>
> When we trying to check whether the target path exists in HDFS through
> webhdfs, if the given path to check doesn't exist, we will always observe
> warning message like:
> ===========================================================
> 13/04/21 04:38:01 WARN web.WebHdfsFileSystem: Original exception is
> org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser = null, path
> = /testWebhdfs
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:552)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:573)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:584)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> ===========================================================
> while actually FileNotFoundException should be expected when the operation is
> GETFILESTATUS and target path doesn't exist. The fact that RemoteException
> didn't include the real exception class(FileNotFoundException) in its
> toString method even make the message more misleading, since from the message
> user won't know what the warning is about
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
