[ https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088906#comment-15088906 ]
Daisuke Kobayashi commented on HDFS-9455: ----------------------------------------- Thanks Yongjun for correcting me! Indeed, distcp runs within a single cluster per the description and I could reproduce the issue as follows. [~archanat], is this what you're also hitting, correct? {noformat} [daisuke@test1 jars]$ hadoop distcp webhdfs://test1:50470/user/daisuke/hosts webhdfs://test1:50470/user/daisuke/dir/ 16/01/08 17:40:22 WARN security.UserGroupInformation: PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS) cause:java.net.SocketException: Unexpected end of file from server 16/01/08 17:40:22 WARN security.UserGroupInformation: PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS) cause:java.net.SocketException: Unexpected end of file from server 16/01/08 17:40:22 ERROR tools.DistCp: Invalid arguments: java.net.SocketException: Unexpected end of file from server at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:336) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:614) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1307) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:239) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:429) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:450) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:697) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:609) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:844) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:859) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1409) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:200) at org.apache.hadoop.tools.DistCp.run(DistCp.java:112) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:429) Invalid arguments: Unexpected end of file from server usage: distcp OPTIONS [source_path...] <target_path> OPTIONS -append Reuse existing data in target files and append new data to them if possible <...snip...> {noformat} So my proposal is catching SocketException and print more meaningful message like {{An error occurs while getting target path}} If this is agreeable, do you allow me to create a patch to fix this, [~archanat]? > In distcp, Invalid Argument Error thrown in case of filesystem operation > failure > -------------------------------------------------------------------------------- > > Key: HDFS-9455 > URL: https://issues.apache.org/jira/browse/HDFS-9455 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp, security > Reporter: Archana T > Assignee: Archana T > Priority: Minor > > When Filesystem Operation failure happens during discp, > Wrong exception : Invalid Argument thrown along with distcp command Usage. > {color:red} > hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp > Invalid arguments: Unexpected end of file from server > usage: distcp OPTIONS [source_path...] <target_path> > OPTIONS > -append Reuse existing data in target files and > append new data to them if possible > -async Should distcp execution be blocking > -atomic Commit all changes or none > -bandwidth <arg> Specify bandwidth per map in MB > -delete Delete from target, files missing in source > -diff <arg> Use snapshot diff report to identify the > difference between source and target > -f <arg> List of files that need to be copied > -filelimit <arg> (Deprecated!) Limit number of files copied > to <= n > -i Ignore failures during copy > ..... > {color} > Instead Proper Exception has to be thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)