[jira] [Commented] (HDFS-10400) hdfs dfs -put exits with zero on error
[ https://issues.apache.org/jira/browse/HDFS-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290165#comment-15290165 ] Jo Desmet commented on HDFS-10400: -- {quote} Jo Desmet, how did you verify that dfs -put failed? {quote} Unix, simply using {{$?}}. I did confirm that I was able to detect other error conditions, like source file non-existing. Additionally when the untrapped exception occurred, I also verified that the file was either missing or incomplete at the target location. [~linyiqun], checking checksum seems to be extreme if an exception is somehow already generated? > hdfs dfs -put exits with zero on error > -- > > Key: HDFS-10400 > URL: https://issues.apache.org/jira/browse/HDFS-10400 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jo Desmet >Assignee: Yiqun Lin > Attachments: HDFS-10400.001.patch, HDFS-10400.002.patch > > > On a filesystem that is about to fill up, execute "hdfs dfs -put" for a file > that is big enough to go over the limit. As a result, the command fails with > an exception, however the command terminates normally (exit code 0). > Expectation is that any detectable failure generates an exit code different > than zero. > Documentation on > https://hadoop.apache.org/docs/r1.2.1/file_system_shell.html#put states: > Exit Code: > Returns 0 on success and -1 on error. > following is the exception generated: > 16/05/11 13:37:07 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.EOFException: Premature EOF: no length prefix available > at > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1352) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464) > 16/05/11 13:37:07 INFO hdfs.DFSClient: Abandoning > BP-1964113808-130.8.138.99-1446787670498:blk_1073835906_95114 > 16/05/11 13:37:08 INFO hdfs.DFSClient: Excluding datanode > DatanodeInfoWithStorage[130.8.138.99:50010,DS-eed7039a-8031-499e-85a5-7216b9d766a8,DISK] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10400) hdfs dfs -put exits with zero on error
[ https://issues.apache.org/jira/browse/HDFS-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283309#comment-15283309 ] Jo Desmet commented on HDFS-10400: -- Hadoop version is 2.7.1.2.3.2.0-2950 Apologies, didn't notice that I used the hadoop 1 link. I doubt that a fix will cause issues, as all other exceptions so far have been returning a valid exit code (non-zero). Also filling up a filesystem will hopefully be a rare event. In my case the event went undetected, and a lot of wasteful processing continued. It does look however that other exceptions are handled - things like a {{put}} of a non existing file - as they do not generate a Java Exception but a clean error report. I think what needs to happen is put a generic handler in place for un-trapped exceptions, still do the exception dump, but then exit with proper exit code. There is precedent for this issue, please see - [HADOOP-4340|https://issues.apache.org/jira/browse/hadoop-4340] - [MAPREDUCE-3179|https://issues.apache.org/jira/browse/MAPREDUCE-3179] > hdfs dfs -put exits with zero on error > -- > > Key: HDFS-10400 > URL: https://issues.apache.org/jira/browse/HDFS-10400 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jo Desmet > > On a filesystem that is about to fill up, execute "hdfs dfs -put" for a file > that is big enough to go over the limit. As a result, the command fails with > an exception, however the command terminates normally (exit code 0). > Expectation is that any detectable failure generates an exit code different > than zero. > Documentation on > https://hadoop.apache.org/docs/r1.2.1/file_system_shell.html#put states: > Exit Code: > Returns 0 on success and -1 on error. > following is the exception generated: > 16/05/11 13:37:07 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.EOFException: Premature EOF: no length prefix available > at > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1352) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464) > 16/05/11 13:37:07 INFO hdfs.DFSClient: Abandoning > BP-1964113808-130.8.138.99-1446787670498:blk_1073835906_95114 > 16/05/11 13:37:08 INFO hdfs.DFSClient: Excluding datanode > DatanodeInfoWithStorage[130.8.138.99:50010,DS-eed7039a-8031-499e-85a5-7216b9d766a8,DISK] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10400) hdfs dfs -put exits with zero on error
Jo Desmet created HDFS-10400: Summary: hdfs dfs -put exits with zero on error Key: HDFS-10400 URL: https://issues.apache.org/jira/browse/HDFS-10400 Project: Hadoop HDFS Issue Type: Bug Reporter: Jo Desmet On a filesystem that is about to fill up, execute "hdfs dfs -put" for a file that is big enough to go over the limit. As a result, the command fails with an exception, however the command terminates normally (exit code 0). Expectation is that any detectable failure generates an exit code different than zero. Documentation on https://hadoop.apache.org/docs/r1.2.1/file_system_shell.html#put states: Exit Code: Returns 0 on success and -1 on error. following is the exception generated: 16/05/11 13:37:07 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1352) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464) 16/05/11 13:37:07 INFO hdfs.DFSClient: Abandoning BP-1964113808-130.8.138.99-1446787670498:blk_1073835906_95114 16/05/11 13:37:08 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[130.8.138.99:50010,DS-eed7039a-8031-499e-85a5-7216b9d766a8,DISK] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org