ZhangBiao created HDFS-12326:
--------------------------------
Summary: What is the correct way of retrying when failure occurs
during writing
Key: HDFS-12326
URL: https://issues.apache.org/jira/browse/HDFS-12326
Project: Hadoop HDFS
Issue Type: Test
Components: hdfs-client
Reporter: ZhangBiao
I'm using hdfs client for golang https://github.com/colinmarc/hdfs to write to
the hdfs.
When the number of files concurrently being opened is larger, for example 200.
I'll always get the 'broken pipe' error.
So I want to retry to continue writing. What is the correct way of retrying?
Because https://github.com/colinmarc/hdfs hasn't been able to recover the
stream status when an error occurs duing writing, so I have to reopen and get a
new stream. So I tried the following steps:
1 close the current stream
2 Append the file to get a new stream
But when I close the stream, I got the error "updateBlockForPipeline call
failed with ERROR_APPLICATION (java.io.IOException"
and it seems the namenode complains:
{code:java}
2017-08-20 03:22:55,598 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 9000, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.updateBlockForPipeline from
192.168.0.39:46827 Call#50183 Retry#-1
java.io.IOException:
BP-1152809458-192.168.0.39-1502261411064:blk_1073825071_111401 does not exist
or is not under Constructionblk_1073825071_111401{UCState=COMMITTED,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-d61914ba-df64-467b-bb75-272875e5e865:NORMAL:192.168.0.39:50010|RBW],
ReplicaUC[[DISK]DS-1314debe-ab08-4001-ab9a-8e234f28f87c:NORMAL:192.168.0.38:50010|RBW]]}
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6241)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6309)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:806)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:955)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2017-08-20 03:22:56,333 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
blk_1073825071_111401{UCState=COMMITTED, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-d61914ba-df64-467b-bb75-272875e5e865:NORMAL:192.168.0.39:50010|RBW],
ReplicaUC[[DISK]DS-1314debe-ab08-4001-ab9a-8e234f28f87c:NORMAL:192.168.0.38:50010|RBW]]}
is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 1) in file
/user/am/scan_task/2017-08-20/192.168.0.38_audience_f/user-bak010-20170820030804.log
{code}
when I Appended to get a new stream, I got the error 'append call failed with
ERROR_APPLICATION
(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException)', and the
corresponding error in namenode is:
{code:java}
2017-08-20 03:22:56,335 WARN org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.append: Failed to APPEND_FILE
/user/am/scan_task/2017-08-20/192.168.0.38_audience_f/user-bak010-20170820030804.log
for go-hdfs-OAfvZiSUM2Eu894p on 192.168.0.39 because go-hdfs-OAfvZiSUM2Eu894p
is already the current lease holder.
2017-08-20 03:22:56,335 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.append from
192.168.0.39:46827 Call#50186 Retry#-1:
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: Failed to
APPEND_FILE
/user/am/scan_task/2017-08-20/192.168.0.38_audience_f/user-bak010-20170820030804.log
for go-hdfs-OAfvZiSUM2Eu894p on 192.168.0.39 because go-hdfs-OAfvZiSUM2Eu894p
is already the current lease holder.
{code}
Could you please suggest the correct way of retrying of the client side when
write fails?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
