[ https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336386#comment-16336386 ]
Andras Czesznak commented on HDFS-12654: ---------------------------------------- Hello [~iwasakims] San, the issue occurred with the FluentD app and described here: https://github.com/fluent/fluent-plugin-webhdfs/issues/46 >From the HTTPFS's log: ``` 2017-10-03 16:20:59,204 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<username> (auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.FileNotFoundException: failed to append to non-existent file <path>/fluentd/process/<process>.log for client <IP address> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2930) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3227) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3191) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:614) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:126) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:416) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) ``` This is what it looks like in FluentD log ``` 2017-10-04 17:23:54 -0700 [warn]: failed to communicate hdfs cluster, path: <path>/fluentd/process/<process>.log 2017-10-04 17:23:54 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-04 17:24:24 -0700 error_class="WebHDFS::ServerError" error="Failed to connect to host <FQDN>:14000, Broken pipe - sendfile" plugin_id="object:3f8778538560" 2017-10-04 17:23:54 -0700 [warn]: suppressed same stacktrace The source code snippets: 1) HTTPFS: direct FS output stream creation /hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java ... 551 /** 552 * Append to an existing file (optional operation). 553 * <p/> 554 * IMPORTANT: The <code>Progressable</code> parameter is not used. 555 * 556 * @param f the existing file to be appended. 557 * @param bufferSize the size of the buffer to be used. 558 * @param progress for reporting progress if it is not null. 559 * 560 * @throws IOException 561 */ 562 @Override 563 public FSDataOutputStream append(Path f, int bufferSize, 564 Progressable progress) throws IOException { 565 Map<String, String> params = new HashMap<String, String>(); 566 params.put(OP_PARAM, Operation.APPEND.toString()); 567 return uploadData(Operation.APPEND.getMethod(), f, params, bufferSize, 568 HttpURLConnection.HTTP_OK); 569 } ... 2) WebHDFS: indirect FS output stream creation thru DFS ... /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java ... 54 import org.apache.hadoop.fs.FSDataOutputStream; ... 795 /** 796 * Handle create/append output streams 797 */ 798 class FsPathOutputStreamRunner extends AbstractFsPathRunner<FSDataOutputStream> { 799 private final int bufferSize; 800 801 FsPathOutputStreamRunner(Op op, Path fspath, int bufferSize, 802 Param<?,?>... parameters) { 803 super(op, fspath, parameters); 804 this.bufferSize = bufferSize; 805 } 806 807 @Override 808 FSDataOutputStream getResponse(final HttpURLConnection conn) 809 throws IOException { 810 return new FSDataOutputStream(new BufferedOutputStream( 811 conn.getOutputStream(), bufferSize), statistics) { 812 @Override 813 public void close() throws IOException { 814 try { 815 super.close(); 816 } finally { 817 try { 818 validateResponse(op, conn, true); 819 } finally { 820 conn.disconnect(); 821 } 822 } 823 } 824 }; 825 } 826 } ... > APPEND API call is different in HTTPFS and NameNode REST > -------------------------------------------------------- > > Key: HDFS-12654 > URL: https://issues.apache.org/jira/browse/HDFS-12654 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, httpfs, namenode > Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1 > Reporter: Andras Czesznak > Priority: Major > > The APPEND REST API call behaves differently in the NameNode REST and the > HTTPFS codes. The NameNode version creates the target file the new data being > appended to if it does not exist at the time of the call issued. The HTTPFS > version assumes the target file exists when APPEND is called and can append > only the new data but does not create the target file it doesn't exist. > The two implementations should be standardized, preferably the HTTPFS version > should be modified to execute an implicit CREATE if the target file does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org