Ting Dai created YARN-7179: ------------------------------ Summary: The log data is corrupted Key: YARN-7179 URL: https://issues.apache.org/jira/browse/YARN-7179 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Reporter: Ting Dai
In hadoop-0.23.0, inside readAcontainerLogs function, when valueStream is corrupted, then writer will be corrupted. while (true) { try { fileType = valueStream.readUTF(); } catch (EOFException e) { return; } fileLengthStr = valueStream.readUTF(); {color:#d04437}//corrupted{color} fileLength = Long.parseLong(fileLengthStr); {color:#d04437}//0{color} writer.write("\n\nLogType:"); writer.write(fileType); writer.write("\nLogLength:"); writer.write(fileLengthStr); writer.write("\nLog Contents:\n"); BoundedInputStream bis = new BoundedInputStream(valueStream, fileLength); {color:#d04437}//empty stream{color} InputStreamReader reader = new InputStreamReader(bis); int currentRead = 0; int totalRead = 0; while ((currentRead = reader.read(cbuf, 0, bufferSize)) != -1) { {color:#d04437}//always return -1{color} writer.write(cbuf); totalRead += currentRead; } } When the fileLengthStr is corrupted, especially when it is "0", but the valueStream is actually not empty. This will cause bis, reader will be empty due to fileLength. The empty reader causes currentRead return -1 immediately, making writer never write. During next iteration, the fileType, fileLength and bis, reader will be corrupted. For example, if I have a DataInputStream like: "text", "0", "This is the content", "16", "Another content". But the writer will write the following log data: "LogType:text LogLength:0 Log Contents: LogType:This is the content LogLength:2 Log Contents:" -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org