Hi Martin, What you are seeing is expected behavior in earlier versions of Hadoop. hadoop-0.18 has fix for this. What happens here is that, when a file is opened, but not yet closed; the namenode does not yet know the locations/block size from all the datanodes. When a block is successfully written by datanode, it reports this to the namenode. If it at this point, that namenode has knowledge of this block, its location and its size. For open files, this is still being updated.
hadoop 0.18 filters out open files and provides and option (-openforwrite) to select such files. Thanks, Lohit ----- Original Message ---- From: Martin Schaaf <[EMAIL PROTECTED]> To: [email protected] Sent: Thursday, May 22, 2008 4:14:20 PM Subject: Hadoop fsck displays open files as corrupt. Hi, we wrote a program that uses a Writer to append keys and values to a file. If we do an fsck during these writing the opened files are reported as corrupt and the file size is zero until they are closed. On the other side if we copy a file from local fs to the hadoop fs the size constantly increases and the files aren't displayed as corrupt. So my question is this the expected behaviour? What is the difference between this two operation. Thanks in advance for your help martin
