FileSystem.rename will fail silently ------------------------------------ Key: HDFS-1309 URL: https://issues.apache.org/jira/browse/HDFS-1309 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux version 2.6.31-302-ec2 (bui...@yellow) (gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu7) ) #7-Ubuntu SMP Tue Oct 13 19:55:22 UTC 2009
Reporter: Kris Nuttycombe Priority: Minor Some filesystem operations (such as rename) will fail silently. In the attached example, a failure message will be written to the hadoop log, but it would be much better if the operation were to "fail fast" by throwing a checked exception and forcing the caller to handle the problem; failing to do so can easily lead to inadvertent data corruption. val coalesceBasePath = new Path(eventLog.basePath, coalesceTo) val backupBasePath = new Path(eventLog.basePath, relocateTo) eventLog.fs.mkdirs(backupBasePath) for (path <- coalesced; time <- HDFSEventLog.timePart(path, eventType)) { val backupPath = HDFSEventLog.path(backupBasePath, eventType, time) log.info("Relocating " + path + " to " + backupPath) eventLog.fs.rename(path, backupPath) } INF [20100715-16:11:20.727] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226067707 to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226067707 INF [20100715-16:11:20.752] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226077707 to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226077707 INF [20100715-16:11:20.754] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226457707 to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226457707 INF [20100715-16:11:20.757] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279229126727 to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279229126727 Complete. [knuttyco...@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/ Found 3 items drwxr-xr-x - knuttycombe supergroup 0 2010-07-15 14:54 /test-batchEventLog/coalesced drwxr-xr-x - knuttycombe supergroup 0 2010-07-15 14:35 /test-batchEventLog/metrics drwxr-xr-x - knuttycombe supergroup 0 2010-07-15 16:11 /test-batchEventLog/pre-coalesce [knuttyco...@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/metrics Found 4 items -rw-r--r-- 3 knuttycombe supergroup 2017122 2010-07-15 14:34 /test-batchEventLog/metrics/metrics_1279226067707 -rw-r--r-- 3 knuttycombe supergroup 4122951 2010-07-15 14:34 /test-batchEventLog/metrics/metrics_1279226077707 -rw-r--r-- 3 knuttycombe supergroup 5999912 2010-07-15 14:35 /test-batchEventLog/metrics/metrics_1279226457707 -rw-r--r-- 3 knuttycombe supergroup 8638301 2010-07-15 14:26 /test-batchEventLog/metrics/metrics_1279229126727 [knuttyco...@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/pre-coalesce [knuttyco...@floorshow reporting (reporting-coalesce)]$ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.