Do not leave that configuration in after your tests are done. It would be very harmful to allow such tiny block sizes from clients, enabling them to flood your NameNode's metadata with a lot of blocks for a small file.
If its instead possible, tune NNBench's block size to be larger perhaps. On Sat, Mar 22, 2014 at 2:26 PM, Lixiang Ao <aolixi...@gmail.com> wrote: > Checked the logs, and turned out to be configuration problem. Just set > dfs.namenode.fs-limits.min-block-size to 1 and it's fixed > > Thanks. > > > On Wed, Mar 19, 2014 at 2:51 PM, Brahma Reddy Battula > <brahmareddy.batt...@huawei.com> wrote: >> >> Seems to be this is issue, which is logged..Please check following jira >> for same....Hope you also facing same issue... >> >> >> >> https://issues.apache.org/jira/browse/HDFS-4929 >> >> >> >> >> >> >> >> Thanks & Regards >> >> >> >> Brahma Reddy Battula >> >> >> >> ________________________________ >> From: Lixiang Ao [aolixi...@gmail.com] >> Sent: Tuesday, March 18, 2014 10:34 AM >> To: user@hadoop.apache.org >> Subject: Re: Benchmark Failure >> >> the version is release 2.2.0 >> >> 2014年3月18日 上午12:26于 "Lixiang Ao" <aolixi...@gmail.com>写道： >>> >>> Hi all, >>> >>> I'm running jobclient tests(on single node), other tests like TestDFSIO, >>> mrbench succeed except nnbench. >>> >>> I got a lot of Exceptions but without any explanation(see below). >>> >>> Could anyone tell me what might went wrong? >>> >>> Thanks! >>> >>> >>> 14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms >>> 14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001 >>> running in uber mode : false >>> 14/03/17 23:54:23 INFO mapreduce.Job: map 0% reduce 0% >>> 14/03/17 23:54:28 INFO mapred.LocalJobRunner: >>> hdfs://0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125 >>> > map >>> 14/03/17 23:54:29 INFO mapreduce.Job: map 6% reduce 0% >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: >>> Create/Write/Close >>> (1000 Exceptions) >>> . >>> . >>> . >>> results: >>> >>> File System Counters >>> FILE: Number of bytes read=18769411 >>> FILE: Number of bytes written=21398315 >>> FILE: Number of read operations=0 >>> FILE: Number of large read operations=0 >>> FILE: Number of write operations=0 >>> HDFS: Number of bytes read=11185 >>> HDFS: Number of bytes written=19540 >>> HDFS: Number of read operations=325 >>> HDFS: Number of large read operations=0 >>> HDFS: Number of write operations=13210 >>> Map-Reduce Framework >>> Map input records=12 >>> Map output records=95 >>> Map output bytes=1829 >>> Map output materialized bytes=2091 >>> Input split bytes=1538 >>> Combine input records=0 >>> Combine output records=0 >>> Reduce input groups=8 >>> Reduce shuffle bytes=0 >>> Reduce input records=95 >>> Reduce output records=8 >>> Spilled Records=214 >>> Shuffled Maps =0 >>> Failed Shuffles=0 >>> Merged Map outputs=0 >>> GC time elapsed (ms)=211 >>> CPU time spent (ms)=0 >>> Physical memory (bytes) snapshot=0 >>> Virtual memory (bytes) snapshot=0 >>> Total committed heap usage (bytes)=4401004544 >>> File Input Format Counters >>> Bytes Read=1490 >>> File Output Format Counters >>> Bytes Written=170 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: -------------- NNBench >>> -------------- : >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> Version: NameNode Benchmark 0.4 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Date & >>> time: 2014-03-17 23:56:18,619 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Test >>> Operation: create_write >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Start >>> time: 2014-03-17 23:56:15,521 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Maps to >>> run: 12 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Reduces to >>> run: 6 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Block Size >>> (bytes): 1 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes to >>> write: 0 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes per >>> checksum: 1 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Number of >>> files: 1000 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Replication >>> factor: 3 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Successful file >>> operations: 0 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> 14/03/17 23:56:18 INFO hdfs.NNBench: # maps that missed the >>> barrier: 11 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: # >>> exceptions: 1000 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> 14/03/17 23:56:18 INFO hdfs.NNBench: TPS: >>> Create/Write/Close: 0 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Avg exec time (ms): >>> Create/Write/Close: Infinity >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms): >>> Create/Write: NaN >>> 14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms): >>> Close: NaN >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: AL Total >>> #1: 0 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: AL Total >>> #2: 0 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: TPS Total >>> (ms): 1131 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: Longest Map Time >>> (ms): 1.395071776653E12 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: Late >>> maps: 11 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: # of >>> exceptions: 1000 >>> 14/03/17 23:56:18 INFO hdfs.NNBench: >>> > -- Harsh J
