Re: Benchmark Failure

[Lixiang Ao]

Checked the logs, and turned out to be configuration problem. Just
set dfs.namenode.fs-limits.min-block-size to 1 and it's fixed

Thanks.


On Wed, Mar 19, 2014 at 2:51 PM, Brahma Reddy Battula 
brahmareddy.batt...@huawei.com wrote:

  Seems to be this is issue, which is logged..Please check following jirafor 
 sameHope you also facing same issue...



 https://issues.apache.org/jira/browse/HDFS-4929







 Thanks  Regards



 Brahma Reddy Battula


   --
 *From:* Lixiang Ao [aolixi...@gmail.com]
 *Sent:* Tuesday, March 18, 2014 10:34 AM
 *To:* user@hadoop.apache.org
 *Subject:* Re: Benchmark Failure

   the version is release 2.2.0
 2014年3月18日 上午12:26于 Lixiang Ao aolixi...@gmail.com写道：

  Hi all,

  I'm running jobclient tests(on single node), other tests like
 TestDFSIO, mrbench succeed except nnbench.

  I got a lot of Exceptions but without any explanation(see below).

  Could anyone tell me what might went wrong?

  Thanks!


  14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms
 14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001
 running in uber mode : false
 14/03/17 23:54:23 INFO mapreduce.Job:  map 0% reduce 0%
 14/03/17 23:54:28 INFO mapred.LocalJobRunner: hdfs://
 0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125
  map
 14/03/17 23:54:29 INFO mapreduce.Job:  map 6% reduce 0%
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 (1000 Exceptions)
 .
 .
 .
 results:

  File System Counters
 FILE: Number of bytes read=18769411
 FILE: Number of bytes written=21398315
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=11185
 HDFS: Number of bytes written=19540
 HDFS: Number of read operations=325
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=13210
 Map-Reduce Framework
 Map input records=12
 Map output records=95
 Map output bytes=1829
 Map output materialized bytes=2091
 Input split bytes=1538
 Combine input records=0
 Combine output records=0
 Reduce input groups=8
 Reduce shuffle bytes=0
 Reduce input records=95
 Reduce output records=8
 Spilled Records=214
 Shuffled Maps =0
 Failed Shuffles=0
 Merged Map outputs=0
 GC time elapsed (ms)=211
 CPU time spent (ms)=0
 Physical memory (bytes) snapshot=0
 Virtual memory (bytes) snapshot=0
 Total committed heap usage (bytes)=4401004544
 File Input Format Counters
 Bytes Read=1490
 File Output Format Counters
 Bytes Written=170
  14/03/17 23:56:18 INFO hdfs.NNBench: -- NNBench
 -- :
 14/03/17 23:56:18 INFO hdfs.NNBench:
  Version: NameNode Benchmark 0.4
 14/03/17 23:56:18 INFO hdfs.NNBench:Date 
 time: 2014-03-17 23:56:18,619
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench: Test
 Operation: create_write
 14/03/17 23:56:18 INFO hdfs.NNBench: Start
 time: 2014-03-17 23:56:15,521
 14/03/17 23:56:18 INFO hdfs.NNBench:Maps to
 run: 12
 14/03/17 23:56:18 INFO hdfs.NNBench: Reduces to
 run: 6
 14/03/17 23:56:18 INFO hdfs.NNBench: Block Size
 (bytes): 1
 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes to
 write: 0
 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes per
 checksum: 1
 14/03/17 23:56:18 INFO hdfs.NNBench:Number of
 files: 1000
 14/03/17 23:56:18 INFO hdfs.NNBench: Replication
 factor: 3
 14/03/17 23:56:18 INFO hdfs.NNBench: Successful file
 operations: 0
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench: # maps that missed the
 barrier: 11
 14/03/17 23:56:18 INFO hdfs.NNBench:   #
 exceptions: 1000
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench:TPS:
 Create/Write/Close: 0
 14/03/17 23:56:18 INFO hdfs.NNBench: Avg exec time (ms):
 Create/Write/Close: Infinity
 14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms):
 Create/Write: NaN
 14/03/17 23:56:18 INFO hdfs.NNBench:Avg Lat (ms):
 Close: NaN
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench:  RAW DATA: AL Total
 #1: 0
 14/03/17

Re: Benchmark Failure

[Harsh J]

Do not leave that configuration in after your tests are done. It would be very
harmful to allow such tiny block sizes from clients, enabling them to
flood your NameNode's metadata with a lot of blocks for a small file.

If its instead possible, tune NNBench's block size to be larger perhaps.

On Sat, Mar 22, 2014 at 2:26 PM, Lixiang Ao aolixi...@gmail.com wrote:
 Checked the logs, and turned out to be configuration problem. Just set
 dfs.namenode.fs-limits.min-block-size to 1 and it's fixed

 Thanks.


 On Wed, Mar 19, 2014 at 2:51 PM, Brahma Reddy Battula
 brahmareddy.batt...@huawei.com wrote:

 Seems to be this is issue, which is logged..Please check following jira
 for sameHope you also facing same issue...



 https://issues.apache.org/jira/browse/HDFS-4929







 Thanks  Regards



 Brahma Reddy Battula



 
 From: Lixiang Ao [aolixi...@gmail.com]
 Sent: Tuesday, March 18, 2014 10:34 AM
 To: user@hadoop.apache.org
 Subject: Re: Benchmark Failure

 the version is release 2.2.0

 2014年3月18日 上午12:26于 Lixiang Ao aolixi...@gmail.com写道：

 Hi all,

 I'm running jobclient tests(on single node), other tests like TestDFSIO,
 mrbench succeed except nnbench.

 I got a lot of Exceptions but without any explanation(see below).

 Could anyone tell me what might went wrong?

 Thanks!


 14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms
 14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001
 running in uber mode : false
 14/03/17 23:54:23 INFO mapreduce.Job:  map 0% reduce 0%
 14/03/17 23:54:28 INFO mapred.LocalJobRunner:
 hdfs://0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125
  map
 14/03/17 23:54:29 INFO mapreduce.Job:  map 6% reduce 0%
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
 Create/Write/Close
 (1000 Exceptions)
 .
 .
 .
 results:

 File System Counters
 FILE: Number of bytes read=18769411
 FILE: Number of bytes written=21398315
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=11185
 HDFS: Number of bytes written=19540
 HDFS: Number of read operations=325
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=13210
 Map-Reduce Framework
 Map input records=12
 Map output records=95
 Map output bytes=1829
 Map output materialized bytes=2091
 Input split bytes=1538
 Combine input records=0
 Combine output records=0
 Reduce input groups=8
 Reduce shuffle bytes=0
 Reduce input records=95
 Reduce output records=8
 Spilled Records=214
 Shuffled Maps =0
 Failed Shuffles=0
 Merged Map outputs=0
 GC time elapsed (ms)=211
 CPU time spent (ms)=0
 Physical memory (bytes) snapshot=0
 Virtual memory (bytes) snapshot=0
 Total committed heap usage (bytes)=4401004544
 File Input Format Counters
 Bytes Read=1490
 File Output Format Counters
 Bytes Written=170
 14/03/17 23:56:18 INFO hdfs.NNBench: -- NNBench
 -- :
 14/03/17 23:56:18 INFO hdfs.NNBench:
 Version: NameNode Benchmark 0.4
 14/03/17 23:56:18 INFO hdfs.NNBench:Date 
 time: 2014-03-17 23:56:18,619
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench: Test
 Operation: create_write
 14/03/17 23:56:18 INFO hdfs.NNBench: Start
 time: 2014-03-17 23:56:15,521
 14/03/17 23:56:18 INFO hdfs.NNBench:Maps to
 run: 12
 14/03/17 23:56:18 INFO hdfs.NNBench: Reduces to
 run: 6
 14/03/17 23:56:18 INFO hdfs.NNBench: Block Size
 (bytes): 1
 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes to
 write: 0
 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes per
 checksum: 1
 14/03/17 23:56:18 INFO hdfs.NNBench:Number of
 files: 1000
 14/03/17 23:56:18 INFO hdfs.NNBench: Replication
 factor: 3
 14/03/17 23:56:18 INFO hdfs.NNBench: Successful file
 operations: 0
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench: # maps that missed the
 barrier: 11
 14/03/17 23:56:18 INFO hdfs.NNBench:   #
 exceptions: 1000
 14/03/17 23:56:18 INFO hdfs.NNBench:
 14/03/17 23:56:18 INFO hdfs.NNBench:TPS:
 Create/Write/Close: 0
 14/03/17 23:56:18 INFO

RE: Benchmark Failure

[Brahma Reddy Battula]

Seems to be this is issue, which is logged..Please check following jira for 
sameHope you also facing same issue...



https://issues.apache.org/jira/browse/HDFS-4929







Thanks  Regards



Brahma Reddy Battula




From: Lixiang Ao [aolixi...@gmail.com]
Sent: Tuesday, March 18, 2014 10:34 AM
To: user@hadoop.apache.org
Subject: Re: Benchmark Failure


the version is release 2.2.0

2014年3月18日 上午12:26于 Lixiang Ao 
aolixi...@gmail.commailto:aolixi...@gmail.com写道：
Hi all,

I'm running jobclient tests(on single node), other tests like TestDFSIO, 
mrbench succeed except nnbench.

I got a lot of Exceptions but without any explanation(see below).

Could anyone tell me what might went wrong?

Thanks!


14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms
14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001 running in 
uber mode : false
14/03/17 23:54:23 INFO mapreduce.Job:  map 0% reduce 0%
14/03/17 23:54:28 INFO mapred.LocalJobRunner: 
hdfs://0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125http://0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125
  map
14/03/17 23:54:29 INFO mapreduce.Job:  map 6% reduce 0%
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: 
Create/Write/Close
(1000 Exceptions)
.
.
.
results:

File System Counters
FILE: Number of bytes read=18769411
FILE: Number of bytes written=21398315
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=11185
HDFS: Number of bytes written=19540
HDFS: Number of read operations=325
HDFS: Number of large read operations=0
HDFS: Number of write operations=13210
Map-Reduce Framework
Map input records=12
Map output records=95
Map output bytes=1829
Map output materialized bytes=2091
Input split bytes=1538
Combine input records=0
Combine output records=0
Reduce input groups=8
Reduce shuffle bytes=0
Reduce input records=95
Reduce output records=8
Spilled Records=214
Shuffled Maps =0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=211
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=4401004544
File Input Format Counters
Bytes Read=1490
File Output Format Counters
Bytes Written=170
14/03/17 23:56:18 INFO hdfs.NNBench: -- NNBench -- :
14/03/17 23:56:18 INFO hdfs.NNBench:Version: 
NameNode Benchmark 0.4
14/03/17 23:56:18 INFO hdfs.NNBench:Date  time: 
2014-03-17 23:56:18,619
14/03/17 23:56:18 INFO hdfs.NNBench:
14/03/17 23:56:18 INFO hdfs.NNBench: Test Operation: 
create_write
14/03/17 23:56:18 INFO hdfs.NNBench: Start time: 
2014-03-17 23:56:15,521
14/03/17 23:56:18 INFO hdfs.NNBench:Maps to run: 12
14/03/17 23:56:18 INFO hdfs.NNBench: Reduces to run: 6
14/03/17 23:56:18 INFO hdfs.NNBench: Block Size (bytes): 1
14/03/17 23:56:18 INFO hdfs.NNBench: Bytes to write: 0
14/03/17 23:56:18 INFO hdfs.NNBench: Bytes per checksum: 1
14/03/17 23:56:18 INFO hdfs.NNBench:Number of files: 
1000
14/03/17 23:56:18 INFO hdfs.NNBench: Replication factor: 3
14/03/17 23:56:18 INFO hdfs.NNBench: Successful file operations: 0
14/03/17 23:56:18 INFO hdfs.NNBench:
14/03/17 23:56:18 INFO hdfs.NNBench: # maps that missed the barrier: 11
14/03/17 23:56:18 INFO hdfs.NNBench:   # exceptions: 
1000
14/03/17 23:56:18 INFO hdfs.NNBench:
14/03/17 23:56:18 INFO hdfs.NNBench:TPS: Create/Write/Close: 0
14/03/17 23:56:18 INFO hdfs.NNBench: Avg exec time (ms): Create/Write/Close: 
Infinity
14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms): Create/Write: NaN
14/03/17 23:56:18 INFO hdfs.NNBench:Avg Lat (ms): Close: NaN
14/03/17 23:56:18 INFO hdfs.NNBench:
14/03/17 23:56:18 INFO hdfs.NNBench:  RAW DATA: AL Total #1: 0
14/03/17 23:56:18 INFO hdfs.NNBench:  RAW DATA: AL Total #2: 0
14/03/17 23:56:18 INFO hdfs.NNBench:   RAW DATA: TPS Total (ms): 
1131
14/03/17 23:56:18 INFO hdfs.NNBench:RAW DATA: Longest Map