Hi, We are using hadoop installation over 6 nodes, that looks like:
172.19.0.10 (master) 172.19.0.11 172.19.0.12 172.19.0.13 172.19.0.14 172.19.0.15 Each node has (4x1.5 TB) hard disk (hdparm -t shows ~125 MB/sec for each other) and 12 GB memory. Connected to 100 Mb switch. I get following results with standard io tests, running as: bin/hadoop jar hadoop-*-test.jar TestDFSIO -read -nrFiles 15 -fileSize 512 -bufferSize 1000000 Write test: ----- TestDFSIO ----- : write Number of files: 15 Total MBytes processed: 7680 Throughput mb/sec: 123.82503264917852 IO rate mb/sec: 129.23692321777344 IO rate std deviation: 22.25608461080549 Test exec time sec: 28.656 Read test: ----- TestDFSIO ----- : read Date & time: Wed Dec 09 13:42:52 CET 2009 Number of files: 15 Total MBytes processed: 7680 Throughput mb/sec: 4.833249737412311 Average IO rate mb/sec: 37.22291946411133 IO rate std deviation: 65.57958375615678 Test exec time sec: 300.542 As you can see read performance is very poor. Also we noticed following: 1) Run ping on all nodes to master. When write test is working ping's times always stable and less than 1 ms, but when read test is working we see increased ping's times: 64 bytes from 172.19.0.10: icmp_seq=4444 ttl=64 time=13.3 ms 64 bytes from 172.19.0.10: icmp_seq=4445 ttl=64 time=18.1 ms 64 bytes from 172.19.0.10: icmp_seq=4446 ttl=64 time=18.3 ms 64 bytes from 172.19.0.10: icmp_seq=4447 ttl=64 time=14.3 ms 2) When running write test iostat shows that there is activity on all 4 disks on node, but when running read test, there is almost no activity. Could you explain where problem could be? Thank you. -- Regards, Dmitriy
