Does anyone run Hadoop in PC cluster?
I just tested WordCount in PC cluster, and my first impression as following:
***
Number of PCs: 7(512M RAM, 2.8G CPU, 100M NIC, CentOS 5.0, Handoop
0.16.1, Sun jre 1.6)
Master(Namenode): 1
Master(Jobtracker): 1
Slaves(Datanode & Tasktracker): 5
1. Writing to HDFS
--
File size: 4,295,341,065 bytes(4.1G)
Time elapsed putting file into HDFS: 7m57.757s
Average rate: 8,990,583 bytes/sec
Average bandwidth usage: 68.59%
I also tested libhdfs, it's just as fine as java.
2. Map/Reduce with Java
--
Time elapsed: 19mins, 56sec
Bytes/time rate: 3,591,422 bytes/sec
Job Counters:
Launched map tasks 67
Launched reduce tasks 7
Data-local map tasks 64
Map-Reduce Framework:
Map input records 65,869,800
Map output records 697,923,360
Map input bytes 4,295,341,065
Map output bytes 6,504,944,565
Combine input records 697,923,360
Combine output records 2,330,048
Reduce input groups 5,201
Reduce input records 2,330,048
Reduce output records 5,201
It's acceptable. The main bottleneck was CPU, keeping 100% usage.
3. Map/Reduce with C++ Pipe(No combiner)
--
Time elapsed: 1hrs, 2mins, 47sec
Bytes/time rate: 1,140,255 bytes/sec
Job Counters:
Launched map tasks 68
Launched reduce tasks 5
Data-local map tasks 64
Map-Reduce Framework:
Map input records 65,869,800
Map output records 697,452,105
Map input bytes 4,295,341,065
Map output bytes 5,107,053,975
Combine input records 0
Combine output records 0
Reduce input groups 5,191
Reduce input records 697,452,105
Reduce output records 5,191
As my first impression, C++ pipe interface is slower than Java. If I add
C++ pipe combiner, the result become even worse: The main bottleneck is
RAM, a great deal of swapping space used, processes blocked, CPU keeping
waiting...
Adding more RAM maybe improve performance, but still slower than Java, I
think.
4. Map/Reduce with Python streaming(No combiner)
--
Time elapsed: 1hrs, 48mins, 53sec
Bytes/time rate: 657,483 bytes/sec
Job Counters:
Launched map tasks 68
Launched reduce tasks 5
Data-local map tasks 64
Map-Reduce Framework:
Map input records 65,869,800
Map output records 697,452,105
Map input bytes 4,295,341,065
Map output bytes 5,107,053,975
Combine input records 0
Combine output records 0
Reduce input groups 5,191
Reduce input records 697,452,105
Reduce output records 5,191
As you see, the result is not as good as C++ pipe interface. Maybe
python is slower, I didn't test other cases.
Are there any suggestions to improve such situation?
--
yingyuan
