Re: Newb question on HDFS

2010-06-23 Thread Mafish Liu
2010/6/24 Blargy : > > Hey all. Im trying to start up a psuedo distributed setup following the > directions in Hadoop In Action however I am having problems. For some reason > my TaskTracker and JobTracker are not starting up but there is nothing in my > log files explaining why. > > FYI My develop

How many nodes are there in the largest hadoop cluster worldwide?

2010-04-11 Thread Mafish Liu
Hi, all: I'm writing an article related to hadoop and want to know how many nodes are there in the largest hadoop cluster worldwide. Regards

Re: Exception occurs when I try to read a file

2010-01-19 Thread Mafish Liu
The node on which this program was running failed to connect to namenode. Check your configuration of hadoop cluster first, eg, the firewall. 2010/1/19 ztesoft : > > Hi, I wrote a program to read a file in HDFS. > The codes are following: > import java.io.IOException; > import java.net.MalformedUR

Re: rmr: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /op. Name node is in safe mode.

2010-01-18 Thread Mafish Liu
2010/1/19 prasenjit mukherjee : >  I run "hadoop fs -rmr .." immediately after start-all.sh    Does the > namenode always start in safemode and after sometime switches to > normal mode ? If that is the problem then your suggestion of waiting > might work. Lemme check. This is the point. Namenode w

Re: hadoop job progress going back

2010-01-04 Thread Mafish Liu
I met this problem too. It does be caused by some failures of reduce task. Check your web UI for details of failure. 2009/12/27 Doopah Shaf : > Running jobs sometimes displays progress as follows: > > map 100% reduce 10% > > and then > > map 100% reduce 0% > > What causes this? Is this speculativ

Re: Question about DFSClient "Could not obtain block" errors

2009-09-14 Thread Mafish Liu
Check if your datanodes are starting up correctly. This error occurs when there are file entries in namenode. By it meets problems to fetch file data from datanodes. 2009/9/15 scott w : > I am trying to read data placed on hdfs in one EC2 cluster from a different > EC2 cluster and am getting the e

Re: Datanode high memory usage

2009-09-01 Thread Mafish Liu
2009/9/1 Mafish Liu : > Both NameNode and DataNode will be affected by number of files greatly. > In my test, almost 60% memory are used in datanodes while storing 1m > files, and the value reach 80% with 2m files. > My test best is with 5 nodes, 1 namenode and 4 datanodes. All nodes

Re: Datanode high memory usage

2009-09-01 Thread Mafish Liu
/9/1 Stas Oskin : > Hi. > > 2009/9/1 Mafish Liu > >> Did you have many small files in your system? >> >> > Yes, quite plenty. > > But this should influence the Namenode, and not the Datanode, correct? > > Regards. > -- maf...@gmail.com

Re: Datanode high memory usage

2009-08-31 Thread Mafish Liu
Did you have many small files in your system? 2009/9/1 Stas Oskin : > Hi. > > >> >>  mapred.child.java.opts >>   >>     -Xmx512M >>   >> >> >> > This has effect even if I not using any reduce tasks? > > Regards. > -- maf...@gmail.com

Re: Building Hadoop from source

2009-07-09 Thread Mafish Liu
Use "ant jar" if you want to jar file. 2009/7/9 Harish Mallipeddi : > Hi, > Are there any instructions on how to build Hadoop from source? Now that the > project seems to have been split into separate projects (common, hdfs, and > mapreduce), there are 3 separate repositories under svn. Informatio