Hello George, Have you looked at your DFS health page (http://NN:50070/)? I believe you have missing or fallen DataNode instances.
I'd start them back up, after checking their (DataNode's) logs to figure out why they died. On Wed, Sep 21, 2011 at 7:28 PM, George Kousiouris <gkous...@mail.ntua.gr> wrote: > > Hi all, > > We are trying to run a mahout job in a hadoop cluster, but we keep getting > the same status. The job passes the initial mahout stages and when it comes > to be executed as a MR job, it seems to be stuck at 0% progress. Through the > UI we see that it is submitted but not running. After a while it gets > killed. In the logs the error shown is this one: > > 2011-09-21 07:47:50,507 INFO org.apache.hadoop.mapred.JobTracker: problem > cleaning system directory: > hdfs://master/var/lib/hadoop-0.20/cache/hdfs/mapred/system > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create > directory /var/lib/hadoop-0.20/cache/hdfs/mapred/system. Name nod$ > The reported blocks 0 needs additional 12 blocks to reach the threshold > 0.9990 of total blocks 13. Safe mode will be turned off automatically. > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:1966) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:1940) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:770) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > > > Some staging files seem to have been created however. > > I was thinking of sending this to the mahout mailing list but it seems a > more core hadoop issue. > > We are using the following command to launch the mahout example: > ./mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job --input > hdfs://master/user/hdfs/testdata/synthetic_control.data --output > hdfs://master/user/hdfs/testdata/output --t1 0.5 --t2 1 --maxIter 50 > > Any clues? > George > > -- > > --------------------------- > > George Kousiouris > Electrical and Computer Engineer > Division of Communications, > Electronics and Information Engineering > School of Electrical and Computer Engineering > Tel: +30 210 772 2546 > Mobile: +30 6939354121 > Fax: +30 210 772 2569 > Email: gkous...@mail.ntua.gr > Site: http://users.ntua.gr/gkousiou/ > > National Technical University of Athens > 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece > > -- Harsh J
