Hadoop issue on 64 bit ubuntu . Native Libraries.
Hi, I am using the kmeans clustering in mahout. It ran fine in my 32 bit machine. But when I try to run it in another 64 bit machine I get the following error: *org.apache.hadoop.util.NativeCodeLoader clinit WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable * I built my native libraries in hadoop. My project uses the hadoop core jar file alone. The algorithm does not run distributively over hadoop but runs only in one system. I am not using the parallelizing capacity of hadoop yet. I am getting the error when using methods in hadoop.util.NativeCodeLoader. Can somebody help me to build the native libraries? I am using ubuntu 64 bit version on a amd processor. -- Sincerely, Ajay Anandan. MSc,Computing Science, University of Alberta.
NullPointerException on namenode
I restarted the cluster after the server was way overload by other task and now I get this 2011-02-23 08:36:18,307 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1088) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1100) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1003) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:206) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:637) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1034) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:845) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:379) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:99) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:347) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:321) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:267) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:461) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1202) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1211) any ideas on how to fix or howtos out there I can not find?
Problem in using Trash
Hello everybody, I have a problem with adapting Trash into the running system. To my knowledge, the jar file is created in the namenode and sent to each datanode during the MapReduce process. And the jar file is removed automatically after the Mapreduce process is completed. But after I adapted the Trash into the system, the jar file is not removed from the datanode. The files were accumulated as I repeat the MapReduce process, and the disk space nearly got full. How could I solve this problem? Do you think it is related with using Trash? Regards, Henny (ahneui...@gmail.com)
Re: Hadoop issue on 64 bit ubuntu . Native Libraries.
Hi Ajay, Hadoop should ship with built artifacts for amd64 in the lib/native/Linux-amd64-64/ subdirectory of your tarball. You just need to put this directory on your java.library.path system property. -Todd You need to run ant -Dcompile.native=1 compile-native fro On Tue, Feb 22, 2011 at 9:14 PM, Ajay Anandan anan...@ualberta.ca wrote: Hi, I am using the kmeans clustering in mahout. It ran fine in my 32 bit machine. But when I try to run it in another 64 bit machine I get the following error: *org.apache.hadoop.util.NativeCodeLoader clinit WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable * I built my native libraries in hadoop. My project uses the hadoop core jar file alone. The algorithm does not run distributively over hadoop but runs only in one system. I am not using the parallelizing capacity of hadoop yet. I am getting the error when using methods in hadoop.util.NativeCodeLoader. Can somebody help me to build the native libraries? I am using ubuntu 64 bit version on a amd processor. -- Sincerely, Ajay Anandan. MSc,Computing Science, University of Alberta. -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop issue on 64 bit ubuntu . Native Libraries.
On Wed, Feb 23, 2011 at 11:21 AM, Todd Lipcon t...@cloudera.com wrote: Hi Ajay, Hadoop should ship with built artifacts for amd64 in the lib/native/Linux-amd64-64/ subdirectory of your tarball. You just need to put this directory on your java.library.path system property. -Todd You need to run ant -Dcompile.native=1 compile-native fro err, disregard this postscript. Started writing how to compile it, and then realized we actually ship built artifacts :) On Tue, Feb 22, 2011 at 9:14 PM, Ajay Anandan anan...@ualberta.ca wrote: Hi, I am using the kmeans clustering in mahout. It ran fine in my 32 bit machine. But when I try to run it in another 64 bit machine I get the following error: *org.apache.hadoop.util.NativeCodeLoader clinit WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable * I built my native libraries in hadoop. My project uses the hadoop core jar file alone. The algorithm does not run distributively over hadoop but runs only in one system. I am not using the parallelizing capacity of hadoop yet. I am getting the error when using methods in hadoop.util.NativeCodeLoader. Can somebody help me to build the native libraries? I am using ubuntu 64 bit version on a amd processor. -- Sincerely, Ajay Anandan. MSc,Computing Science, University of Alberta. -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera
Is it possible to reset task failure counts to 0 without restarting JT, TT or Mapred.
I would like to reset failure count of Task tracker to 0, Jobtracker maintains task failures count for each Task tracker and this count is used in blacklisting Task trackers. In a small cluster if I restart corresponding Task tracker there will be inconsistent state and other task trackers will have hight failure count. To avoid this I would like to reset Task trackers failure count to 0 without restarting Mapred or Jobtracker. Thanks, Ravi
March 2011 San Francisco Hadoop User Meetup (integration)
Hadoop fans, I'm pleased to announce that the third SF Hadoop meetup will be held Wednesday, March 9th, from 6pm to 8pm. (We will hopefully continue using the 2nd Wednesday of the month for successive meetups). This meetup will be hosted by the good folks at Yelp. Their office is at 706 Mission St, San Francisco (3rd and Mission). We have ample space available for a large gathering of Hadoop enthusiasts. As per our emerging tradition, we will continue to use the discussion-based unconference format. At the beginning of the meetup we will collaboratively construct an agenda consisting of several discussion breakout groups. All participants may propose a topic and volunteer to facilitate a discussion. All members of the Hadoop community are welcome to attend. While all Hadoop-related subjects are on topic, I'd like to seed this month's discussion with the theme of integration. Yelp has asked that all attendees RSVP in advance, to comply with their security policy. Please join the meetup group and RSVP at http://www.meetup.com/hadoopsf/events/16678757/ Refreshments will be provided. Regards, - Aaron Kimball
Current available Memory
Hello Everyone, I'm using Runtime.getRuntime().freeMemory() to see current memory available before and after creation of an object, but this doesn't seem to work well with Hadoop? Why? and is there another alternative? Thank you, Maha
Re: Current available Memory
Based on the Java function documentation, it gives approximately the available memory, so I need to tweak it with other functions. So it's a Java issue not Hadoop. Thanks anyways, Maha On Feb 23, 2011, at 6:31 PM, maha wrote: Hello Everyone, I'm using Runtime.getRuntime().freeMemory() to see current memory available before and after creation of an object, but this doesn't seem to work well with Hadoop? Why? and is there another alternative? Thank you, Maha
Library Issues
Dear all, I am confused about the concepts used while running map-reduce jobs in Hadoop Cluster. I attached a program that is used to run in Hadoop Cluster. Please find the attachment. I used to run this program successfully through below command in /home/hadoop/project/hadoop-0.20.2 directory of both master slave node: [hadoop@cuda1 hadoop-0.20.2]$ javac EnumDevices.java [hadoop@cuda1 hadoop-0.20.2]$ java EnumDevices Total number of devices: 1 Name: Tesla C1060 Version: 1.3 Clock rate: 1296000 MHz Threads per block: 512 but when I used the same code ( jcuda libraries in map-reduce job ) , i result in the below errors : [hadoop@ws37-mah-lin hadoop-0.20.2]$ bin/hadoop jar wordcount1.jar org.myorg.WordCount /user/hadoop/gutenberg /user/hadoop/output1 11/02/22 09:59:21 INFO input.FileInputFormat: Total input paths to process : 3 11/02/22 09:59:22 INFO mapred.JobClient: Running job: job_201102220937_0001 11/02/22 09:59:23 INFO mapred.JobClient: map 0% reduce 0% 11/02/22 09:59:33 INFO mapred.JobClient: Task Id : attempt_201102220937_0001_m_00_0, Status : FAILED 11/02/22 09:59:33 INFO mapred.JobClient: Task Id : attempt_201102220937_0001_m_01_0, Status : FAILED java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113) ... 3 more *Caused by: java.lang.UnsatisfiedLinkError: no jcuda in java.library.path* at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734) at java.lang.Runtime.loadLibrary0(Runtime.java:823) at java.lang.System.loadLibrary(System.java:1028) at jcuda.driver.CUDADriver.clinit(CUDADriver.java:909) at jcuda.CUDA.init(CUDA.java:62) at jcuda.CUDA.init(CUDA.java:42) at org.myorg.WordCount$TokenizerMapper.init(WordCount.java:28) ... 8 more 11/02/22 09:59:42 INFO mapred.JobClient: Task Id : attempt_201102220937_0001_m_01_1, Status : FAILED java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113) ... 3 more *Caused by: java.lang.UnsatisfiedLinkError: no jcuda in java.library.path* at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1709) at java.lang.Runtime.loadLibrary0(Runtime.java:823) at java.lang.System.loadLibrary(System.java:1028) at jcuda.driver.CUDADriver.clinit(CUDADriver.java:909) at jcuda.CUDA.init(CUDA.java:62) at jcuda.CUDA.init(CUDA.java:42) at org.myorg.WordCount$TokenizerMapper.init(WordCount.java:28) ... 8 more 11/02/22 09:59:42 INFO mapred.JobClient: Task Id : attempt_201102220937_0001_m_00_1, Status : FAILED java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
Re: Library Issues
Hey, On Thu, Feb 24, 2011 at 10:13 AM, Adarsh Sharma adarsh.sha...@orkash.com wrote: Dear all, I am confused about the concepts used while running map-reduce jobs in Hadoop Cluster. I attached a program that is used to run in Hadoop Cluster. Please find the attachment. My PATH Variable shows that it includes all libraries as [hadoop@cuda1 ~]$ echo $PATH /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/hadoop/project/hadoop-0.20.2/jcuda.jar:/usr/local/cuda/lib:/home/hadoop/bin [hadoop@cuda1 ~]$ AFAIK, on Linux, java.library.path is loaded from LD_LIBRARY_PATH and on Windows it is PATH. Also, build/lib/native or lib/native folders are added if available to Hadoop's java.library.path if launched using the bin scripts. You can place your native libraries there (under the proper Java platform). -- Harsh J www.harshj.com
Re: Current available Memory
I had also encuntered the smae problem a few days ago. any one has another method? 2011/2/24 maha m...@umail.ucsb.edu Based on the Java function documentation, it gives approximately the available memory, so I need to tweak it with other functions. So it's a Java issue not Hadoop. Thanks anyways, Maha On Feb 23, 2011, at 6:31 PM, maha wrote: Hello Everyone, I'm using Runtime.getRuntime().freeMemory() to see current memory available before and after creation of an object, but this doesn't seem to work well with Hadoop? Why? and is there another alternative? Thank you, Maha
is there more smarter way to execute a hadoop cluster?
Hi, I executed my cluster by this way. call a command in shell directly. String runInCommand =/opt/hadoop-0.21.0/bin/hadoop jar testCluster.jar example; Process proc = Runtime.getRuntime().exec(runInCommand); proc.waitFor(); BufferedReader in = new BufferedReader(new InputStreamReader(proc.getErrorStream())); for (String str; (str = in.readLine()) != null;) System.out.println(str); System.exit(0); but, in a hadoop script, it calls the RunJar() class to deploy testCluster.jar file. isn't it? is there more smarter way to execute a hadoop cluster? thanks. -- Junyoung Kim (juneng...@gmail.com)
Re: is there more smarter way to execute a hadoop cluster?
Hello, On Thu, Feb 24, 2011 at 12:25 PM, Jun Young Kim juneng...@gmail.com wrote: Hi, I executed my cluster by this way. call a command in shell directly. What are you doing within your testCluster.jar? If you are simply submitting a job, you can use a Driver method and get rid of all these hassles. JobClient and Job classes both support submitting jobs from Java API itself. Please read the tutorial on submitting application code via code itself: http://developer.yahoo.com/hadoop/tutorial/module4.html#driver Notice the last line in the code presented there, which submits a job itself. Using runJob() also prints your progress/counters etc. The way you've implemented this looks unnecessary when your Jar itself can be made runnable with a Driver! -- Harsh J www.harshj.com
Re: NullPointerException on namenode
You need to further dig down in NameNode logs, also make sure that your edit log and fsimage is not corrupted. Use backup fsimage editlog from Secondary name node to restart the cluster and see if this problem persists. Looking at stack trace I have a feeling that it's due to the corrupt file/directory name in your FSimage. You can use hadoop ovi tool to print FSimage contents. - Ravi On Wed, Feb 23, 2011 at 6:38 AM, gmane.org sa...@pearsonwholesale.comwrote: I restarted the cluster after the server was way overload by other task and now I get this 2011-02-23 08:36:18,307 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1088) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1100) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1003) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:206) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:637) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1034) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:845) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:379) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:99) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:347) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:321) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:267) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:461) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1202) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1211) any ideas on how to fix or howtos out there I can not find?
