Re:Re: Problem with building hadoop 0.21

2011-02-28 Thread 朱韬 
Hi.Simon:
   I modified some coed related to scheduler and designed a  customized 
scheduler .when I built the modified code, then the problems described above 
came up with it. I doubt whether there was something with my code, but after  I 
built the out-of-box code, the same problems still existed. Can you tell me how 
to build and deploy  a  customized hadoop?
 Thank you!

   zhutao
 




At 2011-02-28 11:21:16，Simon  wrote:

>Hey,
>
>Can you let us know why you want to replace all the jar files? That usually
>does not work, especially for development code in the code base.
>So, just use the one you have successfully compiled, don't replace jar
>files.
>
>Hope it can work.
>
>Simon
>
>2011/2/27 朱韬 
>
>> Hi，guys:
>>  I checked out the source code fromhttp://
>> svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using
>> this script:
>>  #!/bin/bash
>> export JAVA_HOME=/usr/share/jdk1.6.0_14
>> export CFLAGS=-m64
>> export CXXFLAGS=-m64
>> export ANT_HOME=/opt/apache-ant-1.8.2
>> export PATH=$PATH:$ANT_HOME/bin
>> ant -Dversion=0.21.0 -Dcompile.native=true
>> -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar
>> It was Ok before these steps. Then I replaced
>> "hadoop-mapred-0.21.0.jar", hadoop-mapred-0.21.0-sources.jar,
>>  hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and
>> hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files
>> from the above step. Also I added my scheduler to lib. When starting the
>> customed hadoop, I encountered the problems as blow:
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/security/RefreshUserMappingsProtocol
>>at java.lang.ClassLoader.defineClass1(Native Method)
>>at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
>>at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
>>at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
>>at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>at java.security.AccessController.doPrivileged(Native Method)
>>at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> 10.61.0.6: starting tasktracker, logging to
>> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out
>> 10.61.0.143: starting tasktracker, logging to
>> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out
>> 10.61.0.7: starting tasktracker, logging to
>> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out
>> 10.61.0.6: Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException
>> 10.61.0.6: Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException
>> 10.61.0.6:  at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> 10.61.0.6:  at java.security.AccessController.doPrivileged(Native
>> Method)
>> 10.61.0.6:  at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> 10.61.0.6:  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> 10.61.0.6:  at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 10.61.0.6:  at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> 10.61.0.6:  at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> 10.61.0.6: Could not find the main class:
>> org.apache.hadoop.mapred.TaskTracker.  Program will exit.
>> 10.61.0.143: Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException
>> 10.61.0.143: Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException
>> 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> 10.61.0.143:at java.security.AccessController.doPrivileged(Native
>> Method)
>> 10.61.0.143:at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> 10.61.0.143:at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 10.61.0.143:at

how to get each task's process?

2011-03-29 Thread 朱韬 
hi,guys:
   I want to monitor each tasks' io usage, do can I get each task's process 
id?
   I used  jvmManager.getPid(tip.getTaskRunner()); but it didn't seem to be 
the task's process.
   Thanks

 zhutao

Re:Re: how to get each task's process

2011-03-29 Thread 朱韬 
Nan:
 Thanks, I mean process, not progress




在 2011-03-29 23:05:29，"Nan Zhu"  写道：

>TaskStatus.getprogress() is helpful
>
>Best,
>Nan
>
>2011/3/29 朱韬 
>
>> hi,guys:
>>   I want to monitor each tasks' io usage, do can I get each task's
>> process id?
>>   I used  jvmManager.getPid(tip.getTaskRunner()); but it didn't seem to
>> be the task's process.
>>   Thanks
>>
>> zhutao
>
>
>
>
>-- 
>Nan Zhu
>School of Software,5501
>Shanghai Jiao Tong University
>800,Dongchuan Road,Shanghai,China
>E-Mail: zhunans...@gmail.com

Re:Re: how to get each task's process

2011-03-30 Thread 朱韬 
Harsh: 
 I  found that jvmManager.getPid(...) returned the pid of 
MapTaskRunner, but I want to get the task's pid. For example, I ran the the 
example randomwrite, the pid of  task which is writing is 8268, but 
jvmManager.getPid(...) seemed to be its parent pid. I can not figure out the 
relationship between the taskrunner and the real task.
Thanks 

 zhutao





At 2011-03-30 01:17:30，"Harsh J"  wrote:

>I think jvmManager.getPid(...) is correct. It should give you the
>launched JVM's PID properly. In fact, the same is even used to kill
>the JVMs.
>
>2011/3/29 朱韬 :
>> hi,guys:
>>   I want to monitor each tasks' io usage, do can I get each task's 
>> process id?
>>   I used  jvmManager.getPid(tip.getTaskRunner()); but it didn't seem to 
>> be the task's process.
>>   Thanks
>>  
>>zhutao
>
>
>
>-- 
>Harsh J
>http://harshj.com