I run entire one node cluster in eclipse by just executing main() (run
or debug menus) for each component.
You need to configure eclipse correctly in order to do that. Can you
compile the whole thing under eclipse?
NameNode example:
= Open NameNode.java in the editor.
= Run / Run
= New Java Application -> will create an entry under "Java Application"
named NameNode
= Select NameNode, go to tab Arguments and enter the following arguments
under "VM Arguments":
-Dhadoop.log.dir=./logs
-Xmx500m
-ea
The first one is required, can point to your log directory, the
other two are optional
= go to the "Classpath" tab, add "hadoop/build" path under "User entries" by
Advanced / New Folder / select "hadoop/build"
That should be it, if the default classpath is configured correctly, and
if I am not forgetting anything.
Let me know if that helped, I 'll send you screenshots of my
configuration if not.
--Konstantin
Mahajan, Neeraj wrote:
There are two sepearete issues you are asking here:
1. How to modify/add to haddop code and execute the changed -
Eclipse is just an IDE, it doesn't matter whether you use eclipse or
some other editor.
I have been using eclipse. What I do is modify the code using eclipse
and then run "ant jar" in the root folder of hadoop (you could also
configure this to work directly from eclipse). This would regenerate the
jars and put them in build/ folder. Now you can either copy these jars
into hadoop root folder (removing "dev" in their name) so that they
replace the original jars or modify the scripts in bin/ to point to the
newly generated jars.
2. How to debug using a IDE -
This page gives a high-level intro to debugging hadoop -
http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms
According to me, there are two ways you can debug hadoop programs: Run
hadoop in local mode and debug in process in the IDE or run hadoop in
distributed mode and remote debug using IDE.
The first way is easy. In the bin/hadoop script at the end there is a
exec command, instead of that put a echo command and run your program.
You can see what the paramters the script passes while starting hadoop.
Use these same parameters in the IDE and you can debug hadoop. Remember
to make change to the conf files so that hadoop runs in local mode. To
be more specific, you will have to set the program arguemnts, VM
arguments and add an entry in the classpath pointing to the conf folder.
The second method is compilcated. You will have to modify the scripts
and put in some extra params like "-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=<port>" for the
java command. Specify the <port> of you choice in it. On the server
where you are running both the namenode/jobnode there will be a conflict
as the same port would be specified. So you will have to do some
intelligent scripting to take care of this. Once the java processes
start you can attach eclipse debugger to that machine's <port> and set
breakpoints. Till this part you can debug all the things before map
reduce tasks. Mapp reduce tasks run in separate process, for debugging
them you will have to figure out yourself.
The best way is to debug using the first approach (as the above link
says). I think by that approach you can fix any map-reduce related
problems and for other purely distributed kind of problems you can
follow the second approach.
~ Neeraj
-----Original Message-----
From: KrzyCube [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 21, 2007 2:08 AM
To: hadoop-user@lucene.apache.org
Subject: How to Start Hadoop Cluster from source code in Eclipse
Hi,all:
I am using Eclipse to View Hadoop source code , and i want to trace to
see how it works, I code a few code to call the FSClient and when i
call into the RPC object, it can not to be deep more .
So i just want to start cluster from source code , which i am holding
them in Eclipse now.
I browse the start-*.sh , and find that it must start several threads ,
such as namenode , datanode,secondnamenode. i just don't know how to
figure out.
or is there any way to attach my code to a running process , just as the
gdb while we are debug c code
Does any body ever use Eclipse to debug these source code , please give
some tip.
Thanks .
KrzyCube
--
View this message in context:
http://www.nabble.com/How-to-Start-Hadoop-Cluster-from-source-code-in-Ec
lipse-tf3957457.html#a11229322
Sent from the Hadoop Users mailing list archive at Nabble.com.
