Re: Probelms getting Eclipse Hadoop plugin to work.
Erik, Try to add following properties into hadoop-site.xml: property namefs.default.name/name valuehdfs://ip_address:9000/value /property property namemapred.job.tracker/name valuehdfs://ip_address:9001/value /property This way your ports become static. Then use port 9001 for MR, 9000 for HDFS in your properties window. If it still doesn't work, try to write ip address instead of host name as target host. Hope this helps, Rasit 2009/2/18 Erik Holstad erikhols...@gmail.com I'm using Eclipse 3.3.2 and want to view my remote cluster using the Hadoop plugin. Everything shows up and I can see the map/reduce perspective but when trying to connect to a location I get: Error: Call failed on local exception I've set the host to for example xx0, where xx0 is a remote machine accessible from the terminal, and the ports to 50020/50040 for M/R master and DFS master respectively. Is there anything I'm missing to set for remote access to the Hadoop cluster? Regards Erik -- M. Raşit ÖZDAŞ
where is the log when using eclipse plugin
The log message can be found from the stdout log file when i am using standard hadoop command. But when using the eclipse plugin and run from the eclipse, no output mess ,age is capture, btw, i use System.out.println to emit log message. -- http://daily.appspot.com/food/
Re: Persistent completed jobs status not showing in jobtracker UI
The idea behind persistent completed jobs status is that they are still available in case of a jobtracker failure, right? The job history is stored as files under the log directory on the jobtracker node. So they won't be available when the jobtraker node fails. For folks who have enabled this feature, how do you retrieve the presistended completed jobs status? Bill On Wed, Feb 18, 2009 at 10:48 PM, Amareshwari Sriramadasu amar...@yahoo-inc.com wrote: Bill Au wrote: I have enabled persistent completed jobs status and can see them in HDFS. However, they are not listed in the jobtracker's UI after the jobtracker is restarted. I thought that jobtracker will automatically look in HDFS if it does not find a job in its memory cache. What am I missing? How to I retrieve the persistent completed job status? Bill JobTracker web ui doesn't look at persistence storage after a restart. You can access the old jobs from job history. History link is accesible from web ui. -Amareshwari
Re: Probelms getting Eclipse Hadoop plugin to work.
What platform are you running Eclipse on? If Windows, see this thread regarding Cygwin: http://www.mail-archive.com/core-user@hadoop.apache.org/msg07669.html For my case, I've never had to touch any of the plugin's advanced parameters. Usually, setting just the Map/Reduce Master and DFS Master (and associated ports) has been sufficient. Norbert On 2/18/09, Erik Holstad erikhols...@gmail.com wrote: I'm using Eclipse 3.3.2 and want to view my remote cluster using the Hadoop plugin. Everything shows up and I can see the map/reduce perspective but when trying to connect to a location I get: Error: Call failed on local exception I've set the host to for example xx0, where xx0 is a remote machine accessible from the terminal, and the ports to 50020/50040 for M/R master and DFS master respectively. Is there anything I'm missing to set for remote access to the Hadoop cluster? Regards Erik
Re: Probelms getting Eclipse Hadoop plugin to work.
Thanks guys! Running Linux and the remote cluster is also Linux. I have the properties set up like that already on my remote cluster, but not sure where to input this info into Eclipse. And when changing the ports to 9000 and 9001 I get: Error: java.io.IOException: Unknown protocol to job tracker: org.apache.hadoop.dfs.ClientProtocol Regards Erik
Re: Map/Recuce Job done locally?
Hey Philipp! MR jobs are run locally if you just run the java file, to get it running in distributed mode you need to create a job jar and run that like ./bin/hadoop jar ... Regards Erik
Re: Map/Recuce Job done locally?
Hi Erik, thank you, this list is really quick and helpful. I will try that. Is it enough to just create a .jar file containing my Class and how would I start such a job from another program, when I want to track for example execution time? Do I run some kind of system.exec(path/hadoop jar MyClass Param)? Best, Philipp Original-Nachricht Datum: Thu, 19 Feb 2009 13:42:41 -0800 Von: Erik Holstad erikhols...@gmail.com An: core-user@hadoop.apache.org Betreff: Re: Map/Recuce Job done locally? Hey Philipp! MR jobs are run locally if you just run the java file, to get it running in distributed mode you need to create a job jar and run that like ./bin/hadoop jar ... Regards Erik -- Pt! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01
Re: Map/Recuce Job done locally?
Hey Philipp! Not sure about your time tracking thing, probably works, I've just used a bash script to start the jar and then you can do the timing in the script. About how to compile the jars, you need to include the dependencies too, but you will see what you are missing when you run the job. Regards Erik
empty log file...
hi, i am setting up hadoop for the first time on multi-node cluster. right now i have two nodes. the two node cluster consists of two laptops connected via ad-hoc wifi network. they they do not have access to the internet. i formated the datanodes on both machines prior to startup... output form the commands /usr/local/hadoop/bin/start-all.sh, jps (on both machines), and /usr/local/hadoop/bin/stop-all.sh all appear normal. however the file /usr/local/hadoop/logs/hadoop-hadoop-datanode-node1.log (the slave node) is empty. the same file for the master node shows the startup and shutdown events as normal and without error. is it okay that the log file on the slave is empty? zander -- View this message in context: http://www.nabble.com/empty-log-file...-tp22113398p22113398.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
How to use Hadoop API to submit job?
Hi, I used to submit Hadoop job with the utility RunJar.main() on hadoop 0.18. On hadoop 0.19, because the commandLineConfig of JobClient was null, I got a NullPointerException error when RunJar.main() calls GenericOptionsParser to get libJars (0.18 didn't do this call). I also tried the class JobShell to submit job, but it catches all exceptions and sends to stderr so that I cann't handle the exceptions myself. I noticed that if I can call JobClient's setCommandLineConfig method, everything goes easy. But this method has default package accessibility, I cann't see the method out of package org.apache.hadoop.mapred. Any advices on using Java APIs to submit job? Wei
