Re: Hadoop-Git-Eclipse
Hi Harsh
I can actually run the jobs (and browse them on localhost:8088) when
executed from command line. The same program gives me problems from
eclipse. :(
Moreover, I cannot even browse DFS locations from eclipse (indigo). I am
using hadoop-0.20.3-dev-eclipse-plugin.jar and following steps from
shaswat's video to setup mapreduce environment in eclipse. It gives me
Error: IPC server version 7 trying to communicate with client version 3.
I believe it is because of using the old (0.20.3) plugin and I could not
find a newer plugin for latest Hadoop versions. Is there a workaround?
I think (not sure) this is the reason that my jobs launched from eclipse
are getting executed by the LocalJobRunner (inspite of specifying details
of classpath in run configurations in eclipse as you suggested).
One more thing, does maven automatically assume jre path even if i specify
JAVA_HOME to my jdk? I had to comment out the following lines and create
the artifact for jdk.tools in ~/.m2/repository manually as eclipse kept
converting the jdk path to jre.
!-- scopesystem/scope
systemPath${java.home}/../lib/tools.jar/systemPath--
The 'effective pom' view for the hadoop-project/pom.xml showed the jre path
instead of jdk.
I saw your old post for such problems at
http://www.harshj.com/2010/07/18/making-the-eclipse-plugin-work-for-hadoop/
but I believe the patch is already committed in the trunk now. Any idea why
eclipse is troubling me like this? :(
Regards,
Prajakta
On Fri, Jun 15, 2012 at 4:43 PM, Harsh J ha...@cloudera.com wrote:
Hey,
The presence of deprecated properties should not really hamper your
submissions. The config for YARN (pseudo or fully distributed) I
usually follow are all listed at the links I provided earlier. When
running programs from within the IDE, I place the directory they are
contained in onto my launcher's (Called 'Run Configuration' in
eclipse, for instance) classpath and they are picked up by my program.
On Fri, Jun 15, 2012 at 12:30 PM, Prajakta Kalmegh pkalm...@gmail.com
wrote:
Hi Harsh
You were right - it was executed via the LocalJobRunner. What settings
do I
need to change to ensure it goes through RM/NM? I did specify to use the
yarn framework in the mapred-site.xml. The eclipse console even shows to
check http://localhost:8080 to track job progress.
I will again cross-check all the parameters today with the ones specified
in the urls you gave me. By the way, I found this link
http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
which gives a list of deprecated properties. I have, in fact, specified a
lot of old properties in my core-site.xml, hdfs-site.xml, mapred-site.xml
files. Will that cause a problem?
Regards,
Prajakta
On Wed, Jun 13, 2012 at 8:18 PM, Harsh J ha...@cloudera.com wrote:
Good to know your progress Prajakta!
Did your submission surely go via the RM/NM or did it execute via the
LocalJobRunner (logs show this classname)?
You would ideally want to set the config mapreduce.framework.name to
value yarn (in either config object before you use it, or in local
mapred-site.xml), for it to use the YARN framework. A set of general
configs for YARN deployment may be found at http://bit.ly/M2Eobz or at
http://bit.ly/LW3Var.
Does this help?
On Wed, Jun 13, 2012 at 6:39 PM, Prajakta Kalmegh pkalm...@gmail.com
wrote:
Hi Harsh
Appreciate the response. I was able to configure and implement basic
JUnits
within eclipse and get some code running. Still getting familiar with
the
new YARN and federation architecture.
I was, however, not able to check the MR jobs submitted within eclipse
for
a sample WordCount program on the
http://localhost:8088/http://localhost:8080/page. I am starting my
namenode/datanode/resourcemanager/nodemanager/historyserver as
instructed
on the wiki page. And then executing JUnit tests from eclipse.
I believe a single MR job will be submitted as a single application in
the
new framework, right? The eclipse console shows a successful execution
(the
details are pretty neat). However, the webpage shows 'No applications
submitted'. Do I have to tweak with any config properties to get this
done?
Please let me know.
Regards,
Prajakta
On Wed, Jun 13, 2012 at 6:09 PM, Harsh J ha...@cloudera.com wrote:
Hi Prajakta,
I have Eclipse setup with M2E plugins. And once thats done, I merely
clone a repo and import projects in via M2E's Import existing maven
projects feature. This seems to work just fine for apache/hadoop's
trunk.
On Thu, Jun 7, 2012 at 5:18 PM, Prajakta Kalmegh
prkal...@in.ibm.com
wrote:
Hi
I have done MapReduce programming using Eclipse before but now I
need
to
learn the Hadoop code internals for one of my projects.
I have forked Hadoop from github
Re: Hadoop-Git-Eclipse
Hi Shant Sorry for the delay in reply (I was not well). Did you manage to install using Shaswat's video? The video gives good demo for setting up basic Mapreduce programming environment. If you need steps for cloning and setting up Hadoop in Eclipse, please follow this link. http://wiki.apache.org/hadoop/EclipseEnvironment For setting up properties, follow this link http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/SingleCluster.html Please ensure to specify appropriate values for host:port combinations. Please let me know if this helps. Regards, Prajakta On Thu, Jun 14, 2012 at 12:32 PM, shashwat shriparv dwivedishash...@gmail.com wrote: Follow this video...: http://www.youtube.com/watch?v=TavehEdfNDk On Thu, Jun 14, 2012 at 10:14 AM, shant. shantlingayya.sw...@gmail.comwrote: Hi Prajakta, can u please tell me the steps to do MapReduce programming using Eclipse . I really appreciate.. ur help. i m new to hadoop . want to learn thanks shant On Thu, Jun 7, 2012 at 6:48 AM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github ( https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Life is Just a dream on the way to death... Regards Shantlingayya -- ∞ Shashwat Shriparv
Re: Hadoop-Git-Eclipse
Hi Harsh You were right - it was executed via the LocalJobRunner. What settings do I need to change to ensure it goes through RM/NM? I did specify to use the yarn framework in the mapred-site.xml. The eclipse console even shows to check http://localhost:8080 to track job progress. I will again cross-check all the parameters today with the ones specified in the urls you gave me. By the way, I found this link http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-project-dist/hadoop-common/DeprecatedProperties.html which gives a list of deprecated properties. I have, in fact, specified a lot of old properties in my core-site.xml, hdfs-site.xml, mapred-site.xml files. Will that cause a problem? Regards, Prajakta On Wed, Jun 13, 2012 at 8:18 PM, Harsh J ha...@cloudera.com wrote: Good to know your progress Prajakta! Did your submission surely go via the RM/NM or did it execute via the LocalJobRunner (logs show this classname)? You would ideally want to set the config mapreduce.framework.name to value yarn (in either config object before you use it, or in local mapred-site.xml), for it to use the YARN framework. A set of general configs for YARN deployment may be found at http://bit.ly/M2Eobz or at http://bit.ly/LW3Var. Does this help? On Wed, Jun 13, 2012 at 6:39 PM, Prajakta Kalmegh pkalm...@gmail.com wrote: Hi Harsh Appreciate the response. I was able to configure and implement basic JUnits within eclipse and get some code running. Still getting familiar with the new YARN and federation architecture. I was, however, not able to check the MR jobs submitted within eclipse for a sample WordCount program on the http://localhost:8088/http://localhost:8080/page. I am starting my namenode/datanode/resourcemanager/nodemanager/historyserver as instructed on the wiki page. And then executing JUnit tests from eclipse. I believe a single MR job will be submitted as a single application in the new framework, right? The eclipse console shows a successful execution (the details are pretty neat). However, the webpage shows 'No applications submitted'. Do I have to tweak with any config properties to get this done? Please let me know. Regards, Prajakta On Wed, Jun 13, 2012 at 6:09 PM, Harsh J ha...@cloudera.com wrote: Hi Prajakta, I have Eclipse setup with M2E plugins. And once thats done, I merely clone a repo and import projects in via M2E's Import existing maven projects feature. This seems to work just fine for apache/hadoop's trunk. On Thu, Jun 7, 2012 at 5:18 PM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github ( https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Harsh J -- Harsh J
Re: Hadoop-Git-Eclipse
Hey, The presence of deprecated properties should not really hamper your submissions. The config for YARN (pseudo or fully distributed) I usually follow are all listed at the links I provided earlier. When running programs from within the IDE, I place the directory they are contained in onto my launcher's (Called 'Run Configuration' in eclipse, for instance) classpath and they are picked up by my program. On Fri, Jun 15, 2012 at 12:30 PM, Prajakta Kalmegh pkalm...@gmail.com wrote: Hi Harsh You were right - it was executed via the LocalJobRunner. What settings do I need to change to ensure it goes through RM/NM? I did specify to use the yarn framework in the mapred-site.xml. The eclipse console even shows to check http://localhost:8080 to track job progress. I will again cross-check all the parameters today with the ones specified in the urls you gave me. By the way, I found this link http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-project-dist/hadoop-common/DeprecatedProperties.html which gives a list of deprecated properties. I have, in fact, specified a lot of old properties in my core-site.xml, hdfs-site.xml, mapred-site.xml files. Will that cause a problem? Regards, Prajakta On Wed, Jun 13, 2012 at 8:18 PM, Harsh J ha...@cloudera.com wrote: Good to know your progress Prajakta! Did your submission surely go via the RM/NM or did it execute via the LocalJobRunner (logs show this classname)? You would ideally want to set the config mapreduce.framework.name to value yarn (in either config object before you use it, or in local mapred-site.xml), for it to use the YARN framework. A set of general configs for YARN deployment may be found at http://bit.ly/M2Eobz or at http://bit.ly/LW3Var. Does this help? On Wed, Jun 13, 2012 at 6:39 PM, Prajakta Kalmegh pkalm...@gmail.com wrote: Hi Harsh Appreciate the response. I was able to configure and implement basic JUnits within eclipse and get some code running. Still getting familiar with the new YARN and federation architecture. I was, however, not able to check the MR jobs submitted within eclipse for a sample WordCount program on the http://localhost:8088/http://localhost:8080/page. I am starting my namenode/datanode/resourcemanager/nodemanager/historyserver as instructed on the wiki page. And then executing JUnit tests from eclipse. I believe a single MR job will be submitted as a single application in the new framework, right? The eclipse console shows a successful execution (the details are pretty neat). However, the webpage shows 'No applications submitted'. Do I have to tweak with any config properties to get this done? Please let me know. Regards, Prajakta On Wed, Jun 13, 2012 at 6:09 PM, Harsh J ha...@cloudera.com wrote: Hi Prajakta, I have Eclipse setup with M2E plugins. And once thats done, I merely clone a repo and import projects in via M2E's Import existing maven projects feature. This seems to work just fine for apache/hadoop's trunk. On Thu, Jun 7, 2012 at 5:18 PM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github ( https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Harsh J -- Harsh J -- Harsh J
Re: Hadoop-Git-Eclipse
Follow this video...: http://www.youtube.com/watch?v=TavehEdfNDk On Thu, Jun 14, 2012 at 10:14 AM, shant. shantlingayya.sw...@gmail.comwrote: Hi Prajakta, can u please tell me the steps to do MapReduce programming using Eclipse . I really appreciate.. ur help. i m new to hadoop . want to learn thanks shant On Thu, Jun 7, 2012 at 6:48 AM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github ( https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Life is Just a dream on the way to death... Regards Shantlingayya -- ∞ Shashwat Shriparv
Re: Hadoop-Git-Eclipse
Hi Prajakta, I have Eclipse setup with M2E plugins. And once thats done, I merely clone a repo and import projects in via M2E's Import existing maven projects feature. This seems to work just fine for apache/hadoop's trunk. On Thu, Jun 7, 2012 at 5:18 PM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github (https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Harsh J
Re: Hadoop-Git-Eclipse
Hi Harsh Appreciate the response. I was able to configure and implement basic JUnits within eclipse and get some code running. Still getting familiar with the new YARN and federation architecture. I was, however, not able to check the MR jobs submitted within eclipse for a sample WordCount program on the http://localhost:8088/http://localhost:8080/page. I am starting my namenode/datanode/resourcemanager/nodemanager/historyserver as instructed on the wiki page. And then executing JUnit tests from eclipse. I believe a single MR job will be submitted as a single application in the new framework, right? The eclipse console shows a successful execution (the details are pretty neat). However, the webpage shows 'No applications submitted'. Do I have to tweak with any config properties to get this done? Please let me know. Regards, Prajakta On Wed, Jun 13, 2012 at 6:09 PM, Harsh J ha...@cloudera.com wrote: Hi Prajakta, I have Eclipse setup with M2E plugins. And once thats done, I merely clone a repo and import projects in via M2E's Import existing maven projects feature. This seems to work just fine for apache/hadoop's trunk. On Thu, Jun 7, 2012 at 5:18 PM, Prajakta Kalmegh prkal...@in.ibm.com wrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github ( https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Harsh J
Re: Hadoop-Git-Eclipse
Hi Prajakta, can u please tell me the steps to do MapReduce programming using Eclipse . I really appreciate.. ur help. i m new to hadoop . want to learn thanks shant On Thu, Jun 7, 2012 at 6:48 AM, Prajakta Kalmegh prkal...@in.ibm.comwrote: Hi I have done MapReduce programming using Eclipse before but now I need to learn the Hadoop code internals for one of my projects. I have forked Hadoop from github (https://github.com/apache/hadoop-common ) and need to configure it to work with Eclipse. All the links I could find list steps for earlier versions of Hadoop. I am right now following instructions given in these links: - http://wiki.apache.org/hadoop/GitAndHadoop - http://wiki.apache.org/hadoop/EclipseEnvironment - http://wiki.apache.org/hadoop/HowToContribute Can someone please give me a link to the steps to be followed for getting Hadoop (latest from trunk) started in Eclipse? I need to be able to commit changes to my forked repository on github. Thanks in advance. Regards, Prajakta -- Life is Just a dream on the way to death... Regards Shantlingayya
Re: Hadoop on Eclipse
Sorry, my bad. It's the 'FAILED DOWNLOADS' error that goes away the second
time. I don't see the 'UNRESOLVED DEPENDENCIES' error.
-Eric
From: Eric Payne eric.payne1...@yahoo.com
To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org
Sent: Friday, October 7, 2011 9:09 AM
Subject: Fw: Hadoop on Eclipse
I have been working in the 0.23 branch. Supposedly, there is not a lot of
difference between 0.23 and 0.24.
I get the 'UNRESOLVED DEPENDENCIES' error sometimes, but then on the second
try, it goes away. Go figure.
I'm an eclipse newbie, but with a lot of help, I was able to get it going on
the 0.23 branch on Linux. Here's what I did:
-- from command line pull down the SVN repo:
export SVNROOT=http://svn.apache.org/repos/asf
svn co $SVNROOT/hadoop/common/branches/branch-0.23
-- just to be safe, I remove my .m2 and .ivy directories from my home
directory.
-- in 'branch-0.23' directory, use maven to build:
mvn clean install -Pdist -Dtar -DskipTests
-- in 'branch-0.23/hadoop-mapreduce-project' directory, build again:
mvn clean install assembly:assembly -DskipTests
-- In same directory, build using ant:
ant -Dresolvers=internal -Dcompile.c+=true -Dcompile.native=true
-Dforrest.home=$FORREST_HOME create-c++-configure tar
-- At this point, I do get the 'UNRESOLVED DEPENDENCIES' error. If I do, I
just try it again and it works.
-- start eclipse
-- I use the following maven plugin: http://m2eclipse.sonatype.org/sites/m2e
-- I go to the project explorer and choose to import a maven project and
select the 'branch-0.23' directory. It works for a while (10 minutes or so),
and then you can edit, navigate, etc.
NOTE: I build from the command line, not from w/i eclipse.
Hope that helps.
-Eric
- Forwarded Message -
From: Eric Payne er...@yahoo-inc.com
To: eric.payne1...@yahoo.com eric.payne1...@yahoo.com
Sent: Friday, October 7, 2011 8:20 AM
Subject: RE: Hadoop on Eclipse
-Original Message-
From: Tim Broberg [mailto:tbrob...@yahoo.com]
Sent: Thursday, October 06, 2011 11:00 PM
To: common-dev@hadoop.apache.org
Subject: Re: Hadoop on Eclipse
After still more puttering, I gave up and just logged this in jira.
https://issues.apache.org/jira/browse/HADOOP-7726
- Tim.
From: Prasanth J buckeye.prasa...@gmail.com
To: common-dev@hadoop.apache.org; Tim Broberg tbrob...@yahoo.com
Sent: Thursday, October 6, 2011 11:02 AM
Subject: Re: Hadoop on Eclipse
Am also facing the exact issue. I think the documentation for setting up
hadoop on eclipse is for older versions of hadoop. Now i am using hadoop
0.24.0 snapshot and with reference to that the documentation seems outdated.
Any help on this regard will be very useful.
On Wed, Oct 5, 2011 at 10:34 PM, Tim Broberg tbrob...@yahoo.com wrote:
I am having issues similar to Alexandre's.
The suggestion to run mvn install -DskipTests -P-cbuild clears up the
initial problems, but problems ensue...
1a - There is no hdfs subdirectory to change into.
1b - If I instead cd to hadoop-hdfs-project and proceed, I get...
[tbroberg@f15 hadoop-common]$ cd hadoop-hdfs-project/
[tbroberg@f15 hadoop-hdfs-project]$ ant compile eclipse
Buildfile: build.xml does not exist!
Build failed
[tbroberg@f15 hadoop-hdfs-project]$
2 - If I skip hdfs altogether and go to hadoop-mapreduce-project problems
much like the original issues crop up:
Output is below, debug output is attached.
Looks like maybe he's failing to expand ${yarn.version}?
Anybody got any better suggestions than go out to dinner?
Thanks,
- Tim.
[tbroberg@f15 hadoop-mapreduce-project]$ ant compile eclipse
Buildfile:
/home/tbroberg/workspace/git/hadoop-common/hadoop-mapreduce-project/build.xml
clover.setup:
clover.info:
[echo]
[echo] Clover not found. Code coverage reports disabled.
[echo]
clover:
ivy-download:
[get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To:
/home/tbroberg/workspace/git/hadoop-common/hadoop-mapreduce-project/ivy/ivy-2.2.0.jar
[get] Not modified - so not downloaded
ivy-init-dirs:
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
[ivy:configure] :: Ivy 2.2.0 - 20100923230623 ::
http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file =
/home/tbroberg/workspace/git/hadoop-common/hadoop-mapreduce-project/ivy/ivysettings.xml
ivy-resolve-common:
[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] WARNINGS
[ivy:resolve] io problem while parsing ivy file:
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-yarn-server-common/0.24.0-SNAPSHOT/hadoop-yarn-server-common-0.24.0-20110914.035045-1.pom:
Impossible to load parent for
file:/home/tbroberg/.ivy2/cache/org.apache.hadoop/hadoop-yarn-server-common/ivy-0.24.0-SNAPSHOT.xml.original.
Parent=org.apache.hadoop#hadoop-yarn-server;${yarn.version
Re: Hadoop on Eclipse
Try to do a mvn install -DskipTests -P-cbuild before mvn eclipse:eclipse..
On Mon, Oct 3, 2011 at 10:08 PM, Alexandre de Assis Bento Lima
as...@cos.ufrj.br wrote:
Hi,
I'm following the steps suggested by the wiki page:
EclipseEnvironment - Hadoop Wiki
(http://wiki.apache.org/hadoop/EclipseEnvironment)
in order to have an Eclipse project for Hadoop. I'm exactly trying to execute
the
following command:
mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
But I always get the following error:
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 12.657s
[INFO] Finished at: Tue Oct 04 02:04:11 BRT 2011
[INFO] Final Memory: 31M/282M
[INFO]
[ERROR] Failed to execute goal on project hadoop-yarn-common: Could not
resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-common:jar:0.24.0-SNAPSHOT:
Failed to collect dependencies for [log4j:log4j:jar:1.2.12 (compile),
org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT (compile),
org.apache.avro:avro:jar:1.5.3 (compile),
com.google.protobuf:protobuf-java:jar:2.4.0a
(compile), org.apache.hadoop:hadoop-common:jar:0.24.0-SNAPSHOT (compile),
org.apache.hadoop:hadoop-annotations:jar:0.24.0-SNAPSHOT (compile),
junit:junit:jar:4.8.2 (compile), org.mockito:mockito-all:jar:1.8.5 (test),
org.apache.hadoop:hadoop-common:jar:tests:0.24.0-SNAPSHOT (test),
org.apache.hadoop:hadoop-hdfs:jar:0.24.0-SNAPSHOT (runtime),
com.google.inject.extensions:guice-servlet:jar:2.0 (compile),
org.jboss.netty:netty:jar:3.2.3.Final (compile), org.slf4j:slf4j-api:jar:1.6.1
(compile), org.slf4j:slf4j-log4j12:jar:1.6.1 (compile)]: Failed to read
artifact
descriptor for org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT: Failure
to find
org.apache.hadoop:hadoop-yarn:pom:${yarn.version} in
http://repository.jboss.org/nexus/content/groups/public/ was cached in the
local
repository, resolution will not be reattempted until the update interval of
repository.jboss.org has elapsed or updates are forced - [Help 1]
Then, I try
mvn -X eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
and get
[DEBUG]
===
[INFO]
[INFO] maven-eclipse-plugin:2.8:eclipse (default-cli) @
hadoop-yarn-common
[DEBUG] Could not find metadata
org.apache.hadoop:hadoop-yarn-api:0.24.0-SNAPSHOT/maven-metadata.xml in local
(/home/assis/.m2/repository)
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for
http://repository.apache.org/snapshots
Downloading:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/maven-metadata.xml
Downloaded:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/maven-metadata.xml
(785 B at 1.0 KB/sec)
[DEBUG] Reading resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/resolver-status.properties
[DEBUG] Writing resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/resolver-status.properties
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Project POM . SUCCESS [1.626s]
[INFO] Apache Hadoop Annotations . SUCCESS [0.289s]
[INFO] Apache Hadoop Project Dist POM SUCCESS [0.255s]
[INFO] Apache Hadoop Assemblies .. SUCCESS [0.207s]
[INFO] Apache Hadoop Auth SUCCESS [0.557s]
[INFO] Apache Hadoop Auth Examples ... SUCCESS [0.384s]
[INFO] Apache Hadoop Common .. SUCCESS [1.912s]
[INFO] Apache Hadoop Common Project .. SUCCESS [0.083s]
[INFO] Apache Hadoop HDFS SUCCESS [3.209s]
[INFO] Apache Hadoop HDFS Project SUCCESS [0.160s]
[INFO] hadoop-yarn-api ... SUCCESS [2.273s]
[INFO] hadoop-yarn-common FAILURE [0.926s]
I check the local repository and realize that there's no maven-metadata.xml
file there.
So, I download it myself, try again, and the same things happen.
Can anyone help me, please? I'm using Ubuntu 11.04 with jdk1.7.0, maven 3.0.3
and
protobuf 2.4.1.
Thanks in advance!
Alexandre A. B. Lima.
---
COPPE / Computer Science
Federal University of Rio de Janeiro
Rio de Janeiro, Brazil.
+55-21-2562-8690
Re: Hadoop on Eclipse
Thanks, Luke! But I'm still having problems.
After importing all projects into Eclipse, I got 1007 erros. The first one:
Project 'MapReduceTools' is missing required library: 'lib/hadoop-core.jar'
Then, many of the following kind:
Unbound classpath variable:
'M2_REPO/aopalliance/aopalliance/1.0/aopalliance-1.0.jar'
in project 'hadoop-mapreduce-client-app'
Is there any place where I can have a complete step list about how to develop
Hadoop?
The wiki seems no to be up-to-date.
Can anyone help?
Thanks in advance!
Alexandre.
On Tue, 4 Oct 2011 03:05:02 -0700, Luke Lu wrote
Try to do a mvn install -DskipTests -P-cbuild before mvn eclipse:eclipse..
On Mon, Oct 3, 2011 at 10:08 PM, Alexandre de Assis Bento Lima
as...@cos.ufrj.br wrote:
Hi,
I'm following the steps suggested by the wiki page:
EclipseEnvironment - Hadoop Wiki
(http://wiki.apache.org/hadoop/EclipseEnvironment)
in order to have an Eclipse project for Hadoop. I'm exactly trying to
execute the
following command:
mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
But I always get the following error:
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 12.657s
[INFO] Finished at: Tue Oct 04 02:04:11 BRT 2011
[INFO] Final Memory: 31M/282M
[INFO]
[ERROR] Failed to execute goal on project hadoop-yarn-common: Could not
resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-common:jar:0.24.0-SNAPSHOT:
Failed to collect dependencies for [log4j:log4j:jar:1.2.12 (compile),
org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT (compile),
org.apache.avro:avro:jar:1.5.3 (compile),
com.google.protobuf:protobuf-java:jar:2.4.0a
(compile), org.apache.hadoop:hadoop-common:jar:0.24.0-SNAPSHOT (compile),
org.apache.hadoop:hadoop-annotations:jar:0.24.0-SNAPSHOT (compile),
junit:junit:jar:4.8.2 (compile), org.mockito:mockito-all:jar:1.8.5 (test),
org.apache.hadoop:hadoop-common:jar:tests:0.24.0-SNAPSHOT (test),
org.apache.hadoop:hadoop-hdfs:jar:0.24.0-SNAPSHOT (runtime),
com.google.inject.extensions:guice-servlet:jar:2.0 (compile),
org.jboss.netty:netty:jar:3.2.3.Final (compile),
org.slf4j:slf4j-api:jar:1.6.1
(compile), org.slf4j:slf4j-log4j12:jar:1.6.1 (compile)]: Failed to read
artifact
descriptor for org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT:
Failure to find
org.apache.hadoop:hadoop-yarn:pom:${yarn.version} in
http://repository.jboss.org/nexus/content/groups/public/ was cached in the
local
repository, resolution will not be reattempted until the update interval of
repository.jboss.org has elapsed or updates are forced - [Help 1]
Then, I try
mvn -X eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
and get
[DEBUG]
===
[INFO]
[INFO] maven-eclipse-plugin:2.8:eclipse (default-cli) @
hadoop-yarn-common
[DEBUG] Could not find metadata
org.apache.hadoop:hadoop-yarn-api:0.24.0-SNAPSHOT/maven-metadata.xml in
local
(/home/assis/.m2/repository)
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for
http://repository.apache.org/snapshots
Downloading:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/maven-metadata.xml
Downloaded:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/maven-metadata.xml
(785 B at 1.0 KB/sec)
[DEBUG] Reading resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/resolver-status.properties
[DEBUG] Writing resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT/resolver-status.properties
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Project POM . SUCCESS [1.626s]
[INFO] Apache Hadoop Annotations . SUCCESS [0.289s]
[INFO] Apache Hadoop Project Dist POM SUCCESS [0.255s]
[INFO] Apache Hadoop Assemblies .. SUCCESS [0.207s]
[INFO] Apache Hadoop Auth SUCCESS [0.557s]
[INFO] Apache Hadoop Auth Examples ... SUCCESS [0.384s]
[INFO] Apache Hadoop Common .. SUCCESS [1.912s]
[INFO] Apache Hadoop Common Project .. SUCCESS [0.083s]
[INFO] Apache Hadoop HDFS SUCCESS [3.209s]
[INFO] Apache Hadoop HDFS Project SUCCESS [0.160s]
[INFO] hadoop-yarn-api ... SUCCESS [2.273s]
[INFO]
RE: Hadoop on Eclipse
Hi Alexandre,
You need to set the M2_REPO variable in your eclipse to the maven repository
which is present in your local system. By default it is
{userhome}/.m2/repository
Regards,
Ravi Teja
-Original Message-
From: Alexandre de Assis Bento Lima [mailto:as...@cos.ufrj.br]
Sent: Wednesday, October 05, 2011 3:55 AM
To: common-dev@hadoop.apache.org
Subject: Re: Hadoop on Eclipse
Thanks, Luke! But I'm still having problems.
After importing all projects into Eclipse, I got 1007 erros. The first one:
Project 'MapReduceTools' is missing required library:
'lib/hadoop-core.jar'
Then, many of the following kind:
Unbound classpath variable:
'M2_REPO/aopalliance/aopalliance/1.0/aopalliance-1.0.jar'
in project 'hadoop-mapreduce-client-app'
Is there any place where I can have a complete step list about how to
develop Hadoop?
The wiki seems no to be up-to-date.
Can anyone help?
Thanks in advance!
Alexandre.
On Tue, 4 Oct 2011 03:05:02 -0700, Luke Lu wrote
Try to do a mvn install -DskipTests -P-cbuild before mvn eclipse:eclipse..
On Mon, Oct 3, 2011 at 10:08 PM, Alexandre de Assis Bento Lima
as...@cos.ufrj.br wrote:
Hi,
I'm following the steps suggested by the wiki page:
EclipseEnvironment - Hadoop Wiki
(http://wiki.apache.org/hadoop/EclipseEnvironment)
in order to have an Eclipse project for Hadoop. I'm exactly trying to
execute the
following command:
mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
But I always get the following error:
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 12.657s
[INFO] Finished at: Tue Oct 04 02:04:11 BRT 2011
[INFO] Final Memory: 31M/282M
[INFO]
[ERROR] Failed to execute goal on project hadoop-yarn-common: Could not
resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-common:jar:0.24.0-SNAPSHOT:
Failed to collect dependencies for [log4j:log4j:jar:1.2.12 (compile),
org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT (compile),
org.apache.avro:avro:jar:1.5.3 (compile),
com.google.protobuf:protobuf-java:jar:2.4.0a
(compile), org.apache.hadoop:hadoop-common:jar:0.24.0-SNAPSHOT
(compile),
org.apache.hadoop:hadoop-annotations:jar:0.24.0-SNAPSHOT (compile),
junit:junit:jar:4.8.2 (compile), org.mockito:mockito-all:jar:1.8.5
(test),
org.apache.hadoop:hadoop-common:jar:tests:0.24.0-SNAPSHOT (test),
org.apache.hadoop:hadoop-hdfs:jar:0.24.0-SNAPSHOT (runtime),
com.google.inject.extensions:guice-servlet:jar:2.0 (compile),
org.jboss.netty:netty:jar:3.2.3.Final (compile),
org.slf4j:slf4j-api:jar:1.6.1
(compile), org.slf4j:slf4j-log4j12:jar:1.6.1 (compile)]: Failed to read
artifact
descriptor for org.apache.hadoop:hadoop-yarn-api:jar:0.24.0-SNAPSHOT:
Failure to find
org.apache.hadoop:hadoop-yarn:pom:${yarn.version} in
http://repository.jboss.org/nexus/content/groups/public/ was cached in
the local
repository, resolution will not be reattempted until the update interval
of
repository.jboss.org has elapsed or updates are forced - [Help 1]
Then, I try
mvn -X eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
and get
[DEBUG]
===
[INFO]
[INFO] maven-eclipse-plugin:2.8:eclipse (default-cli) @
hadoop-yarn-common
[DEBUG] Could not find metadata
org.apache.hadoop:hadoop-yarn-api:0.24.0-SNAPSHOT/maven-metadata.xml in
local
(/home/assis/.m2/repository)
[DEBUG] Using connector WagonRepositoryConnector with priority 0 for
http://repository.apache.org/snapshots
Downloading:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.2
4.0-SNAPSHOT/maven-metadata.xml
Downloaded:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-yarn-api/0.2
4.0-SNAPSHOT/maven-metadata.xml
(785 B at 1.0 KB/sec)
[DEBUG] Reading resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT
/resolver-status.properties
[DEBUG] Writing resolution tracking file
/home/assis/.m2/repository/org/apache/hadoop/hadoop-yarn-api/0.24.0-SNAPSHOT
/resolver-status.properties
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Project POM . SUCCESS
[1.626s]
[INFO] Apache Hadoop Annotations . SUCCESS
[0.289s]
[INFO] Apache Hadoop Project Dist POM SUCCESS
[0.255s]
[INFO] Apache Hadoop Assemblies .. SUCCESS
[0.207s]
[INFO] Apache Hadoop Auth SUCCESS
[0.557s]
[INFO] Apache Hadoop Auth Examples ... SUCCESS
[0.384s]
[INFO] Apache Hadoop Common
