Re: Error: INFO ipc.Client: Retrying connect to server: /192.168.100.11:8020. Already tried 0 time(s).
This just tells that the service which you are trying to conned is not running on that address or port no. just try netstat -nlrt | grep 8020 and check if it showing something. what i can find out from the error that the server is not running or if it running then its running on some other port -- View this message in context: http://hadoop.6.n7.nabble.com/Error-INFO-ipc-Client-Retrying-connect-to-server-192-168-100-11-8020-Already-tried-0-time-s-tp11885p66957.html Sent from the common-user mailing list archive at Nabble.com.
Re: Error:Hdfs Client for hadoop using native java api
there may be a problem with hostname resolution check if it is forward and
backword resolvable...and if you have any ip address anywhere please
replace it with hostname and the try..
On Sun, Jul 22, 2012 at 9:56 PM, Minh Duc Nguyen wrote:
> As Shaswat mentioned previously, you're problem may be related to your
> configuration.
>
> Is core-site.xml on your classpath? For example, what is the value for
> conf.get("fs.default.name")?
>
> Alternatively, you can set this property directly in your code:
>
> conf.set("fs.default.name", "hdfs://hadoop1.devqa.local:8020");
>
> HTH,
> Minh
>
> On Thu, Jul 19, 2012 at 1:18 PM, Sandeep Reddy P <
> sandeepreddy.3...@gmail.com> wrote:
>
> > Hi Shashwat,
> > Here is the snippet of code which is throwing error
> > Path phdfs = new Path(
> > "hdfs://hadoop1.devqa.local:8020/user/hdfs/java/");
> > java.lang.IllegalArgumentException: Wrong FS:
> > hdfs://hadoop1.devqa.local:8020/user/hdfs/java, expected: file:///
> > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
> > at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
> > at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:404)
> > at
> >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:797)
> > at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:349)
> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:205)
> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
> > at
> >
> >
> org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55)
> > at
> > org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1292)
> > at Hdfs.main(Hdfs.java:18)
> >
> > On Thu, Jul 19, 2012 at 12:02 PM, shashwat shriparv <
> > dwivedishash...@gmail.com> wrote:
> >
> > > Can you provide the code(some part of it if you don't want to throw the
> > > full code here) and let us know which part of your code is throwing
> this
> > > error.
> > >
> > > Regards
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> > >
> > >
> > > On Thu, Jul 19, 2012 at 6:46 PM, Sandeep Reddy P <
> > > sandeepreddy.3...@gmail.com> wrote:
> > >
> > > > Hi John,
> > > > We have applications in windows. So our dev's need to connect to HDFS
> > > from
> > > > eclipse installed in windows. I'm trying to put data from
>
> > to
> > > > using java code from windows.
> > > >
> > > > On Thu, Jul 19, 2012 at 5:41 AM, John Hancock <
> jhancock1...@gmail.com
> > > > >wrote:
> > > >
> > > > > Sandeep,
> > > > >
> > > > > I don't understand your situation completely, but why not just use
> > > > > bin/hadoop dfs -copyFromLocal ?
> > > > >
> > > > > -John
> > > > >
> > > > > On Wed, Jul 18, 2012 at 11:33 AM, Sandeep Reddy P <
> > > > > sandeepreddy.3...@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > > I'm trying to load data into hdfs from local linux file system
> > using
> > > > java
> > > > > > code from a windows machine.But i'm getting the error
> > > > > >
> > > > > > java.lang.IllegalArgumentException: Wrong FS:
> > > > > > hdfs://hadoop1.devqa.local:8020/user/hdfs/java, expected:
> file:///
> > > > > > at
> > org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:404)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> > > > > > at
> org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:797)
> > > > > > at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:349)
> > > > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:205)
> > > > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55)
> > > > > > at
> > > > > >
> > > org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1292)
> > > > > > at Hdfs.main(Hdfs.java:18)
> > > > > > File not found
> > > > > >
> > > > > > Can any one please help me with the issue.
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > sandeep
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > sandeep
> > > >
> > >
> > >
> > >
> > > --
> > >
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> >
> >
> >
> > --
> > Thanks,
> > sandeep
> >
>
--
∞
Shashwat Shriparv
Re: Error:Hdfs Client for hadoop using native java api
Or try something like this Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); HFile.Writer hwriter = new HFileWriterV2(conf, new CacheConfig(conf), fs, new Path(fs.getWorkingDirectory() + "/foo")); On Thu, Jul 19, 2012 at 9:35 PM, shashwat shriparv < dwivedishash...@gmail.com> wrote: > By the way you have these two options > > >1. > >Put the configuration files in the classpath, so that the code picks >it. >2. > >Use > Configuration.set()<http://hadoop.apache.org/common/docs/r0.21.0/api/org/apache/hadoop/conf/Configuration.html#set%28java.lang.String,%20java.lang.String%29>to > set the required parameters in the code. > > because the configuration may be directing to local file whice code is nt > abl to find > > > On Thu, Jul 19, 2012 at 9:32 PM, shashwat shriparv < > dwivedishash...@gmail.com> wrote: > >> Can you provide the code(some part of it if you don't want to throw the >> full code here) and let us know which part of your code is throwing this >> error. >> >> Regards >> >> ∞ >> Shashwat Shriparv >> >> >> >> >> On Thu, Jul 19, 2012 at 6:46 PM, Sandeep Reddy P < >> sandeepreddy.3...@gmail.com> wrote: >> >>> Hi John, >>> We have applications in windows. So our dev's need to connect to HDFS >>> from >>> eclipse installed in windows. I'm trying to put data from to >>> using java code from windows. >>> >>> On Thu, Jul 19, 2012 at 5:41 AM, John Hancock >> >wrote: >>> >>> > Sandeep, >>> > >>> > I don't understand your situation completely, but why not just use >>> > bin/hadoop dfs -copyFromLocal ? >>> > >>> > -John >>> > >>> > On Wed, Jul 18, 2012 at 11:33 AM, Sandeep Reddy P < >>> > sandeepreddy.3...@gmail.com> wrote: >>> > >>> > > Hi, >>> > > I'm trying to load data into hdfs from local linux file system using >>> java >>> > > code from a windows machine.But i'm getting the error >>> > > >>> > > java.lang.IllegalArgumentException: Wrong FS: >>> > > hdfs://hadoop1.devqa.local:8020/user/hdfs/java, expected: file:/// >>> > > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410) >>> > > at >>> > > >>> > > >>> > >>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56) >>> > > at >>> > > >>> > > >>> > >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:404) >>> > > at >>> > > >>> > > >>> > >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >>> > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:797) >>> > > at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:349) >>> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:205) >>> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) >>> > > at >>> > > >>> > > >>> > >>> org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55) >>> > > at >>> > > >>> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1292) >>> > > at Hdfs.main(Hdfs.java:18) >>> > > File not found >>> > > >>> > > Can any one please help me with the issue. >>> > > >>> > > -- >>> > > Thanks, >>> > > sandeep >>> > > >>> > >>> >>> >>> >>> -- >>> Thanks, >>> sandeep >>> >> >> >> >> -- >> >> >> ∞ >> Shashwat Shriparv >> >> >> > > > -- > > > ∞ > Shashwat Shriparv > > > -- ∞ Shashwat Shriparv
Re: Error:Hdfs Client for hadoop using native java api
By the way you have these two options 1. Put the configuration files in the classpath, so that the code picks it. 2. Use Configuration.set()<http://hadoop.apache.org/common/docs/r0.21.0/api/org/apache/hadoop/conf/Configuration.html#set%28java.lang.String,%20java.lang.String%29>to set the required parameters in the code. because the configuration may be directing to local file whice code is nt abl to find On Thu, Jul 19, 2012 at 9:32 PM, shashwat shriparv < dwivedishash...@gmail.com> wrote: > Can you provide the code(some part of it if you don't want to throw the > full code here) and let us know which part of your code is throwing this > error. > > Regards > > ∞ > Shashwat Shriparv > > > > > On Thu, Jul 19, 2012 at 6:46 PM, Sandeep Reddy P < > sandeepreddy.3...@gmail.com> wrote: > >> Hi John, >> We have applications in windows. So our dev's need to connect to HDFS from >> eclipse installed in windows. I'm trying to put data from to >> using java code from windows. >> >> On Thu, Jul 19, 2012 at 5:41 AM, John Hancock > >wrote: >> >> > Sandeep, >> > >> > I don't understand your situation completely, but why not just use >> > bin/hadoop dfs -copyFromLocal ? >> > >> > -John >> > >> > On Wed, Jul 18, 2012 at 11:33 AM, Sandeep Reddy P < >> > sandeepreddy.3...@gmail.com> wrote: >> > >> > > Hi, >> > > I'm trying to load data into hdfs from local linux file system using >> java >> > > code from a windows machine.But i'm getting the error >> > > >> > > java.lang.IllegalArgumentException: Wrong FS: >> > > hdfs://hadoop1.devqa.local:8020/user/hdfs/java, expected: file:/// >> > > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410) >> > > at >> > > >> > > >> > >> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56) >> > > at >> > > >> > > >> > >> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:404) >> > > at >> > > >> > > >> > >> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) >> > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:797) >> > > at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:349) >> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:205) >> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) >> > > at >> > > >> > > >> > >> org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55) >> > > at >> > > >> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1292) >> > > at Hdfs.main(Hdfs.java:18) >> > > File not found >> > > >> > > Can any one please help me with the issue. >> > > >> > > -- >> > > Thanks, >> > > sandeep >> > > >> > >> >> >> >> -- >> Thanks, >> sandeep >> > > > > -- > > > ∞ > Shashwat Shriparv > > > -- ∞ Shashwat Shriparv
Re: Error:Hdfs Client for hadoop using native java api
Can you provide the code(some part of it if you don't want to throw the full code here) and let us know which part of your code is throwing this error. Regards ∞ Shashwat Shriparv On Thu, Jul 19, 2012 at 6:46 PM, Sandeep Reddy P < sandeepreddy.3...@gmail.com> wrote: > Hi John, > We have applications in windows. So our dev's need to connect to HDFS from > eclipse installed in windows. I'm trying to put data from to > using java code from windows. > > On Thu, Jul 19, 2012 at 5:41 AM, John Hancock >wrote: > > > Sandeep, > > > > I don't understand your situation completely, but why not just use > > bin/hadoop dfs -copyFromLocal ? > > > > -John > > > > On Wed, Jul 18, 2012 at 11:33 AM, Sandeep Reddy P < > > sandeepreddy.3...@gmail.com> wrote: > > > > > Hi, > > > I'm trying to load data into hdfs from local linux file system using > java > > > code from a windows machine.But i'm getting the error > > > > > > java.lang.IllegalArgumentException: Wrong FS: > > > hdfs://hadoop1.devqa.local:8020/user/hdfs/java, expected: file:/// > > > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410) > > > at > > > > > > > > > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56) > > > at > > > > > > > > > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:404) > > > at > > > > > > > > > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) > > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:797) > > > at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:349) > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:205) > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157) > > > at > > > > > > > > > org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55) > > > at > > > org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1292) > > > at Hdfs.main(Hdfs.java:18) > > > File not found > > > > > > Can any one please help me with the issue. > > > > > > -- > > > Thanks, > > > sandeep > > > > > > > > > -- > Thanks, > sandeep > -- ∞ Shashwat Shriparv
Re: Problem with running hadoop jar demo.jar in local mode.
In addition to Harsh answer, configure the tmp hadoop directory, and set
the full permission for the user who is trying to run jar.
Regards
Shashwat Shriparv
On Sat, Jun 23, 2012 at 2:39 PM, Harsh J wrote:
> Your local /tmp directory needs to be writable by your user, for the
> "hadoop jar" method to execute properly out of the box.
>
> If that is not possible, edit your conf/core-site.xml to change the
> "hadoop.tmp.dir" default of "/tmp/hadoop-${user.name}" to somewhere
> that is writable by you, perhaps "${user.home}/tmp" for your user
> alone.
>
> On Sat, Jun 23, 2012 at 1:10 PM, Sheng Guo wrote:
> > Hi all,
> >
> > sorry to bother, I have a simple hadoop job. It was running well both in
> > local mode and in real hadoop cluster. Recently I try to run it again in
> > single node cluster, and I got the following error:
> >
> > $ hadoop-1.0.1/bin/hadoop jar CarDemo.jar
> >
> > Exception in thread "main" java.io.IOException: Mkdirs failed to create
> > /tmp/hadoop-sguo/hadoop-unjar6763909861121801460/META-INF/license
> > at org.apache.hadoop.util.RunJar.unJar(RunJar.java:47)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:132)
> >
> >
> > I tried this both on 0.20.2 and 1.0.0. Both of them exit with exception
> > like the above.
> > Can anyone help me on this?
> >
> > Thanks!!
> >
> > Sheng
>
>
>
> --
> Harsh J
>
--
∞
Shashwat Shriparv
Re: [Newbie] How to make Multi Node Cluster from Single Node Cluster
Just follow this: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ On Thu, Jun 14, 2012 at 3:39 PM, wrote: > At Newbie level just the same. > > -Original Message- > From: Alpha Bagus Sunggono [mailto:bagusa...@gmail.com] > Sent: jueves, 14 de junio de 2012 12:01 > To: common-user@hadoop.apache.org > Subject: [Newbie] How to make Multi Node Cluster from Single Node Cluster > > Dear All. > > I've been configuring 3 server using Hadoop 1.0.x , Single Node, how to > assembly them into 1 Multi Node Cluster? > > Because when I search for documentation, i've just got configuration for > Hadoop 0.20.x > > Would you mind to assist me? > > > Subject to local law, communications with Accenture and its affiliates > including telephone calls and emails (including content), may be monitored > by our systems for the purposes of security and the assessment of internal > compliance with Accenture policy. > > ______ > > www.accenture.com > > -- ∞ Shashwat Shriparv
Re: master and slaves are running but they seem disconnected
Please send the content of all the hosts file from all the machines. and master and slaves contents from all the machines from master and the slave machines. On Sun, Jun 10, 2012 at 1:39 AM, Joey Krabacher wrote: > Not sure, but I did notice that safe mode is still. I would investigate > that and see if the other nodes show up. > > /* Joey */ > On Jun 9, 2012 2:52 PM, "Pierre Antoine DuBoDeNa" > wrote: > > > Hello everyone.. > > > > I have a cluster of 5 VMs, 1 as master/slave the rest are slaves. I run > > bin/start-all.sh everything seems ok i get no errors.. > > > > I check with jps in all server they run: > > > > master: > > 22418 Jps > > 21497 NameNode > > 21886 SecondaryNameNode > > 21981 JobTracker > > 22175 TaskTracker > > 21688 DataNode > > > > slave: > > 3161 Jps > > 2953 DataNode > > 3105 TaskTracker > > > > But in the web interface i get only 1 server connected.. is like the > > others are ignored.. Any clue why this can happen? where to look for > > errors? > > > > The hdfs web interface: > > > > Live Nodes< > > http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=LIVE> > > : 1 Dead Nodes< > > http://fusemaster.cs.columbia.edu:50070/dfsnodelist.jsp?whatNodes=DEAD> > > : 0 > > it doesn't even show the rest slaves as dead.. > > > > can it be a networking issue? (but i start all processes from master and > it > > starts all processes to all others). > > > > best, > > PA > > > -- ∞ Shashwat Shriparv
Re: Hadoop-Git-Eclipse
Check out these thread : http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/22976 http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201012.mbox/%3c4cff292d.3090...@corp.mail.ru%3E On Fri, Jun 8, 2012 at 6:24 PM, Prajakta Kalmegh wrote: > Hi > > Yes I did configure using the wiki link at > http://wiki.apache.org/hadoop/EclipseEnvironment. > I am facing a new problem while setting up Hadoop in Psuedo-distributed > mode on my laptop. I am trying to execute the following commands for > setting up Hadoop: > hdfs namenode -format > hdfs namenode > hdfs datanode > yarn resourcemanager > yarn nodemanager > > It gives me a "Hadoop Common not found." error for all the commands. When I > try to use "hadoop namenode -format" instead, it gives me a deprecated > command warning. > > I am following the instructions for setting up Hadoop with Eclipse given in > - http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment > - > > http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/SingleCluster.html > > This issue is discussed in JIRA < > https://issues.apache.org/jira/browse/HDFS-2014 > and is resolved. Not > sure > why I am getting the error. > > My environment variables look something like: > > HADOOP_COMMON_HOME=/home/Projects/hadoop-common/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT > > HADOOP_CONF_DIR=/home/Projects/hadoop-common/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-SNAPSHOT/etc/hadoop > > HADOOP_HDFS_HOME=/home/Projects/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0-SNAPSHOT > > HADOOP_MAPRED_HOME=/home/Projects/hadoop-common/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT > > YARN_HOME=/home/Projects/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/hadoop-yarn-common-3.0.0-SNAPSHOT > > YARN_CONF_DIR=/home/Projects/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/conf > > I have included them in the PATH. I am trying to build and setup from > apache-hadoop-common git repository (my own cloned fork). Any idea why > 'Hadoop Common Not found' error is coming? Do I have to add anything to the > hadoop-config.sh or hdfs-config.sh? > > Regards, > Prajakta > > > > > > Deniz Demir > 06/08/2012 05:35 PM > Please respond to > common-user@hadoop.apache.org > To > common-user@hadoop.apache.org, > cc > Subject > Re: Hadoop-Git-Eclipse > > > I did not find that screencast useful. This one worked for me: > > http://wiki.apache.org/hadoop/EclipseEnvironment > > Best, > Deniz > > On Jun 8, 2012, at 1:08 AM, shashwat shriparv wrote: > > > Check out this link: > > > > http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/ > > > > Regards > > > > ∞ > > Shashwat Shriparv > > > > > > > > > > On Fri, Jun 8, 2012 at 1:32 PM, Prajakta Kalmegh >wrote: > > > >> Hi > >> > >> I have done MapReduce programming using Eclipse before but now I need to > >> learn the Hadoop code internals for one of my projects. > >> > >> I have forked Hadoop from github ( > https://github.com/apache/hadoop-common > >> ) and need to configure it to work with Eclipse. All the links I could > >> find list steps for earlier versions of Hadoop. I am right now following > >> instructions given in these links: > >> - http://wiki.apache.org/hadoop/GitAndHadoop > >> - http://wiki.apache.org/hadoop/EclipseEnvironment > >> - http://wiki.apache.org/hadoop/HowToContribute > >> > >> Can someone please give me a link to the steps to be followed for > getting > >> Hadoop (latest from trunk) started in Eclipse? I need to be able to > commit > >> changes to my forked repository on github. > >> > >> Thanks in advance. > >> Regards, > >> Prajakta > > > > > > > > > > -- > > > > > > ∞ > > Shashwat Shriparv > -- ∞ Shashwat Shriparv
Re: Hadoop-Git-Eclipse
Check out this link: http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/ Regards ∞ Shashwat Shriparv On Fri, Jun 8, 2012 at 1:32 PM, Prajakta Kalmegh wrote: > Hi > > I have done MapReduce programming using Eclipse before but now I need to > learn the Hadoop code internals for one of my projects. > > I have forked Hadoop from github (https://github.com/apache/hadoop-common > ) and need to configure it to work with Eclipse. All the links I could > find list steps for earlier versions of Hadoop. I am right now following > instructions given in these links: > - http://wiki.apache.org/hadoop/GitAndHadoop > - http://wiki.apache.org/hadoop/EclipseEnvironment > - http://wiki.apache.org/hadoop/HowToContribute > > Can someone please give me a link to the steps to be followed for getting > Hadoop (latest from trunk) started in Eclipse? I need to be able to commit > changes to my forked repository on github. > > Thanks in advance. > Regards, > Prajakta -- ∞ Shashwat Shriparv
Re: Nutch hadoop integration
Check out these links : http://wiki.apache.org/nutch/NutchHadoopTutorial http://wiki.apache.org/nutch/NutchTutorial http://joey.mazzarelli.com/2007/07/25/nutch-and-hadoop-as-user-with-nfs/ http://stackoverflow.com/questions/5301883/run-nutch-on-existing-hadoop-cluster Regards ∞ Shashwat Shriparv On Fri, Jun 8, 2012 at 1:29 PM, abhishek tiwari < abhishektiwari.u...@gmail.com> wrote: > how can i integrate hadood and nutch ..anyone please brief me . > -- ∞ Shashwat Shriparv
Re: java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
If you have ** * hadoop.tmp.dir* * ../Hadoop/hdfs/tmp in your configuration file then remove it and try Thanks and regards ∞ Shashwat Shriparv * On Thu, Jun 7, 2012 at 1:56 PM, huanchen.zhang wrote: > Hi, > > I coded a map reduce program with hadoop java api. > > When I submitted the job to the cluster, I got the following errors: > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/codehaus/jackson/map/JsonMappingException >at org.apache.hadoop.mapreduce.Job$1.run(Job.java:489) >at java.security.AccessController.doPrivileged(Native Method) >at javax.security.auth.Subject.doAs(Subject.java:396) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) >at org.apache.hadoop.mapreduce.Job.connect(Job.java:487) >at org.apache.hadoop.mapreduce.Job.submit(Job.java:475) >at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506) >at > com.ipinyou.data.preprocess.mapreduce.ExtractFeatureFromURLJob.main(ExtractFeatureFromURLJob.java:52) > Caused by: java.lang.ClassNotFoundException: > org.codehaus.jackson.map.JsonMappingException >at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >... 8 more > > > I found the classes not found here is in "jackson-core-asl-1.5.2" and > "jackson-mapper-asl-1.5.2", so added these two jars to the project and > resubmitted the job. But I got the following errors: > > Jun 7, 2012 4:18:55 PM org.apache.hadoop.metrics.jvm.JvmMetrics init > INFO: Initializing JVM Metrics with processName=JobTracker, sessionId= > Jun 7, 2012 4:18:55 PM org.apache.hadoop.util.NativeCodeLoader > WARNING: Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > Jun 7, 2012 4:18:55 PM org.apache.hadoop.mapred.JobClient > copyAndConfigureFiles > WARNING: Use GenericOptionsParser for parsing the arguments. Applications > should implement Tool for the same. > Jun 7, 2012 4:18:55 PM org.apache.hadoop.mapred.JobClient$2 run > INFO: Cleaning up the staging area > file:/tmp/hadoop-huanchen/mapred/staging/huanchen757608919/.staging/job_local_0001 > Exception in thread "main" > org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path > does not exist: file:/data/huanchen/pagecrawler/url >at > org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231) >at > org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248) >at > org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944) >at > org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) >at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) >at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) >at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) >at java.security.AccessController.doPrivileged(Native Method) >at javax.security.auth.Subject.doAs(Subject.java:396) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) >at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) >at org.apache.hadoop.mapreduce.Job.submit(Job.java:476) >at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506) >at > com.ipinyou.data.preprocess.mapreduce.ExtractFeatureFromURLJob.main(ExtractFeatureFromURLJob.java:51) > > > Note that the error is "Input path does not exist: file:/" instead of > "Input path does not exist: hdfs:/" . So does it mean the job does not > successfully connect to the hadoop cluster? The first > "NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException" error > is also for this reason? > > Any one has any ideas? > > Thank you ! > > > Best, > Huanchen > > 2012-06-07 > > > > huanchen.zhang > -- ∞ Shashwat Shriparv
Re: Pseudo Distributed: ERROR org.apache.hadoop.hbase.HServerAddress: Could not resolve the DNS name of localhost.localdomain
Hey manu which linux distribution you are using?? On Thu, Jun 7, 2012 at 8:18 PM, Manu S wrote: > Thank you Harsh & Shashwat > > I given the hostname in /etc/sysconfig/network as pseudo-distributed. > hostname command returns this name also. I added this name in /etc/hosts > file and changed all the configuration accordingly. But zookeeper is trying > to resolve to localhost.localdomain. There was no entries in any conf files > or hostname related files for localhost.localdomain. > > Yea, everything is pinging as I given the names in /etc/hosts. > > On Thu, Jun 7, 2012 at 7:13 PM, shashwat shriparv < > dwivedishash...@gmail.com > > wrote: > > > Are you able ping to > > > > yourpcipaddress > > domainnameyougaveformachine > > hostnameofthemachine > > > > > > Hbase stops means its not able to start itself on the ip or hostname > which > > you are giving. > > > > > > On Thu, Jun 7, 2012 at 2:48 PM, Manu S wrote: > > > > > Hi All, > > > > > > In pseudo distributed node HBaseMaster is stopping automatically when > we > > > starts HbaseRegion. > > > > > > I have changed all the configuration files of Hadoop,Hbase & Zookeeper > to > > > set the exact hostname of the machine. Also commented the localhost > entry > > > from /etc/hosts & cleared the cache as well. There is no entry of > > > localhost.localdomain entry in these configurations, but this it is > > > resolving to localhost.localdomain. > > > > > > Please find the error: > > > 2012-06-07 12:13:11,995 INFO > > > org.apache.hadoop.hbase.master.MasterFileSystem: No logs to split > > > *2012-06-07 12:13:12,103 ERROR org.apache.hadoop.hbase.HServerAddress: > > > Could not resolve the DNS name of localhost.localdomain > > > 2012-06-07 12:13:12,104 FATAL org.apache.hadoop.hbase.master.HMaster: > > > Unhandled exception. Starting shutdown.* > > > *java.lang.IllegalArgumentException: hostname can't be null* > > >at java.net.InetSocketAddress.(InetSocketAddress.java:121) > > >at > > > > > > > > > org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108) > > >at > > > org.apache.hadoop.hbase.HServerAddress.(HServerAddress.java:64) > > >at > > > > > > > > > org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82) > > >at > > > > > > > > > org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73) > > >at > > > > > > > > > org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:222) > > >at > > > > > > > > > org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRootServerConnection(CatalogTracker.java:240) > > >at > > > > > > > > > org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:487) > > >at > > > > > > org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:455) > > >at > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:406) > > >at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:293) > > > 2012-06-07 12:13:12,106 INFO org.apache.hadoop.hbase.master.HMaster: > > > Aborting > > > 2012-06-07 12:13:12,106 DEBUG org.apache.hadoop.hbase.master.HMaster: > > > Stopping service threads > > > > > > Thanks, > > > Manu S > > > > > > > > > > > -- > > > > > > ∞ > > Shashwat Shriparv > > > -- ∞ Shashwat Shriparv
Re: Pseudo Distributed: ERROR org.apache.hadoop.hbase.HServerAddress: Could not resolve the DNS name of localhost.localdomain
Are you able ping to yourpcipaddress domainnameyougaveformachine hostnameofthemachine Hbase stops means its not able to start itself on the ip or hostname which you are giving. On Thu, Jun 7, 2012 at 2:48 PM, Manu S wrote: > Hi All, > > In pseudo distributed node HBaseMaster is stopping automatically when we > starts HbaseRegion. > > I have changed all the configuration files of Hadoop,Hbase & Zookeeper to > set the exact hostname of the machine. Also commented the localhost entry > from /etc/hosts & cleared the cache as well. There is no entry of > localhost.localdomain entry in these configurations, but this it is > resolving to localhost.localdomain. > > Please find the error: > 2012-06-07 12:13:11,995 INFO > org.apache.hadoop.hbase.master.MasterFileSystem: No logs to split > *2012-06-07 12:13:12,103 ERROR org.apache.hadoop.hbase.HServerAddress: > Could not resolve the DNS name of localhost.localdomain > 2012-06-07 12:13:12,104 FATAL org.apache.hadoop.hbase.master.HMaster: > Unhandled exception. Starting shutdown.* > *java.lang.IllegalArgumentException: hostname can't be null* >at java.net.InetSocketAddress.(InetSocketAddress.java:121) >at > > org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108) >at > org.apache.hadoop.hbase.HServerAddress.(HServerAddress.java:64) >at > > org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82) >at > > org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73) >at > > org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:222) >at > > org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRootServerConnection(CatalogTracker.java:240) >at > > org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:487) >at > org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:455) >at > > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:406) >at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:293) > 2012-06-07 12:13:12,106 INFO org.apache.hadoop.hbase.master.HMaster: > Aborting > 2012-06-07 12:13:12,106 DEBUG org.apache.hadoop.hbase.master.HMaster: > Stopping service threads > > Thanks, > Manu S > -- ∞ Shashwat Shriparv
Re: SecondaryNameNode not connecting to NameNode : PriviledgedActionException
could be the exact issue but when configuring > > > > secondary NN to NN, you need to tell your SNN where the actual NN > > > resides. > > > > Try adding - dfs.http.address on your secondary namenode machine > > > > having value as on hdfs-site.xml Port should be on which > > > > your NN url is opening - means your NN web browser http port. > > > > > > > > Regards, > > > > Praveenesh > > > > On Mon, Jun 4, 2012 at 4:37 PM, wrote: > > > > > > > > > Hello. I'm facing a issue when trying to configure my > > > > > SecondaryNameNode on a different machine than my NameNode. When > > > > > both are on the same machine everything works fine but after > > > > > moving the > > > > secondary to a new machine I get: > > > > > > > > > > 2012-05-28 09:57:36,832 ERROR > > > > > org.apache.hadoop.security.UserGroupInformation: > > > > > PriviledgedActionException as:hadoop > cause:java.net.ConnectException: > > > > > Connection refused > > > > > 2012-05-28 09:57:36,832 ERROR > > > > > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: > > > > > Exception in > > > > > doCheckpoint: > > > > > 2012-05-28 09:57:36,834 ERROR > > > > > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: > > > > > java.net.ConnectException: Connection refused > > > > >at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > >at > > > > > > > > > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl > > > > .j > > > > av > > > > a:327) > > > > >at > > > > > > > > > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSoc > > > > ke > > > > tI > > > > mpl.java:191) > > > > >at > > > > > > > > > > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java: > > > > 180) > > > > >at > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) > > > > >at java.net.Socket.connect(Socket.java:546) > > > > >at java.net.Socket.connect(Socket.java:495) > > > > >at sun.net.NetworkClient.doConnect(NetworkClient.java:178) > > > > >at > sun.net.www.http.HttpClient.openServer(HttpClient.java:409) > > > > >at > sun.net.www.http.HttpClient.openServer(HttpClient.java:530) > > > > >at sun.net.www.http.HttpClient.(HttpClient.java:240) > > > > >at sun.net.www.http.HttpClient.New(HttpClient.java:321) > > > > >at sun.net.www.http.HttpClient.New(HttpClient.java:338) > > > > >at > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpU > > > > RL > > > > Co > > > > nnection.java:935) > > > > >at > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLCo > > > > nn > > > > ec > > > > tion.java:876) > > > > >at > > > > > > > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection. > > > > > java:801) > > > > > > > > > > Is there any configuration I'm missing? At this point my > > > > > mapred-site.xml is very simple just: > > > > > > > > > > > > > > > > > > > > > > > > >mapred.job.tracker > > > > >hadoop00:9001 > > > > > > > > > > > > > > >mapred.system.dir > > > > >/home/hadoop/mapred/system > > > > > > > > > > > > > > >mapred.local.dir > > > > >/home/hadoop/mapred/local > > > > > > > > > > > > > > >mapred.jobtracker.taskScheduler > > > > >org.apache.hadoop.mapred.FairScheduler > > > > > > > > > > > > > > >mapred.fairscheduler.allocation.file > > > > >/home/hadoop/hadoop/conf/fairscheduler.xml > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject to local law, > > > > > communications with Accenture and its affiliates including > > > > > telephone calls and emails (including content), may be monitored > > > > > by our systems for the purposes of security and the assessment > > > > > of internal compliance with Accenture > > policy. > > > > > > > > > > > > > > > __ > > > > > __ > > > > > __ > > > > > > > > > > > > > > > www.accenture.com > > > > > > > > > > > > > > > > > Subject to local law, communications with Accenture and its > > > > affiliates including telephone calls and emails (including > > > > content), may be monitored by our systems for the purposes of > > > > security and the assessment of internal compliance with Accenture > policy. > > > > > > > > __ > > > > __ > > > > __ > > > > > > > > > > > > www.accenture.com > > > > > > > > > > > > > > > > > Subject to local law, communications with Accenture and its > > > affiliates including telephone calls and emails (including content), > > > may be monitored by our systems for the purposes of security and the > > > assessment of internal compliance with Accenture policy. > > > > > > > > > __ > > > > > > > > > www.accenture.com > > > > > > > > > > > > Subject to local law, communications with Accenture and its affiliates > > including telephone calls and emails (including content), may be > > monitored by our systems for the purposes of security and the > > assessment of internal compliance with Accenture policy. > > > > __ > > > > > > www.accenture.com > > > > > > > Subject to local law, communications with Accenture and its affiliates > including telephone calls and emails (including content), may be monitored > by our systems for the purposes of security and the assessment of internal > compliance with Accenture policy. > > __ > > www.accenture.com > > -- ∞ Shashwat Shriparv
Re: Small glitch with setting up two node cluster...only secondary node starts (datanode and namenode don't show up in jps)
Please send you conf file contents and host file contents too.. On Tue, May 29, 2012 at 11:08 PM, Harsh J wrote: > Rohit, > > The SNN may start and run infinitely without doing any work. The NN > and DN have probably not started cause the NN has an issue (perhaps NN > name directory isn't formatted) and the DN can't find the NN (or has > data directory issues as well). > > So this isn't a glitch but a real issue you'll have to take a look at > your logs for. > > On Sun, May 27, 2012 at 10:51 PM, Rohit Pandey > wrote: > > Hello Hadoop community, > > > > I have been trying to set up a double node Hadoop cluster (following > > the instructions in - > > > http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ > ) > > and am very close to running it apart from one small glitch - when I > > start the dfs (using start-dfs.sh), it says: > > > > 10.63.88.53: starting datanode, logging to > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-datanode-ubuntu.out > > 10.63.88.109: starting datanode, logging to > > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-datanode-pandro51-OptiPlex-960.out > > 10.63.88.109: starting secondarynamenode, logging to > > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-secondarynamenode-pandro51-OptiPlex-960.out > > starting jobtracker, logging to > > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-jobtracker-pandro51-OptiPlex-960.out > > 10.63.88.109: starting tasktracker, logging to > > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-tasktracker-pandro51-OptiPlex-960.out > > 10.63.88.53: starting tasktracker, logging to > > /usr/local/hadoop/bin/../logs/hadoop-pandro51-tasktracker-ubuntu.out > > > > which looks like it's been successful in starting all the nodes. > > However, when I check them out by running 'jps', this is what I see: > > 27531 SecondaryNameNode > > 27879 Jps > > > > As you can see, there is no datanode and name node. I have been > > racking my brains at this for quite a while now. Checked all the > > inputs and every thing. Any one know what the problem might be? > > > > -- > > > > Thanks in advance, > > > > Rohit > > > > -- > Harsh J > -- ∞ Shashwat Shriparv
