Re: Error when Using URI in -put command
Cheny, On Thu, Jul 21, 2011 at 7:04 AM, Cheny wrote: > Can I make HDFS to locate the first block of the file on that specific > datanode? No. There is no way to induce this unless you upload from the DN machine itself. -- Harsh J
Re: Error when Using URI in -put command
the fs uri is hdfs://:// . - Original Message - > From: Cheny > To: core-u...@hadoop.apache.org > Cc: > Sent: Wednesday, July 20, 2011 6:34 PM > Subject: Error when Using URI in -put command > > > Not considering replication, if I use following command from a hadoop client > outside the cluster(the client is not a datanode) > > hadoop dfs -put hdfs:// ip>:50010/ > > Can I make HDFS to locate the first block of the file on that specific > datanode? > > I tried to do that and I got this error: > > put: Call to /xxx.xxx.xxx.xxx(ip of my datanode):50010 failed on local > exception: java.io.EOFException > > Any help is greatly appreciated. > > -- > View this message in context: > http://old.nabble.com/Error-when-Using-URI-in--put-command-tp32104146p32104146.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. >
Re: hadoop pipes
When you launch program using bin/hadoop command full cluster info is available to your program like name node, data node etc ..here your just submitting binary but the starting is done by hadoop rather than you running ./a.out On Jun 29, 2011 1:48 AM, "jitter" wrote: > hi i m confused about the execution of hadoop program; > ahat happen when we write the hadoop pipe running command like bin/hadoop > pipes -D pipie.java.record reader =true etc > > i don't know how the program run what does the control do; > I know we compile the c++ program by g++ command and run it by ./a.out . But > in hadoop we dont use the ./a.out command than how this executable run ? > what does the executable do in running command ? > more ever icvhanged the program many time but every time sanre output . > can any body tell me actually how pipes program work > > -- > View this message in context: http://hadoop-common.472056.n3.nabble.com/hadoop-pipes-tp3117626p3117626.html > Sent from the Users mailing list archive at Nabble.com.
Re: Api migration from 0.19.1 to 0.20.20
Hadoop : the definitive guide also talks about migration On Jun 28, 2011 8:31 PM, "Shi Yu" wrote: > On 6/28/2011 7:12 AM, Prashant Sharma wrote: >> Hi , >> >> I have my source code written in 0.19.1 Hadoop API and want to shift >> it to newer API 0.20.20. Any clue on good documentation on migrating >> from older version to newer version will be very helpful. >> >> Thanks. >> Prashant >> >> >> This message was sent using IMP, the Internet Messaging Program. >> >> > In the *Chuck Lam's* book *Hadoop in Action the* upgrade from 0.19.1 to > 0.20.0 is mentioned. Also there are many pieces of information on web, > but they are scattered around. >
Re: Job progress not showing in Hadoop Tasktracker web interface
Although that solves it, its not a solution to the eclipse configuration issue I think. But, good to know you've found a way that works for you! On Thu, Jul 21, 2011 at 6:32 AM, foo_foo_foo wrote: > > Found the issue. I created a JAR and submitted through bin/hadoop and it > distributes the jobs as expected. > -- > View this message in context: > http://old.nabble.com/Job-progress-not-showing-in-Hadoop-Tasktracker--web-interface-tp32096156p32104045.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. > > -- Harsh J
Error when Using URI in -put command
Not considering replication, if I use following command from a hadoop client outside the cluster(the client is not a datanode) hadoop dfs -put hdfs://:50010/ Can I make HDFS to locate the first block of the file on that specific datanode? I tried to do that and I got this error: put: Call to /xxx.xxx.xxx.xxx(ip of my datanode):50010 failed on local exception: java.io.EOFException Any help is greatly appreciated. -- View this message in context: http://old.nabble.com/Error-when-Using-URI-in--put-command-tp32104146p32104146.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Job progress not showing in Hadoop Tasktracker web interface
Found the issue. I created a JAR and submitted through bin/hadoop and it distributes the jobs as expected. -- View this message in context: http://old.nabble.com/Job-progress-not-showing-in-Hadoop-Tasktracker--web-interface-tp32096156p32104045.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Development enviroment problems - eclipse, hadoop 0.20.203
I found source code for 0.20.203 release under "tags" option in svn.. so checkout went smoothly.. Now I am just wondering which compile options I have to choose to build a project without errors. When I choose to run build.xml with options "compile", "compile-core-text", "eclipse-files" there shows some errors not finding some libraries under /build/ivy/lib folder.. Is any other option to take into account? Thank you mertoz wrote: > > I am trying to run some benchmarks test with integrated hadoop schedulers > and make some analysis of its performances. > > Because the current version from svn is not stable I am planning to do > some modification on version 0.20.203. This version is not present(?) in > svn so I am looking to import it from the archive file, downloaded from > http://hadoop.apache.org/common/releases.html here . > > The problem is that when I import them as existing java project, there > appears just libs and sources from java eclipse tool for running jobs. Am > I missing something? How can I establish a working development enviroment > from which I could compile my own scheduler? > > thank you > > (for importing up to date source there is a good guide from > http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/ > cloudera ) > > > -- View this message in context: http://old.nabble.com/Development-enviroment-problems---eclipse%2C-hadoop-0.20.203-tp32101224p32102314.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Development enviroment problems - eclipse, hadoop 0.20.203
Hey, Have you tried running the 'ant eclipse' build target to prepare the eclipse project files before trying to import things? $ svn co hadoop OR $ tar xf hadoop.tar.gz Then, $ cd hadoop $ ant eclipse # sometimes eclipse-files, so try that as well $ # Done, now try to import this directory in Eclipse, would work fine. On 20-Jul-2011, at 10:47 PM, mertoz wrote: > > I am trying to run some benchmarks test with integrated hadoop schedulers and > make some analysis of its performances. > > Because the current version from svn is not stable I am planning to do some > modification on version 0.20.203. This version is not present(?) in svn so > I am looking to import it from the archive file, downloaded from > http://hadoop.apache.org/common/releases.html here . > > The problem is that when I import them as existing java project, there > appears just libs and sources from java eclipse tool for running jobs. Am I > missing something? How can I establish a working development enviroment from > which I could compile my own scheduler? > > thank you > > (for importing up to date source there is a good guide from > http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/ > cloudera ) > > > -- > View this message in context: > http://old.nabble.com/Development-enviroment-problems---eclipse%2C-hadoop-0.20.203-tp32101224p32101224.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. >
Development enviroment problems - eclipse, hadoop 0.20.203
I am trying to run some benchmarks test with integrated hadoop schedulers and make some analysis of its performances. Because the current version from svn is not stable I am planning to do some modification on version 0.20.203. This version is not present(?) in svn so I am looking to import it from the archive file, downloaded from http://hadoop.apache.org/common/releases.html here . The problem is that when I import them as existing java project, there appears just libs and sources from java eclipse tool for running jobs. Am I missing something? How can I establish a working development enviroment from which I could compile my own scheduler? thank you (for importing up to date source there is a good guide from http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/ cloudera ) -- View this message in context: http://old.nabble.com/Development-enviroment-problems---eclipse%2C-hadoop-0.20.203-tp32101224p32101224.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Bad file descriptor with MultipleInputs
Bibudh, Could you post the exact/entire stacktrace, and your client configuration files? (You can use a service like pastebin.com or paste inline into the mail itself, since the list does not allow attachments) On a light hearted note, here's some humor meanwhile: http://imgur.com/jacoj ;) On Wed, Jul 20, 2011 at 3:23 AM, Bibudh Lahiri wrote: > Hi, > I am using MultipleInputs.addInputPath to connect the output of two map > jobs with one reduce job. I've specified the inputformat to be > TextInputFormat, and getting a "bad file descriptor" error. When I ran the > map jobs separately with the reduce job (slightly diff version), it was > fine. The files that these map jobs should read also look alright - these > are outputs of previous reduce jobs in the chain. Any suggestions? Should I > use a different input format? I saw KeyValueInputFormat has become > deprecated. > > Thanks > Bibudh > > -- > Bibudh Lahiri > Senior Analytics Engineer, Kikin Inc > 132 Crosby Street, New York, NY 10012 > http://www.kikin.com > PhD candidate, Dept. of Electrical and Computer Engg. > Iowa State University > http://home.eng.iastate.edu/~bibudh/ > -- Harsh J
Bad file descriptor with MultipleInputs
Hi, I am using MultipleInputs.addInputPath to connect the output of two map jobs with one reduce job. I've specified the inputformat to be TextInputFormat, and getting a "bad file descriptor" error. When I ran the map jobs separately with the reduce job (slightly diff version), it was fine. The files that these map jobs should read also look alright - these are outputs of previous reduce jobs in the chain. Any suggestions? Should I use a different input format? I saw KeyValueInputFormat has become deprecated. Thanks Bibudh -- Bibudh Lahiri Senior Analytics Engineer, Kikin Inc 132 Crosby Street, New York, NY 10012 http://www.kikin.com PhD candidate, Dept. of Electrical and Computer Engg. Iowa State University http://home.eng.iastate.edu/~bibudh/
Utah Hadoop User Group - August 4th
Greetings! Lots of interest in the Utah area has spurred the creation the Utah Hadoop User Group. If you are in the area and want to join us please come. Subject matter will include *Hadoop: Solving your Big Data Woes*, Matt Davies, Tynt, Inc. and *Machine Learning With Big Data*, Richard Wellman, Westminster College We will be meeting August 4th at 7PM. For more details please visit the uhug.org site (http://www.uhug.org/rsvpmaker/uhug-august-4th) Look forward to having you come!
Re: error of loading logging class
The problem is caused by conflict with another jar with org/apache/commons/logging/ path. 2011/7/20 Juwei Shi > Hi, > > We faced a problem of loading logging class when start the name node. It > seems that hadoop can not find commons-logging-*.jar > > We have tried other commons-logging-1.0.4.jar and > commons-logging-api-1.0.4.jar. It does not work! > > The following are error logs from starting console: > > Exception in thread "main" java.lang.NoSuchMethodError: > org/apache/commons/loggi > ng/LogFactory.getClassLoader(Ljava/lang/Class;)Ljava/lang/ClassLoader; > at > org.apache.commons.logging.impl.LogFactoryImpl.getClassLoader(LogFact > oryImpl.java:447) > at > org.apache.commons.logging.impl.LogFactoryImpl.initDiagnostics(LogFac > toryImpl.java:476) > at > org.apache.commons.logging.impl.LogFactoryImpl.(LogFactoryImpl. > java:95) > at java.lang.J9VMInternals.newInstanceImpl(Native Method) > at java.lang.Class.newInstance(Class.java:1345) > at org.apache.commons.logging.LogFactory$2.run(LogFactory.java:531) > at > java.security.AccessController.doPrivileged(AccessController.java:202 > ) > at > org.apache.commons.logging.LogFactory.newFactory(LogFactory.java:517) > at > org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:316) > > > > Thanks in advanced! > -- > - Juwei > -- - Juwei
