contrib/ec2 USER_DATA not used
Hi, can someone tell me what the variable USER_DATA in the launch-hadoop- master is all about. I cant see that it is reused in the script or any other script. Isnt the way those parameters are passed to the nodes the USER_DATA_FILE ? The line is: USER_DATA=MASTER_HOST=master,MAX_MAP_TASKS= $MAX_MAP_TASKS,MAX_REDUCE_TASKS=$MAX_REDUCE_TASKS,COMPRESS=$COMPRESS Any hints? Thanks, Stefan ~~~ Hadoop training and consulting http://www.scaleunlimited.com http://www.101tec.com
who can tell me that if i can run a linpack in hadoop?
Dear, I am thinking that how to know the performance of hadoop. As a kind of cluster, can we run linpack in hadoop? and how can we do that? thanks!
Re: contrib/ec2 USER_DATA not used
Hi Stefan, The USER_DATA line is a hangover from the way that these parameters used to be passed to the node. This line can safely be removed, since the scripts now pass the data in the USER_DATA_FILE as you rightly point out. Tom On Thu, Dec 18, 2008 at 10:09 AM, Stefan Groschupf s...@101tec.com wrote: Hi, can someone tell me what the variable USER_DATA in the launch-hadoop-master is all about. I cant see that it is reused in the script or any other script. Isnt the way those parameters are passed to the nodes the USER_DATA_FILE ? The line is: USER_DATA=MASTER_HOST=master,MAX_MAP_TASKS=$MAX_MAP_TASKS,MAX_REDUCE_TASKS=$MAX_REDUCE_TASKS,COMPRESS=$COMPRESS Any hints? Thanks, Stefan ~~~ Hadoop training and consulting http://www.scaleunlimited.com http://www.101tec.com
Re: who can tell me that if i can run a linpack in hadoop?
Hey, I think you can consider Hadoop as a data processing system which uses the Map-Reduce paradigm to process data. There is a project called Hama (I think) which supports some linear algebra operations on Hadoop. However, it will only be effective for a subset of linear algebra operations. Hadoop is not a general purpose clustering software, and it cannot run linpack. If you want to do large-scale linear algebra operations (without having to write FORTRAN-style code in C/C++), I would recommend investigating PETSc. Brian On Dec 18, 2008, at 6:47 AM, Leeau wrote: Dear, I am thinking that how to know the performance of hadoop. As a kind of cluster, can we run linpack in hadoop? and how can we do that? thanks!
Multiple user permissions
Greetings
I successfully installed a cloud w/ 4 computers logged in with my own account
(jallen). I want to expose the cloud to other users using a generic account.
So, after installing the SW and formatting the file system I went through and
performed a chmod on the installation directory and the root of the hadoop file
system so everyone would have read and write access.
However when I run a command like: bin/hadoop fs -mkdir input from the
generic account I get:
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied:
user=hadoop, access=WRITE, inode=user:jallen:supergroup:rwxr-xr-x
I was wondering if the problem may have been created when I changed the
hadoop.tmp.dir attribute to be just /tmp/hadoop and not based on the user's
name as is the default /tmp/hadoop-${user.name} but after removing that
attribute in hadoop-site.xml, restarting and reformatting, I still get the same
error.
Any thoughts of what's getting accessed or how to solve this?
Jeff
Re: HOD questions
Hemanth, snip Just FYI, at Yahoo! we've set torque to allocate separate nodes for the number specified to HOD. In other words, the number corresponds to the number of nodes, not processors. This has proved simpler to manage. I forget right now, but I think you can make Torque behave like this (to not treat processors as individual nodes). Thanks - I think it's a Maui directive, either on the job level or globally. I'm looking into this currently. However, on inspection of the Jobtracker UI, it tells us that node19 has Max Map Tasks and Max Reduce Tasks both set to 2, when for node19, it should only be allowed one map task. While HOD does not do this automatically, please note that since you are bringing up a Map/Reduce cluster on the allocated nodes, you can submit map/reduce parameters with which to bring up the cluster when allocating jobs. The relevant options are --gridservice-mapred.server-params (or -M in shorthand). Please refer to http://hadoop.apache.org/core/docs/r0.19.0/hod_user_guide.html#Options+for+Configuring+Hadoop for details. I was aware of this, but the issue is that unless you obtain dedicated nodes (as above), this option is not suitable, as it isn't set on a per-node basis. I think it would be /fairly/ straightfoward to add to HOD, as I detailed in my initial email, so that it does the correct thing out the box. (2) In our InputFormat, we use the numSplits to tell us how many map tasks the job's files should be split into. However, HOD does not override the mapred.map.tasks property (nor the mapred.reduce.tasks), while they should be set dependent on the number of available task trackers and/or nodes in the HOD session. Can this not be submitted via the Hadoop job's configuration ? Again, HOD cannot do this automatically currently. But you could use the hod.client-params to set up a client side hadoop-site.xml that would work like this for all jobs submitted to the cluster. According to hadoop-default.xml, the number of maps is Typically set to a prime several times greater than number of available hosts. - Say that we relax this recommendation to read Typically set to a NUMBER several times greater than number of available hosts then it should be straightforward for HOD to set it automatically then? Craig
EC2 Usage?
Hello all, Somewhat of a an off-topic related question, but I know there are Hadoop + EC2 users here. Does anyone know if there is a programmatic API to get find out how many machine time hours have been used by a Hadoop cluster (or anything) running on EC2? I know that you can log into the EC2 web site and see this, but I'm wondering if there's a way to access this data programmaticly via web services? Thanks, Ryan
Re: EC2 Usage?
Hi Ryan, The ec2-describe-instances command in the API tool reports the launch time for each instance, so you could work out the machine hours of your cluster using that information. Tom On Thu, Dec 18, 2008 at 4:59 PM, Ryan LeCompte lecom...@gmail.com wrote: Hello all, Somewhat of a an off-topic related question, but I know there are Hadoop + EC2 users here. Does anyone know if there is a programmatic API to get find out how many machine time hours have been used by a Hadoop cluster (or anything) running on EC2? I know that you can log into the EC2 web site and see this, but I'm wondering if there's a way to access this data programmaticly via web services? Thanks, Ryan
Re: EC2 Usage?
Thanks! On Thu, Dec 18, 2008 at 12:17 PM, Tom White t...@cloudera.com wrote: Hi Ryan, The ec2-describe-instances command in the API tool reports the launch time for each instance, so you could work out the machine hours of your cluster using that information. Tom On Thu, Dec 18, 2008 at 4:59 PM, Ryan LeCompte lecom...@gmail.com wrote: Hello all, Somewhat of a an off-topic related question, but I know there are Hadoop + EC2 users here. Does anyone know if there is a programmatic API to get find out how many machine time hours have been used by a Hadoop cluster (or anything) running on EC2? I know that you can log into the EC2 web site and see this, but I'm wondering if there's a way to access this data programmaticly via web services? Thanks, Ryan
Re: EC2 Usage?
Ryan, A developer on our team wrote some JSP to add to the Job Tracker, so that job times and other stats could be accessed programmatically via web services: https://issues.apache.org/jira/browse/HADOOP-4559 There's another update coming for that patch in JIRA, to get task data. Paco On Thu, Dec 18, 2008 at 11:17, Tom White t...@cloudera.com wrote: Hi Ryan, The ec2-describe-instances command in the API tool reports the launch time for each instance, so you could work out the machine hours of your cluster using that information. Tom On Thu, Dec 18, 2008 at 4:59 PM, Ryan LeCompte lecom...@gmail.com wrote: Hello all, Somewhat of a an off-topic related question, but I know there are Hadoop + EC2 users here. Does anyone know if there is a programmatic API to get find out how many machine time hours have been used by a Hadoop cluster (or anything) running on EC2? I know that you can log into the EC2 web site and see this, but I'm wondering if there's a way to access this data programmaticly via web services? Thanks, Ryan
Re: Hadoop Installation
Hello all The exception was being throw cause the JDK version I had was a lower one. The minute I updated that it worked well! So I guess having the right version of java matters alot!! Thanks for your help! Mithila On Tue, Nov 25, 2008 at 9:57 AM, Steve Loughran ste...@apache.org wrote: Mithila Nagendra wrote: Hey steve The version is: Linux enpc3740.eas.asu.edu 2.6.9-67.0.20.EL #1 Wed Jun 18 12:23:46 EDT 2008 i686 i686 i386 GNU/Linux, this is what I got when I used the command uname -a On Tue, Nov 25, 2008 at 1:50 PM, Steve Loughran ste...@apache.org wrote: Mithila Nagendra wrote: Hey Steve I deleted what ever I needed to.. still no luck.. You said that the classpath might be messed up.. Is there some way I can reset it? For the root user? What path do I set it to. Let's start with what kind of machine is this? Windows? or Linux. If Linux, which one? OK 1. In yum (redhat) or the synaptic package manager, is there any package called log4j installed? or liblog4j? 2. Install ant, and run ant -diagnostics email us the results
Out of Memory error in reduce shuffling phase when compression is turned on
mapred.compress.map.output is set to true, and the job has 6860 mappers and 300 reducers. Several reducers failed because:out of memory error in the shuffling phase. Error log: 2008-12-18 11:42:46,593 WARN org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1 Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:633) at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:95) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(ZlibDecompressor.java:108) at org.apache.hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.java:188) at org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(SequenceFile.java:1458) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1564) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1442) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1363) at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:2989) at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2804) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2556) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:1632) Anybody has seen similar problems? Filed JIRA: https://issues.apache.org/jira/browse/HADOOP-4915 Zheng
Re: Out of Memory error in reduce shuffling phase when compression is turned on
On Dec 18, 2008, at 2:09 PM, Zheng Shao wrote: mapred.compress.map.output is set to true, and the job has 6860 mappers and 300 reducers. Several reducers failed because:out of memory error in the shuffling phase. Error log: 2008-12-18 11:42:46,593 WARN org.apache.hadoop.mapred.ReduceTask: task_200812161126_7976_r_000272_1 Intermediate Merge of the inmemory files threw an exception: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:633) at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:95) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at org .apache .hadoop .io.compress.zlib.ZlibDecompressor.init(ZlibDecompressor.java:108) at org .apache .hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.java:188) at org.apache.hadoop.io.SequenceFile $Reader.getPooledOrNewDecompressor(SequenceFile.java:1458) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java: 1564) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java: 1442) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java: 1363) at org.apache.hadoop.io.SequenceFile$Sorter $SegmentDescriptor.nextRawKey(SequenceFile.java:2989) at org.apache.hadoop.io.SequenceFile$Sorter $MergeQueue.merge(SequenceFile.java:2804) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java: 2556) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $InMemFSMergeThread.run(ReduceTask.java:1632) Anybody has seen similar problems? Filed JIRA: https://issues.apache.org/jira/browse/HADOOP-4915 Hadoop 0.18 fixed a lot of problems with map-output compression using native codecs... HADOOP-2095. Arun
Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)
Hello, I intend to start a mapreduce job from another java app, using ToolRunner.run method. This works fine on a local job. However when distributed i get java.lang.NoClassDefFoundError: org/rosuda/REngine/Rserve/RserveException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) at java.lang.Class.getConstructor0(Class.java:2699) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74) at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) Now, i have a CLASSPATH entry in my .bashrc which contains the location of Rserve.jar (which contains the above classes), yet I still get the above error. Do I have to somehow emulate RunJar.main or is there a simpler way out? Thank you Saptarshi -- Saptarshi Guha - saptarshi.g...@gmail.com
RE: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)
You either need to add the your Rserve.jar to -libjars, or put Rserve.jar in a cluster-accessible NFS mount (and add it to CLASSPATH/HADOOP_CLASSPATH). Zheng -Original Message- From: Saptarshi Guha [mailto:saptarshi.g...@gmail.com] Sent: Thursday, December 18, 2008 3:38 PM To: core-user@hadoop.apache.org Subject: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app) Hello, I intend to start a mapreduce job from another java app, using ToolRunner.run method. This works fine on a local job. However when distributed i get java.lang.NoClassDefFoundError: org/rosuda/REngine/Rserve/RserveException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) at java.lang.Class.getConstructor0(Class.java:2699) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74) at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) Now, i have a CLASSPATH entry in my .bashrc which contains the location of Rserve.jar (which contains the above classes), yet I still get the above error. Do I have to somehow emulate RunJar.main or is there a simpler way out? Thank you Saptarshi -- Saptarshi Guha - saptarshi.g...@gmail.com
Re: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app)
Hello, What if each machine has the identical directory structure and Rserve.jar is present at the same place(i.e the entry in the classpath is identical across machines). The will the latter suggestion work? On Thu, Dec 18, 2008 at 9:02 PM, Zheng Shao zs...@facebook.com wrote: You either need to add the your Rserve.jar to -libjars, or put Rserve.jar in a cluster-accessible NFS mount (and add it to CLASSPATH/HADOOP_CLASSPATH). Zheng -Original Message- From: Saptarshi Guha [mailto:saptarshi.g...@gmail.com] Sent: Thursday, December 18, 2008 3:38 PM To: core-user@hadoop.apache.org Subject: Classes Not Found even when classpath is mentioned (Starting mapreduce from another app) Hello, I intend to start a mapreduce job from another java app, using ToolRunner.run method. This works fine on a local job. However when distributed i get java.lang.NoClassDefFoundError: org/rosuda/REngine/Rserve/RserveException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) at java.lang.Class.getConstructor0(Class.java:2699) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74) at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:402) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) Now, i have a CLASSPATH entry in my .bashrc which contains the location of Rserve.jar (which contains the above classes), yet I still get the above error. Do I have to somehow emulate RunJar.main or is there a simpler way out? Thank you Saptarshi -- Saptarshi Guha - saptarshi.g...@gmail.com -- Saptarshi Guha - saptarshi.g...@gmail.com
