MiniMRCluster not behaving in hadoop 1.0.4

i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the unit tests used to run fine but are now misbehaving. i do a simple setup in the unit test like this: private static MiniDFSCluster dfsCluster; private static FileSystem fs; private static MiniMRCluster mrCluster;

Re: error running pi program

Hi, Ted and Andy I tried both internal and external hostnames. They both worked. But I will elect external host name since this way I can browsing the hdfs file structure and other information from a computer not in the cluster. Thanks! On Fri, Nov 9, 2012 at 1:03 PM, Kartashov, Andy wrote: >

Re: Erro running pi programm

Hi, Andy and Ted Thanks a lot for pointing out my silly error. The problem did go away after I change the hostname from 'master' to the result of “hostname –f”. Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name

Using LZ4_HC in production workload

Hello everyone, Is anyone using LZ4 extensively in production Hadoop workloads? I have a use case that seems like a very good fit for LZ4_HC but since it doesn't appear to have been widely adopted yet, I'm not sure if the headaches associated with using it will be worthwhile. Any information abo

error running pi program

Hello Ted, Thanks yours below. I do run hadoop on EC2 as well. And I use internal host-addresses ($hostname -f) for configuration of core-site/hdfs-site/slaves/master files. The only time I switch to external address i.e. ec2-.. is when I want to access my nodes vie the web,i.e. external

Re: Erro running pi programm

yinghua, One other thing to note here is that on EC2 computer are given two hostnames, one internal and one external, you need to use the external. These host names are listed in the ec2 configuration pane on the AWS management console. Ted Reynolds Technical Support Engineer Hortonworks Work P

Re: mr unit pom file

Yup, your right the mrunit pom file is specified here: https://cwiki.apache.org/confluence/display/MRUNIT/Index%3bjsessionid=C4B0ADDF6B88BBEFCF60136A62441A12 too bad that it needs a classifier :( Unfortunately, it makes the http://mvnrepository.com/artifact/org.apache.mrunit/mrunit/0.9.0-incuba

Re: mr unit pom file

I think you missing the classifier: http://cwiki.apache.org/confluence/display/MRUNIT Brock On Fri, Nov 9, 2012 at 2:33 PM, Matt Goeke wrote: > Have you tried any of the other versions listed in the link below? I haven't > had to pull it down in a while but the dependency block you pasted seems

Re: mr unit pom file

Have you tried any of the other versions listed in the link below? I haven't had to pull it down in a while but the dependency block you pasted seems to match up with at least the latest version. http://mvnrepository.com/artifact/org.apache.mrunit/mrunit -- Matt On Fri, Nov 9, 2012 at 2:26 PM,

Re: mapreduce.job.end-notification settings

Hey Steven, Nice catch - this may be a bug we should fix. There are 4 relevant properties in MR2: Attempts: "mapreduce.job.end-notification.retry.attempts" and "mapreduce.job.end-notification.max.attempts". Retries: "mapreduce.job.end-notification.retry.interval" and "mapreduce.job.end-notifica

Re: Sticky Bit Problem (CDH4.1)

Thanks for following up Brian. I'll try to reproduce this out with some more version hints from your post just in case we aren't missing anything bad. Also, in future, if your problem feels CDH-specific, please do mail the cdh-u...@cloudera.org lists instead of the Apache ones (Groups: https://gro

RE: Erro running pi programm

Try running "hostname -f" on each node, take a note of the fully qualified host address and replace your "master" with the your respective finding. Here are my configuration files core-site.xml fs.default.name hdfs://master:9000

Re: Erro running pi programm

Here are my configuration files core-site.xml fs.default.name hdfs://master:9000 hadoop.tmp.dir /usr/local/hadoop/tmp mapred-site.xml mapreduce.framework.name

Re: Erro running pi programm

Hi, Andy Thanks for suggestions! I am running it on a four node cluster on EC2. All the services started fine, Namenode, Datanode, ResourceManager, NodeManager and JobHistoryServer. Each node can ssh to all the nodes without problem. But problem appears when trying to run any job. On Fri, No

mapreduce.job.end-notification settings

I was looking at: http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml And I noticed that there are some duplicate properties with different values and different descriptions: mapreduce.job.end-notification.retry.attempts 0 Indica

Erro running pi programm

Yinghua, What mode are you running your hadoop in: Local/Pseud/Fully...? Your hostname is not recognised Your configuration setting seems to be wrong. Hi, all Could some help looking at this problem? I am setting up a four node cluster on EC2 and seems that the cluster is set up fine unti

Re: Sticky Bit Problem (CDH4.1)

Very possible it's a mixed version thing, though I'm not sure where in the mess we created it would be. :) As far as I can tell, we were never using Java version 7. Most of these complications arose when we attempted to upgrade to CDH 4.1 via Puppet. Somewhere along the line things were installed

Error running the pi program on a 4 node cluster

Hi, all Could some help looking at this problem? I am setting up a four node cluster on EC2 and seems that the cluster is set up fine until I start testing. I have tried password-less ssh from each node to all the nodes and there is no problem connecting. Any advice will be greatly appreciated!

Re: notorious impersonation ERROR - SOLVED

Andy, For future reference there is a list dedicated to Oozie ( u...@oozie.apache.org) that is full of people who have worked through issues like these. Feel free to ping that next time you have a question specific to the Oozie project. -- Matt On Fri, Nov 9, 2012 at 10:06 AM, Kartashov, Andy w

Re: a few questions

Inline: HTH On Fri, Nov 9, 2012 at 10:13 AM, Kartashov, Andy wrote: > Guys, > > A few questions please. > > 1. When I tried to run Oozie examples I was told to copy to copy /examples > folder into HDFS. However when I tried to run oozie job I was told that the > source file was not found. Well,

a few questions

Guys, A few questions please. 1. When I tried to run Oozie examples I was told to copy to copy /examples folder into HDFS. However when I tried to run oozie job I was told that the source file was not found. Well, until I cd'ed into the local directory on Linux and re-run the job successfully.

RE: notorious impersonation ERROR - SOLVED

Guys, OK, this is what you need to do to enable Oozie impersonation of a User. Forget about modifying oozie-site.xml. The only file I needed to modify was core-site.xml. Explanation: The superuser must be configured on namenode and jobtracker to be allowed to impersonate another user. Followi

Re: notorious impersonation ERROR

Not sure, haven't got that far yet ;) On Nov 9, 2012, at 9:27 AM, "Kartashov, Andy" wrote: > Oleg, > > Thanks. I had added oozie to the hadoop group. Do I also need to restart NN > daemon? > > > -Original Message- > From: Oleg Zhurakousky [mailto:oleg.zhurakou...@gmail.com] > Sent: F

RE: notorious impersonation ERROR

Oleg, Thanks. I had added oozie to the hadoop group. Do I also need to restart NN daemon? -Original Message- From: Oleg Zhurakousky [mailto:oleg.zhurakou...@gmail.com] Sent: Friday, November 09, 2012 9:24 AM To: user@hadoop.apache.org Subject: Re: notorious impersonation ERROR Have you

Re: notorious impersonation ERROR

Have you tried this? Let's say your oozie user is 'oozie' and Name node group is called hadoop sudo adduser --ingroup hadoop oozie Oleg On Nov 9, 2012, at 9:20 AM, "Kartashov, Andy" wrote: > Guys, > > Came across this error like many others who tried to run Ooozie examples. > Searched and r

notorious impersonation ERROR

Guys, Came across this error like many others who tried to run Ooozie examples. Searched and read bunch of posts on this topic. Even came across Harsh's response stipulating that oozie user must be added to the user group on the name node but it wasn't explained how. Any insight please? Thnks,