I think EMR has its own limitation e.g. I want to setup hadoop 2.6.0 with kerberos + hive-1.2.0 to test my hive patch.
How EMR can help me? it supports hadoop up to 2.4.0 (not even 2.4.1) http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-hadoop-version.html On Thu, Mar 5, 2015 at 9:51 PM, Jonathan Aquilina <jaquil...@eagleeyet.net> wrote: > Hi guys I know you guys want to keep costs down, but why go through all > the effort to setup ec2 instances when you deploy EMR it takes the time to > provision and setup the ec2 instances for you. All configuration then for > the entire cluster is done on the master node of the particular cluster or > setting up of additional software that is all done through the EMR console. > We were doing some geospatial calculations and we loaded a 3rd party jar > file called esri into the EMR cluster. I then had to pass a small bootstrap > action (script) to have it distribute esri to the entire cluster. > > Why are you guys reinventing the wheel? > > > > --- > Regards, > Jonathan Aquilina > Founder Eagle Eye T > > On 2015-03-06 03:35, Alexander Pivovarov wrote: > > I found the following solution to this problem > > I registered 2 subdomains (public and local) for each computer on > https://freedns.afraid.org/subdomain/ > e.g. > myhadoop-nn.crabdance.com > myhadoop-nn-local.crabdance.com > > then I added cron job which sends http requests to update public and local > ip on freedns server > hint: public ip is detected automatically > ip address for local name can be set using request parameter &address=10.x.x.x > (don't forget to escape &) > > as a result my nn computer has 2 DNS names with currently assigned ip > addresses , e.g. > myhadoop-nn.crabdance.com 54.203.181.177 > myhadoop-nn-local.crabdance.com 10.220.149.103 > > in hadoop configuration I can use local machine names > to access my cluster outside of AWS I can use public names > > Just curious if AWS provides easier way to name EC2 computers? > > On Thu, Mar 5, 2015 at 5:19 PM, Jonathan Aquilina <jaquil...@eagleeyet.net > > wrote: > >> I dont know how you would do that to be honest. With EMR you have >> destinctions master core and task nodes. If you need to change >> configuration you just ssh into the EMR master node. >> >> >> >> --- >> Regards, >> Jonathan Aquilina >> Founder Eagle Eye T >> >> On 2015-03-06 02:11, Alexander Pivovarov wrote: >> >> What is the easiest way to assign names to aws ec2 computers? >> I guess computer need static hostname and dns name before it can be used >> in hadoop cluster. >> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquil...@eagleeyet.net> >> wrote: >> >>> When I started with EMR it was alot of testing and trial and error. >>> HUE is already supported as something that can be installed from the AWS >>> console. What I need to know is if you need this cluster on all the time or >>> this is goign ot be what amazon call a transient cluster. Meaning you fire >>> it up run the job and tear it back down. >>> >>> >>> >>> --- >>> Regards, >>> Jonathan Aquilina >>> Founder Eagle Eye T >>> >>> On 2015-03-06 01:10, Krish Donald wrote: >>> >>> Thanks Jonathan, >>> >>> I will try to explore EMR option also. >>> Can you please let me know the configuration which you have used it? >>> Can you please recommend for me also? >>> I would like to setup Hadoop cluster using cloudera manager and then >>> would like to do below things: >>> >>> setup kerberos >>> setup federation >>> setup monitoring >>> setup hadr >>> backup and recovery >>> authorization using sentry >>> backup and recovery of individual componenets >>> performamce tuning >>> upgrade of cdh >>> upgrade of CM >>> Hue User Administration >>> Spark >>> Solr >>> >>> >>> Thanks >>> Krish >>> >>> >>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina < >>> jaquil...@eagleeyet.net> wrote: >>> >>>> krish EMR wont cost you much with all the testing and data we ran >>>> through the test systems as well as the large amont of data when everythign >>>> was read we paid about 15.00 USD. I honestly do not think that the specs >>>> there would be enough as java can be pretty ram hungry. >>>> >>>> >>>> >>>> --- >>>> Regards, >>>> Jonathan Aquilina >>>> Founder Eagle Eye T >>>> >>>> On 2015-03-06 00:41, Krish Donald wrote: >>>> >>>> Hi, >>>> >>>> I am new to AWS and would like to setup Hadoop cluster using cloudera >>>> manager for 6-7 nodes. >>>> >>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ? >>>> I would like to use free service as of now. >>>> >>>> Please advise. >>>> >>>> Thanks >>>> Krish >>>> >>>>
