ok, how we can easily put all hadoop computer names and IPs to /etc/hosts
on all computers?
Do you have a script? or I need manually go to each computer, get its ip
and put it to /etc/hosts and then distribute /etc/hosts to all machines?
Don't you think one time effort to configure freedns is easier?
freedns solution works with AWS spot-instances as well.
You need to create snapshot after you configure freedns, hadoop, etc on
particular box.
Next time you need computer you can can go to your saved snapshots and
create spot-instance from it.
On Thu, Mar 5, 2015 at 6:54 PM, max scalf <oracle.bl...@gmail.com> wrote:
> unfortunately without DNS you have to rely on /etc/hosts, so put in entry
> for all your nodes(nn,snn,dn1,dn2 etc..) on all nodes(/etc/hosts file) and
> i have that tested for hortonworks(using ambari) and cloudera manager and i
> am certainly sure it will work for MapR
>
> On Thu, Mar 5, 2015 at 8:47 PM, Alexander Pivovarov <apivova...@gmail.com>
> wrote:
>
>> what about DNS?
>> if you have 2 computers (nn and dn) how nn knows dn ip?
>>
>> The script puts only this computer ip to /etc/hosts
>>
>> On Thu, Mar 5, 2015 at 6:39 PM, max scalf <oracle.bl...@gmail.com> wrote:
>>
>>> Here is a easy way to go about assigning static name to your ec2
>>> instance. When you get the launch an EC2-instance from aws console when
>>> you get to the point of selecting VPC, ip address screen there is a screen
>>> that says "USER DATA"...put the below in with appropriate host name(change
>>> CHANGE_HOST_NAME_HERE to whatever you want) and that should be able to get
>>> you static name.
>>>
>>> #!/bin/bash
>>>
>>> HOSTNAME_TAG=CHANGE_HOST_NAME_HERE
>>> cat > /etc/sysconfig/network << EOF
>>> NETWORKING=yes
>>> NETWORKING_IPV6=no
>>> HOSTNAME=${HOSTNAME_TAG}
>>> EOF
>>>
>>> IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
>>> echo "${IP} ${HOSTNAME_TAG}.localhost ${HOSTNAME_TAG}" >> /etc/hosts
>>>
>>> echo ${HOSTNAME_TAG} > /proc/sys/kernel/hostname
>>> service network restart
>>>
>>>
>>> Also note i was able to do this on couple of spot instance for cheap
>>> price, only thing is once you shut it down or someone outbids you, you
>>> loose that instance but its easy/cheap to play around with.... and i have
>>> used couple of m3.medium for my NN/SNN and couple of them for data nodes...
>>>
>>> On Thu, Mar 5, 2015 at 7:19 PM, Jonathan Aquilina <
>>> jaquil...@eagleeyet.net> wrote:
>>>
>>>> I dont know how you would do that to be honest. With EMR you have
>>>> destinctions master core and task nodes. If you need to change
>>>> configuration you just ssh into the EMR master node.
>>>>
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Jonathan Aquilina
>>>> Founder Eagle Eye T
>>>>
>>>> On 2015-03-06 02:11, Alexander Pivovarov wrote:
>>>>
>>>> What is the easiest way to assign names to aws ec2 computers?
>>>> I guess computer need static hostname and dns name before it can be
>>>> used in hadoop cluster.
>>>> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <jaquil...@eagleeyet.net>
>>>> wrote:
>>>>
>>>>> When I started with EMR it was alot of testing and trial and error.
>>>>> HUE is already supported as something that can be installed from the AWS
>>>>> console. What I need to know is if you need this cluster on all the time
>>>>> or
>>>>> this is goign ot be what amazon call a transient cluster. Meaning you fire
>>>>> it up run the job and tear it back down.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Jonathan Aquilina
>>>>> Founder Eagle Eye T
>>>>>
>>>>> On 2015-03-06 01:10, Krish Donald wrote:
>>>>>
>>>>> Thanks Jonathan,
>>>>>
>>>>> I will try to explore EMR option also.
>>>>> Can you please let me know the configuration which you have used it?
>>>>> Can you please recommend for me also?
>>>>> I would like to setup Hadoop cluster using cloudera manager and then
>>>>> would like to do below things:
>>>>>
>>>>> setup kerberos
>>>>> setup federation
>>>>> setup monitoring
>>>>> setup hadr
>>>>> backup and recovery
>>>>> authorization using sentry
>>>>> backup and recovery of individual componenets
>>>>> performamce tuning
>>>>> upgrade of cdh
>>>>> upgrade of CM
>>>>> Hue User Administration
>>>>> Spark
>>>>> Solr
>>>>>
>>>>>
>>>>> Thanks
>>>>> Krish
>>>>>
>>>>>
>>>>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina <
>>>>> jaquil...@eagleeyet.net> wrote:
>>>>>
>>>>>> krish EMR wont cost you much with all the testing and data we ran
>>>>>> through the test systems as well as the large amont of data when
>>>>>> everythign
>>>>>> was read we paid about 15.00 USD. I honestly do not think that the specs
>>>>>> there would be enough as java can be pretty ram hungry.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Regards,
>>>>>> Jonathan Aquilina
>>>>>> Founder Eagle Eye T
>>>>>>
>>>>>> On 2015-03-06 00:41, Krish Donald wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am new to AWS and would like to setup Hadoop cluster using cloudera
>>>>>> manager for 6-7 nodes.
>>>>>>
>>>>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
>>>>>> I would like to use free service as of now.
>>>>>>
>>>>>> Please advise.
>>>>>>
>>>>>> Thanks
>>>>>> Krish
>>>>>>
>>>>>>
>>>
>>
>
