Dear Jonathan,
Would you please describe the process of running EMR based Hadoop for
$15.00, I tried and my cost were rocketing like $60 for one hour.
Regards
On 05/03/2015 23:57, Jonathan Aquilina wrote:
krish EMR wont cost you much with all the testing and data we ran through
the test
When i was testing I was using default setup 1 master node 2 core and no
task nodes. i would spiin up the cluster then terminate it. The term for
that is a transient cluster.
When the big data was needing to be crunched i changed the setup a bit.
An Important note there is a limitation of 20
@jonathan,
I totaly agree that this is reinventing the wheel, but think about the
folks who wants to do this setup from scratch to better under hadoop or
maybe those folks who are going to do admin realted work...and hence the
need to setting is up from scratch...
@alexandar,
Yes you are right,
krish EMR wont cost you much with all the testing and data we ran
through the test systems as well as the large amont of data when
everythign was read we paid about 15.00 USD. I honestly do not think
that the specs there would be enough as java can be pretty ram hungry.
---
Regards,
Jonathan
Thanks Jonathan,
I will try to explore EMR option also.
Can you please let me know the configuration which you have used it?
Can you please recommend for me also?
I would like to setup Hadoop cluster using cloudera manager and then would
like to do below things:
setup kerberos
setup federation
Hi,
I am new to AWS and would like to setup Hadoop cluster using cloudera
manager for 6-7 nodes.
t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
I would like to use free service as of now.
Please advise.
Thanks
Krish
what about DNS?
if you have 2 computers (nn and dn) how nn knows dn ip?
The script puts only this computer ip to /etc/hosts
On Thu, Mar 5, 2015 at 6:39 PM, max scalf oracle.bl...@gmail.com wrote:
Here is a easy way to go about assigning static name to your ec2
instance. When you get the
I think EMR has its own limitation
e.g. I want to setup hadoop 2.6.0 with kerberos + hive-1.2.0 to test my
hive patch.
How EMR can help me? it supports hadoop up to 2.4.0 (not even 2.4.1)
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-hadoop-version.html
On
The only limitation I know is that of how many nodes you can have and
how many instances of that particular size the host is on can support.
you can load hive in EMR and then any other features of the cluster are
managed at the master node level as you have SSH access there.
What are the
Do a reverse lookup and use the name you find. There are a few areas
of Hadoopo that require reverse name lookup, but in general just
create relevant entries (shared across the cluster, e.g. via Ansible
if more than just a few nodes) in /etc/hosts.
Not hard.
On Thu, Mar 5, 2015 at 6:35 PM,
What is the easiest way to assign names to aws ec2 computers?
I guess computer need static hostname and dns name before it can be used in
hadoop cluster.
On Mar 5, 2015 4:36 PM, Jonathan Aquilina jaquil...@eagleeyet.net wrote:
When I started with EMR it was alot of testing and trial and error.
Here is a easy way to go about assigning static name to your ec2 instance.
When you get the launch an EC2-instance from aws console when you get to
the point of selecting VPC, ip address screen there is a screen that says
USER DATA...put the below in with appropriate host name(change
ok, how we can easily put all hadoop computer names and IPs to /etc/hosts
on all computers?
Do you have a script? or I need manually go to each computer, get its ip
and put it to /etc/hosts and then distribute /etc/hosts to all machines?
Don't you think one time effort to configure freedns is
I dont know how you would do that to be honest. With EMR you have
destinctions master core and task nodes. If you need to change
configuration you just ssh into the EMR master node.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-06 02:11, Alexander Pivovarov wrote:
What is
unfortunately without DNS you have to rely on /etc/hosts, so put in entry
for all your nodes(nn,snn,dn1,dn2 etc..) on all nodes(/etc/hosts file) and
i have that tested for hortonworks(using ambari) and cloudera manager and i
am certainly sure it will work for MapR
On Thu, Mar 5, 2015 at 8:47 PM,
Hi guys I know you guys want to keep costs down, but why go through all
the effort to setup ec2 instances when you deploy EMR it takes the time
to provision and setup the ec2 instances for you. All configuration then
for the entire cluster is done on the master node of the particular
cluster or
When I started with EMR it was alot of testing and trial and error. HUE
is already supported as something that can be installed from the AWS
console. What I need to know is if you need this cluster on all the time
or this is goign ot be what amazon call a transient cluster. Meaning you
fire it
17 matches
Mail list logo
