Jingchen Liu created WHIRR-668:
----------------------------------
Summary: The hadoop data folder /data0 not linked to /mnt in an
EBS instance
Key: WHIRR-668
URL: https://issues.apache.org/jira/browse/WHIRR-668
Project: Whirr
Issue Type: Bug
Components: service/hadoop
Affects Versions: 0.8.0
Environment: My local machine is Ubuntu 12.04 64bit.
Trying to create hadoop cluster in AWS EC2.
Reporter: Jingchen Liu
I tried to create a hadoop cluster in EC2 using Whirr. I used an image of EBS
type. When I access the Namenode Web UI I found the 'Configured Capacity' is
small. It seems that the /data0 folder is not linked to the /mnt correctly, so
that the large storage device of the EC2 instance is not used by hadoop.
How to reproduce this problem:
I found the images here: [http://cloud.ubuntu.com/ami/]
I used the image ami-ce3283cf (Ubuntu lucid 64bit ebs in ap-northeast-1)
My hadoop.properties file is like this:
{code:title=hadoop.properties|borderStyle=solid}
whirr.provider=aws-ec2
whirr.location-id=ap-northeast-1a
whirr.cluster-name=hadoopcluster
whirr.cluster-user=${sys:user.name}
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,12
hadoop-datanode+hadoop-tasktracker
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${whirr.private-key-file}.pub
whirr.hardware-id=c1.medium
whirr.image-id=ap-northeast-1/ami-ce3283cf
{code}
Then I ran the command:
{noformat}
bin/whirr launch-cluster --config hadoop.properties
{noformat}
Sometimes the error 'java: command not found' come out during the procedure. In
this case, if you logged into the master instance and run the command 'hadoop',
it says 'hadoop' is not found too. The namenode Web UI is also not available.
But sometimes, the procedure can success. I can logged into the master instance
and run the hadoop commands. I can also access the Namenode Web UI. But I saw
the 'Configured Capacity' is only about 80GB.
If I change the image to an instance-store type one: ami-be3283bf (Ubuntu lucid
64bit instance-store in ap-northeast-1), all other configs not changed, it all
works fine. The 'Configured Capacity' became 3.91TB.
So I think the problem comes from the EBS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
