Re: Hadoop + Elastic Block Stores
Does anybody have performance statistics on running dfs on EBS instead of local disk ? I think one of the interesting questions would be what is the sustained through-put of EBS. Some general questions on DFS - is the DFS data replicated to more than 1 node ? Has anybody tried running DFS entirely in-memory ? Regards, Alan Ho - Original Message From: Doug Cutting [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Monday, September 8, 2008 10:25:24 AM Subject: Re: Hadoop + Elastic Block Stores Ryan LeCompte wrote: I'd really love to one day see some scripts under src/contrib/ec2/bin that can setup/mount the EBS volumes automatically. :-) The fastest way might be to write contribute such scripts! Doug __ Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your favourite sites. Download it now at http://ca.toolbar.yahoo.com.
Re: Hadoop + Elastic Block Stores
Ryan LeCompte wrote: I'd really love to one day see some scripts under src/contrib/ec2/bin that can setup/mount the EBS volumes automatically. :-) The fastest way might be to write contribute such scripts! Doug
Hadoop + Elastic Block Stores
Hello, I was wondering if anyone has gotten far at all with getting Hadoop up and running with EC2 + EBS? Any luck getting this to work in a way that the HDFS runs on the EBS so that it isn't blown away every time you bring up/down the EC2 Hadoop cluster? I'd like to experiment with this next, and was curious if anyone had any luck. :) Thanks! Ryan
Re: Hadoop + Elastic Block Stores
Ryan, I currently have a Hadoop/HBase setup that uses EBS. It works but using EBS implied an additional overhead of configuration (too bad you can't spawn instances with volumes already attached to it tho I'm sure that'll come). Shutting down instances and bringing others up also requires more micro-management but I think Tom White wrote about it and there was a link to it in another discussion you were part of. Hope this helps, J-D On Fri, Sep 5, 2008 at 7:00 PM, Ryan LeCompte [EMAIL PROTECTED] wrote: Hello, I was wondering if anyone has gotten far at all with getting Hadoop up and running with EC2 + EBS? Any luck getting this to work in a way that the HDFS runs on the EBS so that it isn't blown away every time you bring up/down the EC2 Hadoop cluster? I'd like to experiment with this next, and was curious if anyone had any luck. :) Thanks! Ryan
Re: Hadoop + Elastic Block Stores
Good to know that you got it up and running. I'd really love to one day see some scripts under src/contrib/ec2/bin that can setup/mount the EBS volumes automatically. :-) On Sep 5, 2008, at 11:38 PM, Jean-Daniel Cryans [EMAIL PROTECTED] wrote: Ryan, I currently have a Hadoop/HBase setup that uses EBS. It works but using EBS implied an additional overhead of configuration (too bad you can't spawn instances with volumes already attached to it tho I'm sure that'll come). Shutting down instances and bringing others up also requires more micro-management but I think Tom White wrote about it and there was a link to it in another discussion you were part of. Hope this helps, J-D On Fri, Sep 5, 2008 at 7:00 PM, Ryan LeCompte [EMAIL PROTECTED] wrote: Hello, I was wondering if anyone has gotten far at all with getting Hadoop up and running with EC2 + EBS? Any luck getting this to work in a way that the HDFS runs on the EBS so that it isn't blown away every time you bring up/down the EC2 Hadoop cluster? I'd like to experiment with this next, and was curious if anyone had any luck. :) Thanks! Ryan
