Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-07 Thread Konstantin Boudnik
On Fri, Oct 07, 2011 at 10:17AM, Steve Loughran wrote: > On 06/10/2011 17:49, milind.bhandar...@emc.com wrote: >> Steve, >> >>> Summary: I'm not sure that HDFS is the right FS in this world, as it >>> contains a lot of assumptions about system stability and HDD persistence >>> that aren't valid any

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-07 Thread Milind.Bhandarkar
Steve, > >you can improve Hadoop to make it more agile; my defunct Hadoop >lifecycle branch did a lot of that, but you have to have everyone else >using Hadoop to be willing to let the changes go in -and those changes >mustn't impose a cost or risk to the physical cluster model. Until Hadoop 0.

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-07 Thread Steve Loughran
On 06/10/2011 17:49, milind.bhandar...@emc.com wrote: Steve, Summary: I'm not sure that HDFS is the right FS in this world, as it contains a lot of assumptions about system stability and HDD persistence that aren't valid any more. With the ability to plug in new placers you could do tricks like

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-06 Thread Milind.Bhandarkar
Steve, >Summary: I'm not sure that HDFS is the right FS in this world, as it >contains a lot of assumptions about system stability and HDD persistence >that aren't valid any more. With the ability to plug in new placers you >could do tricks like ensure 1 replica lives in a persistent blockstore >(

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-06 Thread Steve Loughran
On 06/10/11 06:40, Jagane Sundar wrote: On Wed, Oct 5, 2011 at 10:09 PM, Konstantin Boudnik wrote: On Wed, Oct 05, 2011 at 07:00PM, Jagane Sundar wrote: approaches you are familiar with. Chef/Puppet et. al. are not interesting to Is this a technical lack of interest as in these solutions do

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-06 Thread Steve Loughran
On 06/10/11 03:00, Jagane Sundar wrote: Thanks for your input, Milind. It's very useful and interesting. In the interest of brevity, I have truncated most of it except for the point regarding 'cloud friendly'. I have done some research into this, and want to get some more community feedback. 2

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Konstantin Boudnik
On Wed, Oct 05, 2011 at 10:40PM, Jagane Sundar wrote: > On Wed, Oct 5, 2011 at 10:09 PM, Konstantin Boudnik wrote: > > > On Wed, Oct 05, 2011 at 07:00PM, Jagane Sundar wrote: > > > approaches you are familiar with. Chef/Puppet et. al. are not interesting > > to > > > > Is this a technical lack of

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Jagane Sundar
On Wed, Oct 5, 2011 at 10:09 PM, Konstantin Boudnik wrote: > On Wed, Oct 05, 2011 at 07:00PM, Jagane Sundar wrote: > > approaches you are familiar with. Chef/Puppet et. al. are not interesting > to > > Is this a technical lack of interest as in these solutions do not perform > as > you expect the

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Konstantin Boudnik
On Wed, Oct 05, 2011 at 07:00PM, Jagane Sundar wrote: > approaches you are familiar with. Chef/Puppet et. al. are not interesting to Is this a technical lack of interest as in these solutions do not perform as you expect them or this is a policy thing of some kind? > turned out to be slow as sh**

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Roman Shaposhnik
On Wed, Oct 5, 2011 at 7:00 PM, Jagane Sundar wrote: > As far as deployment automation is concerned, I am eager to know what other > approaches you are familiar with. Chef/Puppet et. al. are not interesting to > me. I want this to have end user self-serve service characteristics, not > 'end users

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Roman Shaposhnik
Hi Jagane! On Wed, Oct 5, 2011 at 4:20 PM, Jagane Sundar wrote: > For example, if we had a distro with support for the following features: At the risk of repeating myself I'd like to point out that even though your ideal distro my not exist at the moment we, as a community, have all the tools in

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Jagane Sundar
Thanks for your input, Milind. It's very useful and interesting. In the interest of brevity, I have truncated most of it except for the point regarding 'cloud friendly'. I have done some research into this, and want to get some more community feedback. >2. Built in support for the cloud. (Whirr i

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Milind.Bhandarkar
Jagane, I understand your use case, I think, and so here are my thoughts, inline: >1. Hbase support, i.e. working scale tested Append and Hflush in HDFS Absolutely. Hbase (and other components of the stack that do not follow the MapReduce paradigm) are increasingly important. It is important to

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Jagane Sundar
Hello Milind, A large part of why I sent this email out was to initiate a discussion of the priority of specific features in a Hadoop distro. For example, if we had a distro with support for the following features: 1. Hbase support, i.e. working scale tested Append and Hflush in HDFS 2. Built in

Re: Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-05 Thread Milind.Bhandarkar
Jagane, I think you have forgotten one major deciding factor: Which version is *your* vendor committed to support ? If you are at the same place where you were the last time we met, you have no other choice but to go with 0.20.206. It's in the contract ! :-) - Milind --- Milind Bhandarkar Gree

Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-03 Thread Jagane Sundar
Hello Hadoop experts, I would like to solicit your input in answering this question. Which proposed distro of Hadoop, 0.20.206 or 0.22, is likely to be the better platform for hosting HBase? My requirements are as follows: 1. The Hadoop must support both HBase and MR jobs in the same cluster

Which proposed distro of Hadoop, 0.20.206 or 0.22, will be better for HBase?

2011-10-02 Thread Jagane Sundar
Hello Hadoop experts, I would like to solicit your input in answering this question. Which proposed distro of Hadoop, 0.20.206 or 0.22, is likely to be the better platform for hosting HBase? My requirements are as follows: 1. The Hadoop must support both HBase and MR jobs in the same cluster. At