Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hi Alejandro, On Aug 28, 2012, at 8:50 PM, Alejandro Abdelnur wrote: Chris, thanks for initiating the discussion. No probs! IMO a pre-requisite to this is to figure out how we'll handle the following: To be honest, I don't think any of the below are prereqs. They are technical issues

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Robert Evans
I personally am for splitting up the projects. I think there is a lot of potential that each of the projects could have on their own, and I expect to see them evolve in new and interesting ways when the projects are not tied directly together. But, in order to get there we need to address the

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Suresh Srinivas
I am +1 for splitting up the projects. This is the step in the right direction. There will be challenges along the way. I am confident we can solve them. Robert and Alejandro have brought up good questions. Here are my thoughts: - For first one or two releases all the projects can coordinate and

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Arun C Murthy
On Aug 29, 2012, at 10:02 AM, Suresh Srinivas wrote: I am +1 for splitting up the projects. This is the step in the right direction. There will be challenges along the way. I am confident we can solve them. Robert and Alejandro have brought up good questions. Here are my thoughts: - For

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Alejandro Abdelnur
On Wed, Aug 29, 2012 at 10:02 AM, Suresh Srinivas sur...@hortonworks.com wrote: - As regards to APIs, currently we have LimitedPrivate APIs for related projects. This has been used by HBase as well. We need to think about a timeline by when we can mark these APIs stable. They should remain

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hi Bobby, On Aug 29, 2012, at 8:17 AM, Robert Evans wrote: I personally am for splitting up the projects. I think there is a lot of potential that each of the projects could have on their own, and I expect to see them evolve in new and interesting ways when the projects are not tied

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Suresh Srinivas
- I agree with Arun that the common can move with HDFS. So, this would mean that a bunch of common functionality needed by other TPLs (YARN, MR, HBASE) which is not required by HDFS will end up in HDFS. I'm not necessary against that but it should be well understood/expected/accepted by

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Tom White
On Wed, Aug 29, 2012 at 5:31 PM, Arun C Murthy a...@hortonworks.com wrote: On Aug 28, 2012, at 8:50 PM, Alejandro Abdelnur wrote: Chris, thanks for initiating the discussion. Likewise, thanks Chris! IMO a pre-requisite to this is to figure out how we'll handle the following: Good

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Eric Baldeschwieler
Hi Tom, There are also Hadoop tools like distcp, Hadoop archives, Streaming, etc, which should go with MapReduce. Good point. I agree. The alternative would be to have a Common TLP, which we shouldn't necessarily dismiss, since more important than the size of the codebase is that there's

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Arun C Murthy
On Aug 29, 2012, at 10:04 AM, Arun C Murthy wrote: On Aug 29, 2012, at 10:02 AM, Suresh Srinivas wrote: Robert and Alejandro have brought up good questions. Here are my thoughts: - For first one or two releases all the projects can coordinate and do the releases together. This should help

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Jun Ping Du
Hi Chris and all, Thanks for initiating the discussion. Can I say something in a prospective of contributor but not a committer or PMC member? First, I have a feeling that current hadoop project process is good for contributors to deliver a bug fix but not so easy to deliver a big feature.

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Konstantin Boudnik
Another way around is to produce more than one common's artifacts that will provide some logic split for the downstream projects like MR, and so on. Cos On Wed, Aug 29, 2012 at 10:26AM, Suresh Srinivas wrote: - I agree with Arun that the common can move with HDFS. So, this would mean that

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Eli Collins
Thanks for writing up a proposal Chris. I think it makes sense to have Common live in HDFS at least for now, since it's at the bottom of the stack / dependency chain and it's code is the most intertwined with common, and, per Arun, we tend to work on common stuff more than MR people. The HDFS

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Arun C Murthy
On Aug 28, 2012, at 7:33 PM, Mattmann, Chris A (388J) wrote: Process: 0. [DISCUSS] thread for TLP name in which you talk about #1 and #2 below, potentially draft resolution too. How about something like this... please provide your feedback. This is a very early draft, I'll post this

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Eli Collins
On Wed, Aug 29, 2012 at 11:22 AM, Arun C Murthy a...@hortonworks.com wrote: On Aug 29, 2012, at 10:04 AM, Arun C Murthy wrote: On Aug 29, 2012, at 10:02 AM, Suresh Srinivas wrote: Robert and Alejandro have brought up good questions. Here are my thoughts: - For first one or two releases all

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Alejandro Abdelnur
I volunteer to help cleanup/normalize Maven stuff. Thx On Wed, Aug 29, 2012 at 1:34 PM, Tom White t...@cloudera.com wrote: Eric - I agree with Common being included in HDFS. That's what I meant by Common not having a clear enough mission to be a TLP by itself. Arun - I'm happy to RM some of

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Travis Thompson
+1 not that anyone knows who I am :) On Aug 29, 2012, at 2:22 PM, Jakob Homan wrote: Let's just embrace contention as a fact of life on a high-profile high-stakes project and get back to work. +1

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hi Eli, On Aug 29, 2012, at 11:41 AM, Eli Collins wrote: Thanks for writing up a proposal Chris. NP. I think it makes sense to have Common live in HDFS at least for now, since it's at the bottom of the stack / dependency chain and it's code is the most intertwined with common, and, per

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Arun, great work below. Concrete, and an actual proposal of PMC lists. What do folks think? Cheers, Chris On Aug 29, 2012, at 11:48 AM, Arun C Murthy wrote: On Aug 28, 2012, at 7:33 PM, Mattmann, Chris A (388J) wrote: Process: 0. [DISCUSS] thread for TLP name in which you talk about

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Konstantin Boudnik
On Wed, Aug 29, 2012 at 11:19PM, Mattmann, Chris A (388J) wrote: Hi Eli, On Aug 29, 2012, at 11:41 AM, Eli Collins wrote: Thanks for writing up a proposal Chris. NP. I think it makes sense to have Common live in HDFS at least for now, since it's at the bottom of the stack /

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hi Todd, On Aug 29, 2012, at 2:18 PM, Todd Lipcon wrote: Have we not learned our lessons from the last attempts to split? The issues in our community, which I think Chris is referring to, do not generally revolve around project boundaries. It's not the case that the HDFS community wants to

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hi Cos, On Aug 29, 2012, at 4:27 PM, Konstantin Boudnik wrote: Sounds cool to me. - Hadoop 1.x is is maintenance mode, though it still actively gets patches so we need to consider it. The surgery necessary to split v1 Hadoop is probably not suitable for a sustaining release and not worth

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Todd Lipcon
On Wed, Aug 29, 2012 at 4:20 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Arun, great work below. Concrete, and an actual proposal of PMC lists. What do folks think? Already expressed my opinion above on the thread that whole idea of splitting is crazy. But, I'll comment

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Konstantin Boudnik
On Wed, Aug 29, 2012 at 11:32PM, Mattmann, Chris A (388J) wrote: Hi Cos, On Aug 29, 2012, at 4:27 PM, Konstantin Boudnik wrote: Sounds cool to me. - Hadoop 1.x is is maintenance mode, though it still actively gets patches so we need to consider it. The surgery necessary to split v1

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Todd Lipcon
On Wed, Aug 29, 2012 at 4:29 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: You're right, it's not project boundaries, it's poor community behavior, and general umbrella-project-ness. No doubt there's bad behavior. But splitting into smaller projects won't help anything.

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Konstantin Boudnik
I am curious where the arbitrar numbery 5 is coming from: is it reflected in the bylaws? Cos On Wed, Aug 29, 2012 at 04:35PM, Todd Lipcon wrote: On Wed, Aug 29, 2012 at 4:20 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Arun, great work below. Concrete, and an actual

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Todd Lipcon
On Wed, Aug 29, 2012 at 4:47 PM, Konstantin Boudnik c...@apache.org wrote: I am curious where the arbitrar numbery 5 is coming from: is it reflected in the bylaws? Nope, I picked it based on Arun's earlier picking of the same number in the YARN thread. We have no bylaws about what would happen

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Aaron T. Myers
On Wed, Aug 29, 2012 at 4:35 PM, Todd Lipcon t...@cloudera.com wrote: But I still think this discussion is silly, and we're not ready to do it. +1 Despite many allusions to problems that this project split proposal would purport to solve, I honestly don't see the problems. Yes, Hadoop has had

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
OK I lied and said I wouldn't reply :) On Aug 29, 2012, at 4:44 PM, Todd Lipcon wrote: On Wed, Aug 29, 2012 at 4:29 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: You're right, it's not project boundaries, it's poor community behavior, and general umbrella-project-ness.

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Todd Lipcon
On Wed, Aug 29, 2012 at 4:54 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Please provide examples that show umbrella projects work. Hadoop, in its current form? The code bases are tightly intertwined. We pulled out Pig/Hive/HBase because they were substantial codebases

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Mattmann, Chris A (388J)
Hey Todd, On Aug 29, 2012, at 5:16 PM, Todd Lipcon wrote: On Wed, Aug 29, 2012 at 4:54 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Please provide examples that show umbrella projects work. Hadoop, in its current form? I don't agree that it's working. That's where

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Arun C Murthy
On Aug 29, 2012, at 4:32 PM, Mattmann, Chris A (388J) wrote: Hi Cos, On Aug 29, 2012, at 4:27 PM, Konstantin Boudnik wrote: Sounds cool to me. - Hadoop 1.x is is maintenance mode, though it still actively gets patches so we need to consider it. The surgery necessary to split v1

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Arun C Murthy
On Aug 29, 2012, at 4:48 PM, Todd Lipcon wrote: On Wed, Aug 29, 2012 at 4:47 PM, Konstantin Boudnik c...@apache.org wrote: I am curious where the arbitrar numbery 5 is coming from: is it reflected in the bylaws? Nope, I picked it based on Arun's earlier picking of the same number in the

Re: Updating Hadoop User Groups to include Florida chapter

2012-08-29 Thread Ravishankar Nair
I have been trying my best to get admin rights for the Hadoop User Grops page to edit tit.(http://wiki.apache.org/hadoop/HadoopUserGroups). If anyone has admin rights to edit this page, can you please help me to add below URL (in Moin Moin Wiki format) in the North America section?

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Konstantin Boudnik
On Wed, Aug 29, 2012 at 04:44PM, Todd Lipcon wrote: On Wed, Aug 29, 2012 at 4:29 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: ... I doubt that. Creating TLPs either directly by going to the board, or via going to the Incubator should involve a set of members of the

Re: Updating Hadoop User Groups to include Florida chapter

2012-08-29 Thread Arun C Murthy
Done, thanks for adding this Ravi! On Aug 29, 2012, at 7:35 PM, Ravishankar Nair wrote: I have been trying my best to get admin rights for the Hadoop User Grops page to edit tit.(http://wiki.apache.org/hadoop/HadoopUserGroups). If anyone has admin rights to edit this page, can you please

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Eli Collins
On Wed, Aug 29, 2012 at 4:19 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Eli, On Aug 29, 2012, at 11:41 AM, Eli Collins wrote: Thanks for writing up a proposal Chris. NP. I think it makes sense to have Common live in HDFS at least for now, since it's at the

Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

2012-08-29 Thread Eli Collins
On Wed, Aug 29, 2012 at 4:20 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Arun, great work below. Concrete, and an actual proposal of PMC lists. What do folks think? I don't see how it helps. This substantially *increases* the size of the PMC for HDFS, I don't even