On Fri, Apr 1, 2011 at 08:26, Nigel Daley <nda...@mac.com> wrote: > -1+2. This could potentially allow us to replace Jenkins with Hadoop for our > build and test infrastructure. That would be awesome!
Has anyone checked a calendar lately? > On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote: > >> Experience developing Hadoop has shown that we not only need to >> partition our projects for more active releases, but we also should >> explore speculative project splits. For this, a Hadoop.next() project >> should track the development of a project scheduler that can partition >> the Hadoop subprojects, possibly running a second version of a >> subproject in parallel. Downstream subprojects and TLPs automatically >> accept whichever releases first as a dependency. Implementation should >> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to >> be written). >> >> Of course, not all of these subprojects will succeed. When one fails >> (or is too slow with its project reports), the project scheduler will >> be responsible for respawning it in the Incubator. >> >> The project scheduler will, of course, be pluggable. -C >> >> On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <a...@cloudera.com> wrote: >>> Hello Hadoop Community, >>> >>> Given the tremendous positive feedback we've all had regarding the HDFS, >>> MapReduce, and Common project split, I'd like to propose we take the next >>> step and further separate the existing projects. >>> >>> I propose we begin by splitting the MapReduce project into separate "Map" >>> and "Reduce" sub-projects. This will provide us the opportunity to tease out >>> the complex interdependencies between "map" and "reduce" that exist today, >>> to encourage us to write more modular and isolated code, which should speed >>> releases. This will also aid our users who exclusively run map-only or >>> reduce-only jobs. These are important use-cases, and so should be given high >>> priority. >>> >>> Given that these two portions of the existing MapReduce project share a >>> great deal of code, we will likely need to release these two new projects >>> concurrently at first, but the eventual goal should certainly be to be able >>> to release "Map" and "Reduce" independently. This seems intuitive to me, >>> given the remarkable recent advancements in the academic community regarding >>> "reduce," while the research coming out of the "map" academics has largely >>> stagnated of late. >>> >>> If this proposal is accepted, and it has the success I think it will, then >>> we should strongly consider splitting the other two projects as well. My gut >>> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects, and >>> simply rename the "Common" project to "C'Mon." We can think about the >>> details of what exactly these project splits mean later. >>> >>> Please let me know what you think. >>> >>> Best, >>> Aaron >>> > >
