Some lesson learn during 2.x. WebHDFS, HDFS ACL, QJM HA, rolling upgrade are great features. Mapreduce 1.x uses resources more efficiently, containers have rigid constraint, and applications get killed prematurely. When a node has a lot of containers, YARN takes significant amount of system resources. Existing daemon based application to run on top of YARN without code change is impossible. It is difficult to pinpoint where services will run. Extra routing of client to server code needs to be written for the application. Hence, the existing map reduce approach to spawn off parallelized work load and output result in durable file system is better. Client serving service doesn't need to track states but read from hdfs. Hence some level of HA for external serving service can achieve without YARN. Slider provides a better interface for exposing API to deploy applications.
It would be nice to support the following in 3.x: - JDK 8 - Upgrade to most recent version of Jetty, most hang problems or busy cpu problems comes from Jetty 6.1.x being incompatible with JDK 7 in NIO design. - Improve default security, there is a gap where Default Container Executor vs Linux Container Executor. It would be nicer if default security uses Linux Container Executor to ensure developer remember to run with doAs when designing services to run on top of Hadoop. - Since 3.x is a major release number change. There maybe backward compatible API breakage initially in order to gain new functionality. The backward compatible patches can be added over time. - Reduce YARN framework resource usage - Improve usability of YARN UI. Drill down from application to container then back to application view is almost unusable. - Smarter strategy for containers placement. Some call this anti-affinity support for YARN, but there is only a few types to support. The identified ones are: shared, silo, and dedicated. In shared, containers can co-locate on same node. In silo, where same type of container can only spawn one per node. Dedicated will reserve the entire node for this workload. regards, Eric On Fri, Mar 6, 2015 at 5:20 PM, Chris Douglas <cdoug...@apache.org> wrote: > On Fri, Mar 6, 2015 at 4:32 PM, Vinod Kumar Vavilapalli > <vino...@hortonworks.com> wrote: > > I'd encourage everyone to post their wish list on the Roadmap wiki that > *warrants* making incompatible changes forcing us to go 3.x. > > This is a useful exercise, but not a prerequisite to releasing 3.0.0 > as an alpha off of trunk, right? Andrew summarized the operating > assumptions for anyone working on it: rolling upgrades still work, > wire compat is preserved, breaking changes may get rolled back when > branch-3 is in beta (so be very conservative, notify others loudly). > This applies to branches merged to trunk, also. > > > +1 to Jason's comments on general. We can keep rolling alphas that > downstream can pick up, but I'd also like us to clarify the exit criterion > for a GA release of 3.0 and its relation to the life of 2.x if we are going > this route. This brings us back to the roadmap discussion, and a collective > agreement about a logical step at a future point in time where we say we > have enough incompatible features in 3.x that we can stop putting more of > them and start stabilizing it. > > We'll have this discussion again. We don't need to reach consensus on > the roadmap, just that each artifact reflects the output of the > project. > > > Irrespective of that, here is my proposal in the interim: > > - Run JDK7 + JDK8 first in a compatible manner like I mentioned before > for atleast two releases in branch-2: say 2.8 and 2.9 before we consider > taking up the gauntlet on 3.0. > > - Continue working on the classpath isolation effort and try making it > as compatible as is possible for users to opt in and migrate easily. > > +1 for 2.x, but again I don't understand the sequencing. -C > > > On Mar 5, 2015, at 1:44 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> > wrote: > > > >> I'm OK with a 3.0.0 release as long as we are minimizing the pain of > maintaining yet another release line and conscious of the incompatibilities > going into that release line. > >> For the former, I would really rather not see a branch-3 cut so soon. > It's yet another line onto which to cherry-pick, and I don't see why we > need to add this overhead at such an early phase. We should only create > branch-3 when there's an incompatible change that the community wants and > it should _not_ go into the next major release (i.e.: it's for Hadoop > 4.0). We can develop 3.0 alphas and betas on trunk and release from trunk > in the interim. IMHO we need to stop treating trunk as a place to exile > patches. > >> > >> For the latter, I think as a community we need to evaluate the benefits > of breaking compatibility against the costs of migrating. Each time we > break compatibility we create a hurdle for people to jump when they move to > the new release, and we should make those hurdles worth their time. For > example, wire-compatibility has been mentioned as part of this. Any > feature that breaks wire compatibility better be absolutely amazing, as it > creates a huge hurdle for people to jump. > >> To summarize:+1 for a community-discussed roadmap of what we're > breaking in Hadoop 3 and why it's worth it for users > >> -1 for creating branch-3 now, we can release from trunk until the next > incompatibility for Hadoop 4 arrives > >> +1 for baking classpath isolation as opt-in on 2.x and eventually > default on in 3.0 > >> Jason > >> From: Andrew Wang <andrew.w...@cloudera.com> > >> To: "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org> > >> Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>; " > mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; " > yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org> > >> Sent: Wednesday, March 4, 2015 12:15 PM > >> Subject: Re: Looking to a Hadoop 3 release > >> > >> Let's not dismiss this quite so handily. > >> > >> Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while > we > >> could make classpath isolation opt-in via configuration, what we really > >> want longer term is to have it on by default (or just always on). Stack > in > >> particular points out the practical difficulties in using an opt-in > method > >> in 2.x from a downstream project perspective. It's not pretty. > >> > >> The plan that both Sean and Jason propose (which I support) is to have > an > >> opt-in solution in 2.x, bake it there, then turn it on by default > >> (incompatible) in a new major release. I think this lines up well with > my > >> proposal of some alphas and betas leading up to a GA 3.x. I'm also > willing > >> to help with 2.x release management if that would help with testing this > >> feature. > >> > >> Even setting aside classpath isolation, a new major release is still > >> justified by JDK8. Somehow this is being ignored in the discussion. > Allen, > >> historically the voice of the user in our community, just highlighted > it as > >> a major compatibility issue, and myself and Tucu have also expressed our > >> very strong concerns about bumping this in a minor release. 2.7's bump > is a > >> unique exception, but this is not something to be cited as precedent or > >> policy. > >> > >> Where does this resistance to a new major release stem from? As I've > >> described from the beginning, this will look basically like a 2.x > release, > >> except for the inclusion of classpath isolation by default and target > >> version JDK8. I've expressed my desire to maintain API and wire > >> compatibility, and we can audit the set of incompatible changes in > trunk to > >> ensure this. My proposal for doing alpha and beta releases leading up > to GA > >> also gives downstreams a nice amount of time for testing and validation. > >> > >> Regards, > >> Andrew > >> > >> > >> > >> On Tue, Mar 3, 2015 at 2:32 PM, Arun Murthy <a...@hortonworks.com> > wrote: > >> > >>> Awesome, looks like we can just do this in a compatible manner - > nothing > >>> else on the list seems like it warrants a (premature) major release. > >>> > >>> Thanks Vinod. > >>> > >>> Arun > >>> > >>> ________________________________________ > >>> From: Vinod Kumar Vavilapalli <vino...@hortonworks.com> > >>> Sent: Tuesday, March 03, 2015 2:30 PM > >>> To: common-dev@hadoop.apache.org > >>> Cc: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; > >>> yarn-...@hadoop.apache.org > >>> Subject: Re: Looking to a Hadoop 3 release > >>> > >>> I started pitching in more on that JIRA. > >>> > >>> To add, I think we can and should strive for doing this in a compatible > >>> manner, whatever the approach. Marking and calling it incompatible > before > >>> we see proposal/patch seems premature to me. Commented the same on > JIRA: > >>> > https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=14345875&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14345875 > >>> . > >>> > >>> Thanks > >>> +Vinod > >>> > >>> On Mar 2, 2015, at 8:08 PM, Andrew Wang <andrew.w...@cloudera.com > <mailto: > >>> andrew.w...@cloudera.com>> wrote: > >>> > >>> Regarding classpath isolation, based on what I hear from our customers, > >>> it's still a big problem (even after the MR classloader work). The > latest > >>> Jackson version bump was quite painful for our downstream projects, > and the > >>> HDFS client still leaks a lot of dependencies. Would welcome more > >>> discussion of this on HADOOP-11656, Steve, Colin, and Haohui have > already > >>> chimed in. > >>> > >>> > >> > >> > > >
