Re: [DISCUSS] branch-1
+1 to auto-closing branch-1 issues. -Sandy On Fri, May 8, 2015 at 11:14 AM, Karthik Kambatla wrote: > Closing out the JIRAs as "Auto Closed" or "Closed due to Inactivity" seems > reasonable to me. For branch-1, we can be more aggressive. We should > probably do the same less aggressively for other branches too. > > On Fri, May 8, 2015 at 11:01 AM, Arun C Murthy > wrote: > > > +1 > > > > Arun > > > > On May 8, 2015, at 10:41 AM, Allen Wittenauer wrote: > > > > > > > > May we declare this branch dead and just close bugs (but not > > necessarily concepts, ideas, etc) with won’t fix? I don’t think anyone > has > > any intention of working on the 1.3 release, especially given that 1.2.1 > > was Aug 2013 …. > > > > > > I guess we need a PMC member to declare a vote or whatever…. > > > > > > > > > > > > > -- > Karthik Kambatla > Software Engineer, Cloudera Inc. > > http://five.sentenc.es >
Re: [DISCUSS] Release numbering for stable 2.8 and beyond
My understanding was that the main reason that we labeled 2.0 alpha and 2.1
beta is that we wanted flexibility to make breaking API changes.
Is that the case now with 2.7? I.e. do we have APIs labeled as Public /
Stable that we want freedom to change in 2.8? If not, I definitely don't
think the release merits a special tag. We should just make sure its bugs
are well documented.
-Sandy
On Mon, Apr 27, 2015 at 10:45 AM, Andrew Wang
wrote:
> I think Karthik correctly identified the two reasons we might want to
> denote a release as "unstable":
>
> a) Compatibility. Whether we have freedom to break compatibility before the
> final release, i.e. what "alpha" typically means.
>
> b) Quality. As Konst says, a release can be allowed to break compatibility
> ("alpha") but itself still be a high quality release.
>
> We could try and separate out these two concerns when it comes to
> versioning, but I think in reality users only care about prod vs. non-prod,
> which is why many other projects (Linux, HBase, etc) just have two release
> lines: "stable" (compatible/good-quality) vs. "unstable"
> (unknown-compatible/unknown-quality).
>
> To this end, I don't mind what we choose, as long as the difference between
> stable and unstable is denoted by the version number. I don't like the idea
> of tagging a release as good after the fact (1). The other projects we've
> used as examples make strong statements about their stable release lines,
> and we should too. Our downstreams and end users will appreciate clarity
> from the version number.
>
> Best,
> Andrew
>
> On Sun, Apr 26, 2015 at 9:51 AM, Hitesh Shah wrote:
>
> > There are a couple of different approaches we could take.
> >
> > How about publishing/releasing bits such as “2.8.0-RC0”. Downstream
> > projects can depend on these and use them normally similar to the
> approach
> > that they would have taken with release 2.8.0 or 2.8.0-alpha. After some
> > baking ( or more RC suffixed releases), at some point, a proper 2.8.0
> > release could be made? The RC tag clearly indicates to downstream layers
> > that things will be in flux slightly. Trying to distinguish
> > incompatibilities between 2.8.0 and 2.8.1/2.8.2 ( just because 2.8.0 was
> > tagged alpha in documentation ) are likely to be a nightmare to deal with
> > especially for new features introduced in the 2.8 release ( which might
> get
> > changed slightly based on usage feedback ).
> >
> > Furthermore, considering the release history of hadoop, the likelihood
> > that 2.9.0 will show up before 2.8.2 seems to be very high.
> >
> > With respect to the proposed choice (1), I thought that HBase applied a
> > different approach. The odd-number releases were always unstable and the
> > even number releases were the stable ones. This “could" make things a bit
> > more clear for downstream users without needing to resort to modified
> > versioned numbers ( with alpha/beta tags ) or requiring users to go look
> up
> > the website to find out which particular version of 2.8.x was the first
> > stable release.
> >
> > thanks
> > — Hitesh
> >
> >
> >
> >
> >
> > On Apr 22, 2015, at 2:17 PM, Vinod Kumar Vavilapalli <
> > vino...@hortonworks.com> wrote:
> >
> > > Forking the thread.
> > >
> > > In the previous 2.7.1 thread [1], there were enough yays to my proposal
> > to wait for a bug-fix release or two before calling a 2.x release stable.
> > There were some concerns about the naming.
> > >
> > > We have two options, taking 2.8 as an example
> > > (1) Release 2.8.0, call it as an alpha in documentation and release
> > notes, wait for a 2.8.1/2.8.2 reasonably stable enough to be called as
> the
> > first stable release of 2.8.
> > > (2) Release 2.8.0-alpha, 2.8.0-beta etc before culminating in a 2.8.0
> > stable release.
> > >
> > > (1) is what I preferred first up. This is what HBase used to do, and
> far
> > beyond, in the linux kernel releases. It helps in scenarios where we are
> > forced to downgrade a release, say due to major issues. We can simply
> > announce it as not stable retroactively, change the pointers on our
> website
> > and move on.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > +Vinod
> > >
> > > [1] http://markmail.org/thread/ogzk4phj6wsdpssu
> > >
> > > On Apr 21, 2015, at 4:59 PM, Vinod Kumar Vavilapalli <
> > vino...@hortonworks.com> wrote:
> > >
> > >>
> > >> Sure, I agree it's better to have clear guidelines and scheme. Let me
> > fork this thread about that.
> > >>
> > >> Re 2.7.0, I just forgot about the naming initially though I was clear
> > in the discussion/voting. I so had to end up calling it alpha outside of
> > the release artifact naming.
> > >>
> > >> Thanks
> > >> +Vinod
> > >>
> > >> On Apr 21, 2015, at 4:26 PM, Andrew Wang
> > wrote:
> > >>
> > >>> I would also like to support Karthik's proposal on the release thread
> > about
> > >>> version numbering. 2.7.0 being "alpha" is pretty confusing since all
> > of the
> > >>> other 2.x releases since GA have been stable. I
Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (binding) Ran a pseudo-distributed cluster with the Fair Scheduler and ran some MapReduce example jobs. -Sandy On Wed, Sep 10, 2014 at 12:15 PM, Alejandro Abdelnur wrote: > Thanks Karthik. > > +1. > > + verified MD5 for source tarball > + verified signature for source tarball > + successfully run apache-rat:check > + checked CHANGES, LICENSE, README, NOTICE files. > + built from source tarball > + started pseudo cluster > + run a couple of MR example jobs > + basic test on HttpFS > > On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla > wrote: > > > Thanks for reporting the mistake in the documentation, Akira. While it is > > good to fix it, I am not sure it is big enough to warrant another RC, > > particularly because 2.5.1 is very much 2.5.0 done right. > > > > I just updated the how-to-release wiki to capture this step in the > release > > process, so we don't miss it in the future. > > > > On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA < > ajisa...@oss.nttdata.co.jp > > > > > wrote: > > > > > -0 (non-binding) > > > > > > In the document, "Apache Hadoop 2.5.1 is a minor release in the 2.x.y > > > release line, buliding upon the previous stable release 2.4.1." > > > > > > Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. > > > > > > Regards, > > > Akira > > > > > > > > > (2014/09/09 0:51), Karthik Kambatla wrote: > > > > > >> +1 (non-binding) > > >> > > >> Built the source tarball, brought up a pseudo-distributed cluster and > > ran > > >> a > > >> few MR jobs. Verified documentation and size of the binary tarball. > > >> > > >> On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla > > >> wrote: > > >> > > >> Hi folks, > > >>> > > >>> I have put together a release candidate (RC0) for Hadoop 2.5.1. > > >>> > > >>> The RC is available at: http://people.apache.org/~ > > >>> kasha/hadoop-2.5.1-RC0/ > > >>> The RC git tag is release-2.5.1-RC0 > > >>> The maven artifacts are staged at: > > >>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1010/ > > >>> > > >>> You can find my public key at: > > >>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS > > >>> > > >>> Please try the release and vote. The vote will run for the now usual > 5 > > >>> days. > > >>> > > >>> Thanks > > >>> Karthik > > >>> > > >>> > > >> > > > > > > > > > -- > Alejandro >
Re: [VOTE] Migration from subversion to git for version control
+1 (binding) On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla wrote: > I have put together this proposal based on recent discussion on this topic. > > Please vote on the proposal. The vote runs for 7 days. > >1. Migrate from subversion to git for version control. >2. Force-push to be disabled on trunk and branch-* branches. Applying >changes from any of trunk/branch-* to any of branch-* should be through >"git cherry-pick -x". >3. Force-push on feature-branches is allowed. Before pulling in a >feature, the feature-branch should be rebased on latest trunk and the >changes applied to trunk through "git rebase --onto" or "git cherry-pick >". >4. Every time a feature branch is rebased on trunk, a tag that >identifies the state before the rebase needs to be created (e.g. >tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once >the feature is pulled into trunk and the tags are no longer useful. >5. The relevance/use of tags stay the same after the migration. > > Thanks > Karthik > > PS: Per Andrew Wang, this should be a "Adoption of New Codebase" kind of > vote and will be Lazy 2/3 majority of PMC members. >
Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated
That policy makes sense to me. We should still label things @Private of course so that it can be reflected in the documentation. -Sandy On Tue, Jul 22, 2014 at 2:54 PM, Karthik Kambatla wrote: > Hi devs > > As you might have noticed, we have several classes and methods in them that > are not annotated at all. This is seldom intentional. Avoiding incompatible > changes to all these classes can be considerable baggage. > > I was wondering if we should add an explicit disclaimer in our > compatibility guide that says, "Classes without annotations are to > considered @Private" > > For methods, is it reasonable to say - "Class members without specific > annotations inherit the annotations of the class"? > > Thanks > Karthik >
Re: Where is the map input transfered to the 'map worker'/container?
Hi Christian, I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of getting the bytes from the remote node. -Sandy On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote < cgr...@mail.uni-paderborn.de> wrote: > Hey, > > I'm looking for the place where the actual map input is transfered to the > 'map worker'/container (in case it's assigned to a host that doesn't have > the data already). > > Something similar to the copyFromHost(..) method in Fetcher.java > (org.apache.hadoop.mapreduce.task.reduce), where the map output is > transfered. > > > Best Regards, > Christian Grote > > > > > > > >
Re: Moving to JDK7, JDK8 and new major releases
While we haven't codified this in our compatibility guidelines, dropping a Java version seems to me like change that needs to happen alongside a major release. In plain talk, it has the ability to break everything for users who aren't doing anything particularly unreasonable. I don't think we should accept Hadoop's compatibility behavior 6 years ago as precedent for what we can do now. That was before Hadoop 1.0. And we probably have several orders of magnitude more production users. I also don't think we should accept library upgrades as precedent. While this may make sense in specific situations, I definitely don't think this is OK in general. I'd be very nervous about updating Guava outside of major version upgrade. Lastly, I think the claim that nobody is running in production on Java 6 is unsubstantiated. We need to think about a JDK upgrade in terms of what its implications are for users, not in terms of what other kinds of compatibility we've broken that's loosely analogous. -Sandy On Tue, Jun 24, 2014 at 3:42 PM, Vinod Kumar Vavilapalli wrote: > Tx for the new thread Andrew, hopefully it can attract more eyes. > > Here's what I am behind - a modified proposal C. > - Overall I wouldn't think about EOL of JDK7 and/or JDK8 specifically > given how long it has taken for JDK6 life-cycle to end. We should try to > focus on JDK7 only for now. > - As we have seen, a lot (majority?) of orgs on Hadoop have moved beyond > JDK6 and are already running on JDK7. So upgrading to JDK7 is more of a > reflection of reality (to quote Steve) than it in itself being a disruptive > change. > - We should try decoupling the discussion of major releases from JDK > upgrades. We have seen individual libraries getting updated right in the > 2.x lines as and when necessary. Given the new reality of JDK7, I don't see > the 'JDK change' as much different from the library upgrades. > > We have seen how long it has taken (and still taking) users and > organization to move from Hadoop 1 to Hadoop 2. A Hadoop 3/4 that adds > nothing else other than JDK upgrades will be a big source of confusion for > users. A major version update is also seen an opportunity for devs to break > APIs. Unless we have groundbreaking 'features' (like YARN or > wire-compatibility in Hadoop-2) that a majority of users want and that > specifically warrant incompatible changes in our APIs or wire protocols, we > are better off separating the major-version update discussion into ints own. > > Irrespective of all this, we should actively get behind better isolation > of user classes/jars from MapReduce classpath. This one's been such a long > running concern, it's not funny anymore. > > Thanks, > +Vinod > > On Jun 24, 2014, at 11:17 AM, Andrew Wang > wrote: > > > Hi all, > > > > Forking this thread as requested by Vinod. To help anyone who's catching > up > > with this thread, I've written up a wiki page containing what I think are > > the proposals under discussion. I did my very best to make this as > > fact-based and disinterested as possible; I really appreciate the > > constructive discussion we've had so far. If you believe you have a > > proposal pending, please feel free to edit the wiki. > > > > https://wiki.apache.org/hadoop/MovingToJdk7and8 > > > > I think based on our current compatibility guidelines, Proposal A is the > > most attractive. We're pretty hamstrung by the requirement to keep the > > classpath the same, which would be solved by either OSGI or shading our > > deps (but that's a different discussion). > > > > Thanks, > > Andrew > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 (binding) -Sandy On Tue, Jun 24, 2014 at 7:53 AM, Devaraj K wrote: > +1 > > Thanks > Devaraj K > > > On Tue, Jun 24, 2014 at 2:23 PM, Arun C Murthy > wrote: > > > Folks, > > > > As discussed, I'd like to call a vote on changing our by-laws to change > > release votes from 7 days to 5. > > > > I've attached the change to by-laws I'm proposing. > > > > Please vote, the vote will the usual period of 7 days. > > > > thanks, > > Arun > > > > > > > > [main]$ svn diff > > Index: author/src/documentation/content/xdocs/bylaws.xml > > === > > --- author/src/documentation/content/xdocs/bylaws.xml (revision > 1605015) > > +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) > > @@ -344,7 +344,16 @@ > > Votes are open for a period of 7 days to allow all active > > voters time to consider the vote. Votes relating to code > > changes are not subject to a strict timetable but should be > > -made as timely as possible. > > +made as timely as possible. > > + > > + > > + Product Release - Vote Timeframe > > + Release votes, alone, run for a period of 5 days. All > other > > + votes are subject to the above timeframe of 7 days. > > + > > + > > + > > + > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > > > -- > > > Thanks > Devaraj K >
Re: Plans of moving towards JDK7 in trunk
Andrew, correct me if I'm misunderstanding, but the incompatible change that would require a major version bump is dropping support for JDK6. On Mon, Jun 23, 2014 at 1:53 PM, sanjay Radia wrote: > > On Jun 21, 2014, at 8:01 AM, Andrew Wang wrote: > > > This is why I'd like to keep my original proposal on the table: keep > going > > with branch-2 in the near term, while working towards a JDK8-based > Hadoop 3 > > by April next year. It doesn't need to be a big bang release either. I'd > be > > delighted if we could rolling upgrade from one to the other. I just > didn't > > want to rule out the inclusion of some very compelling feature outright. > > Trust me though, I'd be the first person to ask about compatibility if > such > > a feature does come up. > > > Given your above statement on compatibility (such as rolling upgrades), > it should be fine for the JDK8-based-Hadoop-release to not be 3.0 and > instead merely be 2.x? Or do you have any incompatible changes to Hadoop > protocol or APIs in mind during the same time period? > > sanjay > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: hadoop-2.5 - June end?
That sounds reasonable to me. -Sandy On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy wrote: > Folks, > > As you can see from the Roadmap wiki, it looks like several items are > still a bit away from being ready. > > I think rather than wait for them, it will be useful to create an > intermediate release (2.5) this month - I think ATS security is pretty > close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of > the month, with a branch a couple of weeks prior. > > Thoughts? > > thanks, > Arun > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
[jira] [Created] (MAPREDUCE-5896) Allow InputSplits to indicate which locations have the block cached in memory
Sandy Ryza created MAPREDUCE-5896: - Summary: Allow InputSplits to indicate which locations have the block cached in memory Key: MAPREDUCE-5896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5896 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Thinking ahead
Reference: http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201401.mbox/%3ccajs6+b4tkupvqfaorf_314xveivryqy-vtryd8+36tu_bc5...@mail.gmail.com%3E This is of course open source, and I have neither the ability nor the desire to compel or pressure anybody to work on anything, but I wanted to point out that the proposal to merge AHS into trunk stated an intent to tie ends up on security. On Sun, Apr 13, 2014 at 7:50 PM, Sandy Ryza wrote: > Unfortunately I don't have the bandwidth to take on ATS security at this > time. My (I now understand mistaken) impression that it was being worked > on, as we had discussed it under the initial criteria for including the AHS > in a release. I brought it up as a feature I felt worth waiting for if it > meant extending the target release date a couple weeks - I of course had no > intention to volunteer anybody for it. > > > On Sun, Apr 13, 2014 at 5:50 PM, Arun C Murthy wrote: > >> Sandy, >> >> On Apr 12, 2014, at 10:09 AM, Sandy Ryza wrote: >> >> I'm having trouble editing the wiki, but I think Timeline Server stability >> (e.g. security and locking down APIs) should go on that list. >> >> >> I'm very glad to see you are very passionate about security for ATS since >> you've asked about it a few times around here. >> >> I just opened https://issues.apache.org/jira/browse/YARN-1935. >> >> Would you be willing to volunteer to take this on? Much appreciated. >> >> thanks, >> Arun >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > >
Re: Thinking ahead
Unfortunately I don't have the bandwidth to take on ATS security at this time. My (I now understand mistaken) impression that it was being worked on, as we had discussed it under the initial criteria for including the AHS in a release. I brought it up as a feature I felt worth waiting for if it meant extending the target release date a couple weeks - I of course had no intention to volunteer anybody for it. On Sun, Apr 13, 2014 at 5:50 PM, Arun C Murthy wrote: > Sandy, > > On Apr 12, 2014, at 10:09 AM, Sandy Ryza wrote: > > I'm having trouble editing the wiki, but I think Timeline Server stability > (e.g. security and locking down APIs) should go on that list. > > > I'm very glad to see you are very passionate about security for ATS since > you've asked about it a few times around here. > > I just opened https://issues.apache.org/jira/browse/YARN-1935. > > Would you be willing to volunteer to take this on? Much appreciated. > > thanks, > Arun > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: Thinking ahead
+1 for starting to think about 2.5. Early June seems a little early to me - we had talked about a quarterly release cadence and this would be about half that. I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list. I think we should probably take YARN-1404 off the list - even with 3 months it's unlikely to be complete. On Sat, Apr 12, 2014 at 9:50 AM, Chris Nauroth wrote: > +1 > > The proposed content for 2.5 in the roadmap wiki looks good to me. > On Apr 12, 2014 7:26 AM, "Arun C Murthy" wrote: > > > Gang, > > > > With hadoop-2.4 out, it's time to think ahead. > > > > In the short-term hadoop-2.4.1 is in order; particularly with > > https://issues.apache.org/jira/browse/MAPREDUCE-5830 (it's a break to > > @Private API, unfortunately something Hive is using - sigh!). There are > > some other fixes which testing has uncovered; so it will be nice to pull > > them them in. I'm thinking of an RC by end of the coming week - > committers, > > please be *very* conservative when getting stuff into 2.4.1 (i.e. merging > > to branch-2.4). > > > > Next up, hadoop-2.5. > > > > I've updated https://wiki.apache.org/hadoop/Roadmap with some > candidates > > for consideration - please chime in and say 'aye'/'nay' or add new > content. > > IAC, I suspect that list is too large. > > > > Rather than wait for everything it would be better to plan on releasing > > it on a time-bound manner; particularly around the Hadoop Summit. If that > > makes sense; I think we should target branching for 2.5 by mid-May to get > > it stable and released by early June. > > > > Thoughts? > > > > thanks, > > Arun > > > > > > -- > > Arun C. Murthy > > Hortonworks Inc. > > http://hortonworks.com/ > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: [VOTE] Release Apache Hadoop 2.4.0
While the Scheduler Load Simulator isn't part of YARN's core, it's a tool that YARN includes, and it's broken entirely in the current RC. YARN-1726 seems to me like something worth including in the release. -Sandy On Thu, Apr 3, 2014 at 11:12 AM, Xuan Gong wrote: > +1 non-binding > > Built from source code, tested on a single node cluster. Successfully ran a > few MR sample jobs. > Tested RM failover while job is running. > > Thanks > > Xuan Gong > > > On Wed, Apr 2, 2014 at 10:21 PM, Zhijie Shen > wrote: > > > +1 non-binding > > > > I built from source code, and setup a single node non-secure cluster with > > almost the default configurations and ran a number of MR example jobs and > > distributedshell jobs. I verified the generic and the per-framework (DS > job > > only) historic information has been persisted, and such information is > > accessible via webUI, RESTful APIs and CLI. > > > > Thanks, > > Zhijie > > > > > > On Wed, Apr 2, 2014 at 1:26 PM, Jian He wrote: > > > > > +1 non-binding > > > > > > Built from source code, tested on a single node cluster. Successfully > > ran a > > > few MR sample jobs. > > > Tested RM restart while job is running. > > > > > > Thanks, > > > Jian > > > > > > On Tue, Apr 1, 2014 at 5:42 PM, Travis Thompson < > tthomp...@linkedin.com > > > >wrote: > > > > > > > +1 non-binding > > > > > > > > Built from git. Started with 120 node 2.3.0 cluster with security and > > > > non HA, ran upgrade (non rolling) to 2.4.0. Confirmed fsimage is OK > > and > > > > HDFS successfully upgraded. Also successfully ran some pig jobs and > > > > mapreduce examples. Haven't found any issues yet but will continue > > > > testing. Did not test Timeline Server since I'm using security. > > > > > > > > Thanks, > > > > Travis > > > > > > > > On 03/31/2014 02:24 AM, Arun C Murthy wrote: > > > > > Folks, > > > > > > > > > > I've created a release candidate (rc0) for hadoop-2.4.0 that I > would > > > > like to get released. > > > > > > > > > > The RC is available at: > > > > http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 > > > > > The RC tag in svn is here: > > > > > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 > > > > > > > > > > The maven artifacts are available via repository.apache.org. > > > > > > > > > > Please try the release and vote; the vote will run for the usual 7 > > > days. > > > > > > > > > > thanks, > > > > > Arun > > > > > > > > > > -- > > > > > Arun C. Murthy > > > > > Hortonworks Inc. > > > > > http://hortonworks.com/ > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > Zhijie Shen > > Hortonworks Inc. > > http://hortonworks.com/ > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: [VOTE] Release Apache Hadoop 2.4.0
Thanks Zhijie - where is that document located? On Mon, Mar 31, 2014 at 11:23 AM, Zhijie Shen wrote: > Hi Sandy, > > Application History Server (we prefer to call it Timeline Server instead > now) is going to be shipped in 2.4 without security. I've already wrote a > document about what it is, what's the current status, and how to config and > start it. Via the document, hopefully the users are clear about the > timeline server. There's another jira, YARN-1876, in which I've attached a > patch about the REST APIs of using generic history data and per-framework > data. However, it seems to be to late to get it in. > > For the timeline store, we have already one implementation based on > Leveldb. We have monitored its performance closely and reviewed its > license, and believed it should be a good shape now. Currently the timeline > store is for storing per-framework data only, while we eventually hope to > move the generic data there as well. > > Thanks, > Zhijie > > > > > On Mon, Mar 31, 2014 at 11:02 AM, Sandy Ryza >wrote: > > > What's the state of the application history server? Do we have security, > > documentation, and are APIs stable? If any of these are missing, do we > > have a plan for how to make this clear to users? > > > > What about the timeline store? > > > > thanks, > > Sandy > > > > > > On Mon, Mar 31, 2014 at 2:22 AM, Arun C Murthy > > wrote: > > > > > Folks, > > > > > > I've created a release candidate (rc0) for hadoop-2.4.0 that I would > like > > > to get released. > > > > > > The RC is available at: > > > http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 > > > The RC tag in svn is here: > > > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 > > > > > > The maven artifacts are available via repository.apache.org. > > > > > > Please try the release and vote; the vote will run for the usual 7 > days. > > > > > > thanks, > > > Arun > > > > > > -- > > > Arun C. Murthy > > > Hortonworks Inc. > > > http://hortonworks.com/ > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: [VOTE] Release Apache Hadoop 2.4.0
What's the state of the application history server? Do we have security, documentation, and are APIs stable? If any of these are missing, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Mon, Mar 31, 2014 at 2:22 AM, Arun C Murthy wrote: > Folks, > > I've created a release candidate (rc0) for hadoop-2.4.0 that I would like > to get released. > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 > The RC tag in svn is here: > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 > > The maven artifacts are available via repository.apache.org. > > Please try the release and vote; the vote will run for the usual 7 days. > > thanks, > Arun > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: Thinking ahead to 2.4
Looking forward to the release. What's the state of the application history server? Do we have security, documentation, and are APIs stable? If not, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Fri, Mar 14, 2014 at 8:23 PM, Arun C Murthy wrote: > Update: We are now down to just 8 blockers of which 4 are already PA. > > I know it's getting in good shape per our QE gang too. If things go well, > I plan to create an RC later half of next week. > > thanks, > Arun > > On Mar 11, 2014, at 7:11 AM, Arun C Murthy wrote: > > > Sorry, the previous link had a bug, the correct one is: > http://s.apache.org/hadoop-2.4.0-blockers. > > > > We are currently down to 12 blockers; with several PA. > > > > thanks, > > Arun > > > > On Mar 6, 2014, at 1:40 PM, Arun C Murthy wrote: > > > >> Gang, > >> > >> Most of the big-ticket items are already in, awesome! > >> > >> I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we > get through the list of blockers. Here is a handy link: > http://s.apache.org/hadoop-2.4-blockers > >> > >> If you find more, please set Target Version to 2.4.0 and mark it a > blocker. I'll try nudging people to start closing these soon, appreciate > any help! > >> > >> thanks, > >> Arun > >> > >> > >> On Feb 20, 2014, at 3:45 PM, Arun C Murthy wrote: > >> > >>> Thanks Azuryy & Suresh. I've updated the roadmap wiki to reflect this. > >>> > >>> Arun > >>> > >>> On Feb 20, 2014, at 2:01 PM, Suresh Srinivas > wrote: > >>> > Arun, > > Some of the previously 2.4 targeted features were made available in > 2.3: > - Heterogeneous storage support > - Datanode cache > > The following are being targeted for 2.4: > - Use protobuf for fsimge (already in) > - ACLs (in trunk. In a week or so, this will be merged to branch-2.4) > - Rolling upgrades (last bunch of jiras being worked in feature > branch. > Will be in 2.4 in around two weeks. Currently testing is in progress) > > So HDFS features should be ready in two weeks. > > > On Sat, Feb 15, 2014 at 4:47 PM, Azuryy wrote: > > > Hi， > > I think you omit some key pieces in 2.4 > > > > Protobuf fsimage, rolling upgrade are also targeting 2.4 > > > > > > > > Sent from my iPhone5s > > > >> On 2014年2月16日, at 6:59, Arun C Murthy wrote: > >> > >> Folks, > >> > >> With hadoop-2.3 nearly done, I think it's time to think ahead to > > hadoop-2.4. I think it was a good idea to expedite release of 2.3 > while we > > finished up pieces that didn't make it in such as HDFS Caching & > Support > > for Heterogenous Storage. > >> > >> Now, most of the key pieces incl. Resource Manager Automatic > Failover > > (YARN-149), Application History Server (YARN-321) & Application > Timeline > > Server (YARN-1530) are either complete or very close to done, and I > think > > we will benefit with an extended test-cycle for 2.4 - similar to what > > happened with 2.2. To provide some context: 2.2 went through nearly > 6 weeks > > of extended testing and it really helped us push out a very stable > release. > >> > >> I think it will be good to create a 2.4 branch ASAP and start > testing. > > As such, I plan to cut the branch early next week. With this, we > should be > > good shape sometime to release 2.4 in mid-March. > >> > >> I've updated https://wiki.apache.org/hadoop/Roadmap to reflect > this. > >> > >> Also, we should start thinking ahead to 2.5 and what folks would > like to > > see in it. If we continue our 6-week cycles, we could shoot to get > that out > > in April. > >> > >> Thoughts? > >> > >> thanks, > >> Arun > >> > >> > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity > > to > >> which it is addressed and may contain information that is > confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby > notified > > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > > immediately > >> and delete it from your system. Thank You. > > > > > > -- > http://hortonworks.com/download/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or > entity to > which it is addressed and may contain information that is > confidential, > privileged and exempt from disclosu
[jira] [Created] (MAPREDUCE-5763) Warn message about httpshuffle in NM logs
Sandy Ryza created MAPREDUCE-5763:
-
Summary: Warn message about httpshuffle in NM logs
Key: MAPREDUCE-5763
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5763
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Naren Koneru
{code}
2014-02-20 12:08:45,141 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class
class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send ServiceData
and read Service Meta Data may have issues unless the refer to the name in the
config.
2014-02-20 12:08:45,142 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding
auxiliary service httpshuffle, "mapreduce_shuffle"
{code}
I'm seeing this in my NodeManager logs, even though things work fine. A WARN
is being caused by some sort of mismatch between the name of the service (in
terms of org.apache.hadoop.service.Service.getName()) and the name of the
auxiliary service.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5759) Remove unnecessary conf load in Limits
Sandy Ryza created MAPREDUCE-5759: - Summary: Remove unnecessary conf load in Limits Key: MAPREDUCE-5759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5759 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza This is a continuation if MAPREDUCE-5487. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: [VOTE] Release Apache Hadoop 2.3.0
+1 (non-binding) Built from source and ran jobs on a pseudo-distributed cluster with the Fair Scheduler On Wed, Feb 12, 2014 at 7:56 PM, Xuan Gong wrote: > +1 (non-binding) > > downloaded the source tar ball, built, ran a number of MR jobs on a > single-node cluster and checked the job history from job history server. > > > On Wed, Feb 12, 2014 at 7:53 PM, Gera Shegalov wrote: > > > +1 non-binding > > > > - checked out the rc tag and built from source > > - deployed a pseudo-distributed cluster with > > yarn.resourcemanager.recovery.enabled=true > > - ran a sleep job with multiple map waves and a long reducer > > -- SIGKILL'd AM at various points and verified AM restart > > -- SIGKILL'd RM at various points and verified RM restart > > - checked some ui issues we had fixed. > > - verified the new restful plain text container log NM-WS > > > > Thanks, > > > > Gera > > > > > > On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy > > wrote: > > > > > Folks, > > > > > > I've created a release candidate (rc0) for hadoop-2.3.0 that I would > like > > > to get released. > > > > > > The RC is available at: > > > http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0 > > > The RC tag in svn is here: > > > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0 > > > > > > The maven artifacts are available via repository.apache.org. > > > > > > Please try the release and vote; the vote will run for the usual 7 > days. > > > > > > thanks, > > > Arun > > > > > > PS: Thanks to Andrew, Vinod & Alejandro for all their help in various > > > release activities. > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: The code for the Shuffle phase of MapReduce
Hi Pramod, For the reduce side, also check out Shuffle.java. -Sandy On Mon, Feb 10, 2014 at 2:54 AM, Pramod Biligiri wrote: > Hi, > I'm beginning to look at the code for the Shuffle phase of MapReduce, for > an academic project. > > I wanted to confirm if I have started at the right location: > I'm looking at a file called ShuffleHandler.java in the > hadoop-mapreduce-client-shuffle sub project of hadoop-mapreduce-project. > > Are there other packages/projects I should be examining that are closely > related to this? > > Thanks, > Pramod >
[jira] [Resolved] (MAPREDUCE-5745) thread may hang forever, even after it receives all the expected data
[ https://issues.apache.org/jira/browse/MAPREDUCE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5745. --- Resolution: Invalid > thread may hang forever, even after it receives all the expected data > - > > Key: MAPREDUCE-5745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5745 > Project: Hadoop Map/Reduce > Issue Type: Wish >Reporter: Jinfeng Ni >Priority: Trivial > > Please discard this JIRA issue (I should open it under a different project). > Tried to cancel this issue, but could not find a way to do so. Sorry about > this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Re-swizzle 2.3
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a problem caused by YARN-1493 I think we can revert it in branch-2.3 as well. I think we should leave them in branch-2 for now. We can revert if 2.4 is imminent and they're holding it up, but hopefully the issues they caused will be fixed by then. -Sandy On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur wrote: > Thanks Robert, > > All, > > So it seems that YARN-1493 and YARN-1490 are introducing serious > regressions. > > I would propose to revert them and the follow up JIRAs from the 2.3 branch > and keep working on them on trunk/branch-2 until the are stable (I would > even prefer reverting them from branch-2 not to block a 2.4 if they are not > ready in time). > > As I've mentioned before, the list of JIRAs to revert were: > > YARN-1493 > YARN-1490 > YARN-1166 > YARN-1041 > YARN-1566 > > Plus 2 additional JIRAs committed since my email on this issue 2 days ago: > > *YARN-1661 > *YARN-1689 (not sure if this JIRA is related in functionality to the > previous ones but it is creating conflicts). > > I think we should hold on continuing work on top of something that is > broken until the broken stuff is fixed. > > Quoting Arun, "Committers - Henceforth, please use extreme caution while > committing to branch-2.3. Please commit *only* blockers to 2.3." > > YARN-1661 & YARN-1689 are not blockers. > > Unless there are objections, I'll revert all these JIRAs from branch-2.3 > tomorrow around noon and I'll update fixedVersion in the JIRAs. > > I'm inclined to revert them from branch-2 as well. > > Thoughts? > > Thanks. > > > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter > wrote: > > > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it > was > > causing some strange behavior in the Oozie unit tests: > > > > Basically, we use a single MiniMRCluster and MiniDFSCluster across all > unit > > tests in a module. With YARN-1490 we saw that, regardless of test order, > > the last few tests would timeout waiting for an MR job to finish; on > slower > > machines, the entire test suite would timeout. Through some digging, I > > found that we were getting a ton of "Connection refused" Exceptions on > > LeaseRenewer talking to the NN and a few on the AM talking to the RM. > > > > After a bunch of investigation, I found that the problem went away once > > YARN-1490 was removed. Though I couldn't figure out the exact problem. > > Even though this occurred in unit tests, it does make me concerned that > it > > could indicate some bigger issue in a long-running real cluster (where > > everything isn't running on the same machine) that we haven't seen yet. > > > > > > > > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla > > wrote: > > > > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly. > > Will > > > pull it out of branch-2.3 if anyone objects. > > > > > > > > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal < > aagar...@hortonworks.com > > > >wrote: > > > > > > > Merged HADOOP-10273 to branch-2.3 as r1565456. > > > > > > > > > > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal < > > aagar...@hortonworks.com > > > > >wrote: > > > > > > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3. > > > > > > > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees. > > > > > > > > > > > > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur < > > t...@cloudera.com > > > > >wrote: > > > > > > > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very > > odd > > > > >> ways > > > > >> (to the point it seems un-deterministic). > > > > >> > > > > >> I'd say eiher YARN-1577 is fixed or we revert > > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean > > > reverts) > > > > >> from Hadoop 2.3 branch before doing the release. > > > > >> > > > > >> > > > > >> I've verified that after reverting those JIRAs things work fine > with > > > > >> unmanaged AMs. > > > > >> > > > > >> Thanks. > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy < > a...@hortonworks.com > > > > > > > >> wrote: > > > > >> > > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue. > > > > >> > > > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much > > > > progress > > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any > > > > >> objections? > > > > >> > > > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered. > > > > >> > > > > > >> > Overall, I'll try get this out in next couple of days if we can > > > clear > > > > >> the > > > > >> > list. > > > > >> > > > > > >> > thanks, > > > > >> > Arun > > > > >> > > > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy > > > > > wrote: > > > > >> > > > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we > > are > > > > now > > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN. > > > > >> > > > > > >
Re: Re-swizzle 2.3
Going forward with commits because it seems like others have been doing so On Mon, Jan 27, 2014 at 1:31 PM, Sandy Ryza wrote: > We should hold off commits until that's done, right? > > > On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy wrote: > >> Yep, on it as we speak. :) >> >> >> Arun >> >> On Jan 27, 2014, at 12:36 PM, Jason Lowe wrote: >> >> > Thanks, Arun. Are there plans to update the Fix Versions and >> CHANGES.txt accordingly? There are a lot of JIRAs that are now going to >> ship in 2.3.0 but the JIRA and CHANGES.txt says they're not fixed until >> 2.4.0. >> > >> > Jason >> > >> > On 01/27/2014 08:47 AM, Arun C Murthy wrote: >> >> Done. I've re-created branch-2.3 from branch-2. >> >> >> >> thanks, >> >> Arun >> >> >> >> On Jan 23, 2014, at 2:40 AM, Arun Murthy wrote: >> >> >> >>> Based on the discussion at common-dev@, we've decided to target 2.3 >> >>> off the tip of branch-2 based on the 2 major HDFS features which are >> >>> Heterogenous Storage (HDFS-2832) and HDFS Cache (HDFS-4949). >> >>> >> >>> I'll create a new branch-2.3 on (1/24) at 6pm PST. >> >>> >> >>> thanks, >> >>> Arun >> >> -- >> >> Arun C. Murthy >> >> Hortonworks Inc. >> >> http://hortonworks.com/ >> >> >> >> >> >> >> > >> >> -- >> Arun C. Murthy >> Hortonworks Inc. >> http://hortonworks.com/ >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified >> that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender >> immediately >> and delete it from your system. Thank You. >> > >
Re: Doubt in Yarn Scheduler.
Hi Suresh, The schedulers used in MR1 are not compatible with the schedulers used in MR2 / YARN. Check out ResourceScheduler.java for the updated interface. I believe that speculative execution still works the same way. -Sandy On Sat, Jan 25, 2014 at 9:47 AM, Suresh S wrote: > Hello Friends, > > I have the following doubts. > > Is the schedulers(Capacity Scheduler, Fair Scheduler, etc ) used in MR 1.x > are compatible with 2.x ? > > If it is not compatible means, what are the > changes/modifications/difference between them? > > I am working to implement a modified delay scheduling in HFS for 1.x > version. > How it could be modified for 2.x versions? > > What about Speculative Execution in MRv2. Is the any changes? > > Help me in this regard. Thanks in advance. > > *Regards* > *S.Suresh,* > *+91-9941506562* >
Re: Re-swizzle 2.3
We should hold off commits until that's done, right? On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy wrote: > Yep, on it as we speak. :) > > > Arun > > On Jan 27, 2014, at 12:36 PM, Jason Lowe wrote: > > > Thanks, Arun. Are there plans to update the Fix Versions and > CHANGES.txt accordingly? There are a lot of JIRAs that are now going to > ship in 2.3.0 but the JIRA and CHANGES.txt says they're not fixed until > 2.4.0. > > > > Jason > > > > On 01/27/2014 08:47 AM, Arun C Murthy wrote: > >> Done. I've re-created branch-2.3 from branch-2. > >> > >> thanks, > >> Arun > >> > >> On Jan 23, 2014, at 2:40 AM, Arun Murthy wrote: > >> > >>> Based on the discussion at common-dev@, we've decided to target 2.3 > >>> off the tip of branch-2 based on the 2 major HDFS features which are > >>> Heterogenous Storage (HDFS-2832) and HDFS Cache (HDFS-4949). > >>> > >>> I'll create a new branch-2.3 on (1/24) at 6pm PST. > >>> > >>> thanks, > >>> Arun > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > >> > > > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
[jira] [Created] (MAPREDUCE-5732) Report proper queue when job has been automatically placed
Sandy Ryza created MAPREDUCE-5732: - Summary: Report proper queue when job has been automatically placed Key: MAPREDUCE-5732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5732 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza Some schedulers, such as the Fair Scheduler, provide the ability to automatically place an application into a queue based on attributes such as the user and group of the submitter. In these cases, the JobHistoryServer and AM web UI report the requested queue, not the queue that the app is actually running in. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5725) TestNetworkedJob relies on the Capacity Scheduler
Sandy Ryza created MAPREDUCE-5725: - Summary: TestNetworkedJob relies on the Capacity Scheduler Key: MAPREDUCE-5725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5725 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza We should either make this explicit or make it scheduler-agnostic. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-5712) Backport Fair Scheduler pool placement by secondary group
[ https://issues.apache.org/jira/browse/MAPREDUCE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5712. --- Resolution: Fixed Assignee: Ted Malaska Hadoop Flags: Reviewed I just committed this to branch-1 > Backport Fair Scheduler pool placement by secondary group > - > > Key: MAPREDUCE-5712 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5712 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: scheduler >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5712 > > > YARN-1423 introduced a quue police that support selecting a queue if a > secondary group was found in the defined queues. This functionality would be > useful and minimally invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5651. --- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I just committed this to branch-1. Thanks Ted! > Backport Fair Scheduler queue placement policies to branch-1 > > > Key: MAPREDUCE-5651 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: scheduler > Reporter: Sandy Ryza >Assignee: Ted Malaska > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5651.2.patch, MAPREDUCE-5651.3.patch, > MAPREDUCE-5651.4.patch, MAPREDUCE-5651.5.patch, MAPREDUCE-5651.patch > > > YARN-1392 introduced general policies for assigning applications to queues in > the YARN fair scheduler. This functionality would be useful and minimally > invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5665) Add audience annotations to MiniMRYarnCluster and MiniMRCluster
Sandy Ryza created MAPREDUCE-5665: - Summary: Add audience annotations to MiniMRYarnCluster and MiniMRCluster Key: MAPREDUCE-5665 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5665 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.2.0 Reporter: Sandy Ryza We should make it clear whether these are public interfaces. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1
Sandy Ryza created MAPREDUCE-5651: - Summary: Backport Fair Scheduler queue placement policies to branch-1 Key: MAPREDUCE-5651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza YARN-1392 introduced general policies for assigning applications to queues in the YARN fair scheduler. This functionality would be useful and minimally invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Next releases
Here are few patches that I put into 2.2.1 and are minimally invasive, but I don't think are blockers: YARN-305. Fair scheduler logs too many "Node offered to app" messages. YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication YARN-1333. Support blacklisting in the Fair Scheduler YARN-1109. Demote NodeManager "Sending out status for container" logs to debug (haosdent via Sandy Ryza) YARN-1388. Fair Scheduler page always displays blank fair share +1 to doing releases at some fixed time interval. -Sandy On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy wrote: > > On Nov 12, 2013, at 1:54 PM, Todd Lipcon wrote: > > > On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe >wrote: > > > >> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be > >> there. However, I have only been following the HDFS and common side > >> of things so I may not have the full picture. Arun, can you give a > >> specific example of something you'd like to "blow away"? > > There are bunch of issues in YARN/MapReduce which clearly aren't > *critical*, similarly in HDFS a cursory glance showed up some > *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a > patch release, plus things like: > > HADOOP-9623 > Update jets3t dependency to 0.9.0 > > > > > > > > > > Having said that, the HDFS devs know their code the best. > > > I agree with Colin. If we've been backporting things into a patch release > > (third version component) which don't belong, we should explicitly call > out > > those patches, so we can learn from our mistakes and have a discussion > > about what belongs. > > Good point. > > Here is a straw man proposal: > > > A patch (third version) release should only include *blocker* bugs which > are critical from an operational, security or data-integrity issues. > > This way, we can ensure that a minor series release (2.2.x or 2.3.x or > 2.4.x) is always release-able, and more importantly, deploy-able at any > point in time. > > > > Sandy did bring up a related point about timing of releases and the urge > for everyone to cram features/fixes into a dot release. > > So, we could remedy that situation by doing a release every 4-6 weeks > (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs. > > Thoughts? > > thanks, > Arun > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
[jira] [Created] (MAPREDUCE-5619) Separate out configuration loading from QueueManager in the Fair Scheduler
Sandy Ryza created MAPREDUCE-5619: - Summary: Separate out configuration loading from QueueManager in the Fair Scheduler Key: MAPREDUCE-5619 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5619 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Next releases
Starting afresh with 2.2.1 and keeping it as small as possible sounds reasonable to me. Would love to get 2.3 out soon. To that end, how would people feel about having code and/or feature freeze and/or ship dates? We've been way behind out goals for recent releases. Having actual targets on the calendar would help us achieve the regular release cadence that can really benefit both downstream projects and ourselves. Without this, we get in the habit of stuffing changes into the closest release, even if minor or patch, for fear that the next will be many months away. Vinod, Zhijie, or Mayank, please correct me if I'm wrong, but my intuition is that end-of-year is a little unrealistic for YARN-321. Will the ~30 working days before the end of the year be enough to complete the feature, stabilize it, bring APIs to the point that we won't need to break them in the future, and merge the branch? Could we either push the feature out or aim for the end of January instead? Assuming the latter, some strawman dates: Feature freeze - 1/6 Code freeze - 1/16 Release - 1/30 Lastly, I'd like to add finer-grained CPU scheduling a la YARN-1089 to the target features. thanks, Sandy On Thu, Nov 7, 2013 at 6:42 PM, Arun C Murthy wrote: > Gang, > > Thinking through the next couple of releases here, appreciate f/b. > > # hadoop-2.2.1 > > I was looking through commit logs and there is a *lot* of content here > (81 commits as on 11/7). Some are features/improvements and some are fixes > - it's really hard to distinguish what is important and what isn't. > > I propose we start with a blank slate (i.e. blow away branch-2.2 and > start fresh from a copy of branch-2.2.0) and then be very careful and > meticulous about including only *blocker* fixes in branch-2.2. So, most of > the content here comes via the next minor release (i.e. hadoop-2.3) > > In future, we continue to be *very* parsimonious about what gets into a > patch release (major.minor.patch) - in general, these should be only > *blocker* fixes or key operational issues. > > # hadoop-2.3 > > I'd like to propose the following features for YARN/MR to make it into > hadoop-2.3 and punt the rest to hadoop-2.4 and beyond: > * Application History Server - This is happening in a branch and is > close; with it we can provide a reasonable experience for new frameworks > being built on top of YARN. > * Bug-fixes in RM Restart > * Minimal support for long-running applications (e.g. security) via > YARN-896 > * RM Fail-over via ZKFC > * Anything else? > > HDFS??? > > Overall, I feel like we have a decent chance of rolling hadoop-2.3 by the > end of the year. > > Thoughts? > > thanks, > Arun > > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
[jira] [Created] (MAPREDUCE-5612) Document TaskAttemptCompletionStatuses
Sandy Ryza created MAPREDUCE-5612: - Summary: Document TaskAttemptCompletionStatuses Key: MAPREDUCE-5612 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5612 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Priority: Minor What's the difference between FAILED and TIPFAILED? What is OBSOLETE? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
Sandy Ryza created MAPREDUCE-5608: - Summary: Replace and deprecate mapred.tasktracker.indexcache.mb Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size of the shuffle service index cache. As the tasktracker no longer exists, we should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5601) Fetches when reducer can't fit them result in unnecessary reads on shuffle server
Sandy Ryza created MAPREDUCE-5601: - Summary: Fetches when reducer can't fit them result in unnecessary reads on shuffle server Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5599) JobHistoryServer unnecessarily copies all jobs on each query
Sandy Ryza created MAPREDUCE-5599: - Summary: JobHistoryServer unnecessarily copies all jobs on each query Key: MAPREDUCE-5599 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5599 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Instead, in CachedHistoryStorage, we should only copy jobs that will be returned. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5596) Allow configuring the number of threads used to serve shuffle connections
Sandy Ryza created MAPREDUCE-5596: - Summary: Allow configuring the number of threads used to serve shuffle connections Key: MAPREDUCE-5596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5596 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza MR1 had mapreduce.tasktracker.http.threads. MR2 always uses the Netty default 2 * Runtime.availableProcessors(). We should make this configurable. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5592) Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1
Sandy Ryza created MAPREDUCE-5592: - Summary: Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1 Key: MAPREDUCE-5592 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5592 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task-controller Reporter: Sandy Ryza Assignee: Sandy Ryza MAPREDUCE-1119 dumps stack traces on a task timeout, making it easier this difficult case easier to debug. This made it into 0.21, but never into branch-1, and the backport very very dirty. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: streaming documentation in Hadoop 2?
Doc existed in MR1 http://hadoop.apache.org/docs/stable/streaming.html, but it looks like it and a bunch of other stuff (e.g. Rumen and the MapReduce Tutorial) weren't ported over. On Mon, Oct 14, 2013 at 3:20 PM, Eli Collins wrote: > It probably just needs doc, I'd go ahead and file a jira for it. The > wiki content here could be a good starting point. > > On Mon, Oct 14, 2013 at 2:56 PM, Sandy Ryza > wrote: > > Hi All, > > > > I noticed that the hadoop streaming documentation does not exist in the > > Hadoop 2 source tree, and also cannot be found on the internet. Is this > > on purpose? I found this wiki page > > http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is > > supposed to go? As this page isn't tied to a specific version, how does > it > > work if new options are added? > > > > thanks, > > -Sandy >
streaming documentation in Hadoop 2?
Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't tied to a specific version, how does it work if new options are added? thanks, -Sandy
[jira] [Created] (MAPREDUCE-5578) Miscellaneous Fair Scheduler speedups
Sandy Ryza created MAPREDUCE-5578: - Summary: Miscellaneous Fair Scheduler speedups Key: MAPREDUCE-5578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5578 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
Sandy Ryza created MAPREDUCE-5577: - Summary: Allow querying the JobHistoryServer by job arrival time Key: MAPREDUCE-5577 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Sandy Ryza Assignee: Sandy Ryza The JobHistoryServer REST APIs currently allow querying by job submit time and finish time. However, jobs don't necessarily arrive in order of their finish time, meaning that a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. The arrival time of a job would be defined as the time that it lands in the done directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
pipes not working in MR2?
I'm unable to get a simple hadoop pipes job working in MR2, and got the sense it hasn't been working for a while. Does anybody have any insight into what's going on? Has anybody used them successfully recently? thanks for any help, Sandy
[jira] [Created] (MAPREDUCE-5575) History files deleted from the intermediate directory never get removed from the JobListCache
Sandy Ryza created MAPREDUCE-5575: - Summary: History files deleted from the intermediate directory never get removed from the JobListCache Key: MAPREDUCE-5575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5575 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.2.0 Reporter: Sandy Ryza The JobHistoryServer periodically scans through the intermediate directory. It adds all files to the JobListCache. It deletes job files that are older than the max age and moves all other files to the done directory. Later, when files in the done directory become too old, they're deleted from the JobListCache. Jobs that were deleted in the intermediate directory (and thus never moved to the done directory) end up in the JobListCache but can never be deleted from it. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (non-binding) Built from source and ran a few jobs on a pseudo-distributed cluster with the Fair Scheduler. On Tue, Oct 8, 2013 at 6:48 AM, Thomas Graves wrote: > +1. > > Downloaded, verified signature/md5, CHANGES.txt, NOTICE, LICENSE, README, > release notes, built the source tar ball, and ran a few small jobs on a > pseudo-distributed cluster. > > Tom > > On 10/7/13 2:00 AM, "Arun C Murthy" wrote: > > >Folks, > > > >I've created a release candidate (rc0) for hadoop-2.2.0 that I would like > >to get released - this release fixes a small number of bugs and some > >protocol/api issues which should ensure they are now stable and will not > >change in hadoop-2.x. > > > >The RC is available at: > >http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 > >The RC tag in svn is here: > >http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 > > > >The maven artifacts are available via repository.apache.org. > > > >Please try the release and vote; the vote will run for the usual 7 days. > > > >thanks, > >Arun > > > >P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail > >down the symlinks-related issues. I'll release note the fact that we have > >disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the > >YARN side in the last couple of weeks. > > > > > > > > > > > >-- > >Arun C. Murthy > >Hortonworks Inc. > >http://hortonworks.com/ > > > > > > > >-- > >CONFIDENTIALITY NOTICE > >NOTICE: This message is intended for the use of the individual or entity > >to > >which it is addressed and may contain information that is confidential, > >privileged and exempt from disclosure under applicable law. If the reader > >of this message is not the intended recipient, you are hereby notified > >that > >any printing, copying, dissemination, distribution, disclosure or > >forwarding of this communication is strictly prohibited. If you have > >received this communication in error, please contact the sender > >immediately > >and delete it from your system. Thank You. > >
JobClient.getRootQueues returns default, not root?
Apparently JobClient.getRootQueues returns the default queue, not the root queue. Is this the correct behavior? It might make sense for the FIFO scheduler, but not for the Fair and Capacity schedulers. thanks for any guidance -Sandy
[jira] [Created] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
Sandy Ryza created MAPREDUCE-5544: - Summary: JobClient#getJob loads job conf twice Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Calling JobClient#getJob causes the job conf file to be loaded twice, once in the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We should remove the former. MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive tests, but the problem persists because it only fixed one of the places where the job conf file is loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5527) Add CONTAINERS_MILLIS_MAPS|REDUCES counters
Sandy Ryza created MAPREDUCE-5527: - Summary: Add CONTAINERS_MILLIS_MAPS|REDUCES counters Key: MAPREDUCE-5527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5527 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza It would be helpful to have a counters which report the total wallclock time spent in all map/reduce tasks. This is what SLOTS_MILLIS_MAPS usually did in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
Sandy Ryza created MAPREDUCE-5487: - Summary: In task processes, JobConf is unnecessarily loaded again in Limits Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5484) YarnChild unnecessarily loads job conf twice
Sandy Ryza created MAPREDUCE-5484: - Summary: YarnChild unnecessarily loads job conf twice Key: MAPREDUCE-5484 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5484 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza In MR task processes, a JobConf is instantiated with the same job.xml twice, once at the beginning of main() and once in configureTask. IIUC, the second instantiation is not necessary. These take time reading from disk and parsing XML. Removing the second instantiation shaved a second off the average map task time in a 1,000-map sleep job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5478) TeraInputFormat unnecessarily defines its own FileSplit subclass
Sandy Ryza created MAPREDUCE-5478: - Summary: TeraInputFormat unnecessarily defines its own FileSplit subclass Key: MAPREDUCE-5478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5478 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor TeraInputFormat defines its own TeraFileSplit subclass of FileSplit that adds a locations field, which is already included in FileSplit. This is causing MR2 TeraSort to fail on MR1, which, for a System.arraycopy, requires splits to be of the FileSplit class. While nobody is promising that everything that runs on MR2 should run on MR1, fixing this would be easy and make it possible to compare MR2 TeraSort performance between MR1 and MR2. We should just get rid of TeraFileSplit and use FileSplit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
+1 (non-binding) Built from source, ran jobs on a pseudo-distributed cluster with the Fair Scheduler. -Sandy On Tue, Aug 20, 2013 at 7:27 PM, Arun C Murthy wrote: > Thanks for the heads up Aaron, I've changed fix-version of HDFS-4763 to > 2.1.1-beta for now. > > Committers - please be careful setting fix-versions, this is a good > anti-pattern to avoid… though, I'm willing to bet a lot of dough that this > isn't the first Hadoop release with this issue… *smile* > > Arun > > > On Aug 20, 2013, at 6:09 PM, Aaron T. Myers wrote: > > > I was evaluating the release bits when I noticed that the change done in > > HDFS-4763 to add support for starting the HDFS NFSv3 gateway, which is > > marked with a "fix version" of 2.1.0-beta and included in the release > notes > > of RC2, is not in fact included in the RC2 release bits. It looks to me > > like the change is included in branch-2.1-beta, but not > branch-2.1.0-beta. > > > > Particularly since the release notes in RC2 are incorrect in claiming > that > > this change is in this release, it seems like a pretty serious > > issue. Ordinarily I'd say that this issue should result in a new RC, and > I > > would vote -1 on RC2. But, given the previous discussion that folks are > > interested in releasing 2.1.0-beta with several fairly substantial bugs > > that we already know about, I'll withhold my vote. If RC2 ends up getting > > released as-is, we should be sure to change the fix version field on that > > JIRA to be correct. > > > > -- > > Aaron T. Myers > > Software Engineer, Cloudera > > > > > > On Thu, Aug 15, 2013 at 2:15 PM, Arun C Murthy > wrote: > > > >> Folks, > >> > >> I've created a release candidate (rc2) for hadoop-2.1.0-beta that I > would > >> like to get released - this fixes the bugs we saw since the last > go-around > >> (rc1). > >> > >> The RC is available at: > >> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ > >> The RC tag in svn is here: > >> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 > >> > >> The maven artifacts are available via repository.apache.org. > >> > >> Please try the release and vote; the vote will run for the usual 7 days. > >> > >> thanks, > >> Arun > >> > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or > entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > immediately > >> and delete it from your system. Thank You. > >> > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Vinod, your thinking makes sense to me. My two cents are that we should hold off on fixes until 2.1.1-beta. Unless there are downstream projects that need it to work for integration testing. -Sandy On Mon, Aug 19, 2013 at 2:49 PM, Vinod Kumar Vavilapalli wrote: > > Thanks for the clarification Daryn. That is what I was asking before when > I said "From my limited understanding, this doesn't seem like a API or a > compatibility issue. Can we not fix it in subsequent bug-fix releases? " > > Now, I was wavering as to whether we can omit this or not. There are other > such things like YARN-1082 (RM restart doesn't work in security), YARN-49 > (dist-shell in secure mode) etc. that is essentially a loss of some > functionality in some cases (secure mode). > > Flipping back the question, aren't we okay release now with all the API > changes that are already in and then immediately follow up with a bunch of > bug-fix releases or wait till 'everything' gets fixed. > > Posing the question like that, I am willing to move ahead without waiting > for these fixes, but that's just me. > > What do others think? > > Thanks, > +Vinod > > On Aug 19, 2013, at 8:25 AM, Daryn Sharp wrote: > > I've been OOO (got a call to fix this bug), but just to clarify: > > We're ok with HA _not working at all_ with security enabled in 2.1.0-beta? > That's the ramification of omitting HADOOP-9880. > > Daryn > > On Aug 16, 2013, at 9:20 PM, Arun C Murthy wrote: > > Yep, there are quite a number of such fixes in 2.1.1 ATM, I think it will > serve us better to get 2.1.0 out and then quickly turn around to make 2.1.1. > > > My current plan is to start work on 2.1.1 right after this release gets > complete… hopefully next week. > > > thanks, > > Arun > > > On Aug 16, 2013, at 4:36 PM, Vinod Kumar Vavilapalli < > vino...@hortonworks.com> wrote: > > > > There are other such isolated and well understood bug-fixes that we pushed > to 2.1.1 in the interesting of making progress with 2.1.0 and the > corresponding API changes. > > > 2.1.1 should happen soon enough after this. > > > Thanks, > > +Vinod > > > On Aug 16, 2013, at 4:22 PM, Roman Shaposhnik wrote: > > > What are the downsides of getting this fix into the 2.1? It appears > > that the fix is pretty isolated and well understood. > > > Thoughts? > > > Thanks, > > Roman. > > > On Fri, Aug 16, 2013 at 3:04 PM, Kihwal Lee wrote: > > I've changed the target version of HADOOP-9880 to 2.1.1. Please change it > back, if you feel that it needs to be in 2.1.0-beta. > > > > Kihwal > > > > > > From: Kihwal Lee > > To: Arun Murthy ; "common-...@hadoop.apache.org" < > common-...@hadoop.apache.org> > > Cc: "mapreduce-dev@hadoop.apache.org" ; " > hdfs-...@hadoop.apache.org" ; " > yarn-...@hadoop.apache.org" > > Sent: Friday, August 16, 2013 4:55 PM > > Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta > > > > It's your call, Arun. I.e. as long you believe rc2 meets the expectations > and objectives of 2.1.0-beta. > > > Kihwal > > > > > > From: Arun Murthy > > To: "common-...@hadoop.apache.org" > > Cc: Kihwal Lee ; "mapreduce-dev@hadoop.apache.org" < > mapreduce-dev@hadoop.apache.org>; "hdfs-...@hadoop.apache.org" < > hdfs-...@hadoop.apache.org>; "yarn-...@hadoop.apache.org" < > yarn-...@hadoop.apache.org> > > Sent: Friday, August 16, 2013 3:44 PM > > Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta > > > > That makes sense too. > > > > On Aug 16, 2013, at 10:39 AM, Vinod Kumar Vavilapalli > > wrote: > > > > We need to make a call on what blockers will be. From my limited > understanding, this doesn't seem like a API or a compatibility issue. Can > we not fix it in subsequent bug-fix releases? > > > I do see a lot of follow up releases to 2.1.0. Getting this release out > will help downstream projects start testing with all the API stuff that has > already gone in 2.1.0. > > > Thanks, > > +Vinod > > > On Aug 16, 2013, at 10:13 AM, Kihwal Lee wrote: > > > We have found HADOOP-9880, which prevents Namenode HA from running with > security. > > > Kihwal > > > > > > From: Arun C Murthy > > To: "common-...@hadoop.apache.org" ; " > hdfs-...@hadoop.apache.org" ; " > mapreduce-dev@hadoop.apache.org" ; " > yarn-...@hadoop.apache.org" > > Sent: Thursday, August 15, 2013 4:15 PM > > Subject: [VOTE] Release Apache Hadoop 2.1.0-beta > > > > Folks, > > > I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would > like to get released - this fixes the bugs we saw since the last go-around > (rc1). > > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ > > The RC tag in svn is here: > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 > > > The maven artifacts are available via repository.apache.org. > > > Please try the release and vote; the vote will run for the usual 7 days. > > > thanks, > > Arun > > > -- > > Arun C. Murthy > > Hort
[jira] [Created] (MAPREDUCE-5464) Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter
Sandy Ryza created MAPREDUCE-5464: - Summary: Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter Key: MAPREDUCE-5464 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5464 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Per discussion on MAPREDUCE-5311, it would be good to have analogs for SLOTS_MILLIS that better fit the MR2 resource model. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5463) Deprecate SLOTS_MILLIS counters
Sandy Ryza created MAPREDUCE-5463: - Summary: Deprecate SLOTS_MILLIS counters Key: MAPREDUCE-5463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5463 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza As discussed in MAPREDUCE-5311, the SLOTS_MILLIS_MAPS and SLOTS_MILLIS_REDUCES counters don't really make sense in MR2, and should be deprecated so that they can eventually be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5462) In map-side sort, swap entire meta entries instead of indexes for better cache performance
Sandy Ryza created MAPREDUCE-5462: - Summary: In map-side sort, swap entire meta entries instead of indexes for better cache performance Key: MAPREDUCE-5462 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5462 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5457) Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators
Sandy Ryza created MAPREDUCE-5457: - Summary: Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators Key: MAPREDUCE-5457 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5457 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: Sandy Ryza MR jobs sometimes want to just output lines of text, not key/value pairs. TextOutputFormat handles this by, if a null value is given, outputting only the key with no separator. Streaming jobs are unable to take advantage of this, because they can't output null values. A text output format that ignores values and only outputs keys would allow streaming jobs to output lines of text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5450. --- Resolution: Fixed Hadoop Flags: Reviewed > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > fra
[jira] [Reopened] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5311: --- > Remove slot millis computation logic and deprecate counter constants > > > Key: MAPREDUCE-5311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.0.4-alpha >Reporter: Alejandro Abdelnur >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5311.patch, MAPREDUCE-5311.patch > > > Per discussion in MAPREDUCE-5310 and comments in the code we should remove > all the related logic and just leave the counter constant for backwards > compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5059. --- Resolution: Fixed > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta, 0.23.8 > > Attachments: MAPREDUCE-5059-20130325.patch, > MAPREDUCE-5059-20130410.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5059: --- > Job overview shows average merge time larger than for any reduce attempt > > > Key: MAPREDUCE-5059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, webapps >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.1.0-beta, 0.23.8 > > Attachments: MAPREDUCE-5059-20130325.patch, > MAPREDUCE-5059-20130410.patch > > > When looking at a job overview page on the history server, the Average Merge > Time is often reported with a value that is far larger than the Elapsed Merge > Time shown for any reduce task attempt. The job overview page calculates the > merge time as the time delta between the sort finishing and the job launching > while the attempts page calculates it as the time delta between the sort > finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5420) Remove mapreduce.task.tmp.dir from mapred-default.xml
Sandy Ryza created MAPREDUCE-5420: - Summary: Remove mapreduce.task.tmp.dir from mapred-default.xml Key: MAPREDUCE-5420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5420 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza mapreduce.task.tmp.dir no longer has any effect, so it should no longer be documented in mapred-default. (There is no YARN equivalent for the property. It now is just always ./tmp). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5403) Get rid of yarn.application.classpath
Sandy Ryza created MAPREDUCE-5403: - Summary: Get rid of yarn.application.classpath Key: MAPREDUCE-5403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza yarn.application.classpath is a confusing property because it is used by MapReduce and not YARN, and MapReduce already has mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5383) Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir
Sandy Ryza created MAPREDUCE-5383: - Summary: Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir Key: MAPREDUCE-5383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza This will allow configurations that had previously set mapreduce.jobtracker.staging.root.dir should be able to more easily transition to MR2, as well as make it clear that these properties refer to the same thing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5382) LocalJobRunner should use default FS for system and staging dirs by default
Sandy Ryza created MAPREDUCE-5382: - Summary: LocalJobRunner should use default FS for system and staging dirs by default Key: MAPREDUCE-5382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2, 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza For local jobs, staging dirs and system dirs are currently required to be placed on the local FS. I am continually bitten by permissions errors when I set mapreduce.jobtracker.staging.root.dir to /user, even when the default FS is still HDFS. I think using a different FS for staging than the default FS is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5379) Include FS delegation token ID in job conf
Sandy Ryza created MAPREDUCE-5379: - Summary: Include FS delegation token ID in job conf Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission, security Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5372) ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2
Sandy Ryza created MAPREDUCE-5372: - Summary: ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2 Key: MAPREDUCE-5372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5372 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Sandy Ryza In MR2, the 'd' in Id is lowercase, but in MR1, it is capitalized. While ControlledJob is marked as Evolving, there is no reason to be inconsistent here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5367) Local jobs all use same local working directory
Sandy Ryza created MAPREDUCE-5367: - Summary: Local jobs all use same local working directory Key: MAPREDUCE-5367 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5367 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza This means that local jobs, even in different JVMs, can't run concurrently because they might delete each other's files during work directory setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5365) Set mapreduce.job.classpath to true by default
Sandy Ryza created MAPREDUCE-5365: - Summary: Set mapreduce.job.classpath to true by default Key: MAPREDUCE-5365 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5365 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza MAPREDUCE-1700 introduced the mapreduce.job.classpath option, which uses a custom classloader to separate system classes from user classes. It seems like there are only rare cases when a user would not want this on, and that it should enabled by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5363) Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus
Sandy Ryza created MAPREDUCE-5363:
-
Summary: Fix doc and spelling for
TaskCompletionEvent#getTaskStatus and getStatus
Key: MAPREDUCE-5363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 1.1.2, 2.1.0-beta
Reporter: Sandy Ryza
The doc for TaskCompletionEvent#get(Task)Status in both MR1 and MR2 is
{code}
Returns enum Status.SUCESS or Status.FAILURE.
@return task tracker status
{code}
The actual values that the Status enum can take are
FAILED, KILLED, SUCCEEDED, OBSOLETE, TIPFAILED
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: How to Free-up a Map Slot without Killing the Entire Job?
The fair scheduler does preemption as well. -Sandy On Sat, Jun 29, 2013 at 12:05 PM, Steve Loughran wrote: > there's a scheduler in contrib/ that does pre-emption. look here > > > https://github.com/apache/hadoop-common/tree/branch-0.22/mapreduce/src/contrib/dynamic-scheduler > > On 28 June 2013 22:13, Sreejith Ramakrishnan >wrote: > > > I'm trying to implement a scheduler (EDF). The scheduler should be able > to > > kill or free-up a running map slot so that it can be assigned as a map > slot > > to another job. > > > > I did some looking around and found a kill() method in > > org.apache.hadoop.mapred.JobInProgress. But, this kills the entire job. I > > want the job to still be working after a map slot has been removed from > it. > > > > Can you guys tell me the right method/class to use? > > > > Thanks > > >
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
For YARN-791, if we can come to consensus on the correct approach, I can try to have a patch ASAP. -Sandy On Fri, Jun 28, 2013 at 12:03 PM, Hitesh Shah wrote: > Hi Arun, > > From a YARN perspective, YARN-791 and YARN-727 are 2 jiras that may > potentially change the apis. They can implemented in a backward compat > fashion if committed after 2.1.0. However, this will require adding of > differently-named apis ( different urls in case of the webservices ) and > make the current version of the api deprecated and/or obsolete. YARN-818 > which is currently patch available also changes behavior. > > Assuming that as soon as 2.1.0 is released, we are to follow a very strict > backward-compat retaining approach to all user-facing layers ( > api/webservices/rpc/... ) in common/hdfs/yarn/mapreduce, does it make sense > to try and pull them in and roll out a new RC after they are ready? Perhaps > Vinod can chime in if he is aware of any other such jiras under YARN-386 > which should be considered compat-related blockers for a 2.1.0 RC. > > thanks > -- Hitesh > > On Jun 26, 2013, at 1:17 AM, Arun C Murthy wrote: > > > Folks, > > > > I've created a release candidate (rc0) for hadoop-2.1.0-beta that I > would like to get released. > > > > This release represents a *huge* amount of work done by the community > (639 fixes) which includes several major advances including: > > # HDFS Snapshots > > # Windows support > > # YARN API stabilization > > # MapReduce Binary Compatibility with hadoop-1.x > > # Substantial amount of integration testing with rest of projects in the > ecosystem > > > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/ > > The RC tag in svn is here: > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0 > > > > The maven artifacts are available via repository.apache.org. > > > > Please try the release and vote; the vote will run for the usual 7 days. > > > > thanks, > > Arun > > > > -- > > Arun C. Murthy > > Hortonworks Inc. > > http://hortonworks.com/ > > > > > >
[jira] [Created] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
Sandy Ryza created MAPREDUCE-5351: - Summary: JobTracker memory leak caused by CleanupQueue reopening FileSystem Key: MAPREDUCE-5351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Critical When a job is completed, closeAllForUGI is called to close all the cached FileSystems in the FileSystem cache. However, the CleanupQueue may run after this occurs and call FileSystem.get() to delete the staging directory, adding a FileSystem to the cache that will never be closed. People on the user-list have reported this causing their JobTrackers to OOME every two weeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5350) Expose Fair Scheduler-specific queue metrics
Sandy Ryza created MAPREDUCE-5350: - Summary: Expose Fair Scheduler-specific queue metrics Key: MAPREDUCE-5350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5350 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When the Fair Scheduler is enabled, QueueMetrics should include fair share, minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5338) Bring back mapred.child.ulimit
Sandy Ryza created MAPREDUCE-5338: - Summary: Bring back mapred.child.ulimit Key: MAPREDUCE-5338 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5338 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza In MR1, a ulimit could be set for MapReduce child processes. For parity, this would be good to have in MR2 as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: mapred.child.ulimit in MR2
Thanks Bobby, I filed MAPREDUCE-5338 for it. -Sandy On Wed, Jun 19, 2013 at 8:16 AM, Robert Evans wrote: > Sandy, > > I think it was something that was missed in the port to YARN and the dead > code was cleaned up as part of HADOOP-8288. > > If you have a use case for it or are worried about backwards compatibility > we can add it back in. It is not that hard, all it did was add 'ulimt -v > ' to the shell script that launched the task, except on windows. > > --Bobby > > On 6/18/13 3:56 PM, "Sandy Ryza" wrote: > > >Hi yarn-dev/mapreduce-dev, > > > >Is there a reason that mapred.child.ulimit no longer has an effect in MR2? > > Should it be added back in? > > > >thanks for any help, > >-Sandy > >
mapred.child.ulimit in MR2
Hi yarn-dev/mapreduce-dev, Is there a reason that mapred.child.ulimit no longer has an effect in MR2? Should it be added back in? thanks for any help, -Sandy
[jira] [Created] (MAPREDUCE-5321) Enable better parallelism in the Fair Scheduler
Sandy Ryza created MAPREDUCE-5321: - Summary: Enable better parallelism in the Fair Scheduler Key: MAPREDUCE-5321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5321 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza Currently, the Fair Scheduler is locked on pretty much every operation, node updates, application additions and removals, every time the update thread runs, and every time the RM queries it for information. Most of this locking is unnecessary, especially as only the core scheduling operations like application additions, removals, and node updates need a consistent view of scheduler state. We can probably increase parallelism by using concurrent data structures when applicable, as well as keeping a slightly stale view to serve via the RM APIs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5182. --- Resolution: Won't Fix > LineRecordReader#getProgress throwing IOException breaks compatibility > -- > > Key: MAPREDUCE-5182 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5182 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.1.2, 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > This has been in trunk for a while (since MAPREDUCE-773), but was only > introduced into branch-1 in July. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest
[ https://issues.apache.org/jira/browse/MAPREDUCE-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5252. --- Resolution: Not A Problem This was fixed in YARN-326 > Fair scheduler should use SchedulerUtils.normalizeRequest > - > > Key: MAPREDUCE-5252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha > Reporter: Sandy Ryza >Assignee: Sandy Ryza >Priority: Minor > > The capacity scheduler and the fifo scheduler use the same normalizeRequest > in SchedulerUtils. The fair scheduler has its own version of this method > that does exactly the same thing. It should use the common one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5302) NodeManager throws AvroRuntimeException on failed start
Sandy Ryza created MAPREDUCE-5302: - Summary: NodeManager throws AvroRuntimeException on failed start Key: MAPREDUCE-5302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5302 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza NodeManager wraps exceptions that occur in its start method in AvroRuntimeExceptions, even though it doesn't use Avro anywhere else. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)
+1 (non-binding) Built from source, ran a few sample jobs on a pseudo-distributed cluster On Mon, Jun 3, 2013 at 3:22 PM, Chris Douglas wrote: > +1 > > Checksum and signature match, ran some unit tests, checked diff > against 2.0.4-alpha. > > Thanks for seeing this through, Cos. -C > > On Mon, Jun 3, 2013 at 1:13 PM, Alejandro Abdelnur > wrote: > > +1 RC2. Verified MD5 & signature, checked CHANGES.txt files, built, > > configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. > > > > > > On Mon, Jun 3, 2013 at 12:51 PM, Konstantin Boudnik > wrote: > > > >> I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha. > >> > >> The difference between rc1 and rc2 is the "optimistic release date" is > set > >> for > >> 06/06/2013 in the CHANGES.txt files. > >> > >> The binary artifact is the same - there's no need to rebuild it. The > maven > >> artifacts are the same. > >> > >> The difference between the two RCs: > >> > >> svn diff \ > >> > >> > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1/\ > >> > >> > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2/ > >> > >> New RC builds are uploaded to the web. > >> The RC is available at: > >> http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/ > >> The RC tag in svn is here: > >> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2 > >> > >> I would like to extend the vote for another three days for it is such a > >> minor > >> change that doesn't affect anything but the recorded release date. > Please > >> cast your vote before 06/06/2013 5pm PDT. > >> > >> Thanks for your patience! > >> Cos > >> > >> On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote: > >> > All, > >> > > >> > I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I > >> would > >> > like to release. > >> > > >> > This is a stabilization release that includes fixed for a couple a of > >> issues > >> > discovered in the testing with BigTop 0.6.0 release candidate. > >> > > >> > The RC is available at: > >> http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ > >> > The RC tag in svn is here: > >> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1 > >> > > >> > The maven artifacts will be available via repository.apache.org on > Sat, > >> June > >> > 1st, 2013 at 2 pm PDT as outlined here > >> > http://s.apache.org/WKD > >> > > >> > Please try the release bits and vote; the vote will run for the 3 > days, > >> > because this is just a version name change. The bits are identical to > >> the ones > >> > voted on before in > >> > http://s.apache.org/2041move > >> > > >> > Thanks for your voting > >> > Cos > >> > > >> > > > > > > > > -- > > Alejandro >
Re: [VOTE] Release Apache Hadoop 0.23.8
+1 (non-binding). Did a full build from source and ran a few sample jobs on a pseudo-distributed cluster. -Sandy On Mon, Jun 3, 2013 at 11:29 AM, Kihwal Lee wrote: > +1 I've downloaded & built the RC and ran several tests on a single node > cluster. > > Kihwal > > On 5/28/13 11:00 AM, "Thomas Graves" wrote: > > > > >I've created a release candidate (RC0) for hadoop-0.23.8 that I would like > >to release. > > > >This release is a sustaining release with several important bug fixes in > >it. The most critical one is MAPREDUCE-5211. > > > >The RC is available at: > >http://people.apache.org/~tgraves/hadoop-0.23.8-candidate-0/ > >The RC tag in svn is here: > >http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.8-rc0/ > > > >The maven artifacts are available via repository.apache.org. > > > >Please try the release and vote; the vote will run for the usual 7 days. > > > >I am +1 (binding). > > > >thanks, > >Tom Graves > > > >
[jira] [Created] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
Sandy Ryza created MAPREDUCE-5283: - Summary: Over 10 different tests have near identical implementations of AppContext Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5036: --- > Default shuffle handler port should not be 8080 > --- > > Key: MAPREDUCE-5036 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.3-alpha > Reporter: Sandy Ryza > Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036.patch > > > The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is > a pretty common port for web services, and is likely to cause unnecessary > port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Clarifications on MAPREDUCE-5183
If you're planning to fix it, it would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -Sandy On Tue, May 21, 2013 at 10:55 AM, maisnam ns wrote: > Thanks Sandy Ryza > > > > > On Tue, May 21, 2013 at 11:20 PM, Sandy Ryza >wrote: > > > Hi Niranjan, > > > > Your understanding is correct. > > > > -Sandy > > > > > > On Tue, May 21, 2013 at 1:02 AM, maisnam ns > wrote: > > > > > Hi, > > > > > > I was looking into this issue but would be happy if someone could > clarify > > > some of my doubts. > > > > > > Is the issue related to the given below snapshot of log: > > > > > > org.apache.hadoop.mapred.TaskTracker: Task > > > attempt_201305211246_0001_r_01_0 is in commit-pending, task > > > state:COMMIT_PENDING 2013-05-21 12:48:51,058 INFO > > > org.apache.hadoop.mapred.TaskTracker: > > > attempt_201305211246_0001_r_01_0 0.0% reduce > copy > 2013-05-21 > > > 12:48:51,769 DEBUG org.apache.hadoop.mapred.TaskTracker: > > > Got heartbeatResponse from JobTracker with responseId: 27 and 2 > > > actions 2013-05-21 12:48:51,769 INFO > > org.apache.hadoop.mapred.TaskTracker: > > > Received commit task action for > > > attempt_201305211246_0001_r_00_0 2013-05-21 12:48:51,769 > > > INFO org.apache.hadoop.mapred.TaskTracker: > > > Received commit task action for > > > attempt_201305211246_0001_r_01_0 2013-05-21 12:48:53,695 > > > INFO org.apache.hadoop.mapred.TaskTracker: > > > attempt_201305211246_0001_r_01_0 1.0% reduce > > > > reduce 2013-05-21 12:48:53,777 INFO > > > > > > The issue says 'TaskTracker#reportProgress logging of 0.0-1.0 progress > > > is followed by percent sign' > > > > > > > > > 1.attempt_201305211246_0001_r_01_0 0.0% reduce > > > 2.attempt_201305211246_0001_r_01_0 1.0% reduce > > > > > > If my understanding is correct the percent sign at the end of 0.0 and > 1.0 > > > > > > should be removed. > > > > > > Regards > > > > > > Niranjan Singh > > > > > >
Re: Clarifications on MAPREDUCE-5183
Hi Niranjan, Your understanding is correct. -Sandy On Tue, May 21, 2013 at 1:02 AM, maisnam ns wrote: > Hi, > > I was looking into this issue but would be happy if someone could clarify > some of my doubts. > > Is the issue related to the given below snapshot of log: > > org.apache.hadoop.mapred.TaskTracker: Task > attempt_201305211246_0001_r_01_0 is in commit-pending, task > state:COMMIT_PENDING 2013-05-21 12:48:51,058 INFO > org.apache.hadoop.mapred.TaskTracker: > attempt_201305211246_0001_r_01_0 0.0% reduce > copy > 2013-05-21 > 12:48:51,769 DEBUG org.apache.hadoop.mapred.TaskTracker: > Got heartbeatResponse from JobTracker with responseId: 27 and 2 > actions 2013-05-21 12:48:51,769 INFO org.apache.hadoop.mapred.TaskTracker: > Received commit task action for > attempt_201305211246_0001_r_00_0 2013-05-21 12:48:51,769 > INFO org.apache.hadoop.mapred.TaskTracker: > Received commit task action for > attempt_201305211246_0001_r_01_0 2013-05-21 12:48:53,695 > INFO org.apache.hadoop.mapred.TaskTracker: > attempt_201305211246_0001_r_01_0 1.0% reduce > > reduce 2013-05-21 12:48:53,777 INFO > > The issue says 'TaskTracker#reportProgress logging of 0.0-1.0 progress > is followed by percent sign' > > > 1.attempt_201305211246_0001_r_01_0 0.0% reduce > 2.attempt_201305211246_0001_r_01_0 1.0% reduce > > If my understanding is correct the percent sign at the end of 0.0 and 1.0 > > should be removed. > > Regards > > Niranjan Singh >
Re: [VOTE] Plan to create release candidate for 0.23.8
+1 (non-binding) On Sun, May 19, 2013 at 1:22 PM, Derek Dagit wrote: > +1 (non-binding) > > On May 17, 2013, at 4:14 PM, "Thomas Graves" > wrote: > > > Hello all, > > > > We've had a few critical issues come up in 0.23.7 that I think warrants a > > 0.23.8 release. The main one is MAPREDUCE-5211. There are a couple of > > other issues that I want finished up and get in before we spin it. Those > > include HDFS-3875, HDFS-4805, and HDFS-4835. I think those are on track > > to finish up early next week. So I hope to spin 0.23.8 soon after this > > vote completes. > > > > Please vote '+1' to approve this plan. Voting will close on Friday May > > 24th at 2:00pm PDT. > > > > Thanks, > > Tom Graves > > >
[jira] [Resolved] (MAPREDUCE-5236) references to JobConf.DISABLE_MEMORY_LIMIT don't make sense in the context of MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5236. --- Resolution: Duplicate > references to JobConf.DISABLE_MEMORY_LIMIT don't make sense in the context of > MR2 > - > > Key: MAPREDUCE-5236 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5236 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > In MR1, a special value of -1 could be given for > mapreduce.job.map|reduce.memory.mb when memory limits were disabled. In MR2, > this makes no sense, as with slots gone, this value is used for requesting > resources and scheduling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest
Sandy Ryza created MAPREDUCE-5252: - Summary: Fair scheduler should use SchedulerUtils.normalizeRequest Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor The capacity scheduler and the fifo scheduler use the same normalizeRequest in SchedulerUtils. The fair scheduler has its own version of this method that does exactly the same thing. It should use the common one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk
Sandy Ryza created MAPREDUCE-5238:
-
Summary: TestDistCacheEmulation.testGenerateDistCacheData is
failing in trunk
Key: MAPREDUCE-5238
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
{noformat
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec <<<
FAILURE!
testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
Time elapsed: 16767 sec <<< FAILURE!
java.lang.AssertionError: Wrong permissions for distributed cache file
/user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
expected: but was:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheEmulation.java:109)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.testGenerateDistCacheData(TestDistCacheEmulation.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5236) JobConf should not support disabling memory limits
Sandy Ryza created MAPREDUCE-5236: - Summary: JobConf should not support disabling memory limits Key: MAPREDUCE-5236 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5236 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza In MR1, a special value of -1 could be given for mapreduce.job.map|reduce.memory.mb to disable memory limits. In MR2, this makes no sense, as with slots gone, this value is used for requesting resources and scheduling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
Sandy Ryza created MAPREDUCE-5220: - Summary: Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
Sandy Ryza created MAPREDUCE-5219: - Summary: JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 Key: MAPREDUCE-5219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza We should change it back for compatibility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3946) If a resource requirement is higher than available on any node, job should fail early
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-3946. --- Resolution: Duplicate > If a resource requirement is higher than available on any node, job should > fail early > - > > Key: MAPREDUCE-3946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Todd Lipcon > > If you configure the NMs to have 1GB of RAM each, and then try to submit a > job which has an AM resource requirement of 1.5GB, the job will neither run > nor fail. Instead, it will slowly sop of all of the resources in the cluster > as "reservations" despite the fact that it will never be able to schedule > something. Instead, it should fail early indicating that the required memory > allocation is infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5134) Default settings cause LocalJobRunner to OOME
[ https://issues.apache.org/jira/browse/MAPREDUCE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5134. --- Resolution: Not A Problem > Default settings cause LocalJobRunner to OOME > - > > Key: MAPREDUCE-5134 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5134 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha > Reporter: Sandy Ryza > Assignee: Sandy Ryza > > If I run a job using the local job runner with vanilla settings, I get an out > of memory error. This seems to be because the default client memory maximum > is 128 MB, and the default io.sort.mb is 100 MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
