Re: [DISCUSS] branch-1

2015-05-08 Thread Sandy Ryza
+1 to auto-closing branch-1 issues. -Sandy On Fri, May 8, 2015 at 11:14 AM, Karthik Kambatla wrote: > Closing out the JIRAs as "Auto Closed" or "Closed due to Inactivity" seems > reasonable to me. For branch-1, we can be more aggressive. We should > probably do the same less aggressively for ot

Re: [DISCUSS] Release numbering for stable 2.8 and beyond

2015-04-27 Thread Sandy Ryza
My understanding was that the main reason that we labeled 2.0 alpha and 2.1 beta is that we wanted flexibility to make breaking API changes. Is that the case now with 2.7? I.e. do we have APIs labeled as Public / Stable that we want freedom to change in 2.8? If not, I definitely don't think the

Re: [VOTE] Release Apache Hadoop 2.5.1 RC0

2014-09-10 Thread Sandy Ryza
+1 (binding) Ran a pseudo-distributed cluster with the Fair Scheduler and ran some MapReduce example jobs. -Sandy On Wed, Sep 10, 2014 at 12:15 PM, Alejandro Abdelnur wrote: > Thanks Karthik. > > +1. > > + verified MD5 for source tarball > + verified signature for source tarball > + successful

Re: [VOTE] Migration from subversion to git for version control

2014-08-09 Thread Sandy Ryza
+1 (binding) On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla wrote: > I have put together this proposal based on recent discussion on this topic. > > Please vote on the proposal. The vote runs for 7 days. > >1. Migrate from subversion to git for version control. >2. Force-push to be di

Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated

2014-07-22 Thread Sandy Ryza
That policy makes sense to me. We should still label things @Private of course so that it can be reflected in the documentation. -Sandy On Tue, Jul 22, 2014 at 2:54 PM, Karthik Kambatla wrote: > Hi devs > > As you might have noticed, we have several classes and methods in them that > are not

Re: Where is the map input transfered to the 'map worker'/container?

2014-07-06 Thread Sandy Ryza
Hi Christian, I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of getting the bytes from the remote node. -Sandy On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote < cgr...@mail.uni-paderborn.de> wrote: > Hey, > > I'm looking for the place where the actual map input is tra

Re: Moving to JDK7, JDK8 and new major releases

2014-06-24 Thread Sandy Ryza
While we haven't codified this in our compatibility guidelines, dropping a Java version seems to me like change that needs to happen alongside a major release. In plain talk, it has the ability to break everything for users who aren't doing anything particularly unreasonable. I don't think we sho

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

2014-06-24 Thread Sandy Ryza
+1 (binding) -Sandy On Tue, Jun 24, 2014 at 7:53 AM, Devaraj K wrote: > +1 > > Thanks > Devaraj K > > > On Tue, Jun 24, 2014 at 2:23 PM, Arun C Murthy > wrote: > > > Folks, > > > > As discussed, I'd like to call a vote on changing our by-laws to change > > release votes from 7 days to 5. > >

Re: Plans of moving towards JDK7 in trunk

2014-06-23 Thread Sandy Ryza
Andrew, correct me if I'm misunderstanding, but the incompatible change that would require a major version bump is dropping support for JDK6. On Mon, Jun 23, 2014 at 1:53 PM, sanjay Radia wrote: > > On Jun 21, 2014, at 8:01 AM, Andrew Wang wrote: > > > This is why I'd like to keep my original

Re: hadoop-2.5 - June end?

2014-06-10 Thread Sandy Ryza
That sounds reasonable to me. -Sandy On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy wrote: > Folks, > > As you can see from the Roadmap wiki, it looks like several items are > still a bit away from being ready. > > I think rather than wait for them, it will be useful to create an > intermedia

[jira] [Created] (MAPREDUCE-5896) Allow InputSplits to indicate which locations have the block cached in memory

2014-05-20 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5896: - Summary: Allow InputSplits to indicate which locations have the block cached in memory Key: MAPREDUCE-5896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5896

Re: Thinking ahead

2014-04-13 Thread Sandy Ryza
to point out that the proposal to merge AHS into trunk stated an intent to tie ends up on security. On Sun, Apr 13, 2014 at 7:50 PM, Sandy Ryza wrote: > Unfortunately I don't have the bandwidth to take on ATS security at this > time. My (I now understand mistaken) impression that i

Re: Thinking ahead

2014-04-13 Thread Sandy Ryza
if it meant extending the target release date a couple weeks - I of course had no intention to volunteer anybody for it. On Sun, Apr 13, 2014 at 5:50 PM, Arun C Murthy wrote: > Sandy, > > On Apr 12, 2014, at 10:09 AM, Sandy Ryza wrote: > > I'm having trouble editing the w

Re: Thinking ahead

2014-04-12 Thread Sandy Ryza
+1 for starting to think about 2.5. Early June seems a little early to me - we had talked about a quarterly release cadence and this would be about half that. I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list.

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-04-03 Thread Sandy Ryza
While the Scheduler Load Simulator isn't part of YARN's core, it's a tool that YARN includes, and it's broken entirely in the current RC. YARN-1726 seems to me like something worth including in the release. -Sandy On Thu, Apr 3, 2014 at 11:12 AM, Xuan Gong wrote: > +1 non-binding > > Built fr

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-03-31 Thread Sandy Ryza
eline > store is for storing per-framework data only, while we eventually hope to > move the generic data there as well. > > Thanks, > Zhijie > > > > > On Mon, Mar 31, 2014 at 11:02 AM, Sandy Ryza >wrote: > > > What's the state of the application hist

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-03-31 Thread Sandy Ryza
What's the state of the application history server? Do we have security, documentation, and are APIs stable? If any of these are missing, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Mon, Mar 31, 2014 at 2:22 AM, Arun C Murthy wrote:

Re: Thinking ahead to 2.4

2014-03-20 Thread Sandy Ryza
Looking forward to the release. What's the state of the application history server? Do we have security, documentation, and are APIs stable? If not, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Fri, Mar 14, 2014 at 8:23 PM, Arun C Mur

[jira] [Created] (MAPREDUCE-5763) Warn message about httpshuffle in NM logs

2014-02-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5763: - Summary: Warn message about httpshuffle in NM logs Key: MAPREDUCE-5763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5763 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5759) Remove unnecessary conf load in Limits

2014-02-18 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5759: - Summary: Remove unnecessary conf load in Limits Key: MAPREDUCE-5759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5759 Project: Hadoop Map/Reduce

Re: [VOTE] Release Apache Hadoop 2.3.0

2014-02-13 Thread Sandy Ryza
+1 (non-binding) Built from source and ran jobs on a pseudo-distributed cluster with the Fair Scheduler On Wed, Feb 12, 2014 at 7:56 PM, Xuan Gong wrote: > +1 (non-binding) > > downloaded the source tar ball, built, ran a number of MR jobs on a > single-node cluster and checked the job history

Re: The code for the Shuffle phase of MapReduce

2014-02-10 Thread Sandy Ryza
Hi Pramod, For the reduce side, also check out Shuffle.java. -Sandy On Mon, Feb 10, 2014 at 2:54 AM, Pramod Biligiri wrote: > Hi, > I'm beginning to look at the code for the Shuffle phase of MapReduce, for > an academic project. > > I wanted to confirm if I have started at the right location:

[jira] [Resolved] (MAPREDUCE-5745) thread may hang forever, even after it receives all the expected data

2014-02-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5745. --- Resolution: Invalid > thread may hang forever, even after it receives all the expec

Re: Re-swizzle 2.3

2014-02-06 Thread Sandy Ryza
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a problem caused by YARN-1493 I think we can revert it in branch-2.3 as well. I think we should leave them in branch-2 for now. We can revert if 2.4 is imminent and they're holding it up, but hopefully the issues they caused wil

Re: Re-swizzle 2.3

2014-01-28 Thread Sandy Ryza
Going forward with commits because it seems like others have been doing so On Mon, Jan 27, 2014 at 1:31 PM, Sandy Ryza wrote: > We should hold off commits until that's done, right? > > > On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy wrote: > >> Yep, on it as

Re: Doubt in Yarn Scheduler.

2014-01-27 Thread Sandy Ryza
Hi Suresh, The schedulers used in MR1 are not compatible with the schedulers used in MR2 / YARN. Check out ResourceScheduler.java for the updated interface. I believe that speculative execution still works the same way. -Sandy On Sat, Jan 25, 2014 at 9:47 AM, Suresh S wrote: > Hello Friends

Re: Re-swizzle 2.3

2014-01-27 Thread Sandy Ryza
We should hold off commits until that's done, right? On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy wrote: > Yep, on it as we speak. :) > > > Arun > > On Jan 27, 2014, at 12:36 PM, Jason Lowe wrote: > > > Thanks, Arun. Are there plans to update the Fix Versions and > CHANGES.txt accordingly?

[jira] [Created] (MAPREDUCE-5732) Report proper queue when job has been automatically placed

2014-01-21 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5732: - Summary: Report proper queue when job has been automatically placed Key: MAPREDUCE-5732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5732 Project: Hadoop

[jira] [Created] (MAPREDUCE-5725) TestNetworkedJob relies on the Capacity Scheduler

2014-01-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5725: - Summary: TestNetworkedJob relies on the Capacity Scheduler Key: MAPREDUCE-5725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5725 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-5712) Backport Fair Scheduler pool placement by secondary group

2014-01-13 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5712. --- Resolution: Fixed Assignee: Ted Malaska Hadoop Flags: Reviewed I just

[jira] [Resolved] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1

2014-01-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5651. --- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I just

[jira] [Created] (MAPREDUCE-5665) Add audience annotations to MiniMRYarnCluster and MiniMRCluster

2013-12-03 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5665: - Summary: Add audience annotations to MiniMRYarnCluster and MiniMRCluster Key: MAPREDUCE-5665 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5665 Project

[jira] [Created] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1

2013-11-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5651: - Summary: Backport Fair Scheduler queue placement policies to branch-1 Key: MAPREDUCE-5651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 Project: Hadoop

Re: Next releases

2013-11-13 Thread Sandy Ryza
N-1333. Support blacklisting in the Fair Scheduler YARN-1109. Demote NodeManager "Sending out status for container" logs to debug (haosdent via Sandy Ryza) YARN-1388. Fair Scheduler page always displays blank fair share +1 to doing releases at some fixed time interval. -Sandy On W

[jira] [Created] (MAPREDUCE-5619) Separate out configuration loading from QueueManager in the Fair Scheduler

2013-11-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5619: - Summary: Separate out configuration loading from QueueManager in the Fair Scheduler Key: MAPREDUCE-5619 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5619

Re: Next releases

2013-11-08 Thread Sandy Ryza
Starting afresh with 2.2.1 and keeping it as small as possible sounds reasonable to me. Would love to get 2.3 out soon. To that end, how would people feel about having code and/or feature freeze and/or ship dates? We've been way behind out goals for recent releases. Having actual targets on the

[jira] [Created] (MAPREDUCE-5612) Document TaskAttemptCompletionStatuses

2013-11-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5612: - Summary: Document TaskAttemptCompletionStatuses Key: MAPREDUCE-5612 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5612 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb

2013-11-05 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5608: - Summary: Replace and deprecate mapred.tasktracker.indexcache.mb Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5601) Fetches when reducer can't fit them result in unnecessary reads on shuffle server

2013-10-29 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5601: - Summary: Fetches when reducer can't fit them result in unnecessary reads on shuffle server Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE

[jira] [Created] (MAPREDUCE-5599) JobHistoryServer unnecessarily copies all jobs on each query

2013-10-28 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5599: - Summary: JobHistoryServer unnecessarily copies all jobs on each query Key: MAPREDUCE-5599 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5599 Project: Hadoop

[jira] [Created] (MAPREDUCE-5596) Allow configuring the number of threads used to serve shuffle connections

2013-10-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5596: - Summary: Allow configuring the number of threads used to serve shuffle connections Key: MAPREDUCE-5596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5596

[jira] [Created] (MAPREDUCE-5592) Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1

2013-10-23 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5592: - Summary: Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1 Key: MAPREDUCE-5592 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5592 Project

Re: streaming documentation in Hadoop 2?

2013-10-14 Thread Sandy Ryza
d file a jira for it. The > wiki content here could be a good starting point. > > On Mon, Oct 14, 2013 at 2:56 PM, Sandy Ryza > wrote: > > Hi All, > > > > I noticed that the hadoop streaming documentation does not exist in the > > Hadoop 2 source tree, and also

streaming documentation in Hadoop 2?

2013-10-14 Thread Sandy Ryza
Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't tie

[jira] [Created] (MAPREDUCE-5578) Miscellaneous Fair Scheduler speedups

2013-10-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5578: - Summary: Miscellaneous Fair Scheduler speedups Key: MAPREDUCE-5578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5578 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time

2013-10-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5577: - Summary: Allow querying the JobHistoryServer by job arrival time Key: MAPREDUCE-5577 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 Project: Hadoop Map

pipes not working in MR2?

2013-10-10 Thread Sandy Ryza
I'm unable to get a simple hadoop pipes job working in MR2, and got the sense it hasn't been working for a while. Does anybody have any insight into what's going on? Has anybody used them successfully recently? thanks for any help, Sandy

[jira] [Created] (MAPREDUCE-5575) History files deleted from the intermediate directory never get removed from the JobListCache

2013-10-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5575: - Summary: History files deleted from the intermediate directory never get removed from the JobListCache Key: MAPREDUCE-5575 URL: https://issues.apache.org/jira/browse/MAPREDUCE

Re: [VOTE] Release Apache Hadoop 2.2.0

2013-10-08 Thread Sandy Ryza
+1 (non-binding) Built from source and ran a few jobs on a pseudo-distributed cluster with the Fair Scheduler. On Tue, Oct 8, 2013 at 6:48 AM, Thomas Graves wrote: > +1. > > Downloaded, verified signature/md5, CHANGES.txt, NOTICE, LICENSE, README, > release notes, built the source tar ball, an

JobClient.getRootQueues returns default, not root?

2013-10-02 Thread Sandy Ryza
Apparently JobClient.getRootQueues returns the default queue, not the root queue. Is this the correct behavior? It might make sense for the FIFO scheduler, but not for the Fair and Capacity schedulers. thanks for any guidance -Sandy

[jira] [Created] (MAPREDUCE-5544) JobClient#getJob loads job conf twice

2013-09-26 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5544: - Summary: JobClient#getJob loads job conf twice Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5527) Add CONTAINERS_MILLIS_MAPS|REDUCES counters

2013-09-23 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5527: - Summary: Add CONTAINERS_MILLIS_MAPS|REDUCES counters Key: MAPREDUCE-5527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5527 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits

2013-08-29 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5487: - Summary: In task processes, JobConf is unnecessarily loaded again in Limits Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project

[jira] [Created] (MAPREDUCE-5484) YarnChild unnecessarily loads job conf twice

2013-08-28 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5484: - Summary: YarnChild unnecessarily loads job conf twice Key: MAPREDUCE-5484 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5484 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5478) TeraInputFormat unnecessarily defines its own FileSplit subclass

2013-08-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5478: - Summary: TeraInputFormat unnecessarily defines its own FileSplit subclass Key: MAPREDUCE-5478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5478 Project

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-21 Thread Sandy Ryza
+1 (non-binding) Built from source, ran jobs on a pseudo-distributed cluster with the Fair Scheduler. -Sandy On Tue, Aug 20, 2013 at 7:27 PM, Arun C Murthy wrote: > Thanks for the heads up Aaron, I've changed fix-version of HDFS-4763 to > 2.1.1-beta for now. > > Committers - please be careful

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-19 Thread Sandy Ryza
Vinod, your thinking makes sense to me. My two cents are that we should hold off on fixes until 2.1.1-beta. Unless there are downstream projects that need it to work for integration testing. -Sandy On Mon, Aug 19, 2013 at 2:49 PM, Vinod Kumar Vavilapalli wrote: > > Thanks for the clarificatio

[jira] [Created] (MAPREDUCE-5464) Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5464: - Summary: Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter Key: MAPREDUCE-5464 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5464 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5463) Deprecate SLOTS_MILLIS counters

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5463: - Summary: Deprecate SLOTS_MILLIS counters Key: MAPREDUCE-5463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5463 Project: Hadoop Map/Reduce Issue

[jira] [Created] (MAPREDUCE-5462) In map-side sort, swap entire meta entries instead of indexes for better cache performance

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5462: - Summary: In map-side sort, swap entire meta entries instead of indexes for better cache performance Key: MAPREDUCE-5462 URL: https://issues.apache.org/jira/browse/MAPREDUCE

[jira] [Created] (MAPREDUCE-5457) Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators

2013-08-13 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5457: - Summary: Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators Key: MAPREDUCE-5457 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5457

[jira] [Resolved] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1

2013-08-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5450. --- Resolution: Fixed Hadoop Flags: Reviewed > Unnecessary Configurat

[jira] [Reopened] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants

2013-08-02 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5311: --- > Remove slot millis computation logic and deprecate counter consta

[jira] [Resolved] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt

2013-07-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5059. --- Resolution: Fixed > Job overview shows average merge time larger than for any red

[jira] [Reopened] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt

2013-07-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5059: --- > Job overview shows average merge time larger than for any reduce atte

[jira] [Created] (MAPREDUCE-5420) Remove mapreduce.task.tmp.dir from mapred-default.xml

2013-07-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5420: - Summary: Remove mapreduce.task.tmp.dir from mapred-default.xml Key: MAPREDUCE-5420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5420 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5403) Get rid of yarn.application.classpath

2013-07-18 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5403: - Summary: Get rid of yarn.application.classpath Key: MAPREDUCE-5403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5383) Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir

2013-07-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5383: - Summary: Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir Key: MAPREDUCE-5383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5383

[jira] [Created] (MAPREDUCE-5382) LocalJobRunner should use default FS for system and staging dirs by default

2013-07-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5382: - Summary: LocalJobRunner should use default FS for system and staging dirs by default Key: MAPREDUCE-5382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5382

[jira] [Created] (MAPREDUCE-5379) Include FS delegation token ID in job conf

2013-07-09 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5379: - Summary: Include FS delegation token ID in job conf Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5372) ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2

2013-07-02 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5372: - Summary: ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2 Key: MAPREDUCE-5372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5372

[jira] [Created] (MAPREDUCE-5367) Local jobs all use same local working directory

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5367: - Summary: Local jobs all use same local working directory Key: MAPREDUCE-5367 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5367 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5365) Set mapreduce.job.classpath to true by default

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5365: - Summary: Set mapreduce.job.classpath to true by default Key: MAPREDUCE-5365 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5365 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5363) Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5363: - Summary: Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus Key: MAPREDUCE-5363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5363

Re: How to Free-up a Map Slot without Killing the Entire Job?

2013-06-29 Thread Sandy Ryza
The fair scheduler does preemption as well. -Sandy On Sat, Jun 29, 2013 at 12:05 PM, Steve Loughran wrote: > there's a scheduler in contrib/ that does pre-emption. look here > > > https://github.com/apache/hadoop-common/tree/branch-0.22/mapreduce/src/contrib/dynamic-scheduler > > On 28 June 201

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-06-28 Thread Sandy Ryza
For YARN-791, if we can come to consensus on the correct approach, I can try to have a patch ASAP. -Sandy On Fri, Jun 28, 2013 at 12:03 PM, Hitesh Shah wrote: > Hi Arun, > > From a YARN perspective, YARN-791 and YARN-727 are 2 jiras that may > potentially change the apis. They can implemented

[jira] [Created] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem

2013-06-24 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5351: - Summary: JobTracker memory leak caused by CleanupQueue reopening FileSystem Key: MAPREDUCE-5351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 Project

[jira] [Created] (MAPREDUCE-5350) Expose Fair Scheduler-specific queue metrics

2013-06-24 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5350: - Summary: Expose Fair Scheduler-specific queue metrics Key: MAPREDUCE-5350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5350 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5338) Bring back mapred.child.ulimit

2013-06-21 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5338: - Summary: Bring back mapred.child.ulimit Key: MAPREDUCE-5338 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5338 Project: Hadoop Map/Reduce Issue

Re: mapred.child.ulimit in MR2

2013-06-21 Thread Sandy Ryza
are worried about backwards compatibility > we can add it back in. It is not that hard, all it did was add 'ulimt -v > ' to the shell script that launched the task, except on windows. > > --Bobby > > On 6/18/13 3:56 PM, "Sandy Ryza" wrote: >

mapred.child.ulimit in MR2

2013-06-18 Thread Sandy Ryza
Hi yarn-dev/mapreduce-dev, Is there a reason that mapred.child.ulimit no longer has an effect in MR2? Should it be added back in? thanks for any help, -Sandy

[jira] [Created] (MAPREDUCE-5321) Enable better parallelism in the Fair Scheduler

2013-06-13 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5321: - Summary: Enable better parallelism in the Fair Scheduler Key: MAPREDUCE-5321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5321 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility

2013-06-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5182. --- Resolution: Won't Fix > LineRecordReader#getProgress throwing IOExceptio

[jira] [Resolved] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest

2013-06-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5252. --- Resolution: Not A Problem This was fixed in YARN-326 > Fair schedu

[jira] [Created] (MAPREDUCE-5302) NodeManager throws AvroRuntimeException on failed start

2013-06-03 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5302: - Summary: NodeManager throws AvroRuntimeException on failed start Key: MAPREDUCE-5302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5302 Project: Hadoop Map

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

2013-06-03 Thread Sandy Ryza
+1 (non-binding) Built from source, ran a few sample jobs on a pseudo-distributed cluster On Mon, Jun 3, 2013 at 3:22 PM, Chris Douglas wrote: > +1 > > Checksum and signature match, ran some unit tests, checked diff > against 2.0.4-alpha. > > Thanks for seeing this through, Cos. -C > > On Mon,

Re: [VOTE] Release Apache Hadoop 0.23.8

2013-06-03 Thread Sandy Ryza
+1 (non-binding). Did a full build from source and ran a few sample jobs on a pseudo-distributed cluster. -Sandy On Mon, Jun 3, 2013 at 11:29 AM, Kihwal Lee wrote: > +1 I've downloaded & built the RC and ran several tests on a single node > cluster. > > Kihwal > > On 5/28/13 11:00 AM, "Thomas

[jira] [Created] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext

2013-05-29 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5283: - Summary: Over 10 different tests have near identical implementations of AppContext Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283

[jira] [Reopened] (MAPREDUCE-5036) Default shuffle handler port should not be 8080

2013-05-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5036: --- > Default shuffle handler port should not be 8

Re: Clarifications on MAPREDUCE-5183

2013-05-21 Thread Sandy Ryza
If you're planning to fix it, it would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -Sandy On Tue, May 21, 2013 at 10:55 AM, maisnam ns wrote: > Thanks Sandy Ryza > > > > > On Tue, May 21, 2013 at 11:20 PM, Sandy Ryza &g

Re: Clarifications on MAPREDUCE-5183

2013-05-21 Thread Sandy Ryza
Hi Niranjan, Your understanding is correct. -Sandy On Tue, May 21, 2013 at 1:02 AM, maisnam ns wrote: > Hi, > > I was looking into this issue but would be happy if someone could clarify > some of my doubts. > > Is the issue related to the given below snapshot of log: > > org.apache.hadoop.map

Re: [VOTE] Plan to create release candidate for 0.23.8

2013-05-19 Thread Sandy Ryza
+1 (non-binding) On Sun, May 19, 2013 at 1:22 PM, Derek Dagit wrote: > +1 (non-binding) > > On May 17, 2013, at 4:14 PM, "Thomas Graves" > wrote: > > > Hello all, > > > > We've had a few critical issues come up in 0.23.7 that I think warrants a > > 0.23.8 release. The main one is MAPREDUCE-521

[jira] [Resolved] (MAPREDUCE-5236) references to JobConf.DISABLE_MEMORY_LIMIT don't make sense in the context of MR2

2013-05-16 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5236. --- Resolution: Duplicate > references to JobConf.DISABLE_MEMORY_LIMIT don't ma

[jira] [Created] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest

2013-05-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5252: - Summary: Fair scheduler should use SchedulerUtils.normalizeRequest Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk

2013-05-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5238: - Summary: TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk Key: MAPREDUCE-5238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238 Project

[jira] [Created] (MAPREDUCE-5236) JobConf should not support disabling memory limits

2013-05-09 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5236: - Summary: JobConf should not support disabling memory limits Key: MAPREDUCE-5236 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5236 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2

2013-05-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5220: - Summary: Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220

[jira] [Created] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2

2013-05-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5219: - Summary: JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 Key: MAPREDUCE-5219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 Project

[jira] [Resolved] (MAPREDUCE-3946) If a resource requirement is higher than available on any node, job should fail early

2013-05-03 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-3946. --- Resolution: Duplicate > If a resource requirement is higher than available on

[jira] [Resolved] (MAPREDUCE-5134) Default settings cause LocalJobRunner to OOME

2013-04-29 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5134. --- Resolution: Not A Problem > Default settings cause LocalJobRunner to O

  1   2   >