Re: [DISCUSS] branch-1

2015-05-08 Thread Sandy Ryza
+1 to auto-closing branch-1 issues. -Sandy On Fri, May 8, 2015 at 11:14 AM, Karthik Kambatla ka...@cloudera.com wrote: Closing out the JIRAs as Auto Closed or Closed due to Inactivity seems reasonable to me. For branch-1, we can be more aggressive. We should probably do the same less

Re: [DISCUSS] Release numbering for stable 2.8 and beyond

2015-04-27 Thread Sandy Ryza
My understanding was that the main reason that we labeled 2.0 alpha and 2.1 beta is that we wanted flexibility to make breaking API changes. Is that the case now with 2.7? I.e. do we have APIs labeled as Public / Stable that we want freedom to change in 2.8? If not, I definitely don't think the

Re: [VOTE] Release Apache Hadoop 2.5.1 RC0

2014-09-10 Thread Sandy Ryza
+1 (binding) Ran a pseudo-distributed cluster with the Fair Scheduler and ran some MapReduce example jobs. -Sandy On Wed, Sep 10, 2014 at 12:15 PM, Alejandro Abdelnur t...@cloudera.com wrote: Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball +

Re: [VOTE] Migration from subversion to git for version control

2014-08-09 Thread Sandy Ryza
+1 (binding) On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2.

Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated

2014-07-22 Thread Sandy Ryza
That policy makes sense to me. We should still label things @Private of course so that it can be reflected in the documentation. -Sandy On Tue, Jul 22, 2014 at 2:54 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi devs As you might have noticed, we have several classes and methods in them

Re: Where is the map input transfered to the 'map worker'/container?

2014-07-06 Thread Sandy Ryza
Hi Christian, I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of getting the bytes from the remote node. -Sandy On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote cgr...@mail.uni-paderborn.de wrote: Hey, I'm looking for the place where the actual map input is

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

2014-06-24 Thread Sandy Ryza
+1 (binding) -Sandy On Tue, Jun 24, 2014 at 7:53 AM, Devaraj K deva...@apache.org wrote: +1 Thanks Devaraj K On Tue, Jun 24, 2014 at 2:23 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes

Re: Moving to JDK7, JDK8 and new major releases

2014-06-24 Thread Sandy Ryza
While we haven't codified this in our compatibility guidelines, dropping a Java version seems to me like change that needs to happen alongside a major release. In plain talk, it has the ability to break everything for users who aren't doing anything particularly unreasonable. I don't think we

Re: Plans of moving towards JDK7 in trunk

2014-06-23 Thread Sandy Ryza
Andrew, correct me if I'm misunderstanding, but the incompatible change that would require a major version bump is dropping support for JDK6. On Mon, Jun 23, 2014 at 1:53 PM, sanjay Radia san...@hortonworks.com wrote: On Jun 21, 2014, at 8:01 AM, Andrew Wang andrew.w...@cloudera.com wrote:

Re: hadoop-2.5 - June end?

2014-06-10 Thread Sandy Ryza
That sounds reasonable to me. -Sandy On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create

[jira] [Created] (MAPREDUCE-5896) Allow InputSplits to indicate which locations have the block cached in memory

2014-05-20 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5896: - Summary: Allow InputSplits to indicate which locations have the block cached in memory Key: MAPREDUCE-5896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5896

Re: Thinking ahead

2014-04-13 Thread Sandy Ryza
to point out that the proposal to merge AHS into trunk stated an intent to tie ends up on security. On Sun, Apr 13, 2014 at 7:50 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Unfortunately I don't have the bandwidth to take on ATS security at this time. My (I now understand mistaken) impression

Re: Thinking ahead

2014-04-12 Thread Sandy Ryza
+1 for starting to think about 2.5. Early June seems a little early to me - we had talked about a quarterly release cadence and this would be about half that. I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list.

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-04-03 Thread Sandy Ryza
While the Scheduler Load Simulator isn't part of YARN's core, it's a tool that YARN includes, and it's broken entirely in the current RC. YARN-1726 seems to me like something worth including in the release. -Sandy On Thu, Apr 3, 2014 at 11:12 AM, Xuan Gong xg...@hortonworks.com wrote: +1

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-03-31 Thread Sandy Ryza
What's the state of the application history server? Do we have security, documentation, and are APIs stable? If any of these are missing, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Mon, Mar 31, 2014 at 2:22 AM, Arun C Murthy

Re: [VOTE] Release Apache Hadoop 2.4.0

2014-03-31 Thread Sandy Ryza
, while we eventually hope to move the generic data there as well. Thanks, Zhijie On Mon, Mar 31, 2014 at 11:02 AM, Sandy Ryza sandy.r...@cloudera.com wrote: What's the state of the application history server? Do we have security, documentation, and are APIs stable? If any

Re: Thinking ahead to 2.4

2014-03-20 Thread Sandy Ryza
Looking forward to the release. What's the state of the application history server? Do we have security, documentation, and are APIs stable? If not, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Fri, Mar 14, 2014 at 8:23 PM, Arun C

[jira] [Created] (MAPREDUCE-5763) Warn message about httpshuffle in NM logs

2014-02-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5763: - Summary: Warn message about httpshuffle in NM logs Key: MAPREDUCE-5763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5763 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5759) Remove unnecessary conf load in Limits

2014-02-18 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5759: - Summary: Remove unnecessary conf load in Limits Key: MAPREDUCE-5759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5759 Project: Hadoop Map/Reduce

Re: [VOTE] Release Apache Hadoop 2.3.0

2014-02-13 Thread Sandy Ryza
+1 (non-binding) Built from source and ran jobs on a pseudo-distributed cluster with the Fair Scheduler On Wed, Feb 12, 2014 at 7:56 PM, Xuan Gong xg...@hortonworks.com wrote: +1 (non-binding) downloaded the source tar ball, built, ran a number of MR jobs on a single-node cluster and

Re: The code for the Shuffle phase of MapReduce

2014-02-10 Thread Sandy Ryza
Hi Pramod, For the reduce side, also check out Shuffle.java. -Sandy On Mon, Feb 10, 2014 at 2:54 AM, Pramod Biligiri pramodbilig...@gmail.comwrote: Hi, I'm beginning to look at the code for the Shuffle phase of MapReduce, for an academic project. I wanted to confirm if I have started at

Re: Re-swizzle 2.3

2014-02-06 Thread Sandy Ryza
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a problem caused by YARN-1493 I think we can revert it in branch-2.3 as well. I think we should leave them in branch-2 for now. We can revert if 2.4 is imminent and they're holding it up, but hopefully the issues they caused

[jira] [Resolved] (MAPREDUCE-5745) thread may hang forever, even after it receives all the expected data

2014-02-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5745. --- Resolution: Invalid thread may hang forever, even after it receives all the expected

Re: Re-swizzle 2.3

2014-01-28 Thread Sandy Ryza
Going forward with commits because it seems like others have been doing so On Mon, Jan 27, 2014 at 1:31 PM, Sandy Ryza sandy.r...@cloudera.com wrote: We should hold off commits until that's done, right? On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy a...@hortonworks.comwrote: Yep

Re: Re-swizzle 2.3

2014-01-27 Thread Sandy Ryza
We should hold off commits until that's done, right? On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy a...@hortonworks.com wrote: Yep, on it as we speak. :) Arun On Jan 27, 2014, at 12:36 PM, Jason Lowe jl...@yahoo-inc.com wrote: Thanks, Arun. Are there plans to update the Fix Versions

Re: Doubt in Yarn Scheduler.

2014-01-27 Thread Sandy Ryza
Hi Suresh, The schedulers used in MR1 are not compatible with the schedulers used in MR2 / YARN. Check out ResourceScheduler.java for the updated interface. I believe that speculative execution still works the same way. -Sandy On Sat, Jan 25, 2014 at 9:47 AM, Suresh S suresh...@gmail.com

[jira] [Created] (MAPREDUCE-5732) Report proper queue when job has been automatically placed

2014-01-21 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5732: - Summary: Report proper queue when job has been automatically placed Key: MAPREDUCE-5732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5732 Project: Hadoop

[jira] [Created] (MAPREDUCE-5725) TestNetworkedJob relies on the Capacity Scheduler

2014-01-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5725: - Summary: TestNetworkedJob relies on the Capacity Scheduler Key: MAPREDUCE-5725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5725 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-5712) Backport Fair Scheduler pool placement by secondary group

2014-01-13 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5712. --- Resolution: Fixed Assignee: Ted Malaska Hadoop Flags: Reviewed I just

[jira] [Resolved] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1

2014-01-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5651. --- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I just

[jira] [Created] (MAPREDUCE-5665) Add audience annotations to MiniMRYarnCluster and MiniMRCluster

2013-12-03 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5665: - Summary: Add audience annotations to MiniMRYarnCluster and MiniMRCluster Key: MAPREDUCE-5665 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5665 Project

[jira] [Created] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1

2013-11-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5651: - Summary: Backport Fair Scheduler queue placement policies to branch-1 Key: MAPREDUCE-5651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 Project: Hadoop

Re: Next releases

2013-11-13 Thread Sandy Ryza
blacklisting in the Fair Scheduler YARN-1109. Demote NodeManager Sending out status for container logs to debug (haosdent via Sandy Ryza) YARN-1388. Fair Scheduler page always displays blank fair share +1 to doing releases at some fixed time interval. -Sandy On Wed, Nov 13, 2013 at 10:10 AM

[jira] [Created] (MAPREDUCE-5619) Separate out configuration loading from QueueManager in the Fair Scheduler

2013-11-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5619: - Summary: Separate out configuration loading from QueueManager in the Fair Scheduler Key: MAPREDUCE-5619 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5619

[jira] [Created] (MAPREDUCE-5612) Document TaskAttemptCompletionStatuses

2013-11-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5612: - Summary: Document TaskAttemptCompletionStatuses Key: MAPREDUCE-5612 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5612 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb

2013-11-05 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5608: - Summary: Replace and deprecate mapred.tasktracker.indexcache.mb Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5596) Allow configuring the number of threads used to serve shuffle connections

2013-10-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5596: - Summary: Allow configuring the number of threads used to serve shuffle connections Key: MAPREDUCE-5596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5596

[jira] [Created] (MAPREDUCE-5592) Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1

2013-10-23 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5592: - Summary: Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1 Key: MAPREDUCE-5592 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5592 Project

streaming documentation in Hadoop 2?

2013-10-14 Thread Sandy Ryza
Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't

Re: streaming documentation in Hadoop 2?

2013-10-14 Thread Sandy Ryza
and file a jira for it. The wiki content here could be a good starting point. On Mon, Oct 14, 2013 at 2:56 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found

[jira] [Created] (MAPREDUCE-5578) Miscellaneous Fair Scheduler speedups

2013-10-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5578: - Summary: Miscellaneous Fair Scheduler speedups Key: MAPREDUCE-5578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5578 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5575) History files deleted from the intermediate directory never get removed from the JobListCache

2013-10-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5575: - Summary: History files deleted from the intermediate directory never get removed from the JobListCache Key: MAPREDUCE-5575 URL: https://issues.apache.org/jira/browse/MAPREDUCE

pipes not working in MR2?

2013-10-10 Thread Sandy Ryza
I'm unable to get a simple hadoop pipes job working in MR2, and got the sense it hasn't been working for a while. Does anybody have any insight into what's going on? Has anybody used them successfully recently? thanks for any help, Sandy

[jira] [Created] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time

2013-10-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5577: - Summary: Allow querying the JobHistoryServer by job arrival time Key: MAPREDUCE-5577 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 Project: Hadoop Map

Re: [VOTE] Release Apache Hadoop 2.2.0

2013-10-08 Thread Sandy Ryza
+1 (non-binding) Built from source and ran a few jobs on a pseudo-distributed cluster with the Fair Scheduler. On Tue, Oct 8, 2013 at 6:48 AM, Thomas Graves tgra...@yahoo-inc.com wrote: +1. Downloaded, verified signature/md5, CHANGES.txt, NOTICE, LICENSE, README, release notes, built the

JobClient.getRootQueues returns default, not root?

2013-10-02 Thread Sandy Ryza
Apparently JobClient.getRootQueues returns the default queue, not the root queue. Is this the correct behavior? It might make sense for the FIFO scheduler, but not for the Fair and Capacity schedulers. thanks for any guidance -Sandy

[jira] [Created] (MAPREDUCE-5544) JobClient#getJob loads job conf twice

2013-09-26 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5544: - Summary: JobClient#getJob loads job conf twice Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5527) Add CONTAINERS_MILLIS_MAPS|REDUCES counters

2013-09-23 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5527: - Summary: Add CONTAINERS_MILLIS_MAPS|REDUCES counters Key: MAPREDUCE-5527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5527 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits

2013-08-29 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5487: - Summary: In task processes, JobConf is unnecessarily loaded again in Limits Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project

[jira] [Created] (MAPREDUCE-5484) YarnChild unnecessarily loads job conf twice

2013-08-28 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5484: - Summary: YarnChild unnecessarily loads job conf twice Key: MAPREDUCE-5484 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5484 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5478) TeraInputFormat unnecessarily defines its own FileSplit subclass

2013-08-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5478: - Summary: TeraInputFormat unnecessarily defines its own FileSplit subclass Key: MAPREDUCE-5478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5478 Project

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-21 Thread Sandy Ryza
+1 (non-binding) Built from source, ran jobs on a pseudo-distributed cluster with the Fair Scheduler. -Sandy On Tue, Aug 20, 2013 at 7:27 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks for the heads up Aaron, I've changed fix-version of HDFS-4763 to 2.1.1-beta for now. Committers -

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-19 Thread Sandy Ryza
Vinod, your thinking makes sense to me. My two cents are that we should hold off on fixes until 2.1.1-beta. Unless there are downstream projects that need it to work for integration testing. -Sandy On Mon, Aug 19, 2013 at 2:49 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Thanks

[jira] [Created] (MAPREDUCE-5462) In map-side sort, swap entire meta entries instead of indexes for better cache performance

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5462: - Summary: In map-side sort, swap entire meta entries instead of indexes for better cache performance Key: MAPREDUCE-5462 URL: https://issues.apache.org/jira/browse/MAPREDUCE

[jira] [Created] (MAPREDUCE-5463) Deprecate SLOTS_MILLIS counters

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5463: - Summary: Deprecate SLOTS_MILLIS counters Key: MAPREDUCE-5463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5463 Project: Hadoop Map/Reduce Issue

[jira] [Created] (MAPREDUCE-5464) Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter

2013-08-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5464: - Summary: Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter Key: MAPREDUCE-5464 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5464 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5457) Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators

2013-08-13 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5457: - Summary: Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators Key: MAPREDUCE-5457 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5457

[jira] [Resolved] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1

2013-08-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5450. --- Resolution: Fixed Hadoop Flags: Reviewed Unnecessary Configuration

[jira] [Reopened] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants

2013-08-02 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5311: --- Remove slot millis computation logic and deprecate counter constants

[jira] [Reopened] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt

2013-07-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5059: --- Job overview shows average merge time larger than for any reduce attempt

[jira] [Resolved] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt

2013-07-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5059. --- Resolution: Fixed Job overview shows average merge time larger than for any reduce

[jira] [Created] (MAPREDUCE-5420) Remove mapreduce.task.tmp.dir from mapred-default.xml

2013-07-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5420: - Summary: Remove mapreduce.task.tmp.dir from mapred-default.xml Key: MAPREDUCE-5420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5420 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5403) Get rid of yarn.application.classpath

2013-07-18 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5403: - Summary: Get rid of yarn.application.classpath Key: MAPREDUCE-5403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5382) LocalJobRunner should use default FS for system and staging dirs by default

2013-07-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5382: - Summary: LocalJobRunner should use default FS for system and staging dirs by default Key: MAPREDUCE-5382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5382

[jira] [Created] (MAPREDUCE-5383) Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir

2013-07-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5383: - Summary: Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir Key: MAPREDUCE-5383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5383

[jira] [Created] (MAPREDUCE-5379) Include FS delegation token ID in job conf

2013-07-09 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5379: - Summary: Include FS delegation token ID in job conf Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5372) ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2

2013-07-02 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5372: - Summary: ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2 Key: MAPREDUCE-5372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5372

[jira] [Created] (MAPREDUCE-5363) Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5363: - Summary: Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus Key: MAPREDUCE-5363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5363

[jira] [Created] (MAPREDUCE-5365) Set mapreduce.job.classpath to true by default

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5365: - Summary: Set mapreduce.job.classpath to true by default Key: MAPREDUCE-5365 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5365 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5367) Local jobs all use same local working directory

2013-07-01 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5367: - Summary: Local jobs all use same local working directory Key: MAPREDUCE-5367 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5367 Project: Hadoop Map/Reduce

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-06-28 Thread Sandy Ryza
For YARN-791, if we can come to consensus on the correct approach, I can try to have a patch ASAP. -Sandy On Fri, Jun 28, 2013 at 12:03 PM, Hitesh Shah hit...@apache.org wrote: Hi Arun, From a YARN perspective, YARN-791 and YARN-727 are 2 jiras that may potentially change the apis. They

[jira] [Created] (MAPREDUCE-5350) Expose Fair Scheduler-specific queue metrics

2013-06-24 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5350: - Summary: Expose Fair Scheduler-specific queue metrics Key: MAPREDUCE-5350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5350 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem

2013-06-24 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5351: - Summary: JobTracker memory leak caused by CleanupQueue reopening FileSystem Key: MAPREDUCE-5351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 Project

Re: mapred.child.ulimit in MR2

2013-06-21 Thread Sandy Ryza
for it or are worried about backwards compatibility we can add it back in. It is not that hard, all it did was add 'ulimt -v number' to the shell script that launched the task, except on windows. --Bobby On 6/18/13 3:56 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi yarn-dev/mapreduce-dev

[jira] [Created] (MAPREDUCE-5338) Bring back mapred.child.ulimit

2013-06-21 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5338: - Summary: Bring back mapred.child.ulimit Key: MAPREDUCE-5338 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5338 Project: Hadoop Map/Reduce Issue

mapred.child.ulimit in MR2

2013-06-18 Thread Sandy Ryza
Hi yarn-dev/mapreduce-dev, Is there a reason that mapred.child.ulimit no longer has an effect in MR2? Should it be added back in? thanks for any help, -Sandy

[jira] [Created] (MAPREDUCE-5321) Enable better parallelism in the Fair Scheduler

2013-06-13 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5321: - Summary: Enable better parallelism in the Fair Scheduler Key: MAPREDUCE-5321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5321 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest

2013-06-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5252. --- Resolution: Not A Problem This was fixed in YARN-326 Fair scheduler

[jira] [Resolved] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility

2013-06-07 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5182. --- Resolution: Won't Fix LineRecordReader#getProgress throwing IOException breaks

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

2013-06-03 Thread Sandy Ryza
+1 (non-binding) Built from source, ran a few sample jobs on a pseudo-distributed cluster On Mon, Jun 3, 2013 at 3:22 PM, Chris Douglas cdoug...@apache.org wrote: +1 Checksum and signature match, ran some unit tests, checked diff against 2.0.4-alpha. Thanks for seeing this through, Cos.

[jira] [Created] (MAPREDUCE-5302) NodeManager throws AvroRuntimeException on failed start

2013-06-03 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5302: - Summary: NodeManager throws AvroRuntimeException on failed start Key: MAPREDUCE-5302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5302 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext

2013-05-29 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5283: - Summary: Over 10 different tests have near identical implementations of AppContext Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283

[jira] [Reopened] (MAPREDUCE-5036) Default shuffle handler port should not be 8080

2013-05-28 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5036: --- Default shuffle handler port should not be 8080

Re: Clarifications on MAPREDUCE-5183

2013-05-21 Thread Sandy Ryza
Hi Niranjan, Your understanding is correct. -Sandy On Tue, May 21, 2013 at 1:02 AM, maisnam ns maisnam...@gmail.com wrote: Hi, I was looking into this issue but would be happy if someone could clarify some of my doubts. Is the issue related to the given below snapshot of log:

Re: Clarifications on MAPREDUCE-5183

2013-05-21 Thread Sandy Ryza
If you're planning to fix it, it would probably look prettiest to keep the percentage sign and have the numbers between 0 and 100. -Sandy On Tue, May 21, 2013 at 10:55 AM, maisnam ns maisnam...@gmail.com wrote: Thanks Sandy Ryza On Tue, May 21, 2013 at 11:20 PM, Sandy Ryza sandy.r

Re: [VOTE] Plan to create release candidate for 0.23.8

2013-05-19 Thread Sandy Ryza
+1 (non-binding) On Sun, May 19, 2013 at 1:22 PM, Derek Dagit der...@yahoo-inc.com wrote: +1 (non-binding) On May 17, 2013, at 4:14 PM, Thomas Graves tgra...@yahoo-inc.com wrote: Hello all, We've had a few critical issues come up in 0.23.7 that I think warrants a 0.23.8 release.

[jira] [Created] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest

2013-05-15 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5252: - Summary: Fair scheduler should use SchedulerUtils.normalizeRequest Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk

2013-05-10 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5238: - Summary: TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk Key: MAPREDUCE-5238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238 Project

[jira] [Created] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2

2013-05-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5219: - Summary: JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 Key: MAPREDUCE-5219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 Project

[jira] [Created] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2

2013-05-07 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5220: - Summary: Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220

[jira] [Resolved] (MAPREDUCE-3946) If a resource requirement is higher than available on any node, job should fail early

2013-05-03 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-3946. --- Resolution: Duplicate If a resource requirement is higher than available on any

Re: Hadoop MapReduce High Availability

2013-04-29 Thread Sandy Ryza
Hi Augusto, In Hadoop 2, ResourceManager HA is being worked on under YARN-128 and YARN-149. There's a design doc for RM recovery on the latter. Hadoop 1's MapReduce high availability story is kind of fragmented. Cloudera distribution has JobTracker HA based on the HA libraries available in

[jira] [Resolved] (MAPREDUCE-5134) Default settings cause LocalJobRunner to OOME

2013-04-29 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5134. --- Resolution: Not A Problem Default settings cause LocalJobRunner to OOME

Re: Versions - Confusion

2013-04-27 Thread Sandy Ryza
The 2.0 series contains security as well. -Sandy On Sat, Apr 27, 2013 at 5:30 AM, Suresh S suresh...@gmail.com wrote: Thank you very much sir. great explaination. 0.20.20X is renamed to 1.0. So, 1.0 series is coming with security. What about 2.0 series? It is with security or without?

[jira] [Created] (MAPREDUCE-5185) When log aggregation not enabled, message should point to NM HTTP port, not IPC port

2013-04-26 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5185: - Summary: When log aggregation not enabled, message should point to NM HTTP port, not IPC port Key: MAPREDUCE-5185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5185

[jira] [Created] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility

2013-04-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5182: - Summary: LineRecordReader#getProgress throwing IOException breaks compatibility Key: MAPREDUCE-5182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5182

[jira] [Created] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign

2013-04-25 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5183: - Summary: In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign Key: MAPREDUCE-5183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183

Re: ConfigKeys wrappers for MapReduce source code base

2013-04-22 Thread Sandy Ryza
Hi Jay, In trunk, these keys are in MRJobConfig (extended by JobContext) and MRConfig. I'm not sure what the state of things are in branch-1. Agreed that it would be nice for the naming convention to be consistent with HDFS, but personally I like *Config more than *ConfigKeys better because the

[jira] [Created] (MAPREDUCE-5171) Expose info about blacklisted nodes from MR AM REST API

2013-04-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5171: - Summary: Expose info about blacklisted nodes from MR AM REST API Key: MAPREDUCE-5171 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5171 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-5172) Expose number of non-local maps in MR app metrics

2013-04-22 Thread Sandy Ryza (JIRA)
Sandy Ryza created MAPREDUCE-5172: - Summary: Expose number of non-local maps in MR app metrics Key: MAPREDUCE-5172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5172 Project: Hadoop Map/Reduce

  1   2   >