Re: [Vote] Merge branch-trunk-win to trunk
+1 for the merge. As someone who has been testing the code for many months now, both on singlenode and multinode clusters, I am very confident about the stability and the quality of the code. I have run several regression tests to verify distributed cache, streaming, compression, capacity scheduler, job history and many more features in HDFS and MR. - Ramya On Thu, Feb 28, 2013 at 3:08 PM, sanjay Radia wrote: > +1 > Java has done the bulk of the work in making Hadoop multi-platform. > Windows specific code is a tiny percentage of the code. > Jeninks support for windows is going help us keep the platform portable > going forward. > I expect that the vast majority of new commits have no problems. I > propose that we start by fixing problems that Jenkins raises but not block > new commits for too long if the author does not have a windows box or if a > volunteer does not step up. > > sanjay > > > >
Re: Heads up: branch-2.1-beta
We have started testing branch-2.1-beta and for most parts the code looks very stable. We have deployed both secure and non-secure multinode clusters. We had some minor hiccups with some of our e2e tests breaking due to additional setsid info being logged by the bin scripts and errors while building with snappy turned on. Both are now fixed. Some of the e2e tests had to be tweaked due to configuration changes in branch-2.1-beta such as yarn.nodemanager.delete.debug-delay-sec has to be set to 0 to invoke the cleanup of container tmp files, fs.trash.interval is now a server side config etc. We are now seeing failures while configuring short circuit read and job summary logs(most likely configuration issues) and tests which use hftp:// and webhdfs:// are failing on secure clusters. Will file jiras after initial investigation. Thanks, Ramya On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy wrote: > Folks, > > The vast majority of of the planned features and API work is complete, > thanks to everyone who contributed! > > I've created a branch-2.1-beta branch from which I anticipate I can make > the first of our beta releases very shortly. > > For now the remaining work is to wrap up loose ends i.e. last minute api > work (e.g. YARN-759 showed up last night for consideration), bug-fixes > etc.; then run this through a battery of unit/system/integration tests and > do a final review before we ship. There is more work remaining on > documentation (e.g. HADOOP-9517) and I plan to personally focus on it this > week - obviously help reviewing docs is very welcome. > > Committers, from now, please please exercise your judgement on where you > commit. Typically, features should go into branch-2 with 2.3.0 as the > version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 > will be limited to content in branch-2.1-beta and we stick to stabilizing > it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet). > > thanks, > Arun >
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
-1. Some of the cli and distcp system tests which use hftp:// and webhdfs:// are failing on secure cluster (HDFS-4841 and HDFS-4952/HDFS-4896). This is a regression and we need to make sure they work before we call a release. On Wed, Jun 26, 2013 at 1:17 AM, Arun C Murthy wrote: > Folks, > > I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would > like to get released. > > This release represents a *huge* amount of work done by the community (639 > fixes) which includes several major advances including: > # HDFS Snapshots > # Windows support > # YARN API stabilization > # MapReduce Binary Compatibility with hadoop-1.x > # Substantial amount of integration testing with rest of projects in the > ecosystem > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/ > The RC tag in svn is here: > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0 > > The maven artifacts are available via repository.apache.org. > > Please try the release and vote; the vote will run for the usual 7 days. > > thanks, > Arun > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > >
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 Deployed on secure and nonsecure clusters. Ran some hdfs, MR and yarn tests. Seems good to me. On Mon, Oct 7, 2013 at 12:21 PM, Alejandro Abdelnur wrote: > +1 > > * downloaded source tarball > * verified MD5 > * verified signature > * verified CHANGES.txt files, release # and date > * run 'mvn apache-rat:check' successfully > * built distribution > * setup speudo cluster > * started HDFS/YARN > * run some HTTFS tests > * run a couple of MR examples > * run a few tests using Llama AM > > > > On Mon, Oct 7, 2013 at 12:07 PM, Tassapol Athiapinya < > tathiapi...@hortonworks.com> wrote: > > > +1 for the release. > > > > I have deployed a multinode cluster and extensively tested MR speculative > > execution, YARN CLI and YARN distributed shell. There were couple of > issues > > I encountered while testing MAPREDUCE-5533, YARN-1168, YARN-1167, > > YARN-1157, YARN-1131, YARN-1118, YARN-1117 and all of them have been > fixed. > > > > Thanks, > > Tassapol > > > > On Oct 7, 2013, at 12:00 AM, Arun C Murthy wrote: > > > > > Folks, > > > > > > I've created a release candidate (rc0) for hadoop-2.2.0 that I would > > like to get released - this release fixes a small number of bugs and some > > protocol/api issues which should ensure they are now stable and will not > > change in hadoop-2.x. > > > > > > The RC is available at: > > http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 > > > The RC tag in svn is here: > > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 > > > > > > The maven artifacts are available via repository.apache.org. > > > > > > Please try the release and vote; the vote will run for the usual 7 > days. > > > > > > thanks, > > > Arun > > > > > > P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail > > down the symlinks-related issues. I'll release note the fact that we have > > disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the > > YARN side in the last couple of weeks. > > > > > > > > > > > > > > > > > > -- > > > Arun C. Murthy > > > Hortonworks Inc. > > > http://hortonworks.com/ > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > > > -- > Alejandro > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (MAPREDUCE-858) NPE in heartbeat if "mapred.job.tracker.history.completed.location" is not writable
[ https://issues.apache.org/jira/browse/MAPREDUCE-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-858. --- Resolution: Fixed This is no longer an issue. JT gracefully shuts down if mapred.job.tracker.history.completed.location is not writable. > NPE in heartbeat if "mapred.job.tracker.history.completed.location" is not > writable > > > Key: MAPREDUCE-858 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-858 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1, 0.21.0 >Reporter: Ramya Sunil >Priority: Minor > > If "mapred.job.tracker.history.completed.location" has been configured to > write to a location which is not writable by JT, NullPointerException is > thrown in TT heartbeat. Below is the Exception obtained: > {noformat} > 2009-08-13 07:56:02,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler > on , call > heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1e7a6ae, false, false, > true, 1775) from :: error: java.io.IOException: > java.lang.NullPointerException > java.io.IOException: java.lang.NullPointerException > at > org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager.moveToDone(JobHistory.java:215) > at > org.apache.hadoop.mapred.JobHistory$JobInfo.markCompleted(JobHistory.java:1071) > at > org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2413) > at > org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2729) > at > org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2327) > at > org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2259) > at > org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:957) > at > org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3946) > at > org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3123) > at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2861) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > {noformat} > Instead of an NPE, it would be helpful if an useful error message is logged. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5766) Ping messages from attempts should be moved to DEBUG
Ramya Sunil created MAPREDUCE-5766: -- Summary: Ping messages from attempts should be moved to DEBUG Key: MAPREDUCE-5766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5766 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Ramya Sunil Priority: Minor Fix For: 0.24.0 Messages such as "org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from attempt_1391416522080_0015_m_00_0" in AM logs should be moved to DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-2677) MR-279: 404 error while accessing pages from history server
MR-279: 404 error while accessing pages from history server --- Key: MAPREDUCE-2677 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2677 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Accessing the following pages from the history server, causes 404 HTTP error 1. Cluster-> About 2. Cluster -> Applications 3. Cluster -> Scheduler 4. Application -> About -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2686) NPE while requesting info for a non-existing job
NPE while requesting info for a non-existing job Key: MAPREDUCE-2686 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2686 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 While performing job related operations such as job -kill, -status, -events etc for an unknown job, the following NPE is seen: Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:112) at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:100) at org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:93) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154) at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:254) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1986) History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
[ https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-1986. Resolution: Duplicate This issue is exactly the same as MAPREDUCE-2463. Since there is already a patch available in MAPREDUCE-2463, resolving this as duplicate. Thanks Bhallamudi. > History files fail to move to DONE folder when hadoop.job.history.location is > configured to a HDFS path > --- > > Key: MAPREDUCE-1986 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.2 >Reporter: Ramya Sunil > > When hadoop.job.history.location is configured to a HDFS path and when the > job retires, the history files are never moved to > mapred.job.tracker.history.completed.location and they remain forever in > hadoop.job.history.location . The jobhistory is never viewable thereafter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2687) Non superusers unable to launch apps in a non-secure cluster
Non superusers unable to launch apps in a non-secure cluster Key: MAPREDUCE-2687 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2687 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Apps of non superuser fail to succeed in a non-secure environment. Only the superuser(i.e. one who started/owns the mrv2 cluster) is able to launch apps successfully. However, when a normal user submits a job, the job fails. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2689) InvalidStateTransisiton when AM is not assigned to a job
InvalidStateTransisiton when AM is not assigned to a job Key: MAPREDUCE-2689 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2689 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 In cases where an AM is not being assigned to a job, RELEASED at COMPLETED invalid event is observed. This is easily reproducible in cases such as MAPREDUCE-2687. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2690) Construct the web page for default scheduler
Construct the web page for default scheduler Key: MAPREDUCE-2690 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, the web page for default scheduler reads as "Under construction". This is a long known issue, but could not find a tracking ticket. Hence opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2737) Update the progress of jobs on client side
Update the progress of jobs on client side -- Key: MAPREDUCE-2737 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2737 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The progress of the jobs are not being correctly updated on the client side. The map progress halts at 66% and both map/reduce progress % does not display 100 when the job completes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2738) Missing cluster level stats on the RM UI
Missing cluster level stats on the RM UI Key: MAPREDUCE-2738 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Cluster usage information such as the following are currently not available in the RM UI. - Total number of apps submitted so far - Total number of containers running/total memory usage - Total capacity of the cluster (in terms of memory) - Reserved memory - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward - Excluded NMs - Handle to the jobhistory server -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2754) MR-279: Error messages are being moved to stdout instead of stderr
MR-279: Error messages are being moved to stdout instead of stderr -- Key: MAPREDUCE-2754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Error messages flagging the reason for app failures are currently being moved to stdout of container logs instead of stderr. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2755) MR-279: AM writes logs to stderr
MR-279: AM writes logs to stderr Key: MAPREDUCE-2755 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2755 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently the AM logs are written to $YARN_LOG_DIR/appID/containerID/stderr. In order to maintain consistency with other container logs, it probably should be moved to syslog. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2757) [MR-279] Redundant "file:" directory created in appcache
[MR-279] Redundant "file:" directory created in appcache - Key: MAPREDUCE-2757 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2758) MR-279: Log info about the location of dist cache
MR-279: Log info about the location of dist cache - Key: MAPREDUCE-2758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2758 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Currently, there is no log info available about the actual location of the file/archive in dist cache being used by the task except for the "ln" command in task.sh. We need to log this information to help in debugging esp in those cases where there are more than one archive with the same name. In 0.20.x, in task logs, one could find log info such as the following: INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /archive <- /archive -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[MR-279] - Cleanup staging dir after job completion --- Key: MAPREDUCE-2762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2763) IllegalArgumentException while using the dist cache
IllegalArgumentException while using the dist cache --- Key: MAPREDUCE-2763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2763 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 IllegalArgumentException is seen while using distributed cache to cache some files and custom jars in classpath. A simple way to reproduce this error is by using a streaming job: hadoop jar hadoop-streaming.jar -libjars file:// -input -output out -mapper "cat" -reducer NONE -cacheFile hdfs://#linkname This is a regression introduced and the same command works fine on 0.20.x -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2766) [MR-279] Set correct permissions for files in dist cache
[MR-279] Set correct permissions for files in dist cache Key: MAPREDUCE-2766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2766 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, the files in both public and private dist cache are having 777 permission. Also, the group ownership of files on private cache have to be set to $TT_SPECIAL_GROUP -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2768) [MR-279] NMs not being blacklisted as determined by health scripts
[MR-279] NMs not being blacklisted as determined by health scripts -- Key: MAPREDUCE-2768 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2768 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The NMs are not being blacklisted via the node health script. Below is the configuration used: yarn.server.nodemanager.healthchecker.script.path= yarn.server.nodemanager.healthchecker.interval=10 yarn.server.nodemanager.healthchecker.script.timeout=12 The node continues to be healthy forever. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2774) Add a startup msg while starting RM/NM
Add a startup msg while starting RM/NM -- Key: MAPREDUCE-2774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Add a startup msg while starting NM/RM indicating the version, build details etc. This will help in easier parsing of logs and debugging. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2790) [MR-279] Add additional field for storing the AM/job history info on CLI
[MR-279] Add additional field for storing the AM/job history info on CLI Key: MAPREDUCE-2790 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2790 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/mapred job [-list [all]] displays the AM or job history location in the "SchedulingInfo" field. An additional column has to be added to display the AM/job history information. Currently, the output reads: {noformat} JobId State StartTime UserNameQueue Priority SchedulingInfo jobID FAILED 0 ramya default NORMAL AM information/job history location {noformat} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[MR-279] Missing/incorrect info on job -status CLI --- Key: MAPREDUCE-2791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 There are a couple of details missing/incorrect on the job -status command line output for completed jobs: 1. Incorrect job file 2. map() completion is always 0 3. reduce() completion is always set to 0 4. history URL is empty 5. Missing launched map tasks 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2792) [MR-279] Replace IP addresses with hostnames
[MR-279] Replace IP addresses with hostnames Key: MAPREDUCE-2792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2792 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, all the logs, UI, CLI have IP addresses of the NM/RM, which are difficult to manage. It will be useful to have hostnames like in 0.20.x for easier debugging and maintenance purpose. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2793) [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs
[MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs -- Key: MAPREDUCE-2793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2793 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 appIDs, jobIDs and attempt/container ids are not consistently named in the logs, console and UI. For consistency purpose, they all have to follow a common naming convention. Currently, For appID = On the RM UI: app_1308259676864_5 On the JHS UI: No appID Console/logs: No appID mapred-local dirs are named as: application_1308259676864_0005 For jobID = On the RM UI: job_1308259676864_5_5 JHS UI: job_1308259676864_5_5 Console/logs: job_1308259676864_0005 mapred-local dirs are named as: No jobID For attemptID On the RM UI: attempt_1308259676864_5_5_m_24_0 JHS attempt_1308259676864_5_5_m_24_0 Console/logs: attempt_1308259676864_0005_m_24_0 mapred-local dirs are named as: container_1308259676864_0005_24 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2794) [MR-279] Incorrect metrics value for AvailableGB per queue per user
[MR-279] Incorrect metrics value for AvailableGB per queue per user --- Key: MAPREDUCE-2794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 AvailableGB per queue is not the same as AvailableGB per queue per user when the user limit is set to 100%. i.e. if the total available GB of the cluster is 60, and queue "default" has 92% capacity with 100% as the user limit, AvailableGB per queue default = 55 (i.e. 0.92*60) whereas AvailableGB per queue for user ramya is 56 (however it should be 55 = 0.92*60*1) Also, unlike the AvailableGB/queue, AvailableGB/queue/user is not decremented when user ramya is running apps on the "default" queue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2795) [MR-279] AppsKilled is never incremented
[MR-279] AppsKilled is never incremented Key: MAPREDUCE-2795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2795 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 AppsKilled metrics is never incremented even though there are killed jobs in the system. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2796) [MR-279] Start time for all the apps is set to 0
[MR-279] Start time for all the apps is set to 0 Key: MAPREDUCE-2796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2796 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The start time for all the apps in the output of "job -list" is set to 0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2799) [MR-279] NPE is throwing on job -status
[ https://issues.apache.org/jira/browse/MAPREDUCE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-2799. Resolution: Duplicate Devaraj, there is already a known bug MAPREDUCE-2686 for the same. Hence resolving this as duplicate. > [MR-279] NPE is throwing on job -status > - > > Key: MAPREDUCE-2799 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2799 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Devaraj K >Assignee: Devaraj K > > {code:xml} > Exception in thread "main" java.lang.NullPointerException > at > org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:113) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:101) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:94) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:384) > at > org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515) > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154) > at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:223) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) > at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074) > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2800) clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files
clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files - Key: MAPREDUCE-2800 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2800 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 for all the map tasks for the last 4 progress interval in the jobhistory files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2801) Include the native libs in java.library.path
Include the native libs in java.library.path - Key: MAPREDUCE-2801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[MR-279] Jobhistory filenames should have jobID to help in better parsing -- Key: MAPREDUCE-2802 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 For jobID such as job_1312933838300_0007, jobhistory file names are named as job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2834) [MR-279] Enable dense update for file sink metrics
[MR-279] Enable dense update for file sink metrics -- Key: MAPREDUCE-2834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, if File sink is enabled for MRAppMaster or Resourcemanager, it does not populate the file with all the available attributes. It would be useful for debugging and admin purpose to have all the metrics populated in the file. For eg: MRAppMaster metrics currently logs value only for JobsRunning even though the total available job level metrics are JobsCompleted, JobsFailed, JobsKilled, JobsPreparing etc -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[MR-279] Node entries on the RM UI are not sortable --- Key: MAPREDUCE-2843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The nodemanager entries on the RM UI is not sortable unlike the other web pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2844) [MR-279] Incorrect node ID info
[MR-279] Incorrect node ID info Key: MAPREDUCE-2844 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2844 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The node ID info for the nodemanager entires on the RM UI incorrectly displays the value of $yarn.server.nodemanager.address instead of the ID. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3043) Missing containers info on the nodes page
Missing containers info on the nodes page - Key: MAPREDUCE-3043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3043 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The containers info on the nodes page on the RM seems to be missing. This was useful in understanding the usage on each of the nodemanagers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3044) Pipes jobs stuck without making progress
Pipes jobs stuck without making progress Key: MAPREDUCE-3044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3044 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 A simple example pipes job gets stuck without making any progress. The AM is launched but the maps do not make any progress. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
Elapsed time filter on jobhistory server displays incorrect table entries - Key: MAPREDUCE-3045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 The elapsed time filter on the jobhistory server filters incorrect information. For e.g. on a cluster where the elapsed time of all the tasks is either 7 or 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3051) HADOOP_CONF_DIR exported twice in the classpath
HADOOP_CONF_DIR exported twice in the classpath --- Key: MAPREDUCE-3051 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3051 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 HADOOP_CONF_DIR is exported twice in the classpath during RM, NM and container startup time. Not an issue so far but seems redundant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2763) IllegalArgumentException while using the dist cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-2763. Resolution: Fixed I see this issue being fixed in the latest code base. Hence resolving this. > IllegalArgumentException while using the dist cache > --- > > Key: MAPREDUCE-2763 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2763 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 > Reporter: Ramya Sunil > Fix For: 0.23.0 > > > IllegalArgumentException is seen while using distributed cache to cache some > files and custom jars in classpath. > A simple way to reproduce this error is by using a streaming job: > hadoop jar hadoop-streaming.jar -libjars file:// -input > -output out -mapper "cat" -reducer NONE -cacheFile > hdfs://#linkname > This is a regression introduced and the same command works fine on 0.20.x -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3063) Mapreduce trunk Commit builds are failing
Mapreduce trunk Commit builds are failing - Key: MAPREDUCE-3063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3063 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Ramya Sunil Priority: Critical Mapreduce trunk commit builds are failing due to test failures. See https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/testReport/ for more details. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3080) dfs calls from streaming fails with ExceptionInInitializerError
dfs calls from streaming fails with ExceptionInInitializerError --- Key: MAPREDUCE-3080 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3080 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.205.0 Reporter: Ramya Sunil Fix For: 0.20.205.0 Dfs calls from streaming seem to fail with the following error: {noformat} Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) Caused by: org.apache.commons.logging.LogConfigurationException: User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is not useable. at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874) at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at org.apache.hadoop.conf.Configuration.(Configuration.java:142) ... 3 more {noformat} commons-logging-1.1.1.jar is in the classpath. An easy way to reproduce this is, on a secure deploy, "hadoop --config $HADOOP_CONF_DIR jar hadoop-streaming.jar -input -output Out -mapper "hadoop --config $HADOOP_CONF_DIR dfs -help" -reducer NONE" -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3777) used mem and util have negative values after a queue addition
used mem and util have negative values after a queue addition - Key: MAPREDUCE-3777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3777 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 After a queue addition to capacity scheduler and submission of an application, root queue utilization and used memory have negative values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3780) RM assigns containers to killed applications
RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3781) Fix history for apps which were terminated before the AM launch
Fix history for apps which were terminated before the AM launch --- Key: MAPREDUCE-3781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3781 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Currently the history for applications which were terminated/killed/failed before the AM was launched redirects to a page that does not exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3783) Fix "queue -list" to display the correct capacity of queues
Fix "queue -list" to display the correct capacity of queues --- Key: MAPREDUCE-3783 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3783 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Occasionally, the capacity of the queue as displayed by "queue -list" has incorrect values. For e.g: yarn.scheduler.capacity.root.queues=a,b yarn.scheduler.capacity.root.b.queues=b1,b2,b3 yarn.scheduler.capacity.root.b.b1.capacity=30 yarn.scheduler.capacity.root.b.b2.capacity=30 yarn.scheduler.capacity.root.b.b3.capacity=40 $ mapred queue -list Queue Name : b Queue State : running Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b1 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b2 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3784) maxActiveApplications(|PerUser) per queue is too low for small clusters
maxActiveApplications(|PerUser) per queue is too low for small clusters --- Key: MAPREDUCE-3784 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3784 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil We ran into this issue while testing on small clusters. On a 7node cluster with 8G per node, for a queue with absolute capacity 30%, user limit 100%, maxActiveApplications and maxActiveApplicationsPerUser is calculated to be 1. This means that even though the queue has 17GB(0.3*8*7), only 1 user can run 1 app at a given time queuing up rest of the apps/users. This hurts performance on small clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3786) No checks for misconfigured userlimit
No checks for misconfigured userlimit - Key: MAPREDUCE-3786 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3786 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Currently, there are no checks being made for misconfigured userLimit (such as negative values/values >100) This can potentially be a problem if the RM comes up with incorrect userLimit values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3792) job -list displays only the jobs submitted by a particular user
job -list displays only the jobs submitted by a particular user --- Key: MAPREDUCE-3792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3792 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil "mapred job -list" lists only the jobs submitted by the user who ran the command. This behavior is different from 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3819) Exception while renewing delegation token
Exception while renewing delegation token - Key: MAPREDUCE-3819 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3820) Improve logging when containers run beyond memory limits
Improve logging when containers run beyond memory limits Key: MAPREDUCE-3820 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3820 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor When containers run beyond memory limits, they are killed without logging any useful message. Diagnostics message reads "Task attemptID failed 0 times" and the console output reads "INFO mapreduce.Job: Job jobID failed with state KILLED due to:" which are both not useful. This message has to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3821) NPE while running Shuffle benchmark
NPE while running Shuffle benchmark --- Key: MAPREDUCE-3821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Critical hadoop jar hadoop-mapreduce-test.jar loadgen -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text The tasks fail with the following exception: {noformat} Error: java.lang.NullPointerException at org.apache.hadoop.fs.Path.(Path.java:69) at org.apache.hadoop.fs.Path.(Path.java:58) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209) at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100) at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94) at org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176) at org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248) at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955) at org.apache.hadoop.mapred.Task.done(Task.java:912) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3841) Broken Server metrics and Local logs link under the tools menu
Broken Server metrics and Local logs link under the tools menu -- Key: MAPREDUCE-3841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3841 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Local logs link redirects to the cluster page and Server metrics opens an empty page on the RM/JHS homepage. So does the links from nodemanager UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3122) -libjars with streaming does not create symlink in the cwd of task
-libjars with streaming does not create symlink in the cwd of task -- Key: MAPREDUCE-3122 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3122 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.205.0 Reporter: Ramya Sunil When libjars option is used with streaming, the symlink to the jar file is not created in the working dir of the task. Any map/reduce tasks which uses this jar fails with NoClassDefFoundError For e.g: $ hadoop jar hadoop-streaming.jar -libjars /home/ramya/HelloWorld.jar -input input.txt -output Out -mapper "mapper.sh" -reducer NONE -file /home/ramya/mapper.sh $ cat /home/ramya/mapper.sh #!/bin/sh cat &> /dev/null java -cp HelloWorld.jar HelloWorld $ java -cp /home/ramya/HelloWorld.jar HelloWorld Hello World -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
Invalid tracking URL for streaming jobs --- Key: MAPREDUCE-3243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The tracking URL for streaming jobs currently display "http://N/A"; {noformat} INFO streaming.StreamJob: To kill this job, run: INFO streaming.StreamJob: hadoop job -kill INFO streaming.StreamJob: Tracking URL: http://N/A INFO mapreduce.Job: Running job: INFO mapreduce.Job: map 0% reduce 0% INFO mapred.ClientServiceDelegate: Tracking Url of JOB is {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3254) Streaming jobs failing with PipeMapRunner ClassNotFoundException
Streaming jobs failing with PipeMapRunner ClassNotFoundException Key: MAPREDUCE-3254 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3254 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil ClassNotFoundException: org.apache.hadoop.streaming.PipeMapRunner encountered while running streaming jobs. Stack trace in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3263) compile-mapred-test target fails
compile-mapred-test target fails Key: MAPREDUCE-3263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3263 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Compile mapred test target is broken due to which the builds are not archiving the test jars. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3269) Jobsummary logs not being moved to a separate file
Jobsummary logs not being moved to a separate file -- Key: MAPREDUCE-3269 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3269 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Blocker The jobsummary logs are not being moved to a separate file. Below is the configuration in log4j.properties: {noformat} mapred.jobsummary.logger=INFO,console log4j.logger.org.apache.hadoop.mapreduce.jobhistory.JobSummary=${mapred.jobsummary.logger} log4j.additivity.org.apache.hadoop.mapreduce.jobhistory.JobSummary=false log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender log4j.appender.JSA.File=${hadoop.log.dir}/mapred-jobsummary.log log4j.appender.JSA.layout=org.apache.log4j.PatternLayout log4j.appender.JSA.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.JSA.DatePattern=.-MM-dd {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3270) Decommissioned node not removed from active NM list
Decommissioned node not removed from active NM list --- Key: MAPREDUCE-3270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3270 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Blocker A decommissioned node is not being removed from the "Total nodes" list and is not added to the "Decommissioned nodes" list. The list of nodes to decommission is added in a file defined by "yarn.resourcemanager.nodes.exclude-path" and excluded via refreshNodes CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3271) Lost nodes list and count not updated
Lost nodes list and count not updated - Key: MAPREDUCE-3271 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3271 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 When nodemanagers are lost, the "Lost Nodes" list and the count is not incremented. Either we, 1. Fix the lost nodes list when a nodemanager is lost - The problem with tracking lost nodes is, if the nodemanager joins back, there would be duplicate entries in active and lost nodes with different port numbers. 2. Do not track lost nodemanagers -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3272) Lost NMs fail to rejoin
Lost NMs fail to rejoin --- Key: MAPREDUCE-3272 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Lost nodemanagers fail to join back. When the NM is lost, RM log reads {noformat} INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired: Timed out after 600 secs INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing of type EXPIRE INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node Transitioned from RUNNING to LOST {noformat} When the NM joins back, RM log reads {noformat} INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3282) bin/mapred job -list throws exception
bin/mapred job -list throws exception - Key: MAPREDUCE-3282 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3282 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/mapred job -list throws exception when mapreduce.framework.name is set to "yarn" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3283) yarn classpath CLI does not display the complete classpath
yarn classpath CLI does not display the complete classpath -- Key: MAPREDUCE-3283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3283 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/yarn classpath does not display the complete classpath. Below is how the classpath looks like: {noformat} $HADOOP_CONF_DIR:$HADOOP_CONF_DIR::$TOOLS_JAR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*: $HADOOP_MAPRED_HOME/bin/../modules/*:$HADOOP_MAPRED_HOME/bin/../lib/* {noformat} "*" has to be substituted with the actual jars. Also, $HADOOP_CONF_DIR appears twice in the classpath -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3284) bin/mapred queue fails with JobQueueClient ClassNotFoundException
bin/mapred queue fails with JobQueueClient ClassNotFoundException - Key: MAPREDUCE-3284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3284 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/mapred queue fails with the following exception: {code} -bash$ bin/mapred queue Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobQueueClient Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobQueueClient at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapred.JobQueueClient. Program will exit. {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3288) Mapreduce 23 builds failing
Mapreduce 23 builds failing --- Key: MAPREDUCE-3288 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3288 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Blocker Fix For: 0.23.0 Hadoop mapreduce 0.23 builds are failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3290) list-active-trackers throws NPE
list-active-trackers throws NPE --- Key: MAPREDUCE-3290 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3290 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/mapred -list-active-trackers throws NPE in mrV2. Trace in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3291) App fail to launch due to delegation token not found in cache
App fail to launch due to delegation token not found in cache - Key: MAPREDUCE-3291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3291 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 In secure mode, saw an app failure due to "org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token for ) can't be found in cache" Exception in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3293) Reason for application failure is not correctly reported
Reason for application failure is not correctly reported Key: MAPREDUCE-3293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3293 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 When apps fail, the reason for failure is not correctly reflected in the UI. For one such app failure, the UI reports "Application failed 1 times due to . Failing the application." which is not very helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3294) Log the reason for killing a task during speculative execution
Log the reason for killing a task during speculative execution -- Key: MAPREDUCE-3294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3294 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The reason for killing a speculated task has to be logged. Currently, a speculated task is killed with a note of "Container killed by the ApplicationMaster. Container killed on request. Exit code is 137" which is not very useful. Better logging of this message stating the task was killed due to completion of its speculative task would be useful. Also, this message is lost once the app is moved to history. All we are left with is a list of killed tasks without a reason being notified to the user. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3305) Fix -list-blacklisted-trackers to print the blacklisted NMs
Fix -list-blacklisted-trackers to print the blacklisted NMs --- Key: MAPREDUCE-3305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3305 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 bin/mapred job -list-blacklisted-trackers currently prints "getBlacklistedTrackers - Not implemented yet" This is a long pending issue. Could not find a tracking ticket, hence opening one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3307) Improve logging on the console during job execution
Improve logging on the console during job execution --- Key: MAPREDUCE-3307 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3307 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 There is a lot of redundant information being printed on the console and a not so intuitive flow of events. We should improve the logging on console during job execution. More details in the next comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3308) MR builds failing due to download failure
MR builds failing due to download failure - Key: MAPREDUCE-3308 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3308 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Giridharan Kesavan Priority: Critical Fix For: 0.23.0 MR builds are failing due to unresolved dependencies. [ivy:resolve] :: problems summary :: [ivy:resolve] WARNINGS [ivy:resolve] [FAILED ] org.apache.commons#commons-daemon;1.0.3!commons-daemon.jar: (0ms) [ivy:resolve] fs: tried [ivy:resolve] /home/jenkins/.m2/repository/org/apache/commons/commons-daemon/1.0.3/commons-daemon-1.0.3.jar [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-daemon/1.0.3/commons-daemon-1.0.3.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/commons/commons-daemon/1.0.3/commons-daemon-1.0.3.jar [ivy:resolve] :: [ivy:resolve] :: FAILED DOWNLOADS:: [ivy:resolve] :: ^ see resolution messages for details ^ :: [ivy:resolve] :: [ivy:resolve] :: org.apache.commons#commons-daemon;1.0.3!commons-daemon.jar [ivy:resolve] :: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3309) Report the AM of an application in the UI
Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3368) compile-mapred-test fails
compile-mapred-test fails - Key: MAPREDUCE-3368 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3368 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Critical Fix For: 0.23.0 compile-mapred-test target is failing once again. Details: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-0.23-Build/83/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3605) Allow mr commands to be run via bin/hadoop
Allow mr commands to be run via bin/hadoop -- Key: MAPREDUCE-3605 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3605 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 MR command line options are not supported in bin/hadoop. {noformat} bin/hadoop job Exception in thread "main" java.lang.NoClassDefFoundError: job Caused by: java.lang.ClassNotFoundException: job at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: job. Program will exit. {noformat} A deprecated message like "DEPRECATED: Use of this script to execute mapred command is deprecated. Instead use the mapred command for it." should be displayed along with the correct output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3646) Remove redundant URL info from "mapred job" output
Remove redundant URL info from "mapred job" output -- Key: MAPREDUCE-3646 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3646 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.1 The URL information to track the job is printed for all the "mapred job"mrv2 commands. This information is redundant and has to be removed. E.g: {noformat} -bash-3.2$ mapred job -list Total jobs:3 JobId State StartTime UserNameQueue PriorityMaps Reduces UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info 12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the job: :8088/proxy// RUNNING 1326147596446 ramya default NORMAL 10 10 21 0 22528M 0M 22528M :8088/proxy// 12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the job: :8088/proxy// RUNNING 1326147603726 ramya default NORMAL 10 10 11 0 12288M 0M 12288M :8088/proxy// 12/01/09 22:20:16 INFO mapred.ClientServiceDelegate: The url to track the job: :8088/proxy// RUNNING 1326147520126 ramya default NORMAL 10 10 21 0 22528M 0M 22528M :8088/proxy// {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3647) Pipes job fails with "Illegal text protocol"
Pipes job fails with "Illegal text protocol" Key: MAPREDUCE-3647 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3647 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Pipes job fail with "Hadoop Pipes Exception: Illegal text protocol command" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3746) Nodemanagers are not automatically shut down after decommissioning
Nodemanagers are not automatically shut down after decommissioning -- Key: MAPREDUCE-3746 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3746 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Nodemanagers are not automatically shutdown after decommissioning. MAPREDUCE-2775 does not seem to fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
Memory Total is not refreshed until an app is launched -- Key: MAPREDUCE-3747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Memory Total on the RM UI is not refreshed until an application is launched. This is a problem when the cluster is started for the first time or when there are any lost/decommissioned NMs. When the cluster is started for the first time, Active Nodes is > 0 but the Memory Total=0. Also when there are any lost/decommissioned nodes, Memory Total has wrong value. This is a useful tool for cluster admins and has to be updated correctly without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3748) Move CS related nodeUpdate log messages to DEBUG
Move CS related nodeUpdate log messages to DEBUG Key: MAPREDUCE-3748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.1 Currently, the RM has nodeUpdate logs per NM per second such as the following: 2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: : clusterResources: memory: 57344 2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: : clusterResources: memory: 57344 2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: : clusterResources: memory: 57344 Debugging is difficult with huge amount of logs such as this. These logs need to be moved to DEBUG. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3759) ClassCastException thrown in -list-active-trackers when there are a few unhealthy nodes
ClassCastException thrown in -list-active-trackers when there are a few unhealthy nodes --- Key: MAPREDUCE-3759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3759 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 When there are a few blacklisted nodes in the cluster, "bin/mapred job -list-active-trackers" throws "java.lang.ClassCastException: org.apache.hadoop.yarn.server.resourcemanager.resource.Resources$1 cannot be cast to org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3760) Blacklisted NMs should not appear in Active nodes list
Blacklisted NMs should not appear in Active nodes list -- Key: MAPREDUCE-3760 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3760 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.1 Blacklisted NMs appear in both "Active Nodes" and "Unhealthy nodes" on the RM UI. This should be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3761) AM info in job -list does not reflect the actual AM hostname
AM info in job -list does not reflect the actual AM hostname Key: MAPREDUCE-3761 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3761 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 The AM info field on "bin/mapred job -list" currently has a value :8088/proxy/appID. This info is irrelevant unless it shows the real information of where the AM was launched. This needs to be fixed to show the AM host details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3763) Failed refreshQueues due to misconfiguration prevents further refreshing of queues
Failed refreshQueues due to misconfiguration prevents further refreshing of queues -- Key: MAPREDUCE-3763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3763 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was: 1. Added a new queue "newQueue" without defining its capacity. 2. "bin/mapred queue -refreshQueues" fails correctly with "Illegal capacity of -1 for queue root.newQueue" 3. However, after defining the capacity of "newQueue" followed by a second "bin/mapred queue -refreshQueues" throws "org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists!" Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added. The expected behavior would be to refresh the queues correctly and allow addition of "newQueue". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira