[jira] Updated: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J Chouraria updated MAPREDUCE-2170: - Attachment: mapreduce.loadaverage.r4.diff Updating patch to fix a findbugs BX warning on ResourceCalculatorPlugin:46 > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[
https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934065#action_12934065
]
Ramkumar Vadali commented on MAPREDUCE-1783:
Latest patch TEST RESULTS:
One test fails, but that also fails on a clean checkout
{code}
[junit] Test org.apache.hadoop.mapred.TestControlledMapReduceJob FAILED
(timeout)
{code}
ant test-patch succeeds:
{code}
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
[exec]
[exec]
BUILD SUCCESSFUL
Total time: 13 minutes 6 seconds
Test results are in /tmp/rvadali.hadoopQA
{code}
> Task Initialization should be delayed till when a job can be run
>
>
> Key: MAPREDUCE-1783
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
>Affects Versions: 0.20.1
>Reporter: Ramkumar Vadali
>Assignee: Ramkumar Vadali
> Fix For: 0.22.0
>
> Attachments: 0001-Pool-aware-job-initialization.patch,
> 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch,
> submit-mapreduce-1783.patch
>
>
> The FairScheduler task scheduler uses PoolManager to impose limits on the
> number of jobs that can be running at a given time. However, jobs that are
> submitted are initiaiized immediately by EagerTaskInitializationListener by
> calling JobInProgress.initTasks. This causes the job split file to be read
> into memory. The split information is not needed until the number of running
> jobs is less than the maximum specified. If the amount of split information
> is large, this leads to unnecessary memory pressure on the Job Tracker.
> To ease memory pressure, FairScheduler can use another implementation of
> JobInProgressListener that is aware of PoolManager limits and can delay task
> initialization until the number of running jobs is below the maximum.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2195) New property for local conf directory in system-test-mapreduce.xml file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2195. --- Tags: herriot Resolution: Fixed Fix Version/s: 0.22.0 Same as for HDFS, no need to review. I have just committed it. > New property for local conf directory in system-test-mapreduce.xml file. > > > Key: MAPREDUCE-2195 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2195 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 > Environment: herriot >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > > As its counter-part HDFS-1167: new parameter needs to be added to the > system-test configuration file to serve 'cluster restart with new > configuration' feature -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2195) New property for local conf directory in system-test-mapreduce.xml file.
New property for local conf directory in system-test-mapreduce.xml file. Key: MAPREDUCE-2195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2195 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Environment: herriot Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik As its counter-part HDFS-1167: new parameter needs to be added to the system-test configuration file to serve 'cluster restart with new configuration' feature -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-1783: --- Status: Patch Available (was: Open) > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1783) Task Initialization should be delayed till when a job can be run
[ https://issues.apache.org/jira/browse/MAPREDUCE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-1783: --- Attachment: MAPREDUCE-1783.patch Patch after svn up > Task Initialization should be delayed till when a job can be run > > > Key: MAPREDUCE-1783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Affects Versions: 0.20.1 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Fix For: 0.22.0 > > Attachments: 0001-Pool-aware-job-initialization.patch, > 0001-Pool-aware-job-initialization.patch.1, MAPREDUCE-1783.patch, > submit-mapreduce-1783.patch > > > The FairScheduler task scheduler uses PoolManager to impose limits on the > number of jobs that can be running at a given time. However, jobs that are > submitted are initiaiized immediately by EagerTaskInitializationListener by > calling JobInProgress.initTasks. This causes the job split file to be read > into memory. The split information is not needed until the number of running > jobs is less than the maximum specified. If the amount of split information > is large, this leads to unnecessary memory pressure on the Job Tracker. > To ease memory pressure, FairScheduler can use another implementation of > JobInProgressListener that is aware of PoolManager limits and can delay task > initialization until the number of running jobs is below the maximum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933867#action_12933867 ] Nigel Daley commented on MAPREDUCE-2170: Sorry for the long precommit comment by Hadoop QA. I'm trying to get this running on MR again. Lots of tests failing. Looks like trunk wasn't properly updated after 0.22 branching. I still see references to 0.22 in the ivy/libraries.properties file on trunk. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce.loadaverage.r3.diff > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2194) Local mode seems to be broken in Cloudera's 737 release
[
https://issues.apache.org/jira/browse/MAPREDUCE-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved MAPREDUCE-2194.
Resolution: Invalid
Hi Alex. I transferred this ticket to the Cloudera DISTRO
JIRA:https://issues.cloudera.org/browse/DISTRO-63
For CDH specific bugs it's best to either use that JIRA or our support portal,
unless you can reproduce upstream or in trunk.
> Local mode seems to be broken in Cloudera's 737 release
> ---
>
> Key: MAPREDUCE-2194
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2194
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2
>Reporter: Alex Rovner
>
> We have upgraded our dev environment from Cloudera's 0.20.2-228-cloudera to
> 0.20.2-737-cloudera
> Version 228 worked great for us. In version 737 we are getting the following
> exception:
> (LocalJobRunner.java:295) - job_local_0005
> java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit
> cannot be cast to org.apache.hadoop.mapred.InputSplit
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:357)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> We have tried to set
> job.getConfiguration().setBoolean("mapred.reducer.new-api", true) but that
> did not resolve the issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2194) Local mode seems to be broken in Cloudera's 737 release
Local mode seems to be broken in Cloudera's 737 release
---
Key: MAPREDUCE-2194
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2194
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Alex Rovner
We have upgraded our dev environment from Cloudera's 0.20.2-228-cloudera to
0.20.2-737-cloudera
Version 228 worked great for us. In version 737 we are getting the following
exception:
(LocalJobRunner.java:295) - job_local_0005
java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit
cannot be cast to org.apache.hadoop.mapred.InputSplit
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:357)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
We have tried to set
job.getConfiguration().setBoolean("mapred.reducer.new-api", true) but that did
not resolve the issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933733#action_12933733 ] Hadoop QA commented on MAPREDUCE-2170: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459720/mapreduce.loadaverage.r3.diff against trunk revision 1036745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.conf.TestJobConf org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.examples.pi.math.TestSummation org.apache.hadoop.examples.terasort.TestTeraSort org.apache.hadoop.examples.TestBaileyBorweinPlouffe org.apache.hadoop.fs.slive.TestSlive org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.fs.TestHarFileSystem org.apache.hadoop.fs.TestJHLA org.apache.hadoop.io.TestSequenceFileMergeProgress org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.pipes.TestPipesAsDifferentUser org.apache.hadoop.mapred.pipes.TestPipes org.apache.hadoop.mapred.TestAdminOperationsProtocolWithServiceAuthorization org.apache.hadoop.mapred.TestAuditLogger org.apache.hadoop.mapred.TestBadRecords org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestClusterStatus org.apache.hadoop.mapred.TestCompositeTaskTrackerInstrumentation org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestDebugScriptWithLinuxTaskController org.apache.hadoop.mapred.TestDebugScript org.apache.hadoop.mapred.TestEmptyJob org.apache.hadoop.mapred.TestFieldSelection org.apache.hadoop.mapred.TestIFile org.apache.hadoop.mapred.TestInputPath org.apache.hadoop.mapred.TestIsolationRunner org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapred.TestJobDirCleanup org.apache.hadoop.mapred.TestJobExecutionAsDifferentUser org.apache.hadoop.mapred.TestJobHistory org.apache.hadoop.mapred.TestJobInProgressListener org.apache.hadoop.mapred.TestJobKillAndFail org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestJobQueueClient org.apache.hadoop.mapred.TestJobQueueInformation org.apache.hadoop.mapred.TestJobRetire org.apache.hadoop.mapred.TestJobStatusPersistency org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestJobTrackerInstrumentation org.apache.hadoop.mapred.TestJobTrackerStart org.apache.hadoop.mapred.TestJobTrackerXmlJsp org.apache.hadoop.mapred.TestJvmManager org.apache.hadoop.mapred.TestJvmReuse org.apache.hadoop.mapred.TestKeyValueTextInputFormat org.apache.hadoop.mapred.TestKillSubProcessesWithLinuxTaskController org.apache.hadoop.mapred.TestKillSubProcesses org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestLimitTasksPerJobTaskScheduler org.apache.hadoop.mapred.TestLinuxTaskController org.apache.hadoop.mapred.TestLocalizationWithLinuxTaskController org.apache.hadoop.mapred.TestMapProgress org.apache.hadoop.mapred.TestMapredHeartbeat org.apache.hadoop.mapred.TestMapredSystemDir org.apac
