[jira] Commented: (MAPREDUCE-2073) TestTrackerDistributedCacheManager should be up-front about requirements on build environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930491#action_12930491 ] Tom White commented on MAPREDUCE-2073: -- +1 This looks good to me. Can you run test-patch on it please? > TestTrackerDistributedCacheManager should be up-front about requirements on > build environment > - > > Key: MAPREDUCE-2073 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2073 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: distributed-cache, test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: mapreduce-2073-0.20.txt, mapreduce-2073.txt > > > TestTrackerDistributedCacheManager will fail on a system where the build > directory is in any path where an ancestor doesn't have a+x permissions. On > one of our hudson boxes, for example, hudson's workspace had 700 permissions > and caused this test to fail reliably, but not in an obvious manner. It would > be helpful if the test failed with a more obvious error message during > setUp() when the build environment is misconfigured. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-461) Enable ServicePlugins for the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-461: Attachment: MAPREDUCE-461.patch Here's an updated patch which includes a unit test. It also addresses Amar's feedback. > Enable ServicePlugins for the JobTracker > > > Key: MAPREDUCE-461 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-461 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Fredrik Hedberg >Priority: Minor > Attachments: MAPREDUCE-461.patch, sp-jt-1.diff > > > Allow ServicePlugins (see HADOOP-5257) for the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-461) Enable ServicePlugins for the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-461: Status: Patch Available (was: Open) > Enable ServicePlugins for the JobTracker > > > Key: MAPREDUCE-461 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-461 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Fredrik Hedberg >Priority: Minor > Attachments: MAPREDUCE-461.patch, sp-jt-1.diff > > > Allow ServicePlugins (see HADOOP-5257) for the JobTracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930454#action_12930454 ] Todd Lipcon commented on MAPREDUCE-2181: I think you're mixing up staging dir (a rather new config) and system dir (one that's been around a long time). staging dir is per-user, and written to by the user submitting the job (from the submitting machine) > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930439#action_12930439 ] Allen Wittenauer commented on MAPREDUCE-2181: - Actually, now that I think about it. I don't really care. It is a system tunable.There are so many other bad defaults, one more won't hurt. > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930438#action_12930438 ] Allen Wittenauer commented on MAPREDUCE-2181: - Why would staging be 777? IIRC, it should be owned and is only written to by the user that the JT runs as. > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930437#action_12930437 ] Todd Lipcon commented on MAPREDUCE-2181: bq. In any case, I'm still left with /user being not a good place to put system resources I fail to see how the job staging directory is considered a "system resource". It's per-user temporary data during the job submission process. Much like how web browsers store per-user caches in $HOME/.mozilla, the job submitter should put its data in $HOME/.staging. Putting a big quota on /mapred and making /mapred/staging mode 777 (or mode 1777 on trunk) just gives users one more place they can potentially abuse to store more data than they should be allowed. > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930425#action_12930425 ] Allen Wittenauer commented on MAPREDUCE-2181: - I realize this is an HDFS dir. Let be more obvious: What I'm worried about is that many sites with multiple users do: ... dfsadmin -setQuota value /user/* ... so that all users have the same quota values. [Making variable sizes of quotas is makes Hadoop nearly impossible support since there is no real quota reporting capabilities, short of traversing the file system looking for them.] In this case, it would basically mean that the JobTracker would be forced to contend with the same quota size as users. Even given your scenario above, this would mean that the JT space quota would need to be usersize*number of users, which is a bit ridiculous to maintain. [If anyone actually sets /user explicitly... well, I hope they aren't multi-user or have some sort of Plan.] In any case, I'm still left with /user being not a good place to put system resources. There are reasons why everyone in the UNIX world doesn't put home directories under /usr anymore. Mixing system bits and user bits is just bad practice. > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930412#action_12930412 ] Todd Lipcon commented on MAPREDUCE-2181: bq. I can see major problems with sites that set quotas. It seems to me that quotas should include temporary space used during job submission. If your job jar or dcache resources are very large, by all means it should count against your quota, don't you think? On any _reasonable_ workload, the space used in the staging directory will be many orders of magnitude lower than any user quotas, anyway. bq. Why not just make it an explicit /tmp? Because /tmp is not necessarily created on a fresh HDFS either. Note that this is an *HDFS* directory, not local fs. > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2034) TestSubmitJob triggers NPE instead of permissions error
[ https://issues.apache.org/jira/browse/MAPREDUCE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930404#action_12930404 ] Todd Lipcon commented on MAPREDUCE-2034: Test patch result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. I checked the release audit and findbugs, and it seems like an issue with the test-patch script -- in both cases the "new" warnings were unrelated to the patch. > TestSubmitJob triggers NPE instead of permissions error > --- > > Key: MAPREDUCE-2034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2034 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: mapreduce-2034.txt > > > TestSubmitJob.testSecureJobExecution catches _any_ IOException and assumes a > permissions error has been caught. In fact, it was passing an invalid path > name to the NameNode and triggering an NPE, not a Permission denied error, in > one case, but the test was not specific enough to detect this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930403#action_12930403 ] Allen Wittenauer commented on MAPREDUCE-2181: - I'm not a fan of /user. I can see major problems with sites that set quotas. Why not just make it an explicit /tmp? [FWIW, I use /system to keep it completely separate from everything.] > mapreduce.jobtracker.staging.root.dir default is unreasonable > - > > Key: MAPREDUCE-2181 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > The default for mapreduce.jobtracker.staging.root.dir is set to > ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal > cluster. hadoop.tmp.dir is overloaded in different places where sometimes it > is a local path and sometimes it is a path on HDFS, which makes things even > more confusing. > We should change the default for the staging directory to /user (as is > suggested by the description of that configuration) and then fix > LocalJobRunner to use a different configuration -- perhaps > mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a > *local* path. That one could legitimately default to something inside > hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2181) mapreduce.jobtracker.staging.root.dir default is unreasonable
mapreduce.jobtracker.staging.root.dir default is unreasonable - Key: MAPREDUCE-2181 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2181 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, jobtracker Affects Versions: 0.22.0 Reporter: Todd Lipcon The default for mapreduce.jobtracker.staging.root.dir is set to ${hadoop.tmp.dir}/mapred/staging, which doesn't really work on a normal cluster. hadoop.tmp.dir is overloaded in different places where sometimes it is a local path and sometimes it is a path on HDFS, which makes things even more confusing. We should change the default for the staging directory to /user (as is suggested by the description of that configuration) and then fix LocalJobRunner to use a different configuration -- perhaps mapreduce.localjobrunner.staging.root.dir -- to make it clear that it's a *local* path. That one could legitimately default to something inside hadoop.tmp.dir. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2141) Add an "extra data" field to Task for use by Mesos
[ https://issues.apache.org/jira/browse/MAPREDUCE-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930373#action_12930373 ] Owen O'Malley commented on MAPREDUCE-2141: -- +1, looks good. > Add an "extra data" field to Task for use by Mesos > -- > > Key: MAPREDUCE-2141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2141 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Matei Zaharia >Assignee: Matei Zaharia >Priority: Minor > Attachments: mapreduce-2141-v1.patch, mapreduce-2141-v2.patch > > > In order to support running Hadoop on the Mesos cluster manager > (http://mesos.berkeley.edu/), I'd like to add an extra String field to the > Task class to allow extra data (a Mesos task ID) to be associated with each > task. This should have no impact on normal operation other than making the > serialized form of Task a few bytes longer. In the Mesos support patch for > Hadoop, this field is set by a pluggable Hadoop scheduler implementation to > allow code on the TaskTracker side to see which Mesos task corresponds to > each Hadoop task. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2167) Faster directory traversal for raid node
[ https://issues.apache.org/jira/browse/MAPREDUCE-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-2167: --- Attachment: MAPREDUCE-2167.4.patch Fixed a broken test. TEST RESULTS: ant test-patch has the same number of failures as a clean checkout {code} [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] {code} ant test succeeds: {code} test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 47.071 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 124.583 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 9.337 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 24.481 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.392 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.485 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 71.136 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 471.072 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 107.828 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 25.714 sec test: BUILD SUCCESSFUL Total time: 15 minutes 6 seconds {code} > Faster directory traversal for raid node > > > Key: MAPREDUCE-2167 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2167 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-2167.2.patch, MAPREDUCE-2167.3.patch, > MAPREDUCE-2167.4.patch, MAPREDUCE-2167.patch > > > The RaidNode currently iterates over the directory structure to figure out > which files to RAID. With millions of files, this can take a long time - > especially if some files are already RAIDed and the RaidNode needs to look at > parity files / parity file HARs to determine if the file needs to be RAIDed. > The directory traversal is encapsulated inside the class DirectoryTraversal, > which examines one file at a time, using the caller's thread. > My proposal is to make this multi-threaded as follows: > * use a pool of threads inside DirectoryTraversal > * The caller's thread is used to retrieve directories, and each new > directory is assigned to a thread in the pool. The worker thread examines all > the files the directory. > * If there sub-directories, those are added back as workitems to the pool. > Comments? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2180) Add coverage of fair scheduler servlet to system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2180: --- Fix Version/s: 0.22.0 Status: Patch Available (was: Open) > Add coverage of fair scheduler servlet to system test > - > > Key: MAPREDUCE-2180 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2180 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce-2180.txt > > > MAPREDUCE-2051 added a system test for the fair scheduler which starts a > minicluster and runs a couple jobs with preemption on. I recently found a > deadlock in a previous version of the scheduler that was due to lock > inversion between the scheduler servlet and some JT internals. I'd like to > modify the existing system test to also hit the /scheduler servlet, allowing > jcarder to detect such lock inversions in the future. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2180) Add coverage of fair scheduler servlet to system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2180: --- Attachment: mapreduce-2180.txt Attached patch exercises the servlet at /scheduler while running the jobs. Under jcarder, this shows the lock inversion in the previous release and shows that it is gone in trunk. > Add coverage of fair scheduler servlet to system test > - > > Key: MAPREDUCE-2180 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2180 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: mapreduce-2180.txt > > > MAPREDUCE-2051 added a system test for the fair scheduler which starts a > minicluster and runs a couple jobs with preemption on. I recently found a > deadlock in a previous version of the scheduler that was due to lock > inversion between the scheduler servlet and some JT internals. I'd like to > modify the existing system test to also hit the /scheduler servlet, allowing > jcarder to detect such lock inversions in the future. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2180) Add coverage of fair scheduler servlet to system test
Add coverage of fair scheduler servlet to system test - Key: MAPREDUCE-2180 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2180 Project: Hadoop Map/Reduce Issue Type: Test Components: contrib/fair-share Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor MAPREDUCE-2051 added a system test for the fair scheduler which starts a minicluster and runs a couple jobs with preemption on. I recently found a deadlock in a previous version of the scheduler that was due to lock inversion between the scheduler servlet and some JT internals. I'd like to modify the existing system test to also hit the /scheduler servlet, allowing jcarder to detect such lock inversions in the future. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2169) Integrated Reed-Solomon code with RaidNode
[ https://issues.apache.org/jira/browse/MAPREDUCE-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-2169: --- Attachment: MAPREDUCE-2169.2.patch TEST RESULTS: ant test under raid: {code} test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 373.594 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 138.885 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 15.061 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 24.491 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.39 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.809 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.229 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 461.174 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 218.163 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 24.31 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 14.96 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.368 sec test: BUILD SUCCESSFUL Total time: 22 minutes 53 seconds ant test-patch has the same result as a clean checkout (see MAPREDUCE-2176) {code} [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 28 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == [exec] [exec] {code} > Integrated Reed-Solomon code with RaidNode > -- > > Key: MAPREDUCE-2169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2169 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/raid >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-2169.2.patch, MAPREDUCE-2169.patch > > > Scott Chen recently checked in an implementation of the Reed Solomon code. > This task will track the integration of the code with RaidNode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930227#action_12930227 ] Scott Chen commented on MAPREDUCE-2170: --- Hey Harsh, Why do we need to override getSystemLoadAverage()? The native one should also work for Linux case, right? {code} @Override public float getSystemLoadAverage() { readProcLoadAverageFile(); return loadAverage; } {code} > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930194#action_12930194 ] Arun C Murthy commented on MAPREDUCE-2170: -- Harsh, can you please put up the patch here on jira and grant license to ASF for inclusion? Thanks. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930068#action_12930068 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- I've updated the patch on the review board. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1592) Generate Eclipse's .classpath file from Ivy config
[ https://issues.apache.org/jira/browse/MAPREDUCE-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930018#action_12930018 ] Hudson commented on MAPREDUCE-1592: --- Integrated in Hadoop-Mapreduce-trunk-Commit #533 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/533/]) > Generate Eclipse's .classpath file from Ivy config > -- > > Key: MAPREDUCE-1592 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1592 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Tom White >Assignee: Tom White > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1592.patch, MAPREDUCE-1592.patch, > MAPREDUCE-1592.patch, MAPREDUCE-1592.patch > > > MapReduce companion issue for HADOOP-6407. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2179) RaidBlockSender.java compilation fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930019#action_12930019 ] Hudson commented on MAPREDUCE-2179: --- Integrated in Hadoop-Mapreduce-trunk-Commit #533 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/533/]) MAPREDUCE-2179. Fix RaidBlockSender compilation failure. (Ramkumar Vadali via schen) > RaidBlockSender.java compilation fails > -- > > Key: MAPREDUCE-2179 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2179 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.22.0 >Reporter: Giridharan Kesavan >Assignee: Ramkumar Vadali >Priority: Blocker > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2179.patch > > > https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/490/consoleFull > Mapreduce trunk compilation is broken with > compile: > [echo] contrib: raid > [javac] Compiling 27 source files to > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/contrib/raid/classes > [javac] > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:71: > cannot find symbol > [javac] symbol : class BlockTransferThrottler > [javac] location: class > org.apache.hadoop.hdfs.server.datanode.RaidBlockSender > [javac] private BlockTransferThrottler throttler; > [javac] ^ > [javac] > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:377: > cannot find symbol > [javac] symbol : class BlockTransferThrottler > [javac] location: class > org.apache.hadoop.hdfs.server.datanode.RaidBlockSender > [javac] BlockTransferThrottler throttler) throws > IOException { > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] Note: Some input files use unchecked or unsafe operations. > [javac] Note: Recompile with -Xlint:unchecked for details. > [javac] 2 errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2127) mapreduce trunk builds are filing on hudson ..
[ https://issues.apache.org/jira/browse/MAPREDUCE-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930015#action_12930015 ] Ranjit Mathew commented on MAPREDUCE-2127: -- bq. Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch. Yes, but at least on Fedora 13, this patch *is* required due to the linker changes I referred to earlier. > mapreduce trunk builds are filing on hudson .. > --- > > Key: MAPREDUCE-2127 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2127 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, pipes >Reporter: Giridharan Kesavan > Attachments: MAPREDUCE-2127.patch > > > https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console > [exec] checking for pthread.h... yes > [exec] checking for pthread_create in -lpthread... yes > [exec] checking for HMAC_Init in -lssl... no > [exec] configure: error: Cannot find libssl.so > [exec] > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: > line 4250: exit: please: numeric argument required > [exec] > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: > line 4250: exit: please: numeric argument required > BUILD FAILED > /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: > exec returned: 255 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.