[jira] Commented: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784657#action_12784657
]
Hadoop QA commented on MAPREDUCE-181:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12426616/181-3.patch
against trunk revision 885530.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 75 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/284/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/284/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/284/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/284/console
This message is automatically generated.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Affects Version/s: (was: 0.21.0) 0.22.0 > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Vinod K V >Assignee: Sharad Agarwal > Fix For: 0.22.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-1245: -- Resolution: Fixed Fix Version/s: (was: 0.21.0) 0.22.0 Assignee: Sharad Agarwal Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Vinod for catching this. > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Sharad Agarwal > Fix For: 0.22.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1190) Add package.html to pi and pi.math packages.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784653#action_12784653
]
Chris Douglas commented on MAPREDUCE-1190:
--
+1
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
> Add package.html to pi and pi.math packages.
>
>
> Key: MAPREDUCE-1190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1190
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: documentation
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: m1190_20091105.patch
>
>
> package.html is missing in the pi and pi.math packages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1190) Add package.html to pi and pi.math packages.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved MAPREDUCE-1190. -- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] I committed this. Thanks, Nicholas! > Add package.html to pi and pi.math packages. > > > Key: MAPREDUCE-1190 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1190 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 0.22.0 > > Attachments: m1190_20091105.patch > > > package.html is missing in the pi and pi.math packages. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1245) TestFairScheduler fails with "too many open files" error
[ https://issues.apache.org/jira/browse/MAPREDUCE-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784649#action_12784649 ] Vinod K V commented on MAPREDUCE-1245: -- Patch looks good. +1. > TestFairScheduler fails with "too many open files" error > > > Key: MAPREDUCE-1245 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1245 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Vinod K V > Fix For: 0.21.0 > > Attachments: 1245_v1.patch > > > This was caused by MAPREDUCE-1103 and was observed after MAPREDUCE-1239. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784650#action_12784650 ] Jothi Padmanabhan commented on MAPREDUCE-1185: -- +1, patch looks good! > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, 1185_v7.patch, > patch-1185-1-ydist.txt, patch-1185-2-ydist.txt, patch-1185-3-ydist.txt, > patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1258) Fair scheduler event log not logging job info
Fair scheduler event log not logging job info - Key: MAPREDUCE-1258 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.21.0 Reporter: Matei Zaharia Assignee: Matei Zaharia Priority: Minor The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before. It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784645#action_12784645 ] Matei Zaharia commented on MAPREDUCE-698: - The patch looks good to me. The only other thing I'd ask for is some documentation of what happens if a pool's max share is set to lower than its min share. The best way to do this might be to log a warning using LOG.warn before the "commit" part of PoolManager.reloadAllocs and say that when a pool's max share is lower than its min share, the max share takes precedence. You could also display this warning on the web UI when pools with max share < min share exist, because admins are likely to be looking at the UI after they modify their config file. The unit test failures seem to be unrelated to the patch - in particular, the one in the fair scheduler is due to MAPREDUCE-1245. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-1.txt Patch for trunk, with changes done in task-controller.c > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-1186-1.txt, patch-1186-3-ydist.txt, > patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, > patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Status: Patch Available (was: Open) > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, 1185_v7.patch, > patch-1185-1-ydist.txt, patch-1185-2-ydist.txt, patch-1185-3-ydist.txt, > patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Attachment: 1185_v7.patch Patch for trunk > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, 1185_v7.patch, > patch-1185-1-ydist.txt, patch-1185-2-ydist.txt, patch-1185-3-ydist.txt, > patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Attachment: patch-1185-3-ydist.txt Y!20 patch incorporating review comments. > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-3-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Status: Open (was: Patch Available) > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1075) getQueue(String queue) in JobTracker would return NPE for invalid queue name
[ https://issues.apache.org/jira/browse/MAPREDUCE-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784634#action_12784634 ] Hemanth Yamijala commented on MAPREDUCE-1075: - Vinod, your approach will work if we can throw specific remote exceptions. For e.g. we could have a QueueNotFoundException and throw that remotely. This is supported by reasonable RPC implementations. I am not aware if this is supported in Hadoop. Is it ? > getQueue(String queue) in JobTracker would return NPE for invalid queue name > > > Key: MAPREDUCE-1075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1075 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: V.V.Chaitanya Krishna >Assignee: V.V.Chaitanya Krishna > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1075-1.patch, MAPREDUCE-1075-2.patch, > MAPREDUCE-1075-3.patch, MAPREDUCE-1075-4.patch, MAPREDUCE-1075-5.patch, > MAPREDUCE-1075-6.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1256) org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment (from TestFairScheduler) is failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V resolved MAPREDUCE-1256. -- Resolution: Duplicate bq. MAPREDUCE-1245? Yes, it is. > org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment (from > TestFairScheduler) is failing in trunk > -- > > Key: MAPREDUCE-1256 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1256 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > > Trunk build is failing. The unit testcase that fail is: > org.apache.hadoop.mapred.TestFairScheduler.testPoolAssignment (from > TestFairScheduler) > http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/160/testReport/org.apache.hadoop.mapred/TestFairScheduler/testPoolAssignment/ > Error Message > Timeout occurred. Please note the time in the report does not reflect the > time until the timeout. > Stacktrace > junit.framework.AssertionFailedError: Timeout occurred. Please note the time > in the report does not reflect the time until the timeout -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[
https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784632#action_12784632
]
Jothi Padmanabhan commented on MAPREDUCE-1185:
--
A couple of minor nits on v6:
# The comment before removing the {{MovedFileInfo}} has to be fixed
# The method {{getJobId}} is no longer required
> URL to JT webconsole for running job and job history should be the same
> ---
>
> Key: MAPREDUCE-1185
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Reporter: Sharad Agarwal
>Assignee: Sharad Agarwal
> Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch,
> 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt,
> patch-1185-2-ydist.txt, patch-1185-ydist.txt
>
>
> The tracking url for running jobs and the jobs which are retired is
> different. This creates problem for clients which caches the job running url
> because soon it becomes invalid when job is retired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1124) TestGridmixSubmission fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1124: - Status: Patch Available (was: Open) > TestGridmixSubmission fails sometimes > - > > Key: MAPREDUCE-1124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Chris Douglas > Fix For: 0.21.0 > > Attachments: M1124-0.patch, M1124-1.patch, M1124-2.patch, > M1124-3.patch, TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt > > > TestGridmixSubmission fails sometimes with following error : > Mismatched output bytes 4547848/4561267 > junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267 > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1124) TestGridmixSubmission fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1124: - Status: Open (was: Patch Available) > TestGridmixSubmission fails sometimes > - > > Key: MAPREDUCE-1124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Chris Douglas > Fix For: 0.21.0 > > Attachments: M1124-0.patch, M1124-1.patch, M1124-2.patch, > M1124-3.patch, TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt > > > TestGridmixSubmission fails sometimes with following error : > Mismatched output bytes 4547848/4561267 > junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267 > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1124) TestGridmixSubmission fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1124: - Attachment: M1124-3.patch * Refactored GridmixJob, et al. * TestGridmixSubmission passes reliably, as do new component tests. > TestGridmixSubmission fails sometimes > - > > Key: MAPREDUCE-1124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Chris Douglas > Fix For: 0.21.0 > > Attachments: M1124-0.patch, M1124-1.patch, M1124-2.patch, > M1124-3.patch, TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt > > > TestGridmixSubmission fails sometimes with following error : > Mismatched output bytes 4547848/4561267 > junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267 > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) > at > org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1253) Making Mumak work with Capacity-Scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784627#action_12784627
]
Vinod K V commented on MAPREDUCE-1253:
--
No idea how the Mumak test code is written, but various test-cases in
{{TestCapacityScheduler}} already replace the initialization thread with a
{{ControlledInitializationPoller}} for the sake of deterministic testing.
Please see {{testJobInitialization()}} if it helps.
> Making Mumak work with Capacity-Scheduler
> -
>
> Key: MAPREDUCE-1253
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1253
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
>Affects Versions: 0.21.0, 0.22.0
>Reporter: Anirban Dasgupta
>Assignee: Anirban Dasgupta
>Priority: Minor
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> In order to make the capacity-scheduler work in the mumak simulation
> environment, we have to replace the job-initialization threads of the
> capacity scheduler with classes that perform event-based initialization. We
> propose to use aspectj to disable the threads of the JobInitializationPoller
> class used by the Capacity Scheduler, and then perform the corresponding
> initialization tasks through a simulation job-initialization class that
> receives periodic wake-up calls from the simulator engine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1075) getQueue(String queue) in JobTracker would return NPE for invalid queue name
[
https://issues.apache.org/jira/browse/MAPREDUCE-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784622#action_12784622
]
Vinod K V commented on MAPREDUCE-1075:
--
Eventually, I think we should do ourselves away with these special
interpretations on the client-side and simply return the result on success and
throw exceptions in every failure possible. Not too strongly biased though to
do or not to do this for {{getQueue()}} API in this JIRA issue itself to deal
with the case of non-existent queue.
> getQueue(String queue) in JobTracker would return NPE for invalid queue name
>
>
> Key: MAPREDUCE-1075
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1075
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: V.V.Chaitanya Krishna
>Assignee: V.V.Chaitanya Krishna
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1075-1.patch, MAPREDUCE-1075-2.patch,
> MAPREDUCE-1075-3.patch, MAPREDUCE-1075-4.patch, MAPREDUCE-1075-5.patch,
> MAPREDUCE-1075-6.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784616#action_12784616 ] Hudson commented on MAPREDUCE-1229: --- Integrated in Hadoop-Mapreduce-trunk-Commit #139 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/139/]) . Allow customization of job submission policy in Mumak. Contributed by Hong Tang > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1176) Contribution: FixedLengthInputFormat and FixedLengthRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BitsOfInfo updated MAPREDUCE-1176:
--
Attachment: MAPREDUCE-1176-v3.patch
- Got rid of tabs, cleaned up formatting
- Added static setters for config
- throw exception if record length property is not set
- handle EOF -1 in loop
> Contribution: FixedLengthInputFormat and FixedLengthRecordReader
>
>
> Key: MAPREDUCE-1176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 0.20.1, 0.20.2
> Environment: Any
>Reporter: BitsOfInfo
>Priority: Minor
> Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch,
> MAPREDUCE-1176-v3.patch
>
>
> Hello,
> I would like to contribute the following two classes for incorporation into
> the mapreduce.lib.input package. These two classes can be used when you need
> to read data from files containing fixed length (fixed width) records. Such
> files have no CR/LF (or any combination thereof), no delimiters etc, but each
> record is a fixed length, and extra data is padded with spaces. The data is
> one gigantic line within a file.
> Provided are two classes first is the FixedLengthInputFormat and its
> corresponding FixedLengthRecordReader. When creating a job that specifies
> this input format, the job must have the
> "mapreduce.input.fixedlengthinputformat.record.length" property set as follows
> myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
> OR
> myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH,
> [myFixedRecordLength]);
> This input format overrides computeSplitSize() in order to ensure that
> InputSplits do not contain any partial records since with fixed records there
> is no way to determine where a record begins if that were to occur. Each
> InputSplit passed to the FixedLengthRecordReader will start at the beginning
> of a record, and the last byte in the InputSplit will be the last byte of a
> record. The override of computeSplitSize() delegates to FileInputFormat's
> compute method, and then adjusts the returned split size by doing the
> following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength)
> * fixedRecordLength)
> This suite of fixed length input format classes, does not support compressed
> files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1229: - Resolution: Fixed Fix Version/s: (was: 0.22.0) Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Hong! > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Status: Patch Available (was: Open)
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Attachment: 181-3.patch
Corrected patch.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Status: Open (was: Patch Available)
My bad. My last patch had a silly change that led to the test failures.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784585#action_12784585
]
Hadoop QA commented on MAPREDUCE-181:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12426590/181-3.patch
against trunk revision 885530.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 75 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/158/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/158/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/158/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/158/console
This message is automatically generated.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784541#action_12784541 ] Hadoop QA commented on MAPREDUCE-1229: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426574/mapreduce-1229-20091201.patch against trunk revision 885530. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/283/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/283/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/283/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/283/console This message is automatically generated. > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784504#action_12784504 ] Dick King commented on MAPREDUCE-1229: -- +1 I took a look at the revised patch [ mapreduce-1229-20091201.patch ] and I like it :-) . -dk > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Attachment: 181-3.patch
This patch fixes the findbugs warning and does some cleanup of the testcases.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Status: Patch Available (was: Open)
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch, 181-3.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Status: Open (was: Patch Available)
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-879) TestTaskTrackerLocalization fails on MAC OS
[
https://issues.apache.org/jira/browse/MAPREDUCE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784430#action_12784430
]
Steve Loughran commented on MAPREDUCE-879:
--
underlying cause is that -c isn't an option for stat on OS/X; instead the
arguments are taken as paths, which then fail
{code}
stat . -c %A:%U:%G
234881026 4327070 drwxr-xr-x 20 slo staff 0 680 "Dec 1 20:49:15 2009" "Dec 1
20:59:19 2009" "Dec 1 20:59:19 2009" "Aug 21 12:12:38 2009" 4096 0 0 .
stat: -c: stat: No such file or directory
stat: %A:%U:%G: stat: No such file or directory
{code}
> TestTaskTrackerLocalization fails on MAC OS
> ---
>
> Key: MAPREDUCE-879
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-879
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
> Environment: Mac OS X 10.5.7
>Reporter: Devaraj Das
>Assignee: Vinod K V
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments:
> TEST-org.apache.hadoop.mapred.TestTaskTrackerLocalization.txt
>
>
> TestTaskTrackerLocalization failed on an 'ant test' run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[
https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784391#action_12784391
]
Hong Tang commented on MAPREDUCE-1229:
--
Sample test output after the patch - as we can see, the jobs are submitted to
the cluster within the first 20ms.
{noformat}
Job job_200904211745_0002 is submitted at 1259697521583
1259697521583 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(20.0 <= 2.0*4635)
Job job_200904211745_0003 is submitted at 1259697521584
1259697521584 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(23.0 <= 2.0*4635)
Job job_200904211745_0004 is submitted at 1259697521585
1259697521585 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(154.0 <= 2.0*4635)
Job job_200904211745_0005 is submitted at 1259697521586
1259697521586 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(174.0 <= 2.0*4635)
Job job_200904211745_0006 is submitted at 1259697521587
1259697521587 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(209.0 <= 2.0*4635)
Job job_200904211745_0007 is submitted at 1259697521588
1259697521588 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(3413.0 <= 2.0*4635)
Job job_200904211745_0008 is submitted at 1259697521589
1259697521589 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(6617.0 <= 2.0*4635)
Job job_200904211745_0009 is submitted at 1259697521590
1259697521590 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(6718.0 <= 2.0*4635)
Job job_200904211745_0010 is submitted at 1259697521591
1259697521591 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(6719.0 <= 2.0*4635)
Job job_200904211745_0011 is submitted at 1259697521592
1259697521592 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7219.0 <= 2.0*4635)
Job job_200904211745_0012 is submitted at 1259697521593
1259697521593 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7220.0 <= 2.0*4635)
Job job_200904211745_0013 is submitted at 1259697521594
1259697521594 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7240.0 <= 2.0*4635)
Job job_200904211745_0015 is submitted at 1259697521595
1259697521595 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7241.0 <= 2.0*4635)
Job job_200904211745_0014 is submitted at 1259697521596
1259697521596 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7242.0 <= 2.0*4635)
Job job_200904211745_0016 is submitted at 1259697521597
1259697521597 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(7742.0 <= 2.0*4635)
Job job_200904211745_0018 is submitted at 1259697521598
1259697521598 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(8242.0 <= 2.0*4635)
Job job_200904211745_0019 is submitted at 1259697521599
1259697521599 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(8243.0 <= 2.0*4635)
Job job_200904211745_0017 is submitted at 1259697521600
1259697521600 Overloaded is false: incompleteMapTasks <= 2.0*mapSlotCapacity
(8244.0 <= 2.0*4635)
Job job_200904211745_0020 is submitted at 1259697521601
{noformat}
> [Mumak] Allow customization of job submission policy
>
>
> Key: MAPREDUCE-1229
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mumak
>Affects Versions: 0.21.0, 0.22.0
>Reporter: Hong Tang
>Assignee: Hong Tang
> Fix For: 0.21.0, 0.22.0
>
> Attachments: mapreduce-1229-20091121.patch,
> mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch,
> mapreduce-1229-20091201.patch
>
>
> Currently, mumak replay job submission faithfully. To make mumak useful for
> evaluation purposes, it would be great if we can support other job submission
> policies such as sequential job submission, or stress job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1229: - Status: Patch Available (was: Open) > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1229: - Attachment: mapreduce-1229-20091201.patch New patch incorporate the ideas outlined in my previous comments. > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch, > mapreduce-1229-20091201.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1229) [Mumak] Allow customization of job submission policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784382#action_12784382 ] Hong Tang commented on MAPREDUCE-1229: -- After sitting on it overnight, I think I can simplify isOverloaded() by eliminating the check of occupied map slot percentage and use mainly pending map task count, which seems to be updated by job tracker right after job submission. This would allow me to ramping up the load to the cluster at a rapid rate of one job per mili-second without worrying about overshooting. Will upload a patch shortly. > [Mumak] Allow customization of job submission policy > > > Key: MAPREDUCE-1229 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1229 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/mumak >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1229-20091121.patch, > mapreduce-1229-20091123.patch, mapreduce-1229-20091130.patch > > > Currently, mumak replay job submission faithfully. To make mumak useful for > evaluation purposes, it would be great if we can support other job submission > policies such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784278#action_12784278 ] Owen O'Malley commented on MAPREDUCE-1252: -- To be clearer: 1. The previous patch fixes the warnings, but not the problem. Thus, it is necessary but not sufficient. 2. On a one node cluster with 0.21.0-dev from yesterday: a. Have mapred.map.tasks set to 12 in hadoop-site.xml. b. Submit a word count example on a two file input directory. 3. The 2 maps run fine. 4. The 2 reduces both lock up at 5.5% with a status of 2 of 12 maps fetched. Looking at the system directory the job.xml has mapreduce.job.maps = 2, but the task and the web ui show it as 12. If I change the config to use mapreduce.job.maps to set the default, everything works correctly. > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784271#action_12784271 ] Hadoop QA commented on MAPREDUCE-1185: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426531/1185_v6.patch against trunk revision 885530. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/157/console This message is automatically generated. > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-1252: - Status: Open (was: Patch Available) > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784215#action_12784215
]
Hadoop QA commented on MAPREDUCE-181:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12426518/181-2.patch
against trunk revision 885530.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 78 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/282/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/282/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/282/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/282/console
This message is automatically generated.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784207#action_12784207 ] Hadoop QA commented on MAPREDUCE-698: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426456/mapreduce-698-trunk-3.patch against trunk revision 885530. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/console This message is automatically generated. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784187#action_12784187 ] Amareshwari Sriramadasu commented on MAPREDUCE-1185: Uploaded the patch for trunk with Jothi's comment incorporated. > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Status: Patch Available (was: Open) > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Attachment: 1185_v6.patch > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, 1185_v6.patch, patch-1185-1-ydist.txt, > patch-1185-2-ydist.txt, patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784173#action_12784173 ] Iyappan Srinivasan commented on MAPREDUCE-1185: --- Testing completed with new patch. All testcases pased: +1 for QA. Scenarios tested: Change the mapred.job.tracker.history.completed.location to point to a local file system directory 1) Run some jobs like sleep, randomwriter, streaming job and see if their job URI gets redirected to the done folder after the jobs are retired. 2) Run multiple jobs and see if still the jobs are getting redirected properly. 3) Do a JT restart and then type the jobid URL (not the done folder URL) of the job before the JT restart and see if it gives proper error. 4) Change the mapred.job.tracker.history.completed.location to point to a dfs directory and then run steps 1,2,3 and see if it works properly. > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, patch-1185-1-ydist.txt, patch-1185-2-ydist.txt, > patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1185) URL to JT webconsole for running job and job history should be the same
[ https://issues.apache.org/jira/browse/MAPREDUCE-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1185: --- Attachment: patch-1185-2-ydist.txt Y!20 Patch incorporating Jothi's comment. Also fixed a bug in jobdetails.jsp > URL to JT webconsole for running job and job history should be the same > --- > > Key: MAPREDUCE-1185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1185 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: 1185_v1.patch, 1185_v2.patch, 1185_v3.patch, > 1185_v4.patch, 1185_v5.patch, patch-1185-1-ydist.txt, patch-1185-2-ydist.txt, > patch-1185-ydist.txt > > > The tracking url for running jobs and the jobs which are retired is > different. This creates problem for clients which caches the job running url > because soon it becomes invalid when job is retired. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-353) Allow shuffle read and connection timeouts to be configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784151#action_12784151
]
Amareshwari Sriramadasu commented on MAPREDUCE-353:
---
test-patch for Y!20 patch :
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
{noformat}
Ttested manually by setting small values for these timeouts and saw getting the
socketTimeoutException as in
https://issues.apache.org/jira/browse/MAPREDUCE-353?focusedCommentId=12730697&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12730697
All unit tests passed except TestHdfsProxy
> Allow shuffle read and connection timeouts to be configurable
> -
>
> Key: MAPREDUCE-353
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-353
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.21.0
>Reporter: Arun C Murthy
>Assignee: Ravi Gummadi
> Fix For: 0.21.0
>
> Attachments: MR-353.patch, MR-353.v1.patch, patch-353-ydist.txt
>
>
> It would be good for latency-sensitive applications to tune the shuffle
> read/connection timeouts... in fact this made a huge difference to terasort
> since we were seeing individual shuffles stuck for upwards of 60s and had to
> have a very small read timeout.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Assignee: Kevin Peterson Release Note: Per-pool map and reduce caps for Fair Scheduler. Status: Patch Available (was: Open) > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1067) Default state of queues is undefined when unspecified
[ https://issues.apache.org/jira/browse/MAPREDUCE-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V.V.Chaitanya Krishna updated MAPREDUCE-1067: - Attachment: MAPREDUCE-1067-1.patch Had an offline discussion with Rahul and Vinod and came up with the following conclusions: * The default state of container queue is to be RUNNING. * Also, state of a container queue should not be allowed to change once the hierarchy is built. * The state UNDEFINED is to be removed and only two states (RUNNING and STOPPED) are maintained. Uploading patch with the above implemented. > Default state of queues is undefined when unspecified > - > > Key: MAPREDUCE-1067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1067 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: V.V.Chaitanya Krishna >Assignee: V.V.Chaitanya Krishna >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1067-1.patch > > > Currently, if the state of a queue is not specified, it is being set to > "undefined" state instead of running state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1257) Ability to grab the number of spills
Ability to grab the number of spills Key: MAPREDUCE-1257 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1257 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Sriranjan Manjunath The counters should have information about the number of spills in addition to the number of spill records. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Fix Version/s: 0.22.0
Status: Patch Available (was: Open)
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated MAPREDUCE-181:
--
Attachment: 181-2.patch
Quite close i think.
> Secure job submission
> --
>
> Key: MAPREDUCE-181
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Amar Kamat
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 181-1.patch, 181-2.patch,
> hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
> HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
> MAPRED-181-v3.8.patch
>
>
> Currently the jobclient accesses the {{mapred.system.dir}} to add job
> details. Hence the {{mapred.system.dir}} has the permissions of
> {{rwx-wx-wx}}. This could be a security loophole where the job files might
> get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1249) mapreduce.reduce.shuffle.read.timeout's default value should be 3 minutes, in mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784118#action_12784118 ] Amareshwari Sriramadasu commented on MAPREDUCE-1249: bq. -1 contrib tests Is due to MAPREDUCE-1124 and MAPREDUCE-1245 bq. -1 tests included. The change is in mapred-default.xml, Ran a job manually and verified the value is correctly picked up. > mapreduce.reduce.shuffle.read.timeout's default value should be 3 minutes, in > mapred-default.xml > > > Key: MAPREDUCE-1249 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1249 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-1249-1.txt, patch-1249.txt > > > mapreduce.reduce.shuffle.read.timeout has a value of 30,000 (30 seconds) in > mapred-default.xml, whereas the default value in Fetcher code is 3 minutes. > It should be 3 minutes by default, as it was in pre MAPREDUCE-353. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1226) Granularity Variable Task Pre-Scheduler in Heterogeneous Environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaoning Zhang updated MAPREDUCE-1226: -- Component/s: tasktracker task Priority: Minor (was: Major) > Granularity Variable Task Pre-Scheduler in Heterogeneous Environment > - > > Key: MAPREDUCE-1226 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1226 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker, task, tasktracker > Environment: Heterogeneous Cluster >Reporter: Zhaoning Zhang >Priority: Minor > > As we deploy the LATE scheduler of the OSDI08 paper, upon some of our cluster > enviroments, some slow nodes may be assigned tasks that every time run slowly > and be re-executed then killed, so we found these nodes are always with no > use and waste the assigned task slots. > In the LATE mechanism, we re-execute some of the tasks, so these tasks run on > different node twice or more, then this cause some waste of the calculating > resources. > Easily, we can remove these node out of the cluster or split the cluster into > two or more. But I think it's useful and significant to design a mechanism to > help low utility nodes to be effect. > > We want to pre-schedule the tasks with the utility based on node historical > logs, then assign larger size tasks to the fast nodes. In Hadoop task > scheduler, we assign the map task in default splits of 64M. Some may split it > into 128M. But, most of them are of the same granularity. So I want to alter > this mechanism to a granularity variable one. > As we know, the Map task granularity depends on the DFS file size, while the > Reduce task's depends on the Partitioner to split the intermediate results. > So I think this is feasible to get the granularity variable mechanism. > If we use the pre-schedule model, then we can expect all the tasks can start > at a nearly same time and finish at a nearly same time, and the job can fill > a specific time slot. > History-Log-Based nodes Utility description > This is the fundamental description of nodes for the pre-scheduler. And in > the heterogeneous environment, the cluster can be split into different > sub-cluster, and within the sub-cluster the nodes are homogeneous and between > the sub-cluster the nodes are heterogeneous. > Nodes Utility Stability > We think this is important for the pre-scheduler depends on the stability of > the nodes. And we could pick the bad stability nodes up and treat them > differently, but we haven't have good method to handle this. > Error tolerant > I think the original scheduler in the homogeneous cluster is designed to > handle the error nodes, if some nodes get exceptions, the JobTracker > re-execute them, and handle these exceptions dynamically. > So if we use the pre-scheduler, we must face the problem of the exceptions. > I propose that if some tasks got exceptions, we split the task into more than > one part and execute them on more than one different nodes, then the expected > finish time will be shorten, and the total job response time will not be too > long. > Job Priorities > If we use this pre-scheduler, single job will fill the time slot, and if then > will be some other high-priority jobs, they will wait. And I don't get effect > methods to solve this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784114#action_12784114 ] Owen O'Malley commented on MAPREDUCE-1252: -- This patch is necessary to suppress the warnings, but doesn't fix the problem. The problem is that I was using a configuration with the old attribute name. The attribute name isn't being translated during the xml loading. I'm not sure how it is happening, but in some context, we get 2 and others 12. When I change the config to use the new attribute name, it all goes away. I suspect this is caused by a bug in HADOOP-6105. > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
