[jira] Commented: (MAPREDUCE-653) distcp can support bandwidth limiting
[ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842218#action_12842218 ] eric baldeschwieler commented on MAPREDUCE-653: --- So how will this behave if distcp gets 100 slots and is limited to 100mb? Will you use 100 slots badly? Typically we've handled the need for throttling by controlling the bandwidth / client and the number of clients. This lets you reason about the utilization of the clients & bandwidth. This would imply a fixed number of maps that pull files to copy off a queue maintained by the distcp client. You'd need to handle recovery carefully, but that could probably be done at the distcp client (the slave signals done, the client does a final move and updates the queue). > distcp can support bandwidth limiting > - > > Key: MAPREDUCE-653 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-653 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch > > > distcp should support an option for user to specify the bandwidth limit for > the distcp job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1568) TrackerDistributedCacheManager should do deleteLocalPath asynchronously
TrackerDistributedCacheManager should do deleteLocalPath asynchronously --- Key: MAPREDUCE-1568 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 TrackerDistributedCacheManager.deleteCache() has been improved: MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the main thread and then delete them in the background MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of directories can also takes a long time) But the deleteLocalCache is still in the main thread of TaskRunner.run(). So it will still slow down the task which triggers the deletion (originally this will blocks all tasks, but it is fixed by MAPREDUCE-1098). Other tasks do not wait for the deletion. The task which triggers the deletion should not wait for this either. TrackerDistributedCacheManager should do deleteLocalPath() asynchronously. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1567) Sharing Credentials between JobConfs leads to unintentional sharing of credentials
Sharing Credentials between JobConfs leads to unintentional sharing of credentials -- Key: MAPREDUCE-1567 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1567 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.22.0 Currently, if code does new JobConf(jobConf), it will share the Credentials. That leads to unintentional sharing. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1566) Need to add a mechanism to import tokens and secrets into a submitted job.
Need to add a mechanism to import tokens and secrets into a submitted job. -- Key: MAPREDUCE-1566 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1566 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.22.0 We need to include tokens and secrets into a submitted job. I propose adding a configuration attribute that when pointed at a token storage file will include the tokens and secrets from that token storage file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1558) specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842090#action_12842090 ] Owen O'Malley commented on MAPREDUCE-1558: -- There is no need to create the JobConf. Just do the get out of the conf. > specify correct server principal for RefreshAuthorizationPolicyProtocol and > RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612) > - > > Key: MAPREDUCE-1558 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1558 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1558.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842078#action_12842078
]
Chris Douglas commented on MAPREDUCE-1466:
--
Talked with Luke. Since {{INPUT_NUM_FILES}} is shared between both
implementations of {{FileInputFormat}}, most config keys were moved to
JobContext in MAPREDUCE-849 anyway, and config keys are undocumented in our
current regime: the current patch is correct.
+1
> FileInputFormat should save #input-files in JobConf
> ---
>
> Key: MAPREDUCE-1466
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 0.22.0
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1466_yhadoop20-1.patch,
> MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20-3.patch,
> MAPREDUCE-1466_yhadoop20.patch, mr-1466-trunk-v1.patch
>
>
> We already track the amount of data consumed by MR applications
> (MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the
> client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy
> to stick in the JobConf during job-submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842076#action_12842076
]
Hudson commented on MAPREDUCE-1522:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #265 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/265/])
. FileInputFormat may use the default FileSystem for the
input path. Contributed by Tsz Wo (Nicholas), SZE
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1428) Make block size and the size of archive created files configurable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1428: -- Attachment: BinaryFileGenerator.java BinaryFileGenerator.java: added a verifier. > Make block size and the size of archive created files configurable. > --- > > Key: MAPREDUCE-1428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1428 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: BinaryFileGenerator.java, BinaryFileGenerator.java, > BinaryFileGenerator.java > > > Currently the block size used by archives is the default block size of the > hdfs filesystem. We need to make it configurable so that the block size can > be higher for the part files that archives create. > Also, we need to make the size of part files in archives configurable again > to make it bigger in size and create less number of such files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Nicholas!
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842060#action_12842060
]
Chris Douglas commented on MAPREDUCE-1522:
--
All test failures are due to Hudson's recent abject uselessness:
{noformat}
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
{noformat}
The former test results hold, as:
# The original run failed only {{TestMRLocalFS}} due to MAPREDUCE-1520
# The only changes are in the unit test, which passes
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841957#action_12841957
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1522:
---
+1 The changes look good.
Thanks, Chris.
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1565) Need human-readable counter groups for framework job-counters and task-counters
[ https://issues.apache.org/jira/browse/MAPREDUCE-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1565: - Description: Currently the 'group' is just the class-names of the counters, which is very unfortunate - we'll probably need to have both for a while to maintain compatibility, sigh. > Need human-readable counter groups for framework job-counters and > task-counters > --- > > Key: MAPREDUCE-1565 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1565 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker, task >Reporter: Arun C Murthy > Fix For: 0.22.0 > > > Currently the 'group' is just the class-names of the counters, which is very > unfortunate - we'll probably need to have both for a while to maintain > compatibility, sigh. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1565) Need human-readable counter groups for framework job-counters and task-counters
Need human-readable counter groups for framework job-counters and task-counters --- Key: MAPREDUCE-1565 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1565 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker, task Reporter: Arun C Murthy Fix For: 0.22.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1564) Document the framework counters
Document the framework counters --- Key: MAPREDUCE-1564 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1564 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Arun C Murthy Fix For: 0.22.0 We need to document the framework counters in the mapred-tutorial. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841879#action_12841879 ] Hadoop QA commented on MAPREDUCE-1415: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437990/patch-1415-1.txt against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/503/console This message is automatically generated. > With streaming jobs and LinuxTaskController, the localized streaming binary > has 571 permissions instead of 570 > -- > > Key: MAPREDUCE-1415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming, security >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1415-1.txt, patch-1415.txt > > > After MAPREDUCE-856, all localized files are expected to have **0 permissions > for the sake of security. > This was found by Karam while testing LinuxTaskController functionality after > MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1533) reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1533: -- Attachment: mapreduce-1533-v1.4.patch Attaching a patch for review. Instead of doing an in-line String.format(), a wrapper object is passed with all the necessary values to construct the scheduling-info string. The scheduling info is created lazily when toString() method is invoked on the wrapper object. > reduce or remove usage of String.format() usage in > CapacityTaskScheduler.updateQSIObjects > - > > Key: MAPREDUCE-1533 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1533 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Rajesh Balamohan >Assignee: Amar Kamat > Attachments: mapreduce-1533-v1.4.patch > > > When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT > executes heartBeat() method heavily. This internally makes a call to > CapacityTaskScheduler.updateQSIObjects(). > CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() > for setting the job scheduling information. Based on the datastructure size > of "jobQueuesManager" and "queueInfoMap", the number of times String.format() > gets executed becomes very high. String.format() internally does pattern > matching which turns to be out very heavy (This was revealed while profiling > JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of > which String.format() took 46%. > Would it be possible to do String.format() only at the time of invoking > JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while > processing heartbeats. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841846#action_12841846
]
Hadoop QA commented on MAPREDUCE-1522:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437996/M1522-1.patch
against trunk revision 919335.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/343/console
This message is automatically generated.
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841839#action_12841839 ] Hadoop QA commented on MAPREDUCE-890: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console This message is automatically generated. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1550) UGI.doAs should not be used for getting the history file of jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V resolved MAPREDUCE-1550. -- Resolution: Invalid This bug is there not on trunk but on the ydist branch for which patch was attached at MAPREDUCE-1455. Closing this as invalid. > UGI.doAs should not be used for getting the history file of jobs > > > Key: MAPREDUCE-1550 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1550 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das > Fix For: 0.22.0 > > Attachments: 1550-1.patch, 1550-2.1.patch, 1550-2.patch > > > When the jobtracker tries to open a job history file it does a doAs to get > the filesystem for the user (that had submitted the job). This should not be > done since the job history files are owned by the jobtracker. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1.patch
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Status: Patch Available (was: Open)
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1v20.patch
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Attachment: M1522-1v21.patch
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: M1522-1.patch, M1522-1v20.patch, M1522-1v21.patch,
> m1522_20100224.patch, m1522_20100224_0.20.patch, m1522_20100301.patch,
> m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1522) FileInputFormat may change the file system of an input path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1522:
-
Status: Open (was: Patch Available)
* Setting {{fs.default.name}} prior to creating the {{Job}} should make the
mockito part of the test unnecessary, avoiding pushing the ivy changes through
0.21 and 0.20.
* Consider JUnit4 annotations instead of extending {{TestCase}}
> FileInputFormat may change the file system of an input path
> ---
>
> Key: MAPREDUCE-1522
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1522
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Blocker
> Fix For: 0.20.3
>
> Attachments: m1522_20100224.patch, m1522_20100224_0.20.patch,
> m1522_20100301.patch, m1522_20100301_0.20.patch, m1522_20100301_0.21.patch,
> m1552_20100223_0.20.patch
>
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job,
> Path path) uses the default FileSystem but not the FileSystem specified in
> the path.
> {code}
> //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> public static void addInputPath(Job job,
> Path path) throws IOException {
> Configuration conf = job.getConfiguration();
> FileSystem fs = FileSystem.get(conf);
> path = path.makeQualified(fs); // the original FileSystem is lost.
> ...
> }
> {code}
> There is a similar problem in FileInputFormat.setInputPaths(..).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1493) Authorization for job-history pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841807#action_12841807 ] Hadoop QA commented on MAPREDUCE-1493: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437853/MAPREDUCE-1493-20100304.txt against trunk revision 919277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/502/console This message is automatically generated. > Authorization for job-history pages > --- > > Key: MAPREDUCE-1493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1493 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security >Reporter: Vinod K V >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1493-20100222.1.txt, > MAPREDUCE-1493-20100225.2.txt, MAPREDUCE-1493-20100226.1.txt, > MAPREDUCE-1493-20100227.2-ydist.txt, MAPREDUCE-1493-20100227.3-ydist.txt, > MAPREDUCE-1493-20100301.1.txt, MAPREDUCE-1493-20100304.txt > > > MAPREDUCE-1455 introduces authorization for most of the Map/Reduce jsp pages > and servlets, but left history pages. This JIRA will make sure that > authorization checks are made while accessing job-history pages also. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Attachment: patch-1415-1.txt Patch updated to trunk. > With streaming jobs and LinuxTaskController, the localized streaming binary > has 571 permissions instead of 570 > -- > > Key: MAPREDUCE-1415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming, security >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1415-1.txt, patch-1415.txt > > > After MAPREDUCE-856, all localized files are expected to have **0 permissions > for the sake of security. > This was found by Karam while testing LinuxTaskController functionality after > MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1415) With streaming jobs and LinuxTaskController, the localized streaming binary has 571 permissions instead of 570
[ https://issues.apache.org/jira/browse/MAPREDUCE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1415: --- Status: Patch Available (was: Open) > With streaming jobs and LinuxTaskController, the localized streaming binary > has 571 permissions instead of 570 > -- > > Key: MAPREDUCE-1415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1415 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming, security >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1415-1.txt, patch-1415.txt > > > After MAPREDUCE-856, all localized files are expected to have **0 permissions > for the sake of security. > This was found by Karam while testing LinuxTaskController functionality after > MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1466:
-
Status: Open (was: Patch Available)
* Please retain javadoc for {{INPUT_NUM_FILES}}
* Since {{INPUT_NUM_FILES}} will only be set for {{FileInputFormat}}, putting
the constant there- instead of {{JobContext}}- is probably correct.
> FileInputFormat should save #input-files in JobConf
> ---
>
> Key: MAPREDUCE-1466
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 0.22.0
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1466_yhadoop20-1.patch,
> MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20-3.patch,
> MAPREDUCE-1466_yhadoop20.patch, mr-1466-trunk-v1.patch
>
>
> We already track the amount of data consumed by MR applications
> (MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the
> client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy
> to stick in the JobConf during job-submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1403:
-
Status: Open (was: Patch Available)
* The patch includes a whitespace change to {{Job}}
* Can you explain the addition of {{getConfiguration}} to the {{RunningJob}}
interface? Is the relevant copy in {{JobContextImpl}}?
* Please retain javadoc for {{CACHE_FILES_SIZES}} and {{CACHE_ARCHIVES_SIZES}},
rather than code comments
* javadoc referring to classes (as
{{TrackerDistributedCacheManager::getFileStatus}}) should probably use
{...@link org.class.name}}} instead of {{name}}
* The actual/expected in the error message for {{assertEquals}} is redundant
> Save file-sizes of each of the artifacts in DistributedCache in the JobConf
> ---
>
> Key: MAPREDUCE-1403
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 0.22.0
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1403_yhadoop20-1.patch,
> MAPREDUCE-1403_yhadoop20-2.patch, MAPREDUCE-1403_yhadoop20.patch,
> MR-1403-trunk-1.patch
>
>
> It would be a useful metric to collect... potentially GridMix could use it to
> emulate jobs which use the DistributedCache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-653) distcp can support bandwidth limiting
[ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841778#action_12841778 ] Ravi Gummadi commented on MAPREDUCE-653: We need to experiment and check if this way of asking JobTracker for counters from each map task will be a considerable overhead on JobTracker(and would this make JobTracker heavily loaded serving these many requests, with lot more new connections to JobTracker). > distcp can support bandwidth limiting > - > > Key: MAPREDUCE-653 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-653 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch > > > distcp should support an option for user to specify the bandwidth limit for > the distcp job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1518) On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841773#action_12841773 ] Hudson commented on MAPREDUCE-1518: --- Integrated in Hadoop-Mapreduce-trunk-Commit #264 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/264/]) . RaidNode does not run the deletion check on the directory that stores the parity files. (Rodrigo Schmidt via dhruba) > On contrib/raid, the RaidNode currently runs the deletion check for parity > files on directories too. It would be better if it didn't. > - > > Key: MAPREDUCE-1518 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1518 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid > Environment: On contrib/raid, the RaidNode currently runs the > deletion check for parity files on directories too. It runs okay because the > directory is not empty and trying to delete it non-recursively fails, but > such failure messages only polute the log file. > My proposal is the following: > If recursePurge is checking a directory, it should call itself recursively. > If it's checking a file, it should do the deletion check. >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1518.0.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1421) LinuxTaskController tests failing on trunk after the commit of MAPREDUCE-1385
[ https://issues.apache.org/jira/browse/MAPREDUCE-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1421: --- Attachment: patch-1421-1-ydist.txt Patch for Yahoo! distribution. One file was missed in earlier patch. > LinuxTaskController tests failing on trunk after the commit of MAPREDUCE-1385 > - > > Key: MAPREDUCE-1421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker, test >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1421-1-ydist.txt, patch-1421-1.txt, > patch-1421-2.txt, patch-1421-ydist.txt, patch-1421.txt, > TestJobExecutionAsDifferentUser.patch > > > The following tests fail, in particular: > - TestDebugScriptWithLinuxTaskController > - TestJobExecutionAsDifferentUser > - TestPipesAsDifferentUser > - TestKillSubProcessesWithLinuxTaskController -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841766#action_12841766 ] Amar Kamat commented on MAPREDUCE-1422: --- Hemanth, I am confused w.r.t point #3 and #4. I added test-code in TestTaskTrackerLocalization for testing job-work-dir cleanup because there is a call to _tracker.removeJobFiles()_. Now TestLocalizationWithLinuxTaskController (that extends TestTaskTrackerLocalization) will also test the job-work-dir cleanup code with LinuxTaskController. As you are suggesting, if we move this job cleanup test-code into a separate testcase, then we need TestJobLocalDirCleanupWithLinuxTaskController, no? Another way to do this would be to introduce a new testcase (say TestJobDirCleanup) as you suggested which will test the job-dir and task-dirs upon job completion. Also add a testcase (say TestJobDirCleanupWithLinuxTaskController) extending TestJobDirCleanup but configured with LinuxTaskController. Thoughts? > Changing permissions of files/dirs under job-work-dir may be needed sothat > cleaning up of job-dir in all mapred-local-directories succeeds always > - > > Key: MAPREDUCE-1422 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Ravi Gummadi >Assignee: Amar Kamat > Attachments: mapreduce-1422-test-v1.0.patch, > mapreduce-1422-v1.4.2.patch > > > After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller > binary is launched for changing permissions of taskAttemptDir and taskWorkDir > before cleaning up of these directories sothat cleanup will be succeeded even > if user had created files/dirs under taskAttemptDir or taskWorkDir with > non-writable permissions. Users can't create files/dirs under job-dir > directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions > and user can create files/dirs under job-work-dir with non-writable > permissions, Changing permissions of files/dirs under job-work-dir may be > needed sothat cleaning up of job-dir in all mapred-local-directories succeeds > always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Status: Patch Available (was: Open) Allowing it to go through Hudson > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Attachment: MR890.v1.2.patch Syncing with the latest trunk(as MAPREDUCE-1435 went in). > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1527) QueueManager should issue warning if mapred-queues.xml is skipped.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841761#action_12841761 ] Hudson commented on MAPREDUCE-1527: --- Integrated in Hadoop-Mapreduce-trunk-Commit #263 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/263/]) . Better warning logged when mapred.queue.names is overshadowed by mapred-queues.xml. Contributed by Hong Tang. > QueueManager should issue warning if mapred-queues.xml is skipped. > -- > > Key: MAPREDUCE-1527 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1527 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.22.0 > > Attachments: mr-1527-20100224-2.patch, mr-1527-20100224.patch > > > MAPREDUCE-861 added support for hierarchical queues and used a new file > mapred-queues.xml to configure queues. But the QueueManager will silently > skip the file if property "mapred.queue.names" is defined in Configuration. > This tripped us when we are testing MAPREDUCE-1235 and copied configuration > files from a Hadoop 0.20 cluster. > I suggest QueueManager issue a friendly warning if both of the following are > true: (1) "mapred.queue.names" exists in Configuration; (2) > "mapred-queues.xml" is found in the classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1562) TestBadRecords fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841760#action_12841760 ] Amareshwari Sriramadasu commented on MAPREDUCE-1562: bq. I think we should fail any test immediately if MiniMRCluster fails to start even a single TT. MAPREDUCE-1366 is open for this. > TestBadRecords fails sometimes > -- > > Key: MAPREDUCE-1562 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1562 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Vinod K V > Fix For: 0.22.0 > > Attachments: consoleText > > > TestBadRecords.testMapRed fails sometimes. One instance of this was seen by > Hudson while testing MAPREDUCE-890: > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-890: --- Status: Open (was: Patch Available) > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1455: Attachment: 1455.20S.2.fix1.patch Patch fixing testcase for earlier version of hadoop. Not for commit here. > Authorization for servlets > -- > > Key: MAPREDUCE-1455 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security, tasktracker >Reporter: Devaraj Das >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1455.20S.2.fix.patch, 1455.20S.2.fix1.patch, > 1455.20S.2.patch, 1455.patch, 1455.v1.patch, 1455.v2.patch, 1455.v3.patch, > 1455.v4.1.patch, 1455.v4.2.patch, 1455.v4.patch > > > This jira is about building the authorization for servlets (on top of > MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization > checks on web requests based on the configured job permissions. For e.g., if > the job permission is 600, then no one except the authenticated user can look > at the job details via the browser. The authenticated user in the servlet can > be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1518) On contrib/raid, the RaidNode currently runs the deletion check for parity files on directories too. It would be better if it didn't.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-1518: Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Rodrigo! > On contrib/raid, the RaidNode currently runs the deletion check for parity > files on directories too. It would be better if it didn't. > - > > Key: MAPREDUCE-1518 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1518 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid > Environment: On contrib/raid, the RaidNode currently runs the > deletion check for parity files on directories too. It runs okay because the > directory is not empty and trying to delete it non-recursively fails, but > such failure messages only polute the log file. > My proposal is the following: > If recursePurge is checking a directory, it should call itself recursively. > If it's checking a file, it should do the deletion check. >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1518.0.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1560) Better diagnostic message for tasks killed for going over vmem limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841757#action_12841757 ] Arun C Murthy commented on MAPREDUCE-1560: -- Thanks Vinod. > Better diagnostic message for tasks killed for going over vmem limit > > > Key: MAPREDUCE-1560 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1560 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.22.0 > > > Currently the user has no indication of his tasks getting killed due to vmem > limit, the only way to know is by looking at TT logs. We should get the TT to > insert a diagnostic string for the task to indicate this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1560) Better diagnostic message for tasks killed for going over vmem limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V resolved MAPREDUCE-1560. -- Resolution: Invalid We already have this feature. It is just that because of a bug in the diagnostics framework, we don't always record the information. Head to MAPREDUCE-1563 for the bug in question. Resolving this as invalid. > Better diagnostic message for tasks killed for going over vmem limit > > > Key: MAPREDUCE-1560 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1560 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.22.0 > > > Currently the user has no indication of his tasks getting killed due to vmem > limit, the only way to know is by looking at TT logs. We should get the TT to > insert a diagnostic string for the task to indicate this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1563) Task diagnostic info would get missed sometimes.
Task diagnostic info would get missed sometimes. Key: MAPREDUCE-1563 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1563 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Amareshwari Sriramadasu Fix For: 0.22.0 TaskStatus's diagnostic info is reset in TaskTracker.cloneAndResetRunningTaskStatuses() method while constructing TaskTrackerStatus to send the heartbeat. It is also reset after receiving the response in TaskTracker.transmitHeartbeat() method. If any task sets diagnostic info between both the clearStatus() calls, the info is lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1527) QueueManager should issue warning if mapred-queues.xml is skipped.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1527: - Resolution: Fixed Fix Version/s: 0.22.0 Status: Resolved (was: Patch Available) The test failure is unrelated, this is a trivial change to a LOG.warn. I just committed this. Thanks Hong! > QueueManager should issue warning if mapred-queues.xml is skipped. > -- > > Key: MAPREDUCE-1527 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1527 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.22.0 > > Attachments: mr-1527-20100224-2.patch, mr-1527-20100224.patch > > > MAPREDUCE-861 added support for hierarchical queues and used a new file > mapred-queues.xml to configure queues. But the QueueManager will silently > skip the file if property "mapred.queue.names" is defined in Configuration. > This tripped us when we are testing MAPREDUCE-1235 and copied configuration > files from a Hadoop 0.20 cluster. > I suggest QueueManager issue a friendly warning if both of the following are > true: (1) "mapred.queue.names" exists in Configuration; (2) > "mapred-queues.xml" is found in the classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841747#action_12841747 ] Vinod K V commented on MAPREDUCE-890: - The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why? Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1562) TestBadRecords fails sometimes
[
https://issues.apache.org/jira/browse/MAPREDUCE-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841746#action_12841746
]
Vinod K V commented on MAPREDUCE-1562:
--
Two things I immediately can identify:
- One TT crashed because of jetty port problem, possibly related to HADOOP-6528
{code}
[exec] [junit] 2010-03-04 09:40:24,493 ERROR mapred.TaskTracker
(TaskTracker.java:offerService(1460)) - Caught exception: java.io.IOException:
Jetty problem. Jetty didn't bind to a valid port
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.checkJettyPort(TaskTracker.java:1276)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1438)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2256)
[exec] [junit] at
org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:228)
[exec] [junit] at java.lang.Thread.run(Thread.java:619)
{code}
I think we should fail *any* test immediately if MiniMRCluster fails to start
even a single TT.
- Two, I see this 15 min wait (!!)
{code}
else if(MAPPER_BAD_RECORDS.get(2).equals(str)) {
try {
LOG.warn("MAP Encountered BAD record");
Thread.sleep(15*60*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
{code}
Not sure why this is not causing the test to fail all the time.
> TestBadRecords fails sometimes
> --
>
> Key: MAPREDUCE-1562
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1562
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Vinod K V
> Fix For: 0.22.0
>
> Attachments: consoleText
>
>
> TestBadRecords.testMapRed fails sometimes. One instance of this was seen by
> Hudson while testing MAPREDUCE-890:
> http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1562) TestBadRecords fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1562: - Attachment: consoleText Attaching Hudson's console-output for the test from MAPREDUCE-890. > TestBadRecords fails sometimes > -- > > Key: MAPREDUCE-1562 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1562 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Vinod K V > Fix For: 0.22.0 > > Attachments: consoleText > > > TestBadRecords.testMapRed fails sometimes. One instance of this was seen by > Hudson while testing MAPREDUCE-890: > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1542) Deprecate mapred.permissions.supergroup in favor of hadoop.cluster.administrators
[
https://issues.apache.org/jira/browse/MAPREDUCE-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-1542:
Attachment: 1542.patch
Attaching patch that makes mapreduce.cluster.permissions.supergroup deprecated
and makes JobTracker ans TaskTracker use hadoop.cluster.administrators.
Please review and provide your comments.
> Deprecate mapred.permissions.supergroup in favor of
> hadoop.cluster.administrators
> -
>
> Key: MAPREDUCE-1542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Reporter: Vinod K V
>Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: 1542.patch
>
>
> HADOOP-6568 added the configuration {{hadoop.cluster.administrators}} through
> which admins can configure who the superusers/supergroups for the cluster
> are. MAPREDUCE itself already has {{mapred.permissions.supergroup}} (which is
> just a single group). As agreed upon at HADOOP-6568, this should be
> deprecated in favor of {{hadoop.cluster.administrators}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
