[jira] Updated: (MAPREDUCE-1418) LinuxTaskController binary misses validation of arguments passed for relative components in some cases.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala updated MAPREDUCE-1418:
Attachment: MAPREDUCE-1418.patch
Attaching a patch implementing the changes.
Details:
- Fixed all the initialization calls (for user, job, task, log and distributed
cache files) to call check_path_for_relative_components.
- Modified the way in which error code was tracked and handled in the methods
so I could return a relevant error code when relative components are found.
- Added tests for all the conditions. For testing initialization of job, task
and distributed cache, the approach followed is that we create the original
directories and some dummy directories. Then we launch the task controller exe
with some of the arguments tweaked to have relative components. We verify the
call fails with an ExitCodeException with the right error code. For user and
log initialization, I write a taskcontroller.cfg with relative paths in mapred
local directories and hadoop.log.dir and verify the same failures. The latter
approach is needed because passing relative paths in the command line arguments
will not trigger the code path we want to exercise in these two cases.
All task-controller tests are passing with this patch, as also test-patch.
LinuxTaskController binary misses validation of arguments passed for relative
components in some cases.
---
Key: MAPREDUCE-1418
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1418
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security, tasktracker
Reporter: Vinod K V
Assignee: Hemanth Yamijala
Attachments: MAPREDUCE-1418.patch
The function {{int check_path_for_relative_components(char * path)}} should
be used to validate the absence of relative components before any operation
is done on those paths. This is missed in all the {{initialize*()}}
functions, as Hemanth pointed out offline.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1599) MRBench reuses jobConf and credentials there in.
MRBench reuses jobConf and credentials there in. Key: MAPREDUCE-1599 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1599 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey MRBench reuses the jobconf and therefore credentials are re-used, but JobTracker cancels the delegation tokens therefore the test fails sometimes. The fix is to pass the mapreduce.job.complete.cancel.delegation.tokens=false in the jobconf so that JobTracker does not cancel the tokens. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1599) MRBench reuses jobConf and credentials there in.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-1599: Attachment: MR-1599-y20.1.patch Patch for 20 uploaded MRBench reuses jobConf and credentials there in. Key: MAPREDUCE-1599 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1599 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MR-1599-y20.1.patch MRBench reuses the jobconf and therefore credentials are re-used, but JobTracker cancels the delegation tokens therefore the test fails sometimes. The fix is to pass the mapreduce.job.complete.cancel.delegation.tokens=false in the jobconf so that JobTracker does not cancel the tokens. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1420) TestTTResourceReporting failing in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12845450#action_12845450 ] Scott Chen commented on MAPREDUCE-1420: --- @Vinod, The patch has passed the test twice. I think it is ready to go :) TestTTResourceReporting failing in trunk Key: MAPREDUCE-1420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1420 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1420-v1.patch, MAPREDUCE-1420-v1.patch, output.rtf TestTTResourceReporting failing in trunk. The most specific issue from the logs seems to be : Error executing shell command org.apache.hadoop.util.Shell$ExitCodeException: kill: No such process Link : http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/217/ Attaching output in a file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1579) archive: check and possibly replace the space charater in paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-1579: -- Attachment: m1579_20100311_y0.20.patch m1579_20100311_y0.20.patch: for y0.20 archive: check and possibly replace the space charater in paths --- Key: MAPREDUCE-1579 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1579 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Blocker Fix For: 0.22.0 Attachments: m1579_20100310.patch, m1579_20100310b.patch, m1579_20100311.patch, m1579_20100311_y0.20.patch Since the space character is used as a separator in the index files, it won't work if there are spaces in the path (see also HADOOP-6591). The archive tools should # detect if there are spaces in the paths and # provide an option to replace it with some other characters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1600) o.a.h.mapreduce.FileOutputFormat should qualify the output path
o.a.h.mapreduce.FileOutputFormat should qualify the output path --- Key: MAPREDUCE-1600 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1600 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.20.3 Same as HADOOP-4746, but for the new api. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12845603#action_12845603 ] Luke Lu commented on MAPREDUCE-1545: Hong Tang mentioned that first tasks are setup tasks and we'd better add firstMapTaskLaunchTime and firstReduceTaskLaunchTime besides the firstTaskLaunchTime while we're at it. Add 'first-task-launched' to job-summary Key: MAPREDUCE-1545 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.22.0 It would be useful to track 'first-task-launched' time to job-summary for better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1585: --- Attachment: MAPREDUCE-1585.1.patch Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1585.1.patch, MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HADOOP-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HADOOP-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1585) Create Hadoop Archives version 2 with filenames URL-encoded
[ https://issues.apache.org/jira/browse/MAPREDUCE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12845622#action_12845622 ] Rodrigo Schmidt commented on MAPREDUCE-1585: Attached a new patch, but it cannot be tested while pending patch HADOOP-6591 is not committed. Create Hadoop Archives version 2 with filenames URL-encoded --- Key: MAPREDUCE-1585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1585 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Reporter: Rodrigo Schmidt Assignee: Rodrigo Schmidt Fix For: 0.22.0 Attachments: MAPREDUCE-1585.1.patch, MAPREDUCE-1585.patch Hadoop Archives version 1 don't cope with files that have spaces on their names. One proposal is to URLEncode filenames inside the index file (version 2, refers to HADOOP-6591). This task is to allow the creation of version 2 files that have file names encoded appropriately. It currently depends on HADOOP-6591 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1600) o.a.h.mapreduce.FileOutputCommitter should qualify the output path
[ https://issues.apache.org/jira/browse/MAPREDUCE-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1600: - Summary: o.a.h.mapreduce.FileOutputCommitter should qualify the output path (was: o.a.h.mapreduce.FileOutputFormat should qualify the output path) Unfortunately, we can't do it o.a.h.mapreduce.FileOutputFormat.checkOutputSpecs as done by HADOOP-4746 since the new api only gets a copy of the JobConf and not the real one, rendering the modification useless. So, the only way to fix this is to qualify the path in o.a.h.mapreduce.FileOutputCommitter. o.a.h.mapreduce.FileOutputCommitter should qualify the output path -- Key: MAPREDUCE-1600 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1600 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.20.3 Same as HADOOP-4746, but for the new api. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1320) StringBuffer - StringBuilder occurence
[ https://issues.apache.org/jira/browse/MAPREDUCE-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Kay updated MAPREDUCE-1320: --- Status: Patch Available (was: Open) StringBuffer - StringBuilder occurence Key: MAPREDUCE-1320 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1320 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Kay Kay Fix For: 0.22.0 Attachments: MAPREDUCE-1320.patch, MAPREDUCE-1320.patch A good number of toString() implementations use StringBuffer when the reference clearly does not go out of scope of the method and no concurrency is needed. Patch contains replacing those occurences from StringBuffer to StringBuilder. Created against map/reduce project trunk . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1320) StringBuffer - StringBuilder occurence
[ https://issues.apache.org/jira/browse/MAPREDUCE-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Kay updated MAPREDUCE-1320: --- Status: Open (was: Patch Available) StringBuffer - StringBuilder occurence Key: MAPREDUCE-1320 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1320 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Kay Kay Fix For: 0.22.0 Attachments: MAPREDUCE-1320.patch, MAPREDUCE-1320.patch A good number of toString() implementations use StringBuffer when the reference clearly does not go out of scope of the method and no concurrency is needed. Patch contains replacing those occurences from StringBuffer to StringBuilder. Created against map/reduce project trunk . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1320) StringBuffer - StringBuilder occurence
[ https://issues.apache.org/jira/browse/MAPREDUCE-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12845710#action_12845710 ] Kay Kay commented on MAPREDUCE-1320: org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testGroupMappingRefresh (from TestMapredGroupMappingServiceRefresh) java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.init(InetSocketAddress.java:118) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:385) at org.apache.hadoop.hdfs.server.namenode.NameNode.activate(NameNode.java:323) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:310) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:430) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:418) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1333) at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:278) at org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.setUp(TestMapredGroupMappingServiceRefresh.java:85) Does not seem to do with this patch though. StringBuffer - StringBuilder occurence Key: MAPREDUCE-1320 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1320 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Kay Kay Fix For: 0.22.0 Attachments: MAPREDUCE-1320.patch, MAPREDUCE-1320.patch A good number of toString() implementations use StringBuffer when the reference clearly does not go out of scope of the method and no concurrency is needed. Patch contains replacing those occurences from StringBuffer to StringBuilder. Created against map/reduce project trunk . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
