[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881583#action_12881583 ] Todd Lipcon commented on MAPREDUCE-1887: Ravi: this log message only runs once at tasktracker startup, and the number of volumes is usually <12. Not worth guarding in isDebugEnabled since it's not going to ever be a bottleneck. > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.3.patch, > MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[
https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881571#action_12881571
]
Ravi Gummadi commented on MAPREDUCE-1887:
-
LOG.debug() statement is not in LOG.isDebugEnabled() check and causes
unnecessary string concatenations in the for loop. Would you please add
if(LOG.isDebugEnabled()) { } around that ?
> MRAsyncDiskService does not properly absolutize volume root paths
> -
>
> Key: MAPREDUCE-1887
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.3.patch,
> MAPREDUCE-1887.patch
>
>
> In MRAsyncDiskService, volume names are sometimes specified as relative
> paths, which are not converted to absolute paths. This can cause errors of
> the form "cannot delete since it is outside of
> " even though the actual path is inside the root.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1851) Document configuration parameters in streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881563#action_12881563 ] Ravi Gummadi commented on MAPREDUCE-1851: - Latest patch looks good. +1 > Document configuration parameters in streaming > -- > > Key: MAPREDUCE-1851 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1851 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming, documentation >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1851-1.txt, patch-1851-2.txt, patch-1851.txt > > > There are several streaming options such as > stream.map.output.field.separator, stream.num.map.output.key.fields, > stream.map.input.field.separator, stream.reduce.input.field.separator, > stream.map.input.ignoreKey, stream.non.zero.exit.is.failure etc which are > spread everywhere. These should be documented at single place with > description and default-value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881560#action_12881560 ] Hadoop QA commented on MAPREDUCE-1887: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447778/MAPREDUCE-1887.3.patch against trunk revision 957086. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/260/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/260/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/260/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/260/console This message is automatically generated. > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.3.patch, > MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1851) Document configuration parameters in streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1851: --- Attachment: patch-1851-2.txt Modified the description for the property. Ran ant docs with the patch successfully. > Document configuration parameters in streaming > -- > > Key: MAPREDUCE-1851 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1851 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming, documentation >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1851-1.txt, patch-1851-2.txt, patch-1851.txt > > > There are several streaming options such as > stream.map.output.field.separator, stream.num.map.output.key.fields, > stream.map.input.field.separator, stream.reduce.input.field.separator, > stream.map.input.ignoreKey, stream.non.zero.exit.is.failure etc which are > spread everywhere. These should be documented at single place with > description and default-value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-978) -file option in streaming does not preserve execute permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu resolved MAPREDUCE-978. --- Resolution: Invalid Files passed using -file option are packaged into a jar and unjarred on the computing node. So, it wont preserve the execute permissions on the file. > -file option in streaming does not preserve execute permissions > --- > > Key: MAPREDUCE-978 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-978 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Chris Dyer > > For a streaming application I used the -file option to move some executable > files to the slave nodes. On the submit node, they had +x permissions but on > the destination node they were created with -x permissions. This probably > has to do with the umask settings on the various nodes, but streaming should > preserve the original permissions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Fix For: 0.22.0
The following code in StreamJob.java overrides user given output key and value
types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1850: Attachment: mapred-1850-2.patch submitting a revision > Include job submit host information (name and ip) in jobconf and jobdetails > display > --- > > Key: MAPREDUCE-1850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Attachments: mapred-1850-1.patch, mapred-1850-2.patch, > mapred-1850.patch, mapred-1850.patch > > > Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1850: Status: Patch Available (was: Open) revised patch > Include job submit host information (name and ip) in jobconf and jobdetails > display > --- > > Key: MAPREDUCE-1850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Attachments: mapred-1850-1.patch, mapred-1850-2.patch, > mapred-1850.patch, mapred-1850.patch > > > Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1850: Status: Open (was: Patch Available) revised JobSubmitter after a svn sync > Include job submit host information (name and ip) in jobconf and jobdetails > display > --- > > Key: MAPREDUCE-1850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Attachments: mapred-1850-1.patch, mapred-1850.patch, mapred-1850.patch > > > Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1868) Add read timeout on userlog pull
[ https://issues.apache.org/jira/browse/MAPREDUCE-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1868: Status: Patch Available (was: Open) No automated unit test included as it will require manual steps to test this change > Add read timeout on userlog pull > > > Key: MAPREDUCE-1868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1868 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Fix For: 0.22.0 > > Attachments: mapred-1868-1.patch, mapred-1868.20S.patch, > mapred-1868.patch > > > Add read and connection timeout to prevent job client hangs > jobclient can block indefinitely during task log pull if read or connect > fails > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049) > - locked <0xeed0f0f0> (a sun.net.www.protocol.http.HttpURLConnection) > at org.apache.hadoop.mapred.JobClient.getTaskLogs(JobClient.java:1396) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1868) Add read timeout on userlog pull
[ https://issues.apache.org/jira/browse/MAPREDUCE-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1868: Attachment: mapred-1868-1.patch Revised patch. > Add read timeout on userlog pull > > > Key: MAPREDUCE-1868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1868 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Fix For: 0.22.0 > > Attachments: mapred-1868-1.patch, mapred-1868.20S.patch, > mapred-1868.patch > > > Add read and connection timeout to prevent job client hangs > jobclient can block indefinitely during task log pull if read or connect > fails > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049) > - locked <0xeed0f0f0> (a sun.net.www.protocol.http.HttpURLConnection) > at org.apache.hadoop.mapred.JobClient.getTaskLogs(JobClient.java:1396) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1868) Add read timeout on userlog pull
[ https://issues.apache.org/jira/browse/MAPREDUCE-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1868: Status: Open (was: Patch Available) > Add read timeout on userlog pull > > > Key: MAPREDUCE-1868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1868 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 0.22.0 >Reporter: Krishna Ramachandran >Assignee: Krishna Ramachandran > Fix For: 0.22.0 > > Attachments: mapred-1868.20S.patch, mapred-1868.patch > > > Add read and connection timeout to prevent job client hangs > jobclient can block indefinitely during task log pull if read or connect > fails > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049) > - locked <0xeed0f0f0> (a sun.net.www.protocol.http.HttpURLConnection) > at org.apache.hadoop.mapred.JobClient.getTaskLogs(JobClient.java:1396) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881485#action_12881485
]
Hadoop QA commented on MAPREDUCE-1876:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447688/mapreduce-1876-v1.2.patch
against trunk revision 956779.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/259/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/259/console
This message is automatically generated.
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1855) refreshSuperUserGroupsConfiguration for MR should use server side configuration for the refresh (for HADOOP-6815)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved MAPREDUCE-1855. Resolution: Fixed I just committed this. Thanks, Boris! > refreshSuperUserGroupsConfiguration for MR should use server side > configuration for the refresh (for HADOOP-6815) > - > > Key: MAPREDUCE-1855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1855 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1855-1.patch, MAPREDUCE-1855-5.patch, > MAPREDUCE-1855-6.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1887: - Status: Patch Available (was: Open) > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.3.patch, > MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1887: - Attachment: MAPREDUCE-1887.3.patch Agree that the javadoc comment was confusing. Updated. Thanks for the fast review! > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.3.patch, > MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1870) Harmonize MapReduce JAR library versions with Common and HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1870: - Status: Patch Available (was: Open) > Harmonize MapReduce JAR library versions with Common and HDFS > - > > Key: MAPREDUCE-1870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1870 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1870.patch, MAPREDUCE-1870.patch > > > MapReduce part of HADOOP-6800. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1870) Harmonize MapReduce JAR library versions with Common and HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1870: - Attachment: MAPREDUCE-1870.patch New patch to use Jetty's ant dependency. > Harmonize MapReduce JAR library versions with Common and HDFS > - > > Key: MAPREDUCE-1870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1870 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Tom White >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1870.patch, MAPREDUCE-1870.patch > > > MapReduce part of HADOOP-6800. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1887: -- Status: Open (was: Patch Available) > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1855) refreshSuperUserGroupsConfiguration for MR should use server side configuration for the refresh (for HADOOP-6815)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881428#action_12881428 ] Jitendra Nath Pandey commented on MAPREDUCE-1855: - +1 for the patch. > refreshSuperUserGroupsConfiguration for MR should use server side > configuration for the refresh (for HADOOP-6815) > - > > Key: MAPREDUCE-1855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1855 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1855-1.patch, MAPREDUCE-1855-5.patch, > MAPREDUCE-1855-6.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1665) kill and modify should not be the same acl
[ https://issues.apache.org/jira/browse/MAPREDUCE-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-1665: Fix Version/s: 0.22.0 Affects Version/s: 0.21.0 > kill and modify should not be the same acl > -- > > Key: MAPREDUCE-1665 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1665 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 0.21.0, 0.22.0 >Reporter: Allen Wittenauer > Fix For: 0.22.0 > > > The permission to kill a job/task should be split out from modification. > There are definitely instances where someone who can kill a job should not be > able to modify it. [Third person job monitoring, for example, such as we > have here at LinkedIn.] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[
https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881397#action_12881397
]
Zheng Shao commented on MAPREDUCE-1887:
---
Code looks good.
Can we change
{code}
+ * @param nonCanonicalVols The roots of the file system volumes, which may
not
+ * be canonical paths.
{code}
to
{code}
+ * @param nonCanonicalVols The roots of the file system volumes, which can
be
+ * absolute paths from root or relative path from cwd.
{code}
?
I think the second one is easier to understand.
> MRAsyncDiskService does not properly absolutize volume root paths
> -
>
> Key: MAPREDUCE-1887
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.patch
>
>
> In MRAsyncDiskService, volume names are sometimes specified as relative
> paths, which are not converted to absolute paths. This can cause errors of
> the form "cannot delete since it is outside of
> " even though the actual path is inside the root.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1845) FairScheduler.tasksToPeempt() can return negative number
[ https://issues.apache.org/jira/browse/MAPREDUCE-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1845: -- Status: Patch Available (was: Open) I am submitting this to hudson again because testReport is gone. > FairScheduler.tasksToPeempt() can return negative number > > > Key: MAPREDUCE-1845 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1845 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1845.20100717.txt > > > This method can return negative number. This will cause the preemption to > under-preempt. > The bug was discovered by Joydeep. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1845) FairScheduler.tasksToPeempt() can return negative number
[ https://issues.apache.org/jira/browse/MAPREDUCE-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1845: -- Status: Open (was: Patch Available) > FairScheduler.tasksToPeempt() can return negative number > > > Key: MAPREDUCE-1845 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1845 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1845.20100717.txt > > > This method can return negative number. This will cause the preemption to > under-preempt. > The bug was discovered by Joydeep. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881369#action_12881369 ] Todd Lipcon commented on MAPREDUCE-1887: lgtm. +1 > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1887: - Attachment: MAPREDUCE-1887.2.patch new patch to address Todd's comments. > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.2.patch, MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881347#action_12881347 ] Todd Lipcon commented on MAPREDUCE-1887: Couple notes: - Rather than changing all the instances of "volumes" to "this.volumes" can you rename one of the variables? like rename the constructor argument to unnormalizedVolumes, and then you can just use "volumes" throughout? That should reduce changes that someone changing this code later makes the mistake again. - Missing @Test annotation on testRelativeToWorking() - Might be worth adding a comment to testVolumeNormalization stating it's a regression test for this JIRA number. - Either remove or change the log message "Volume: foo -> bar" - eg "Normalized MRAsyncDiskService volume foo -> bar" or just get rid of it. > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1887: - Attachment: MAPREDUCE-1887.patch This has caused intermittent failure of TestMRServerPorts on our hudson instance. I am attaching a patch with a test case that reliably reproduces this error as well as the fix. > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
[ https://issues.apache.org/jira/browse/MAPREDUCE-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1887: - Status: Patch Available (was: Open) > MRAsyncDiskService does not properly absolutize volume root paths > - > > Key: MAPREDUCE-1887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-1887.patch > > > In MRAsyncDiskService, volume names are sometimes specified as relative > paths, which are not converted to absolute paths. This can cause errors of > the form "cannot delete since it is outside of > " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1887) MRAsyncDiskService does not properly absolutize volume root paths
MRAsyncDiskService does not properly absolutize volume root paths - Key: MAPREDUCE-1887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1887 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Aaron Kimball Assignee: Aaron Kimball In MRAsyncDiskService, volume names are sometimes specified as relative paths, which are not converted to absolute paths. This can cause errors of the form "cannot delete since it is outside of " even though the actual path is inside the root. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1545: --- Attachment: mr-1545-y20s-v4.patch Looks like it's not committed to the y20s branch yet. Updated the y20s branch patch and committing. > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-1545-trunk-v1.patch, mr-1545-trunk-v2.patch, > mr-1545-y20s-v1.patch, mr-1545-y20s-v2.patch, mr-1545-y20s-v3.patch, > mr-1545-y20s-v4.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1863) [Rumen] Null failedMapAttemptCDFs in job traces generated by Rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881276#action_12881276 ] Dick King commented on MAPREDUCE-1863: -- The code change looks right. Please add the binary files to the patch, which you need to do to get the Hudson job to work correctly. > [Rumen] Null failedMapAttemptCDFs in job traces generated by Rumen > -- > > Key: MAPREDUCE-1863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1863 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.22.0 > > Attachments: counters-test-trace.json.gz, > dispatch-trace-output.json.gz, rumen-npe-v1.1.patch > > > All the traces generated by Rumen for jobs having failed task attempts has > null value for failedMapAttemptCDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1855) refreshSuperUserGroupsConfiguration for MR should use server side configuration for the refresh (for HADOOP-6815)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1855: -- Attachment: MAPREDUCE-1855-6.patch > refreshSuperUserGroupsConfiguration for MR should use server side > configuration for the refresh (for HADOOP-6815) > - > > Key: MAPREDUCE-1855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1855 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1855-1.patch, MAPREDUCE-1855-5.patch, > MAPREDUCE-1855-6.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1876:
---
Status: Patch Available (was: Open)
+1 latest patch looks fine.
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated MAPREDUCE-1876:
--
Attachment: mapreduce-1876-v1.2.patch
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated MAPREDUCE-1876:
--
Attachment: (was: mapreduce-1876-v1.2.patch)
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated MAPREDUCE-1876:
--
Attachment: mapreduce-1876-v1.2.patch
Attaching a patch with Amareshwari's comment incorporated.
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1886) org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails with timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1886: Summary: org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails with timeout (was: test org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails trunk builds) Description: testcase failure is seen here: Link : http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/355/testReport/junit/org.apache.hadoop.mapred/TestMiniMRChildTask/testTaskEnv/ was: mapreduce trunk build is failing: Link : http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/355/testReport/junit/org.apache.hadoop.mapred/TestMiniMRChildTask/testTaskEnv/ > org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails with timeout > --- > > Key: MAPREDUCE-1886 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1886 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Giridharan Kesavan > > testcase failure is seen here: > Link : > http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/355/testReport/junit/org.apache.hadoop.mapred/TestMiniMRChildTask/testTaskEnv/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1851) Document configuration parameters in streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881133#action_12881133 ] Ravi Gummadi commented on MAPREDUCE-1851: - The description of stream.joindelay.milli "Timeout in milliseconds for error thread and output thread to die" seems to be misleading if this is number of milli seconds for which the error thread and output thread are alive(time from start of thread to die). May be we should mention that this is the amount of time we wait for joining the error and output threads at the end of mapper/reducer ? > Document configuration parameters in streaming > -- > > Key: MAPREDUCE-1851 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1851 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming, documentation >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1851-1.txt, patch-1851.txt > > > There are several streaming options such as > stream.map.output.field.separator, stream.num.map.output.key.fields, > stream.map.input.field.separator, stream.reduce.input.field.separator, > stream.map.input.ignoreKey, stream.non.zero.exit.is.failure etc which are > spread everywhere. These should be documented at single place with > description and default-value. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1886) test org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails trunk builds
test org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv fails trunk builds Key: MAPREDUCE-1886 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1886 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Giridharan Kesavan mapreduce trunk build is failing: Link : http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/355/testReport/junit/org.apache.hadoop.mapred/TestMiniMRChildTask/testTaskEnv/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-577) Duplicate Mapper input when using StreamXmlRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-577:
---
Attachment: 577.v1.patch
Attaching patch for trunk by fixing the testcases so that the configuration
used when FileSystem object is created will have fs.local.block.size set to the
proper value needed.
Both testcases fail without the fix of the patch and pass with the fix.
> Duplicate Mapper input when using StreamXmlRecordReader
> ---
>
> Key: MAPREDUCE-577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Environment: HADOOP 0.17.0, Java 6.0
>Reporter: David Campbell
>Assignee: Ravi Gummadi
> Attachments: 0001-test-to-demonstrate-HADOOP-3484.patch,
> 0002-patch-for-HADOOP-3484.patch, 577.20S.patch, 577.patch, 577.v1.patch,
> HADOOP-3484.combined.patch, HADOOP-3484.try3.patch
>
>
> I have an XML file with 93626 rows. A row is marked by
> I've confirmed this with grep and the Grep example program included with
> HADOOP.
> Here is the grep example output. 93626
> I've setup my job configuration as follows:
> conf.set("stream.recordreader.class",
> "org.apache.hadoop.streaming.StreamXmlRecordReader");
> conf.set("stream.recordreader.begin", "");
> conf.set("stream.recordreader.end", "");
> conf.setInputFormat(StreamInputFormat.class);
> I have a fairly simple test Mapper.
> Here's the map method.
> public void map(Text key, Text value, OutputCollector
> output, Reporter reporter) throws IOException {
> try {
> output.collect(totalWord, one);
> if (key != null && key.toString().indexOf("01852") != -1) {
> output.collect(new Text("01852"), one);
> }
> } catch (Exception ex) {
> Logger.getLogger(TestMapper.class.getName()).log(Level.SEVERE,
> null, ex);
> System.out.println(value);
> }
> }
> For totalWord ("TOTAL"), I get:
> TOTAL 140850
> and for 01852 I get.
> 01852 86
> There are 43 instances of 01852 in the file.
> I have the following setting in my config.
>conf.setNumMapTasks(1);
> I have a total of six machines in my cluster.
> If I run without this, the result is 12x the actual value, not 2x.
> Here's some info from the cluster web page.
> Maps Reduces Total Submissions Nodes Map Task Capacity Reduce
> Task CapacityAvg. Tasks/Node
> 0 0 1 6 12 12 4.00
> I've also noticed something really strange in the job's output. It looks
> like it's starting over or redoing things.
> This was run using all six nodes and no limitations on map or reduce tasks.
> I haven't seen this behavior in any other case.
> 08/06/03 10:50:35 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> 08/06/03 10:50:36 INFO mapred.JobClient: Running job: job_200806030916_0018
> 08/06/03 10:50:37 INFO mapred.JobClient: map 0% reduce 0%
> 08/06/03 10:50:42 INFO mapred.JobClient: map 2% reduce 0%
> 08/06/03 10:50:45 INFO mapred.JobClient: map 12% reduce 0%
> 08/06/03 10:50:47 INFO mapred.JobClient: map 31% reduce 0%
> 08/06/03 10:50:48 INFO mapred.JobClient: map 49% reduce 0%
> 08/06/03 10:50:49 INFO mapred.JobClient: map 68% reduce 0%
> 08/06/03 10:50:50 INFO mapred.JobClient: map 100% reduce 0%
> 08/06/03 10:50:54 INFO mapred.JobClient: map 87% reduce 0%
> 08/06/03 10:50:55 INFO mapred.JobClient: map 100% reduce 0%
> 08/06/03 10:50:56 INFO mapred.JobClient: map 0% reduce 0%
> 08/06/03 10:51:00 INFO mapred.JobClient: map 0% reduce 1%
> 08/06/03 10:51:05 INFO mapred.JobClient: map 28% reduce 2%
> 08/06/03 10:51:07 INFO mapred.JobClient: map 80% reduce 4%
> 08/06/03 10:51:08 INFO mapred.JobClient: map 100% reduce 4%
> 08/06/03 10:51:09 INFO mapred.JobClient: map 100% reduce 7%
> 08/06/03 10:51:10 INFO mapred.JobClient: map 90% reduce 9%
> 08/06/03 10:51:11 INFO mapred.JobClient: map 100% reduce 9%
> 08/06/03 10:51:12 INFO mapred.JobClient: map 100% reduce 11%
> 08/06/03 10:51:13 INFO mapred.JobClient: map 90% reduce 11%
> 08/06/03 10:51:14 INFO mapred.JobClient: map 97% reduce 11%
> 08/06/03 10:51:15 INFO mapred.JobClient: map 63% reduce 11%
> 08/06/03 10:51:16 INFO mapred.JobClient: map 48% reduce 11%
> 08/06/03 10:51:17 INFO mapred.JobClient: map 21% reduce 11%
> 08/06/03 10:51:19 INFO mapred.JobClient: map 0% reduce 11%
> 08/06/03 10:51:20 INFO mapred.JobClient: map 15% reduce 12%
> 08/06/03 10:51:21 INFO mapred.JobClient: map 27% reduce 13%
> 08/06/03 10:51:22 INFO mapred.JobClient: map 67% reduce 13%
> 08/06/03 10:51:24 INFO mapred.JobClient: map 22% reduce 16%
> 08/06/03 10:51:25 INFO mapred.JobClient: map 46% reduce 16%
> 08/06/03 10:51:26 INFO mapred.JobCli
[jira] Commented: (MAPREDUCE-577) Duplicate Mapper input when using StreamXmlRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881122#action_12881122
]
Ravi Gummadi commented on MAPREDUCE-577:
In trunk, testcases were not picking the block size because in
TestStreaming(the base class of the 2 tests of this patch) is creating input
file by creating FileSystem object. As we were setting the config
fs.local.block.size later, it is not effective for the FileSystem --- causing
single split in both tests.
> Duplicate Mapper input when using StreamXmlRecordReader
> ---
>
> Key: MAPREDUCE-577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Environment: HADOOP 0.17.0, Java 6.0
>Reporter: David Campbell
>Assignee: Ravi Gummadi
> Attachments: 0001-test-to-demonstrate-HADOOP-3484.patch,
> 0002-patch-for-HADOOP-3484.patch, 577.20S.patch, 577.patch,
> HADOOP-3484.combined.patch, HADOOP-3484.try3.patch
>
>
> I have an XML file with 93626 rows. A row is marked by
> I've confirmed this with grep and the Grep example program included with
> HADOOP.
> Here is the grep example output. 93626
> I've setup my job configuration as follows:
> conf.set("stream.recordreader.class",
> "org.apache.hadoop.streaming.StreamXmlRecordReader");
> conf.set("stream.recordreader.begin", "");
> conf.set("stream.recordreader.end", "");
> conf.setInputFormat(StreamInputFormat.class);
> I have a fairly simple test Mapper.
> Here's the map method.
> public void map(Text key, Text value, OutputCollector
> output, Reporter reporter) throws IOException {
> try {
> output.collect(totalWord, one);
> if (key != null && key.toString().indexOf("01852") != -1) {
> output.collect(new Text("01852"), one);
> }
> } catch (Exception ex) {
> Logger.getLogger(TestMapper.class.getName()).log(Level.SEVERE,
> null, ex);
> System.out.println(value);
> }
> }
> For totalWord ("TOTAL"), I get:
> TOTAL 140850
> and for 01852 I get.
> 01852 86
> There are 43 instances of 01852 in the file.
> I have the following setting in my config.
>conf.setNumMapTasks(1);
> I have a total of six machines in my cluster.
> If I run without this, the result is 12x the actual value, not 2x.
> Here's some info from the cluster web page.
> Maps Reduces Total Submissions Nodes Map Task Capacity Reduce
> Task CapacityAvg. Tasks/Node
> 0 0 1 6 12 12 4.00
> I've also noticed something really strange in the job's output. It looks
> like it's starting over or redoing things.
> This was run using all six nodes and no limitations on map or reduce tasks.
> I haven't seen this behavior in any other case.
> 08/06/03 10:50:35 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> 08/06/03 10:50:36 INFO mapred.JobClient: Running job: job_200806030916_0018
> 08/06/03 10:50:37 INFO mapred.JobClient: map 0% reduce 0%
> 08/06/03 10:50:42 INFO mapred.JobClient: map 2% reduce 0%
> 08/06/03 10:50:45 INFO mapred.JobClient: map 12% reduce 0%
> 08/06/03 10:50:47 INFO mapred.JobClient: map 31% reduce 0%
> 08/06/03 10:50:48 INFO mapred.JobClient: map 49% reduce 0%
> 08/06/03 10:50:49 INFO mapred.JobClient: map 68% reduce 0%
> 08/06/03 10:50:50 INFO mapred.JobClient: map 100% reduce 0%
> 08/06/03 10:50:54 INFO mapred.JobClient: map 87% reduce 0%
> 08/06/03 10:50:55 INFO mapred.JobClient: map 100% reduce 0%
> 08/06/03 10:50:56 INFO mapred.JobClient: map 0% reduce 0%
> 08/06/03 10:51:00 INFO mapred.JobClient: map 0% reduce 1%
> 08/06/03 10:51:05 INFO mapred.JobClient: map 28% reduce 2%
> 08/06/03 10:51:07 INFO mapred.JobClient: map 80% reduce 4%
> 08/06/03 10:51:08 INFO mapred.JobClient: map 100% reduce 4%
> 08/06/03 10:51:09 INFO mapred.JobClient: map 100% reduce 7%
> 08/06/03 10:51:10 INFO mapred.JobClient: map 90% reduce 9%
> 08/06/03 10:51:11 INFO mapred.JobClient: map 100% reduce 9%
> 08/06/03 10:51:12 INFO mapred.JobClient: map 100% reduce 11%
> 08/06/03 10:51:13 INFO mapred.JobClient: map 90% reduce 11%
> 08/06/03 10:51:14 INFO mapred.JobClient: map 97% reduce 11%
> 08/06/03 10:51:15 INFO mapred.JobClient: map 63% reduce 11%
> 08/06/03 10:51:16 INFO mapred.JobClient: map 48% reduce 11%
> 08/06/03 10:51:17 INFO mapred.JobClient: map 21% reduce 11%
> 08/06/03 10:51:19 INFO mapred.JobClient: map 0% reduce 11%
> 08/06/03 10:51:20 INFO mapred.JobClient: map 15% reduce 12%
> 08/06/03 10:51:21 INFO mapred.JobClient: map 27% reduce 13%
> 08/06/03 10:51:22 INFO mapred.JobClient: map 67% reduce 13%
> 08/06/03 10:51:24 INFO mapred.JobClient: map 22% reduce 16%
> 08/06/03 10:51:25 INFO
[jira] Resolved: (MAPREDUCE-1885) Trunk compilation is broken because of FileSystem api change in HADOOP-6826
[ https://issues.apache.org/jira/browse/MAPREDUCE-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-1885. -- Hadoop Flags: [Reviewed] Fix Version/s: 0.21.0 (was: 0.22.0) Resolution: Fixed I've just committed this. Thanks, Ravi! > Trunk compilation is broken because of FileSystem api change in HADOOP-6826 > --- > > Key: MAPREDUCE-1885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.21.0 > > Attachments: 1885.patch > > > Trunk compilation is broken because of FileSystem api change in HADOOP-6826. > Here are the error messages: > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:277 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] FSDataOutputStream out = logDirFs.create(logFile, > [iajc] ^ > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:297 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] jobFileOut = logDirFs.create(logDirConfPath, > [iajc] > [iajc] > [iajc] 2 errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1885) Trunk compilation is broken because of FileSystem api change in HADOOP-6826
[ https://issues.apache.org/jira/browse/MAPREDUCE-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881081#action_12881081 ] Tom White commented on MAPREDUCE-1885: -- +1 This fixes MapReduce trunk for me. I'll commit it presently. > Trunk compilation is broken because of FileSystem api change in HADOOP-6826 > --- > > Key: MAPREDUCE-1885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1885.patch > > > Trunk compilation is broken because of FileSystem api change in HADOOP-6826. > Here are the error messages: > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:277 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] FSDataOutputStream out = logDirFs.create(logFile, > [iajc] ^ > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:297 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] jobFileOut = logDirFs.create(logDirConfPath, > [iajc] > [iajc] > [iajc] 2 errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1885) Trunk compilation is broken because of FileSystem api change in HADOOP-6826
[ https://issues.apache.org/jira/browse/MAPREDUCE-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881075#action_12881075 ] dhruba borthakur commented on MAPREDUCE-1885: - +1 > Trunk compilation is broken because of FileSystem api change in HADOOP-6826 > --- > > Key: MAPREDUCE-1885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1885.patch > > > Trunk compilation is broken because of FileSystem api change in HADOOP-6826. > Here are the error messages: > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:277 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] FSDataOutputStream out = logDirFs.create(logFile, > [iajc] ^ > [iajc] > /home/gravi/workspace/gitMR/hadoop-mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java:297 > [error] The method create(Path, FsPermission, boolean, int, short, long, > Progressable) in the type FileSystem is not applicable for the arguments > (Path, FsPermission, EnumSet, int, short, long, null) > [iajc] jobFileOut = logDirFs.create(logDirConfPath, > [iajc] > [iajc] > [iajc] 2 errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881074#action_12881074
]
Amareshwari Sriramadasu commented on MAPREDUCE-1876:
Fix looks fine. I have one comment in the testcase though.
The last assert check in testcase should take TaskType as REDUCE, as we would
never have a case where map attempt is run with reduce-id.
> TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
> type for reduce tasks
>
>
> Key: MAPREDUCE-1876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1876-v1.1.patch
>
>
> {{TaskAttemptStartedEvent}} is used to log the start time of both the map and
> reduce task attempts to {{JobHistory}}. Following is the implementation of
> _getEventType()_ method of {{TaskAttemptStartedEvent}}
> {code}
> /** Get the event type */
> public EventType getEventType() {
> return EventType.MAP_ATTEMPT_STARTED;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
