[jira] [Created] (MAPREDUCE-3581) [Rumen] Rumen anonymizer should handle composite string data
[Rumen] Rumen anonymizer should handle composite string data
Key: MAPREDUCE-3581
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3581
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tools/rumen
Affects Versions: 0.24.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Fix For: 0.24.0
Rumen's Anonymizer currently considers string as a single entity. At times,
strings can be composed of smaller sub-strings which can be anonymized
individually. Anonymizing sub-strings separately will result in retaining
certain statistics like frequency ('daily', 'weekly' etc). This was brought up
by Chris while developing the Anonymizer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3580) [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc
[Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc - Key: MAPREDUCE-3580 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3580 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.24.0 Currently, the Rumen anonymizer only parses and anonymizes job properties enumerated in MRJobConfig. Other properties like 1. compression codecs 2. output files 3. Task and Attempt level properties should also be considered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3539) possible Cases for NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172040#comment-13172040
]
kavita sharma commented on MAPREDUCE-3539:
--
Hi,
No testcase is present because for closing streams i have used IOUtils that is
a exiting and tested utility.
Thanks
Kavita
> possible Cases for NullPointerException
> ---
>
> Key: MAPREDUCE-3539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3539
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.20.2, 0.20.205.0, 0.21.0, 1.0.0
>Reporter: kavita sharma
>Priority: Trivial
> Fix For: 0.20.205.0, 1.0.0
>
> Attachments: MAPREDUCE-3539-branch-1.patch, MAPREDUCE-3539.patch
>
>
> in DistCh.java
> {noformat}
> in setup method
> opWriter.close();
> if opWriter is null then if we will try to close will throw NPE.
> {noformat}
> {noformat}
> in checkDuplication method
> in.close();
> if in is null then if we will try to close will throw NPE.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3539) possible Cases for NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172039#comment-13172039
]
kavita sharma commented on MAPREDUCE-3539:
--
Hi,
No testcase is present because for closing streams i have used IOUtils that is
a exiting and tested utility.
Thanks
Kavita Sharma
> possible Cases for NullPointerException
> ---
>
> Key: MAPREDUCE-3539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3539
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.20.2, 0.20.205.0, 0.21.0, 1.0.0
>Reporter: kavita sharma
>Priority: Trivial
> Fix For: 0.20.205.0, 1.0.0
>
> Attachments: MAPREDUCE-3539-branch-1.patch, MAPREDUCE-3539.patch
>
>
> in DistCh.java
> {noformat}
> in setup method
> opWriter.close();
> if opWriter is null then if we will try to close will throw NPE.
> {noformat}
> {noformat}
> in checkDuplication method
> in.close();
> if in is null then if we will try to close will throw NPE.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3539) possible Cases for NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kavita sharma updated MAPREDUCE-3539:
-
Attachment: MAPREDUCE-3539-branch-1.patch
MAPREDUCE-3539.patch
> possible Cases for NullPointerException
> ---
>
> Key: MAPREDUCE-3539
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3539
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.20.2, 0.20.205.0, 0.21.0, 1.0.0
>Reporter: kavita sharma
>Priority: Trivial
> Fix For: 0.20.205.0, 1.0.0
>
> Attachments: MAPREDUCE-3539-branch-1.patch, MAPREDUCE-3539.patch
>
>
> in DistCh.java
> {noformat}
> in setup method
> opWriter.close();
> if opWriter is null then if we will try to close will throw NPE.
> {noformat}
> {noformat}
> in checkDuplication method
> in.close();
> if in is null then if we will try to close will throw NPE.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3579) ConverterUtils should not include a port in a path for a URL with no port
[
https://issues.apache.org/jira/browse/MAPREDUCE-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171868#comment-13171868
]
Hudson commented on MAPREDUCE-3579:
---
Integrated in Hadoop-Mapreduce-trunk #931 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/])
MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a
url without a port (Contributed by atm)
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1215528
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestConverterUtils.java
> ConverterUtils should not include a port in a path for a URL with no port
> -
>
> Key: MAPREDUCE-3579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.23.1, 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3579.patch
>
>
> In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a
> URL includes a valid host it must also include a valid port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3579) ConverterUtils should not include a port in a path for a URL with no port
[
https://issues.apache.org/jira/browse/MAPREDUCE-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171861#comment-13171861
]
Hudson commented on MAPREDUCE-3579:
---
Integrated in Hadoop-Mapreduce-0.23-Build #131 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/131/])
merge MAPREDUCE-3579
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1215534
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestConverterUtils.java
> ConverterUtils should not include a port in a path for a URL with no port
> -
>
> Key: MAPREDUCE-3579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.23.1, 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3579.patch
>
>
> In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a
> URL includes a valid host it must also include a valid port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2592) TT should fail task immediately if userlog dir cannot be created
[
https://issues.apache.org/jira/browse/MAPREDUCE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171859#comment-13171859
]
Hadoop QA commented on MAPREDUCE-2592:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483859/MAPREDUCE-2592.r1.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1474//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1474//console
This message is automatically generated.
> TT should fail task immediately if userlog dir cannot be created
>
>
> Key: MAPREDUCE-2592
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2592
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Harsh J
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2592.r1.diff
>
>
> Currently, TaskRunner will log the message "mkdirs failed. Ignoring" if it
> fails to mkdir the userlog directory for a task. Then, it goes on to spawn
> taskjvm.sh which tries to redirect output into the userlogs dir, thus failing
> with exit code 1. This leads to error messages that are very hard to diagnose
> ("task failed with exit status 1") in cases where the userlog directory has
> either become inaccessible or has reached the maximum number of dirents
> (32000 in ext3)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3579) ConverterUtils should not include a port in a path for a URL with no port
[
https://issues.apache.org/jira/browse/MAPREDUCE-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171854#comment-13171854
]
Hudson commented on MAPREDUCE-3579:
---
Integrated in Hadoop-Hdfs-0.23-Build #111 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/111/])
merge MAPREDUCE-3579
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1215534
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestConverterUtils.java
> ConverterUtils should not include a port in a path for a URL with no port
> -
>
> Key: MAPREDUCE-3579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.23.1, 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3579.patch
>
>
> In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a
> URL includes a valid host it must also include a valid port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3579) ConverterUtils should not include a port in a path for a URL with no port
[
https://issues.apache.org/jira/browse/MAPREDUCE-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171851#comment-13171851
]
Hudson commented on MAPREDUCE-3579:
---
Integrated in Hadoop-Hdfs-trunk #898 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/])
MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a
url without a port (Contributed by atm)
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1215528
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestConverterUtils.java
> ConverterUtils should not include a port in a path for a URL with no port
> -
>
> Key: MAPREDUCE-3579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.23.1, 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3579.patch
>
>
> In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a
> URL includes a valid host it must also include a valid port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1720) 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171846#comment-13171846 ] Hadoop QA commented on MAPREDUCE-1720: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467289/mapred.failed.killed.difference.png against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1475//console This message is automatically generated. > 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker > UI > > > Key: MAPREDUCE-1720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1720 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan >Assignee: Harsh J > Labels: critical-0.22.0 > Attachments: mapred.failed.killed.difference.png, > mapreduce.unsuccessfuljobs.ui.r1.diff > > > The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job > status has been separated from Failed as part of HADOOP-3924, so the UI needs > to be updated to reflect the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2592) TT should fail task immediately if userlog dir cannot be created
[
https://issues.apache.org/jira/browse/MAPREDUCE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-2592:
---
Status: Patch Available (was: Open)
> TT should fail task immediately if userlog dir cannot be created
>
>
> Key: MAPREDUCE-2592
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2592
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Harsh J
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2592.r1.diff
>
>
> Currently, TaskRunner will log the message "mkdirs failed. Ignoring" if it
> fails to mkdir the userlog directory for a task. Then, it goes on to spawn
> taskjvm.sh which tries to redirect output into the userlogs dir, thus failing
> with exit code 1. This leads to error messages that are very hard to diagnose
> ("task failed with exit status 1") in cases where the userlog directory has
> either become inaccessible or has reached the maximum number of dirents
> (32000 in ext3)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2384) Can MR make error response Immediately?
[ https://issues.apache.org/jira/browse/MAPREDUCE-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2384: --- Status: Patch Available (was: Open) > Can MR make error response Immediately? > --- > > Key: MAPREDUCE-2384 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2384 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 0.21.0 >Reporter: Denny Ye >Assignee: Harsh J > Fix For: 0.24.0 > > Attachments: MAPREDUCE-2384.r1.diff, MAPREDUCE-2384.r2.diff, > MAPREDUCE-2384.r3.diff > > > When I read the source code of MapReduce in Hadoop 0.21.0, sometimes it made > me confused about error response. For example: > 1. JobSubmitter checking output for each job. MapReduce makes rule to > limit that each job output must be not exist to avoid fault overwrite. In my > opinion, MR should verify output at the point of client submitting. Actually, > it copies related files to specified target and then, doing the verifying. > 2. JobTracker. Job has been submitted to JobTracker. In first step, > JT create JIT object that is very "huge" . Next step, JT start to verify job > queue authority and memory requirements. > > In normal case, verifying client input then response immediately if > any cases in fault. Regular logic can be performed if all the inputs have > passed. > It seems like that those code does not make sense for understanding. > Is only my personal opinion? Wish someone help me to explain the details. > Thanks! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2769) TT should give more info for failed file operations
[ https://issues.apache.org/jira/browse/MAPREDUCE-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2769: --- Assignee: (was: Harsh J) > TT should give more info for failed file operations > --- > > Key: MAPREDUCE-2769 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2769 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.203.0 >Reporter: Eli Collins >Priority: Minor > > The TT should give more info when it fail a NativeIO file operation (eg the > file name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1932) record skipping doesn't work with the new map/reduce api
[ https://issues.apache.org/jira/browse/MAPREDUCE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-1932. Resolution: Won't Fix Won't Fix, per Tom and Owen's comments above. > record skipping doesn't work with the new map/reduce api > > > Key: MAPREDUCE-1932 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1932 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.20.1 >Reporter: Owen O'Malley >Assignee: Harsh J > Attachments: mapreduce.1932.skippingreader.r1.diff > > > The new HADOOP-1230 map/reduce api doesn't support the record skipping > features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1720) 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1720: --- Status: Patch Available (was: Open) > 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker > UI > > > Key: MAPREDUCE-1720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1720 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan >Assignee: Harsh J > Labels: critical-0.22.0 > Attachments: mapred.failed.killed.difference.png, > mapreduce.unsuccessfuljobs.ui.r1.diff > > > The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job > status has been separated from Failed as part of HADOOP-3924, so the UI needs > to be updated to reflect the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1720) 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171835#comment-13171835 ] Harsh J commented on MAPREDUCE-1720: Looks like this may be required/useful for 0.22 sub releases per Arun's label. > 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker > UI > > > Key: MAPREDUCE-1720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1720 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan >Assignee: Harsh J > Labels: critical-0.22.0 > Attachments: mapred.failed.killed.difference.png, > mapreduce.unsuccessfuljobs.ui.r1.diff > > > The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job > status has been separated from Failed as part of HADOOP-3924, so the UI needs > to be updated to reflect the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1720) 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1720: --- Target Version/s: 0.22.0 (was: 0.22.0, 1.1.0) > 'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker > UI > > > Key: MAPREDUCE-1720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1720 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan >Assignee: Harsh J > Labels: critical-0.22.0 > Attachments: mapred.failed.killed.difference.png, > mapreduce.unsuccessfuljobs.ui.r1.diff > > > The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job > status has been separated from Failed as part of HADOOP-3924, so the UI needs > to be updated to reflect the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-397) In JobInProgress, failed TIP should be added back to the non-running queue only if the tip has not failed.
[
https://issues.apache.org/jira/browse/MAPREDUCE-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-397.
---
Resolution: Not A Problem
Fix Version/s: (was: 0.24.0)
Not a problem, per Arun's comment, in MR2.
Not a severe problem for 1.0.
> In JobInProgress, failed TIP should be added back to the non-running queue
> only if the tip has not failed.
> --
>
> Key: MAPREDUCE-397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
>
> In case of a task failure, the corresponding TIP is added back to the
> non-running list via {{failMap()/failReduce()}} api. This reentry should be
> done only for TIPs that have not failed. Reentry is useless for failed TIPs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-423) Remove getNumResolvedTaskTrackers() api from JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-423. --- Resolution: Not A Problem Fix Version/s: (was: 0.24.0) Not a worry in 0.23+. > Remove getNumResolvedTaskTrackers() api from JobTracker > --- > > Key: MAPREDUCE-423 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-423 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Amar Kamat >Assignee: Harsh J >Priority: Trivial > Attachments: MAPREDUCE-423.r1.diff > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171825#comment-13171825 ] Harsh J commented on MAPREDUCE-2584: (Er, 1.0) > Check for serializers early, and give out more information regarding missing > serializers > > > Key: MAPREDUCE-2584 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.20.2 >Reporter: Harsh J >Assignee: Harsh J > Labels: serializers, tasks > Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, > MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, > MAPREDUCE-2584.r5.diff > > > As discussed on HADOOP-7328, MapReduce can handle serializers in a much > better way in case of bad configuration, improper imports (Some odd Text > class instead of the Writable Text set as key), etc.. > This issue covers the MapReduce parts of the improvements (made to IFile, > MapOutputBuffer, etc. and possible early-check of serializer availability > pre-submit) that provide more information than just an NPE as is the current > case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2584: --- Target Version/s: 0.23.1, 0.24.0, 1.1.0 Fix Version/s: (was: 0.24.0) > Check for serializers early, and give out more information regarding missing > serializers > > > Key: MAPREDUCE-2584 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.20.2 >Reporter: Harsh J >Assignee: Harsh J > Labels: serializers, tasks > Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, > MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, > MAPREDUCE-2584.r5.diff > > > As discussed on HADOOP-7328, MapReduce can handle serializers in a much > better way in case of bad configuration, improper imports (Some odd Text > class instead of the Writable Text set as key), etc.. > This issue covers the MapReduce parts of the improvements (made to IFile, > MapOutputBuffer, etc. and possible early-check of serializer availability > pre-submit) that provide more information than just an NPE as is the current > case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171824#comment-13171824 ] Harsh J commented on MAPREDUCE-2584: HADOOP-7328 was committed to 0.23/trunk. This one still awaits. I re-ran the tests and they seem to pass. I'd also like this committed to branch-0.1 (patch was attached earlier). Let me know if anything further is required! > Check for serializers early, and give out more information regarding missing > serializers > > > Key: MAPREDUCE-2584 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.20.2 >Reporter: Harsh J >Assignee: Harsh J > Labels: serializers, tasks > Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, > MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, > MAPREDUCE-2584.r5.diff > > > As discussed on HADOOP-7328, MapReduce can handle serializers in a much > better way in case of bad configuration, improper imports (Some odd Text > class instead of the Writable Text set as key), etc.. > This issue covers the MapReduce parts of the improvements (made to IFile, > MapOutputBuffer, etc. and possible early-check of serializer availability > pre-submit) that provide more information than just an NPE as is the current > case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2703) Add a doc note about changes to io.sort.mb and io.sort.factor config params for SequenceFile sorting in DistCp
[ https://issues.apache.org/jira/browse/MAPREDUCE-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-2703. Resolution: Won't Fix Appears to be that DistCp uses Sorter only for filename sorting, not much other. Defaults suffice here. Docs not absolutely necessary. > Add a doc note about changes to io.sort.mb and io.sort.factor config params > for SequenceFile sorting in DistCp > -- > > Key: MAPREDUCE-2703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2703 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: distcp, documentation >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Fix For: 0.24.0 > > > Carrying over work from MAPREDUCE-2622 and HADOOP-6801 that impacts how > sequencefiles will utilize the now deprecated io.sort.mb and io.sort.factor > configs (new configs are made available for it to use, see HADOOP-6801). > Since DistCp is the remaining lone framework user of SequenceFile.Sorter, it > will require additions to docs about new properties to control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
