[jira] [Updated] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-279: -- Component/s: (was: jobtracker) (was: tasktracker) mrv2 Description: Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. was:Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Tags: mr2,mapreduce-2.0 Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move.txt, capacity-scheduler-dark-theme.png, multi-column-stable-sort-default-theme.png, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2599) Gridmix system tests are failing due to misconfiguration of task memory in simulated cluster when high ram job emulation enabled.
Gridmix system tests are failing due to misconfiguration of task memory in simulated cluster when high ram job emulation enabled. - Key: MAPREDUCE-2599 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2599 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Reporter: Vinay Kumar Thota Assignee: Amar Kamat Gridmix system test are failing due to misconfiguration of task memory in simulated cluster when high ram job emulation enabled. Girdmix scale the task memory for emulation of high ram jobs based on simulated cluster configuration and it reserved the slots based on scaled memory information of each task. Here, I found an issue, suppose if total reserved slots of either map or reduce exceeds the slot capacity then it hangs the job and failed after timeout.So, Gridmix should show either error message or bailing out before submitting the job in this condition. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050380#comment-13050380
]
jirapos...@reviews.apache.org commented on MAPREDUCE-2584:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
---
(Updated 2011-06-16 12:24:09.519284)
Review request for hadoop-mapreduce.
Changes
---
IOException catching ('stead of null).
Summary
---
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better
way in case of bad configuration, improper imports (Some odd Text class instead
of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to
MapOutputBuffer and possible early-check of serializer availability pre-submit)
that provide more information than just an NPE as is the current case.
This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584
Diffs (updated)
-
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
Diff: https://reviews.apache.org/r/885/diff
Testing
---
Added a test case that expects a failure if no io.serializers are present.
Thanks,
Harsh
Check for serializers early, and give out more information regarding missing
serializers
Key: MAPREDUCE-2584
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
Labels: serializers, tasks
Fix For: 0.23.0
Attachments: MAPREDUCE-2584.r3.diff
As discussed on HADOOP-7328, MapReduce can handle serializers in a much
better way in case of bad configuration, improper imports (Some odd Text
class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to IFile,
MapOutputBuffer, etc. and possible early-check of serializer availability
pre-submit) that provide more information than just an NPE as is the current
case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2584: --- Attachment: (was: MAPREDUCE-2584.r3.diff) Check for serializers early, and give out more information regarding missing serializers Key: MAPREDUCE-2584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.20.2 Reporter: Harsh J Assignee: Harsh J Labels: serializers, tasks Fix For: 0.23.0 Attachments: MAPREDUCE-2584.r2.diff As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2584: --- Attachment: MAPREDUCE-2584.r2.diff Patch updated as per HADOOP-7328 changes. (Still blocked by HADOOP-7328 inclusion and will not compile until) Check for serializers early, and give out more information regarding missing serializers Key: MAPREDUCE-2584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.20.2 Reporter: Harsh J Assignee: Harsh J Labels: serializers, tasks Fix For: 0.23.0 Attachments: MAPREDUCE-2584.r2.diff As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2600) MR-279: simplify the jars
MR-279: simplify the jars -- Key: MAPREDUCE-2600 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2600 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Owen O'Malley Currently the MR-279 mapreduce project generates 59 jars from 59 source roots, which can be dramatically simplified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2600) MR-279: simplify the jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050559#comment-13050559 ] Owen O'Malley commented on MAPREDUCE-2600: -- I'd propose that we have: mr-client/* - src/java, src/test yarn/yarn-api,yarn-common - yarn/client yarn/yarn-server/* - yarn/server so that we end up withyarn-client, yarn-server, and mapreduce jars. Of course the Java package structure will still separate the different servers from each other. MR-279: simplify the jars -- Key: MAPREDUCE-2600 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2600 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Owen O'Malley Currently the MR-279 mapreduce project generates 59 jars from 59 source roots, which can be dramatically simplified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050576#comment-13050576 ] Todd Lipcon commented on MAPREDUCE-2550: Sorry for slow response, I missed this in my jira box. Eric: my point is that the next thing after export HADOOP_PREFIX=... is to source hadoop-config.sh (wherever it might be found). The first export in that file is to export HADOOP_PREFIX again. So regardless of whether this is a dev env or installed env, the export in mapred-config.sh should be redundant, no? bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: Scott Chen Priority: Minor It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050672#comment-13050672 ] Hadoop QA commented on MAPREDUCE-2584: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482789/MAPREDUCE-2584.r2.diff against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//console This message is automatically generated. Check for serializers early, and give out more information regarding missing serializers Key: MAPREDUCE-2584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.20.2 Reporter: Harsh J Assignee: Harsh J Labels: serializers, tasks Fix For: 0.23.0 Attachments: MAPREDUCE-2584.r2.diff As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
[ https://issues.apache.org/jira/browse/MAPREDUCE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Schober updated MAPREDUCE-2601: -- Attachment: patchfile.txt Whoops, it seems I 'attached a patch' without actually attaching it. Here it is. Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.1 Reporter: Scott Chen Priority: Minor Attachments: patchfile.txt It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
[ https://issues.apache.org/jira/browse/MAPREDUCE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050741#comment-13050741 ] Hadoop QA commented on MAPREDUCE-2601: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482861/patchfile.txt against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/401//console This message is automatically generated. Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.1 Reporter: Scott Chen Priority: Minor Attachments: patchfile.txt It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1638) Divide MapReduce into API and implementation source trees
[ https://issues.apache.org/jira/browse/MAPREDUCE-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1638: - Attachment: MAPREDUCE-1638.patch This patch just does some more clean up to allow the separation to happen. This part can be committed now, while the actual separation of the trees would be done later, probably as a part of the process of getting MR-279 into trunk. Divide MapReduce into API and implementation source trees - Key: MAPREDUCE-1638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1638.patch, MAPREDUCE-1638.patch, MAPREDUCE-1638.sh I think it makes sense to separate the MapReduce source into public API and implementation trees. The public API could be broken further into kernel and library trees. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1638) Divide MapReduce into API and implementation source trees
[ https://issues.apache.org/jira/browse/MAPREDUCE-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050755#comment-13050755 ] Owen O'Malley commented on MAPREDUCE-1638: -- I can understand splitting up the client and server jars, but splitting up the API and implementation only makes sense if you have different implementations and a test suite to test them. Cleaning up the dependencies is a good thing, especially removing dependencies from the client on the server code. Divide MapReduce into API and implementation source trees - Key: MAPREDUCE-1638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1638.patch, MAPREDUCE-1638.patch, MAPREDUCE-1638.sh I think it makes sense to separate the MapReduce source into public API and implementation trees. The public API could be broken further into kernel and library trees. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1638) Divide MapReduce into API and implementation source trees
[ https://issues.apache.org/jira/browse/MAPREDUCE-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050765#comment-13050765 ] Tom White commented on MAPREDUCE-1638: -- Yes, these patches are about removing client dependencies on the server. Divide MapReduce into API and implementation source trees - Key: MAPREDUCE-1638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1638.patch, MAPREDUCE-1638.patch, MAPREDUCE-1638.sh I think it makes sense to separate the MapReduce source into public API and implementation trees. The public API could be broken further into kernel and library trees. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2547) TestDFSIO fails on a physical cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050788#comment-13050788
]
Konstantin Boudnik commented on MAPREDUCE-2547:
---
I don't remember exact build numbers but they all were happening after 5/23 for
sure.
TestDFSIO fails on a physical cluster
-
Key: MAPREDUCE-2547
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2547
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Environment: Physical cluster based on 0.22-SNAPSHOT
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
Priority: Blocker
An attempt to run TestDSFIO on cluster fails because TestDFSIO tries to run
MR job with local runner. If JT is explicitly specified via {{-jt}} cmd. arg.
then everything is working as expected.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2593) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050806#comment-13050806 ] Hadoop QA commented on MAPREDUCE-2593: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482070/RndRead-TestDFSIO-061011.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/400//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/400//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/400//console This message is automatically generated. Random read benchmark for DFS - Key: MAPREDUCE-2593 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2593 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Raghu Angadi Assignee: Dave Thompson Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, RndRead-TestDFSIO.patch We should have at least one random read benchmark that can be run with rest of Hadoop benchmarks regularly. Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2601) Add a filter text box to FairSchedulerServlet page
[ https://issues.apache.org/jira/browse/MAPREDUCE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050832#comment-13050832 ] Scott Chen commented on MAPREDUCE-2601: --- Markus: Hadoop QA doesn't accept git patch. It just does patch -p0 -i file. You have to do --no-prefix. Add a filter text box to FairSchedulerServlet page -- Key: MAPREDUCE-2601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2601 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.1 Reporter: Scott Chen Priority: Minor Attachments: patchfile.txt It will be useful if we can filter pool in the fairscheduler UI page. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2602: Status: Patch Available (was: Open) This batch is backward compatible. Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2602: Status: Patch Available (was: Open) This patch is backward compatible. Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2602.patch Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2602: Attachment: MAPREDUCE-2602.patch Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2602.patch Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2602: Status: Open (was: Patch Available) Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2602.patch Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2603) Gridmix system tests are failing due to high ram emulation enable by default for normal mr jobs in the trace which exceeds the solt capacity.
Gridmix system tests are failing due to high ram emulation enable by default for normal mr jobs in the trace which exceeds the solt capacity. - Key: MAPREDUCE-2603 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2603 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota In Gridmix high ram emulation enable by default.Because of this feature, some of the gridmix system tests are hanging for some time and then failing after timeout. Actually the failure case was occurring whenever reserved slot capacity exceeds the cluster slot capacity.So for fixing the issue by disabling the high ram emulation in the tests which are using the normal mr jobs in the traces. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2603) Gridmix system tests are failing due to high ram emulation enable by default for normal mr jobs in the trace which exceeds the solt capacity.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-2603: - Attachment: MAPREDUCE-2603.patch Please review the patch. Gridmix system tests are failing due to high ram emulation enable by default for normal mr jobs in the trace which exceeds the solt capacity. - Key: MAPREDUCE-2603 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2603 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-2603.patch In Gridmix high ram emulation enable by default.Because of this feature, some of the gridmix system tests are hanging for some time and then failing after timeout. Actually the failure case was occurring whenever reserved slot capacity exceeds the cluster slot capacity.So for fixing the issue by disabling the high ram emulation in the tests which are using the normal mr jobs in the traces. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2602) Allow setting of end-of-record delimiter for TextInputFormat (for the old API)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050901#comment-13050901 ] Hadoop QA commented on MAPREDUCE-2602: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482890/MAPREDUCE-2602.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/402//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/402//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/402//console This message is automatically generated. Allow setting of end-of-record delimiter for TextInputFormat (for the old API) -- Key: MAPREDUCE-2602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2602 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2602.patch Since there are users who are still using the old MR API, it will be useful to modify the org.apache.hadoop.mapred.LineRecordReader and org.apache.hadoop.mapred.TextInputFormat to be able to use custom (user-specified) end-of-record delimiters. This will make use of the LineReader improvement introduced in HADOOP-7096 that enables the LineReader to break lines at user-specified delimiters. Note: MAPREDUCE-2254 already added this improvement to the new API (but not the old API). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
