[jira] Updated: (MAPREDUCE-587) Stream test TestStreamingExitStatus fails with Out of Memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-587: - Attachment: MAPREDUCE-587-v1.0.patch Attaching a patch that solves the problem on my box. > Stream test TestStreamingExitStatus fails with Out of Memory > > > Key: MAPREDUCE-587 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-587 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming > Environment: OS/X, 64-bit x86 imac, 4GB RAM. >Reporter: Steve Loughran >Priority: Minor > Attachments: MAPREDUCE-587-v1.0.patch > > > contrib/streaming tests are failing a test with an Out of Memory error on an > OS/X Mac -same problem does not surface on Linux. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-815) Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro Serialization
[
https://issues.apache.org/jira/browse/MAPREDUCE-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736993#action_12736993
]
Ravi Gummadi commented on MAPREDUCE-815:
This could have something like
public class AvroInputFormat extends
FileInputFormat {
@Override
public RecordReader
createRecordReader(InputSplit split,
TaskAttemptContext context) {
return new AvroRecordReader();
}
//...
}
and
public class AvroRecordReader extends
RecordReader {
//implements the methods of RecordReader for KEY and VALUE of avro types
}
Does this look fine ?
> Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro
> Serialization
> --
>
> Key: MAPREDUCE-815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-815
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Ravi Gummadi
>Assignee: Ravi Gummadi
>
> MapReduce needs AvroInputFormat similar to other InputFormats like
> TextInputFormat to be able to use avro serialization in hadoop. Similarly
> AvroOutputFormat is needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Status: Patch Available (was: Open) > Create target for 10 minute patch test build for mapreduce > --- > > Key: MAPREDUCE-670 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: build >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670.patch > > > Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Attachment: mapred-670-v1.patch Patch fixing the indentation issue pointed out by Konstantin. A few additional points: Code coverage (for the mapred package) of all-tests list is 76%. Code coverage (for the mapred package) of commit-tests list is 59% I have left out two tests (TestJobTrackerRestart and TestQueueManager) out of this list as these take about 7 and 6 minutes respectively. Separate effort is underway to refactor these to become unit tests and when completed, these tests will be added to the commit-tests list. The current tests run in less than 9 minutes, so adding these two tests, after they have been refactored, should still keep the run time down to 10 minutes. Code coverage (for the mapred package) of commit-tests + TestJobTrackerRestart + TestQueueManager (as they exist now) is 63% > Create target for 10 minute patch test build for mapreduce > --- > > Key: MAPREDUCE-670 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: build >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: FastTestsInfo.xls, mapred-670-v1.patch, mapred-670.patch > > > Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Status: Open (was: Patch Available) > Create target for 10 minute patch test build for mapreduce > --- > > Key: MAPREDUCE-670 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: build >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: FastTestsInfo.xls, mapred-670.patch > > > Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-815) Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro Serialization
Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro Serialization -- Key: MAPREDUCE-815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-815 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ravi Gummadi Assignee: Ravi Gummadi MapReduce needs AvroInputFormat similar to other InputFormats like TextInputFormat to be able to use avro serialization in hadoop. Similarly AvroOutputFormat is needed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-813) Streaming Doc and M/R-Tutorial Doc - updates
[
https://issues.apache.org/jira/browse/MAPREDUCE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736987#action_12736987
]
Amareshwari Sriramadasu commented on MAPREDUCE-813:
---
Some minor comments:
1. mapred_tutorial.xml has some tabs instead of spaces.
2.
{code}
-Hadoop streaming is a utility that comes with the Hadoop distribution. The
utility allows you to create and run Map/Reduce jobs with any executable or
script as the mapper and/or the reducer. For example:
+Hadoop streaming is a utility that comes with the Hadoop distribution. The
utility allows you to create and run map/reduce jobs with any executable or
+script as the mapper and/or the reducer. For example:
{code}
The documentation change here is the change from *Map/Reduce* to *map/reduce*.
But HADOOP-3691 made sure in all the places, it should be Map/Reduce. I see the
same change in many places. Was this intentional?
> Streaming Doc and M/R-Tutorial Doc - updates
> -
>
> Key: MAPREDUCE-813
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-813
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: documentation
>Affects Versions: 0.21.0
>Reporter: Corinne Chandel
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: mapred_tutorial.pdf, MAPREDUCE-813.patch, streaming.pdf
>
>
> This JIRA addresses issues in the Streaming doc that also require a
> cross-link to and update in the M/R Tutorial doc. All changes approved by the
> reviewing engineer.
> Streaming issues:
> 1. During the execution of a streaming job, the names of the "mapred"
> parameters are transformed. The dots ( . ) become underscores ( _ ).
> Docs affected: streaming and m/r tutorial (new sub-sections added under Task
> Execution & Environment section)
> 2. For -files and -archives options, Hadoop now creates symlink with same
> name as file (user-defined symlinks, #mysymlink, currently not supported)
> Docs affected:streaming
> 3. Streaming supports streaming command options and generic command options.
> Generic options must be placed before streaming options, otherwise command
> fails.
> Docs affected: streaming (reorganized the streaming doc to make distinctions
> between 2 sets of command options more clear)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-629) Modify TestQueueManager to improve execution time
[ https://issues.apache.org/jira/browse/MAPREDUCE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan reassigned MAPREDUCE-629: --- Assignee: (was: Jothi Padmanabhan) > Modify TestQueueManager to improve execution time > - > > Key: MAPREDUCE-629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Priority: Minor > > With a few small changes, the run time of this test can be brought down by > half. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-809) Job summary logs show status of completed jobs as RUNNING
[ https://issues.apache.org/jira/browse/MAPREDUCE-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-809: Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. > Job summary logs show status of completed jobs as RUNNING > -- > > Key: MAPREDUCE-809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-809 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-809_0_20090728.patch, > MAPREDUCE-809_0_20090728_yhadoop20.patch > > > MAPREDUCE-740 added job summary logs. During testing our QA folks noticed > that completed jobs show up as RUNNING in the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-809) Job summary logs show status of completed jobs as RUNNING
[ https://issues.apache.org/jira/browse/MAPREDUCE-809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736909#action_12736909 ] Arun C Murthy commented on MAPREDUCE-809: - All test cases pass, 'ant test-patch' does too and I've not included any new testcases since it's essentially the same fix as MAPREDUCE-740 and I can't add more tests for the reasons elaborated there. Also, this patch only fixes logging for FAILED/KILLED tasks and Suman hasn't been able to reproduce the error for SUCCEEDED ones, I'll commit this patch and we can open a different jira if she can reproduce it later. > Job summary logs show status of completed jobs as RUNNING > -- > > Key: MAPREDUCE-809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-809 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-809_0_20090728.patch, > MAPREDUCE-809_0_20090728_yhadoop20.patch > > > MAPREDUCE-740 added job summary logs. During testing our QA folks noticed > that completed jobs show up as RUNNING in the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-809) Job summary logs show status of completed jobs as RUNNING
[ https://issues.apache.org/jira/browse/MAPREDUCE-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-809: Status: Patch Available (was: Open) > Job summary logs show status of completed jobs as RUNNING > -- > > Key: MAPREDUCE-809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-809 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-809_0_20090728.patch, > MAPREDUCE-809_0_20090728_yhadoop20.patch > > > MAPREDUCE-740 added job summary logs. During testing our QA folks noticed > that completed jobs show up as RUNNING in the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-801) MAPREDUCE framework should issue warning with too many locations for a split
[ https://issues.apache.org/jira/browse/MAPREDUCE-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736698#action_12736698 ] Doug Cutting commented on MAPREDUCE-801: > I'd like to see us discard anything beyond the first 5 options in the JT [ > ... ] Truncating is probably fine. The original proposal was to "discard location information completely". > MAPREDUCE framework should issue warning with too many locations for a split > > > Key: MAPREDUCE-801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-801 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Hong Tang > > Customized input-format may be buggy and report misleading locations through > input-split, an example of which is PIG-878. When an input split returns too > many locations, it would not only artificially inflate the percentage of data > local or rack local maps, but also force scheduler to use more memory and > work harder to conduct task assignment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736636#action_12736636
]
Konstantin Boudnik commented on MAPREDUCE-670:
--
It seems that indentation isn't consistent across the changes.
{code}
{code}
or
{code}
{code}
Looks good otherwise
> Create target for 10 minute patch test build for mapreduce
> ---
>
> Key: MAPREDUCE-670
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-670
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: build
>Reporter: Jothi Padmanabhan
>Assignee: Jothi Padmanabhan
> Attachments: FastTestsInfo.xls, mapred-670.patch
>
>
> Creating a new Jira to track HADOOP-5628 for MapReduce
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: MAPREDUCE-745-v1.2.patch Attaching a patch that fixes the common issue to do with jobtracker instances coming up in same minute. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. The findbug warning is due to refactoring. > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-372: -- Fix Version/s: 0.21.0 Status: Patch Available (was: Open) > Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. > --- > > Key: MAPREDUCE-372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-372.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Attachment: mapred-670.patch Changes to build.xml for the run-commit-tests target. The changes are similar to run-commit-tests target of HDFS. > Create target for 10 minute patch test build for mapreduce > --- > > Key: MAPREDUCE-670 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: build >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: FastTestsInfo.xls, mapred-670.patch > > > Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-670) Create target for 10 minute patch test build for mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan updated MAPREDUCE-670: Status: Patch Available (was: Open) > Create target for 10 minute patch test build for mapreduce > --- > > Key: MAPREDUCE-670 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-670 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: build >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: FastTestsInfo.xls, mapred-670.patch > > > Creating a new Jira to track HADOOP-5628 for MapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-372) Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-372: -- Attachment: patch-372.txt Attaching patch for review. Patch implements ChainMapper and ChainReducer as proposed by Owen. The key and values are always passed *by value* from one element of the chain to the next, because we clone the keys and values between steps. This removes the option to pass by value or by reference, that was available in old api. > Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new api. > --- > > Key: MAPREDUCE-372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-372.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-425) NPE in TaskInProgress.cleanup
[
https://issues.apache.org/jira/browse/MAPREDUCE-425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu resolved MAPREDUCE-425.
---
Resolution: Duplicate
Fixed as part of HADOOP-5233
> NPE in TaskInProgress.cleanup
> -
>
> Key: MAPREDUCE-425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Steve Loughran
>Priority: Minor
>
> This may be something that only my code triggers; an NPE in
> TaskTracker$TaskInProgress.cleanup
> {code}
> [sf-startdaemon-debug] 09/01/28 11:41:06 [TaskLauncher for task] INFO
> mapred.TaskTracker : Error cleaning up task runner:
> java.lang.NullPointerException
> [sf-startdaemon-debug]at
> org.apache.hadoop.mapred.TaskTracker$TaskInProgress.cleanup(TaskTracker.java:2487)
> [sf-startdaemon-debug]at
> org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1825)
> [sf-startdaemon-debug]at
> org.apache.hadoop.mapred.TaskTracker.access$1100(TaskTracker.java:104)
> [sf-startdaemon-debug]at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1779)
> {code}
> Looking at the code, the only source of NPE's on that line is localJobConf
> {code}
> if (localJobConf.getNumTasksToExecutePerJvm() == 1) {
> {code}
> It looks like if TaskInProgress.cleanup() ever gets called with no valid
> localJobConf, then an NPE is the result. The exception gets logged and
> discarded, but it does appear in the logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-688) TestLostTracker sometimes fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jothi Padmanabhan resolved MAPREDUCE-688. - Resolution: Fixed TestLostTracker got rewritten as a Unit test. This bug is no longer valid > TestLostTracker sometimes fails > > > Key: MAPREDUCE-688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-688 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Amareshwari Sriramadasu > Attachments: TEST-org.apache.hadoop.mapred.TestLostTracker.txt > > > Observed that TestLostTracker failed once with follwing assertion failure: > Invalid start time 0 > junit.framework.AssertionFailedError: Invalid start time 0 > at > org.apache.hadoop.mapred.TestLostTracker.testTaskStatuses(TestLostTracker.java:109) > at > org.apache.hadoop.mapred.TestLostTracker.testLostTracker(TestLostTracker.java:99) > at > org.apache.hadoop.mapred.TestLostTracker.testLostTracker(TestLostTracker.java:161) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-814) Move completed Job history files to HDFS
Move completed Job history files to HDFS Key: MAPREDUCE-814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-814 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Sharad Agarwal Currently completed job history files remain on the jobtracker node. Having the files available on HDFS will enable clients to access these files more easily. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-814) Move completed Job history files to HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal reassigned MAPREDUCE-814: Assignee: Sharad Agarwal > Move completed Job history files to HDFS > > > Key: MAPREDUCE-814 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-814 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > > Currently completed job history files remain on the jobtracker node. Having > the files available on HDFS will enable clients to access these files more > easily. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-793) Create a new test that consolidates a few tests to be included in the commit-test list
[ https://issues.apache.org/jira/browse/MAPREDUCE-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-793: -- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks, Jothi! This test runs jobs and tests multiple things before and after the job runs. In the long run, we should move to using mock objects and create more manageable unit tests. > Create a new test that consolidates a few tests to be included in the > commit-test list > -- > > Key: MAPREDUCE-793 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-793 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: mapred-793-v1.patch, mapred-793.patch > > > There are few tests that just run similar jobs and test different > functionality. It would be useful to have a test that runs one job and tests > several of these functionality together so that this test can be included in > the fast commit-tests target. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-628) TestJobInProgress brings up MinMR/DFS clusters for every test
[ https://issues.apache.org/jira/browse/MAPREDUCE-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-628: -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks, Jothi! > TestJobInProgress brings up MinMR/DFS clusters for every test > - > > Key: MAPREDUCE-628 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-628 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan >Priority: Minor > Fix For: 0.21.0 > > Attachments: hadoop-5970.patch, mapred-628-v1.patch, mapred-628.patch > > > TestJobInProgress brings up MiniMR clusters in setUp and brings it down in > tearDown methods. Since these methods are called before each test, the test > brings up/down the cluster several times. Instead, the cluster should just be > brought up once, all tests run and then brought down -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-760) TestNodeRefresh might not work as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal reassigned MAPREDUCE-760: Assignee: Amar Kamat > TestNodeRefresh might not work as expected > -- > > Key: MAPREDUCE-760 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-760 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amar Kamat >Assignee: Amar Kamat > Attachments: MAPREDUCE-760-v1.0.patch > > > MAPREDUCE-677 fixed one part of the problem. It is possible that the > tasktracker might not have joined the jobtracker and hence the asserts might > fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-760) TestNodeRefresh might not work as expected
[ https://issues.apache.org/jira/browse/MAPREDUCE-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736542#action_12736542 ] Sharad Agarwal commented on MAPREDUCE-760: -- +1 Patch looks fine. > TestNodeRefresh might not work as expected > -- > > Key: MAPREDUCE-760 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-760 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Amar Kamat >Assignee: Amar Kamat > Attachments: MAPREDUCE-760-v1.0.patch > > > MAPREDUCE-677 fixed one part of the problem. It is possible that the > tasktracker might not have joined the jobtracker and hence the asserts might > fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-513) Prior code fix in Capacity Scheduler prevents speculative execution in jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan resolved MAPREDUCE-513. -- Resolution: Fixed The TestQueueCapacities have been fixed in HADOOP-5869 > Prior code fix in Capacity Scheduler prevents speculative execution in jobs > --- > > Key: MAPREDUCE-513 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-513 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vivek Ratan >Assignee: Sreekanth Ramakrishnan > Attachments: 4981.1.patch, 4981.2.patch, HADOOP-4981-1.patch, > HADOOP-4981-2.patch, HADOOP-4981-3.patch, HADOOP-4981-4.patch, > HADOOP-4981-5-br20.patch, HADOOP-4981-5.patch > > > As part of the code fix for HADOOP-4035, the Capacity Scheduler obtains a > task from JobInProgress (calling obtainNewMapTask() or obtainNewReduceTask()) > only if the number of pending tasks for a job is greater than zero (see the > if-block in TaskSchedulingMgr.getTaskFromJob()). So, if a job has no pending > tasks and only has running tasks, it will never be given a slot, and will > never have a chance to run a speculative task. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-12) Tasks execed by the task controller shouldn't inherit tasktracker groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan resolved MAPREDUCE-12. - Resolution: Fixed Fixed as part of HADOOP-5420 > Tasks execed by the task controller shouldn't inherit tasktracker groups > > > Key: MAPREDUCE-12 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-12 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: hadoo 0.20 + patches, Linux Task controller >Reporter: Rajiv Chittajallu >Assignee: Sreekanth Ramakrishnan > Attachments: hadoop-5686-1.patch > > > Mapred tasks process seem to inherit the group list from the TaskTracker > daemon instead of the task owner. > tom 26633 15736 0 21:33 ?00:00:02 /usr/bin/java ... > org.apache.hadoop.mapred.Child 127.0.0.1 51207 .. > mapred 15736 1 2 Apr08 ?03:54:59 /usr/bin/java ... > org.apache.hadoop.mapred.TaskTracker > hadoop1:~$ id mapred > uid=50589(mapred) gid=100(users) groups=100(users),20001(hadoop) > hadoop1:~$ fgrep Groups /proc/26633/status > Groups: 100 20001 > hadoop1:~$ id tom > uid=47765(tom) gid=100(users) groups=100(users),10764(ninjas) > org.apache.hadoop.mapred.LinuxTaskController should set the user > supplimentary group list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-539) Implement a config validator tool for the capacity scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan resolved MAPREDUCE-539. -- Resolution: Duplicate Will be fixed as part of MAPREDUCE-768 > Implement a config validator tool for the capacity scheduler > > > Key: MAPREDUCE-539 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-539 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hemanth Yamijala >Assignee: Sreekanth Ramakrishnan > Attachments: HADOOP-4809-1.patch > > > The capacity scheduler sanity checks configuration when it starts and halts > if there are any problems found. For ease of deployment, it would help to > have a simple utility that will validate the configuration before the > capacity scheduler can be started, and report errors / warnings to the user > about (possible) misconfigurations. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736528#action_12736528
]
Amar Kamat commented on MAPREDUCE-805:
--
All contrib tests except TestStreamingExitStatus passed.
> Deadlock in Jobtracker
> --
>
> Key: MAPREDUCE-805
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Michael Tamm
> Attachments: MAPREDUCE-805-v1.1.patch, MAPREDUCE-805-v1.2.patch,
> MAPREDUCE-805-v1.3.patch
>
>
> We are running a hadoop cluster (version 0.20.0) and have detected the
> following deadlock on our jobtracker:
> {code}
> "IPC Server handler 51 on 9001":
> at
> org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
> - waiting to lock <0x7f2b6fb46130> (a
> org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
> - locked <0x7f2b5f026000> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> "pool-1-thread-2":
> at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
> - waiting to lock <0x7f2b5f026000> (a
> org.apache.hadoop.mapred.JobTracker)
> at
> org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
> - locked <0x7f2b6fb46130> (a org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
> - locked <0x7f2b6fb46130> (a org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
> - locked <0x7f2b6fb46130> (a org.apache.hadoop.mapred.JobInProgress)
> at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
> - locked <0x7f2b6fb46130> (a org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
