[jira] [Resolved] (MAPREDUCE-1413) Improve logging of progress/errors in the job and task trackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-1413. --- Resolution: Won't Fix > Improve logging of progress/errors in the job and task trackers > --- > > Key: MAPREDUCE-1413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.203.0, 0.22.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Trivial > Fix For: 0.22.1 > > Attachments: MAPREDUCE-1413.patch, MAPREDUCE-1413.patch > > > I have code that improves the logging of the trackers as they start stop and > fail, through > # More logging of events > # including exception strings and stacks when things go wrong > People's whose JTs and TTs aren't behaving may appreciate this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-512) Add tests for the DFS HTML and JSP pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-512. -- Resolution: Won't Fix > Add tests for the DFS HTML and JSP pages > > > Key: MAPREDUCE-512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-512 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Original Estimate: 1h > Remaining Estimate: 1h > > Add some basic tests to look for the standard JSP pages on a locally deployed > MiniMR cluster > 1. namenode: check that dfshealth is present > 2. datanode: check that all the datanode JSPs load > 3. GET the standard servlets. > The initial checks can just use httpclient to GET the pages; no need (yet) > for HtmlUnit. > If the tests were designed to take optional URLs (e.g test.namenode.url and > test.datanode.url) they could be run against processes brought up > externally/remotely > They would > * help test that the JSP pages are being compiled down and bundled into the > JARS > * verify the classpath is getting set up right > * check that the Jasper engine is working > * check the servlets are all registering > I've effectively had to do this in my own code; having a set of these tests > inside hadoop would make it easier to point the blame at the classpath setup > or something else. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6228) Add truncate operation to SLive
Konstantin Shvachko created MAPREDUCE-6228: -- Summary: Add truncate operation to SLive Key: MAPREDUCE-6228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6228 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Reporter: Konstantin Shvachko Add truncate into the mix of operations for SLive test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6227) DFSIO for truncate
Konstantin Shvachko created MAPREDUCE-6227: -- Summary: DFSIO for truncate Key: MAPREDUCE-6227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6227 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, test Reporter: Konstantin Shvachko Create a benchmark and a test for truncate within the framework of TestDFSIO. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6220) To forbid stderr and stdout for MapReduce job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-6220. - Resolution: Not a Problem > To forbid stderr and stdout for MapReduce job > - > > Key: MAPREDUCE-6220 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6220 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.6.0 >Reporter: Yang Hao >Assignee: Yang Hao > Attachments: MAPREDUCE-6220.patch > > > System.out and System.error is a ugly way to print log, and many times it > would do harm to Hadoop cluster. So we can forbid it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6226) Findbugs warnings in mapreduce-client-core
Varun Saxena created MAPREDUCE-6226: --- Summary: Findbugs warnings in mapreduce-client-core Key: MAPREDUCE-6226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6226 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena Attachments: newPatchFindbugsWarningshadoop-mapreduce-client-core.html Found findbugs warnings reported for mapreduce-client-core while fixing MAPREDUCE-6223 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6225) Fix new findbug warnings in hadoop-mapreduce-client-core
Jason Lowe created MAPREDUCE-6225: - Summary: Fix new findbug warnings in hadoop-mapreduce-client-core Key: MAPREDUCE-6225 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6225 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Jason Lowe Recent precommit builds in hadoop-mapreduce-client-core are flagging findbug warnings that appear to be new with the recent findbugs upgrade. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Mapreduce-trunk - Build # 2036 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2036/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 30989 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.721 sec - in org.apache.hadoop.ipc.TestMRCJCSocketFactory Results : Failed tests: TestJobConf.testNegativeValueForTaskVmem:111 expected:<1024> but was:<-1> TestLargeSort.testLargeSort:61 Large sort failed for 128 expected:<0> but was:<1> Tests run: 519, Failures: 2, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.991 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [01:28 min] [INFO] hadoop-mapreduce-client-common SUCCESS [ 32.093 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.840 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [09:46 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [05:30 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:40 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:57 h [INFO] Finished at: 2015-01-26T15:24:19+00:00 [INFO] Final Memory: 31M/619M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #2031 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20275081 bytes Compression is 0.0% Took 5.2 sec Recording test results Updating YARN-3024 Updating HADOOP-11450 Updating HADOOP-11419 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 2 tests failed. FAILED: org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem Error Message: expected:<1024> but was:<-1> Stack Trace: java.lang.AssertionError: expected:<1024> but was:<-1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111) FAILED: org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort Error Message: Large sort failed for 128 expected:<0> but was:<1> Stack Trace: java.lang.AssertionError: Large sort failed for 128 expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort(TestLargeSort.java:61)
Hadoop-Mapreduce-trunk-Java8 - Build # 86 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/86/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 10555 lines...] Results : Failed tests: TestJobConf.testNegativeValueForTaskVmem:111 expected:<1024> but was:<-1> TestLargeSort.testLargeSort:61 Large sort failed for 128 expected:<0> but was:<1> TestCombineFileInputFormat.testSplitPlacementForCompressedFiles:911 expected:<2> but was:<1> TestCombineFileInputFormat.testSplitPlacement:368 expected:<2> but was:<1> Tests run: 519, Failures: 4, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.066 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [01:58 min] [INFO] hadoop-mapreduce-client-common SUCCESS [ 29.867 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 3.870 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [10:37 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [06:09 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:37 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:56 h [INFO] Finished at: 2015-01-26T15:04:08+00:00 [INFO] Final Memory: 31M/158M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk-Java8 #28 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20283243 bytes Compression is 0.0% Took 5.3 sec Recording test results Updating YARN-3024 Updating HADOOP-11450 Updating HADOOP-11419 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 4 tests failed. FAILED: org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem Error Message: expected:<1024> but was:<-1> Stack Trace: java.lang.AssertionError: expected:<1024> but was:<-1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111) FAILED: org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort Error Message: Large sort failed for 128 expected:<0> but was:<1> Stack Trace: java.lang.AssertionError: Large sort failed for 128 expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort(TestLargeSort.java:61) FAILED: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles Error Message: expected:<2> but was:<1> Stack Trace: junit.framework.AssertionFailedError: expected:<2> but was:<1> at junit.framework.Assert.fail(Assert.java:57) at junit
[jira] [Reopened] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao reopened MAPREDUCE-5718: - > MR job will fail after commit fail > -- > > Key: MAPREDUCE-5718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.3.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: ha > Attachments: mr-5718-0.patch > > > when any of this happens: > * While testing RM HA, if the RM fails over while an MR AM is in the middle > of a commit, > * When testing preempting, if the MR AM fails over during the middle of a > commit > the subsequent AM gets spawned but dies with a diagnostic message - "We > crashed durring a commit". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
