Re: 2.7 status
On 17 February 2015 at 21:52:47, Colin McCabe (cmcc...@alumni.cmu.edumailto:cmcc...@alumni.cmu.edu) wrote: Re: building Windows binaries. Do we release binaries for all the Linux and UNIX architectures? I thought we didn't. It seems a little inconsistent to release binaries just for Windows, but not for those other architectures and OSes. I wonder if we can improve this situation? Windows has less variance; one CPU family; if you target one version its there; if you test on that then you can say works on 64-bit server 2012. Win32 isn't handled no-one would ever use it in production (RAM limitations alone). There's clearly interest in JIRA for a version of the client-side code there There's a more subtle detail: we don't expect all windows users to have the build tools. Whereas on Linux it's pretty much all there. Now, if someone wanted to produce native libs for other platforms, and help test them, we'd probably have to think about what to do then. On that topic, Colin -can you take a look at https://issues.apache.org/jira/browse/HADOOP-10846 ; PowerPC native CRC. I Think the latest patch isn't going to have any adverse effects on x86, but more native-code reviews would help.
Re: 2.7 status
On Wed, Feb 18, 2015 at 1:49 AM, Steve Loughran ste...@hortonworks.com wrote: On 17 February 2015 at 21:52:47, Colin McCabe (cmcc...@alumni.cmu.edumailto:cmcc...@alumni.cmu.edu) wrote: Re: building Windows binaries. Do we release binaries for all the Linux and UNIX architectures? I thought we didn't. It seems a little inconsistent to release binaries just for Windows, but not for those other architectures and OSes. I wonder if we can improve this situation? Windows has less variance; one CPU family; if you target one version its there; if you test on that then you can say works on 64-bit server 2012. Win32 isn't handled no-one would ever use it in production (RAM limitations alone). There's clearly interest in JIRA for a version of the client-side code there There's a more subtle detail: we don't expect all windows users to have the build tools. Whereas on Linux it's pretty much all there. Now, if someone wanted to produce native libs for other platforms, and help test them, we'd probably have to think about what to do then. I guess one more issue is that Windows requires the native bits, whereas Linux does not. I suppose that's a good argument for doing it. On that topic, Colin -can you take a look at https://issues.apache.org/jira/browse/HADOOP-10846 ; PowerPC native CRC. I Think the latest patch isn't going to have any adverse effects on x86, but more native-code reviews would help. I'll try to take a look tomorrow or maybe Friday. Long backlog right now. C.
[jira] [Resolved] (MAPREDUCE-6203) if log-aggregation is enable and run some MR job, the AM log will be aggregated.then disable it,then in MR JobHistoryServer and YARN RM UI link,the AM log cannot be
[ https://issues.apache.org/jira/browse/MAPREDUCE-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved MAPREDUCE-6203. - Resolution: Not a Problem if log-aggregation is enable and run some MR job, the AM log will be aggregated.then disable it,then in MR JobHistoryServer and YARN RM UI link,the AM log cannot be visible Key: MAPREDUCE-6203 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6203 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: huangyitian Priority: Minor as the Summary description, I think for the MR JobHistory Server, should not let “yarn.log-aggregation-enable” affect the history job,even “yarn.log-aggregation-enable” is false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Mapreduce-trunk-Java8 - Build # 109 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/109/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 10603 lines...] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 3.974 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [10:26 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [06:04 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:40 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:58 h [INFO] Finished at: 2015-02-18T15:06:17+00:00 [INFO] Final Memory: 33M/149M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk-Java8 #28 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20303130 bytes Compression is 0.0% Took 5.5 sec Recording test results Updating HADOOP-11522 Updating HDFS-7795 Updating HADOOP-11521 Updating HDFS-7797 Updating HADOOP-11570 Updating HADOOP-11593 Updating HADOOP-11596 Updating HADOOP-11599 Updating MAPREDUCE-6234 Updating MAPREDUCE-4286 Updating MAPREDUCE-6260 Updating HADOOP-11575 Updating HDFS-7780 Updating HDFS-6662 Updating YARN-3207 Updating HDFS-4266 Updating HADOOP-11295 Updating MAPREDUCE-5785 Updating HDFS-7803 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 3 tests failed. FAILED: org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem Error Message: expected:1024 but was:-1 Stack Trace: java.lang.AssertionError: expected:1024 but was:-1 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111) FAILED: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles Error Message: expected:2 but was:1 Stack Trace: junit.framework.AssertionFailedError: expected:2 but was:1 at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911) FAILED: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement Error Message: expected:2 but was:1 Stack Trace: junit.framework.AssertionFailedError: expected:2 but was:1 at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at
Hadoop-Mapreduce-trunk - Build # 2059 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2059/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31136 lines...] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.790 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [01:29 min] [INFO] hadoop-mapreduce-client-common SUCCESS [ 28.415 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.781 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [09:22 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [05:25 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:46 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:03 h [INFO] Finished at: 2015-02-18T15:30:16+00:00 [INFO] Final Memory: 33M/736M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #2031 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 20299592 bytes Compression is 0.0% Took 5.6 sec Recording test results Updating HADOOP-11522 Updating HDFS-7795 Updating HADOOP-11521 Updating HDFS-7797 Updating HADOOP-11570 Updating HADOOP-11593 Updating HADOOP-11596 Updating HADOOP-11599 Updating MAPREDUCE-6234 Updating MAPREDUCE-4286 Updating MAPREDUCE-6260 Updating HADOOP-11575 Updating HDFS-7780 Updating HDFS-6662 Updating YARN-3207 Updating HDFS-4266 Updating HADOOP-11295 Updating MAPREDUCE-5785 Updating HDFS-7803 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem Error Message: expected:1024 but was:-1 Stack Trace: java.lang.AssertionError: expected:1024 but was:-1 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
[jira] [Created] (MAPREDUCE-6263) Large jobs can lose history when killed due to brief client timeout
Jason Lowe created MAPREDUCE-6263: - Summary: Large jobs can lose history when killed due to brief client timeout Key: MAPREDUCE-6263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.6.0 Reporter: Jason Lowe YARNRunner connects to the AM to send the kill job command then waits a hardcoded 10 seconds for the job to enter a terminal state. If the job fails to enter a terminal state in that time then YARNRunner will tell YARN to kill the application forcefully. The latter type of kill usually results in no job history, since the AM process is killed forcefully. Ten seconds can be too short for large jobs in a large cluster, as it takes time to connect to all the nodemanagers, process the state machine events, and copy a large jhist file. The timeout should be more lenient or configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6264) Remove httpclient dependency from hadoop-mapreduce-client
Akira AJISAKA created MAPREDUCE-6264: Summary: Remove httpclient dependency from hadoop-mapreduce-client Key: MAPREDUCE-6264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6264 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Akira AJISAKA Priority: Minor Sub-task of HADOOP-10105. Remove httpclient dependency from TestAMWebApp.java and JobEndNotifier.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
