Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (binding) - verified signatures and digests - built from source - examined CHANGES.txt for items fixed in 2.5.1 - deployed to a single-node cluster and ran some sample MR jobs Jason On 09/05/2014 07:18 PM, Karthik Kambatla wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1010/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik
Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1010/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik
[jira] [Resolved] (MAPREDUCE-6081) MapReduce should take cpu into account when doing headroom calculations
[ https://issues.apache.org/jira/browse/MAPREDUCE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev resolved MAPREDUCE-6081. -- Resolution: Duplicate MapReduce should take cpu into account when doing headroom calculations --- Key: MAPREDUCE-6081 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6081 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Currently the MapReduce AM only uses memory when doing headroom calculation as well calculations about launching reducers. It would be preferable to account for CPU as well if the scheduler on the YARN side is using CPU when scheduling. YARN-2448 lets AMs know what resources are being considered when scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Mapreduce-trunk - Build # 1892 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 34941 lines...] [INFO] hadoop-mapreduce-client ... SUCCESS [ 0.096 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [ 53.200 s] [INFO] hadoop-mapreduce-client-common SUCCESS [ 24.298 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.450 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [08:25 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [05:07 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:44 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] Apache Hadoop MapReduce Streaming . SKIPPED [INFO] Apache Hadoop Distributed Copy SKIPPED [INFO] Apache Hadoop Archives SKIPPED [INFO] Apache Hadoop Rumen ... SKIPPED [INFO] Apache Hadoop Gridmix . SKIPPED [INFO] Apache Hadoop Data Join ... SKIPPED [INFO] Apache Hadoop Extras .. SKIPPED [INFO] Apache Hadoop Pipes ... SKIPPED [INFO] Apache Hadoop OpenStack support ... SKIPPED [INFO] Apache Hadoop Azure support ... SKIPPED [INFO] Apache Hadoop Client .. SKIPPED [INFO] Apache Hadoop Mini-Cluster SKIPPED [INFO] Apache Hadoop Scheduler Load Simulator SKIPPED [INFO] Apache Hadoop Tools Dist .. SKIPPED [INFO] Apache Hadoop Amazon Web Services support . SKIPPED [INFO] Apache Hadoop Tools ... SKIPPED [INFO] Apache Hadoop Distribution SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 05:25 h [INFO] Finished at: 2014-09-10T18:45:23+00:00 [INFO] Final Memory: 78M/331M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating HADOOP-11077 Updating YARN-2526 Updating HADOOP-11057 Updating HDFS-6776 Updating YARN-1471 Updating HDFS-6482 Updating HADOOP-10925 Updating HADOOP-9989 Updating HADOOP-9374 Updating HDFS-6940 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball + successfully run apache-rat:check + checked CHANGES, LICENSE, README, NOTICE files. + built from source tarball + started pseudo cluster + run a couple of MR example jobs + basic test on HttpFS On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1010/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (binding) Ran a pseudo-distributed cluster with the Fair Scheduler and ran some MapReduce example jobs. -Sandy On Wed, Sep 10, 2014 at 12:15 PM, Alejandro Abdelnur t...@cloudera.com wrote: Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball + successfully run apache-rat:check + checked CHANGES, LICENSE, README, NOTICE files. + built from source tarball + started pseudo cluster + run a couple of MR example jobs + basic test on HttpFS On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1010/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (non-binding). Downloaded the source tarball, built binaries from the source, deployed a one-node cluster, and ran some RM examples successfully. On Wed, Sep 10, 2014 at 12:15 PM, Alejandro Abdelnur t...@cloudera.com wrote: Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball + successfully run apache-rat:check + checked CHANGES, LICENSE, README, NOTICE files. + built from source tarball + started pseudo cluster + run a couple of MR example jobs + basic test on HttpFS On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1010/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik -- Alejandro -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-6082) Excessive logging by org.apache.hadoop.util.Progress when value is NaN
Mit Desai created MAPREDUCE-6082: Summary: Excessive logging by org.apache.hadoop.util.Progress when value is NaN Key: MAPREDUCE-6082 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6082 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mit Desai Assignee: Mit Desai MAPREDUCE-5671 fixed the illegal progress values that do not fall into (0,1) interval when the progress value is given. Whenever illegal value was encountered, LOG.warn would log that incident. As a result, each of the task's syslog will be full of WARN [main] org.apache.hadoop.util.Progress: Illegal progress value found, progress is Float.NaN. Progress will be changed to 0 Each input record will contribute to one line of such log, leading to most of the tasks' syslog 1GB. We will need to change the log level to debug to avoid such excessive logging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6083) Map/Reduce dangerously adds Guava @Beta class to CryptoUtils
Christopher Tubbs created MAPREDUCE-6083: Summary: Map/Reduce dangerously adds Guava @Beta class to CryptoUtils Key: MAPREDUCE-6083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6083 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Christopher Tubbs See HDFS-7040 for more background/details. In recent 2.6.0-SNAPSHOTs, the use of LimitInputStream was added to CryptoUtils. This is part of the API components of Hadoop, which severely impacts users who were utilizing newer versions of Guava, where the @Beta and @Deprecated class, LimitInputStream, has been removed (removed in version 15 and later), beyond the impact already experienced in 2.4.0 as identified in HDFS-7040. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
RE: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (non-binding) * built from source tarball * Ran a pseudo-distributed cluster and Rand some MR example jobs * basic test on HttpFS and WebHDFS * basic test for XAttr. * checked CHANGES.txt Regards, Yi Liu -Original Message- From: Alejandro Abdelnur [mailto:t...@cloudera.com] Sent: Thursday, September 11, 2014 3:16 AM To: yarn-...@hadoop.apache.org Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.5.1 RC0 Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball + successfully run apache-rat:check + checked CHANGES, LICENSE, README, NOTICE files. + built from source tarball + started pseudo cluster + run a couple of MR example jobs + basic test on HttpFS On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-101 0/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik -- Alejandro
[jira] [Created] (MAPREDUCE-6084) Load MR Configuration from distributed cache rather than individual node
Junping Du created MAPREDUCE-6084: - Summary: Load MR Configuration from distributed cache rather than individual node Key: MAPREDUCE-6084 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6084 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Junping Du Assignee: Junping Du For rolling upgrade YARN MapReduce component, we already support to submit jobs against different versions of mapreduce jars based on distributed cache (please refer MAPREDUCE-4421). However, this could bring up some potential issues if old jar cannot find a new added class with new configurations that upgrade together with Hadoop components. For details, please refer the proposal in attachments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
