[VOTE] Release Apache Hadoop 2.2.0
Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Hadoop-Mapreduce-trunk - Build # 1571 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1571/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 34350 lines...] TestBlockLimits.testWithLimits:53-runCustomFormat:67 » OutOfMemory unable to ... TestJobSysDirWithDFS.testWithDFS:130 » YarnRuntime java.lang.OutOfMemoryError:... TestMerge.testMerge:81 » YarnRuntime java.lang.OutOfMemoryError: unable to cre... TestNetworkedJob.testGetJobStatus:113 » IO Cannot run program stat: java.io TestNetworkedJob.testNetworkedJob:133 » IO Cannot run program stat: java.io TestNetworkedJob.testJobQueueClient:319 » IO Cannot run program stat: java.i... TestMiniMRChildTask.setup:320 » IO Cannot run program stat: java.io.IOExcept... TestReduceFetchFromPartialMem$1TestSetup.run:27-setUp:61 » YarnRuntime java TestLazyOutput.testLazyOutput:146 » YarnRuntime java.lang.OutOfMemoryError: un... TestJobNameClusterMapReduceTestCase.setUp:56-ClusterMapReduceTestCase.startCluster:86 » YarnRuntime TestJobNameClusterMapReduceTestCase.setUp:56-ClusterMapReduceTestCase.startCluster:86 » YarnRuntime Tests run: 166, Failures: 4, Errors: 17, Skipped: 6 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.521s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [40.313s] [INFO] hadoop-mapreduce-client-common SUCCESS [24.872s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [2.421s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [6:47.692s] [INFO] hadoop-mapreduce-client-hs SUCCESS [1:58.974s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [48:27.027s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 58:24.481s [INFO] Finished at: Mon Oct 07 14:17:33 UTC 2013 [INFO] Final Memory: 24M/93M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? [ERROR] Command was/bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient /home/jenkins/tools/java/jdk1.6.0_26/jre/bin/java -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter2307803079229197403.jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire2882592282491910928tmp /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1651119122714291817168tmp [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating HADOOP-10022 Updating YARN-1278 Updating HDFS-5299 Updating YARN-1277 Updating MAPREDUCE-5562 Updating HDFS-4817 Updating HDFS-5259 Updating YARN-1149 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: [VOTE] Release Apache Hadoop 2.2.0
I ran some basic sanity tests and it seems OK to me. +1 On 10/7/13 2:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5569) FloatSplitter is not generating correct splits
Nathan Roberts created MAPREDUCE-5569:
-
Summary: FloatSplitter is not generating correct splits
Key: MAPREDUCE-5569
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5569
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.0-beta, trunk, 1.3.0
Reporter: Nathan Roberts
Assignee: Nathan Roberts
The closing split is not calculated correctly:
{code}
// Catch any overage and create the closed interval for the last split.
if (curLower = maxVal || splits.size() == 1) {
splits.add(new DataDrivenDBInputFormat.DataDrivenDBInputSplit(
- lowClausePrefix + Double.toString(curUpper),
+ lowClausePrefix + Double.toString(curLower),
colName + = + Double.toString(maxVal)));
}
{code}
For the case of min=5.0, max=7.0, 2 splits, the current code returns splits of
(column1 =5.0, column1 6.0), (column1 =7.0, column1 =7.0). The second split
is obviously not correct.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
Jason Lowe created MAPREDUCE-5570: - Summary: Map task attempt with fetch failure has incorrect attempt finish time Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 2.1.1-beta, 0.23.9 Reporter: Jason Lowe If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [VOTE] Release Apache Hadoop 2.2.0
On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 Great! As usual, Bigtop is now building the entire Hadoop ecosystem on top of this RC: http://bigtop01.cloudera.org:8080/view/Upstream-tests/job/Hadoop-2.2.0/ Feel free to grab packages from there once the job is done. I will be also deploying clusters and running the usual Bigtop battery of tests on them. Stay tuned! Thanks, Roman.
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 for the release. I have deployed a multinode cluster and extensively tested MR speculative execution, YARN CLI and YARN distributed shell. There were couple of issues I encountered while testing MAPREDUCE-5533, YARN-1168, YARN-1167, YARN-1157, YARN-1131, YARN-1118, YARN-1117 and all of them have been fixed. Thanks, Tassapol On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 * downloaded source tarball * verified MD5 * verified signature * verified CHANGES.txt files, release # and date * run 'mvn apache-rat:check' successfully * built distribution * setup speudo cluster * started HDFS/YARN * run some HTTFS tests * run a couple of MR examples * run a few tests using Llama AM On Mon, Oct 7, 2013 at 12:07 PM, Tassapol Athiapinya tathiapi...@hortonworks.com wrote: +1 for the release. I have deployed a multinode cluster and extensively tested MR speculative execution, YARN CLI and YARN distributed shell. There were couple of issues I encountered while testing MAPREDUCE-5533, YARN-1168, YARN-1167, YARN-1157, YARN-1131, YARN-1118, YARN-1117 and all of them have been fixed. Thanks, Tassapol On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 Deployed on secure and nonsecure clusters. Ran some hdfs, MR and yarn tests. Seems good to me. On Mon, Oct 7, 2013 at 12:21 PM, Alejandro Abdelnur t...@cloudera.comwrote: +1 * downloaded source tarball * verified MD5 * verified signature * verified CHANGES.txt files, release # and date * run 'mvn apache-rat:check' successfully * built distribution * setup speudo cluster * started HDFS/YARN * run some HTTFS tests * run a couple of MR examples * run a few tests using Llama AM On Mon, Oct 7, 2013 at 12:07 PM, Tassapol Athiapinya tathiapi...@hortonworks.com wrote: +1 for the release. I have deployed a multinode cluster and extensively tested MR speculative execution, YARN CLI and YARN distributed shell. There were couple of issues I encountered while testing MAPREDUCE-5533, YARN-1168, YARN-1167, YARN-1157, YARN-1131, YARN-1118, YARN-1117 and all of them have been fixed. Thanks, Tassapol On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5572) Provide alternative logic for getPos() implementation in custom RecordReader
jay vyas created MAPREDUCE-5572: --- Summary: Provide alternative logic for getPos() implementation in custom RecordReader Key: MAPREDUCE-5572 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5572 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 1.2.1, 1.2.0, 1.1.1, 1.1.0, 1.1.3, 1.2.2 Reporter: jay vyas Priority: Minor The custom RecordReader class defines the getPos() as follows: long currentOffset = currentStream == null ? 0 : currentStream.getPos(); ... This is meant to prevent errors when underlying stream is null. But it doesn't gaurantee to work: The RawLocalFileSystem, for example, currectly will close the underlying file stream once it is consumed, and the currentStream will thus throw a NullPointerException when trying to access the null stream. This is only seen when running this in the context where the MapTask class, which is only relevant in mapred.* API, calls getPos() twice in tandem, before and after reading a record. This custom record reader should be gaurded, or else eliminated, since it assumes something which is not in the FileSystem contract: That a getPos will always return a integral value. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5573) /ws/v1/mapreduce/blacklistednodes MR AM webservice link is broken.
Omkar Vinit Joshi created MAPREDUCE-5573:
Summary: /ws/v1/mapreduce/blacklistednodes MR AM webservice link
is broken.
Key: MAPREDUCE-5573
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5573
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Omkar Vinit Joshi
run a map reduce job (sleep) then Retrieve MR AM tracking url by running
yarn applications -list
try accessing
trackingurl + /ws/v1/mapreduce/blacklistednodes
{code}
RemoteException
exceptionClassCastException/exception
message
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter cannot
be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor
/message
javaClassNamejava.lang.ClassCastException/javaClassName
/RemoteException
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
