-------- Forwarded Message --------
Subject: Re: h2o failures on jenkins
Date: Fri, 17 Apr 2015 18:35:42 +0000
From: Anand Avati <[email protected]>
To: Andrew Palumbo <[email protected]>
I'm testing this patch:
commit fafa597e0637a59249d754fb5a99d9746b1032f3
Author: Anand Avati <[email protected] <mailto:[email protected]>>
Date: Fri Apr 17 11:34:45 2015 -0700
h2o-test: use unique session id strings
Signed-off-by: Anand Avati <[email protected] <mailto:[email protected]>>
diff --git
a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/Distribute
index 45c77a5..abb4289 100644
---
a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
+++
b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
@@ -30,7 +30,7 @@ trait DistributedH2OSuite extends DistributedMahoutSuite with
LoggerConfiguratio
override protected def beforeEach() {
super.beforeEach()
- mahoutCtx = mahoutH2OContext("mah2out")
+ mahoutCtx = mahoutH2OContext("mah2out" + System.currentTimeMillis())
}
override protected def afterEach() {
On Fri, 17 Apr 2015 at 11:31 Anand Avati <[email protected]
<mailto:[email protected]>> wrote:
That might be a quick fix. Just FYI - for a multi-node test
(whenever we start testing that), all nodes of a session should
share the exact string.
On Fri, 17 Apr 2015 at 11:27 Andrew Palumbo <[email protected]
<mailto:[email protected]>> wrote:
I suppose than an easy fix would be to name the session
something like "mah2out"+getcurrenttimemilliseconds () or
something like that.
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Anand Avati
Date:04/17/2015 2:23 PM (GMT-05:00)
To: Andrew Palumbo
Subject: Re: h2o failures on jenkins
It is quite possible that the two tests (based on hadoop2 and
hadoop1) are accidentally discovering each other when run in
parallel. I think giving each test a separate session id might
fix this problem. Currently the session id string "mah2out" is
hardcoded in the test/ code. If that can be somehow set to a
unique string at runtime (through env variable?), that might do
the trick. Thoughts on what's the best way we could try this?
On Fri, 17 Apr 2015 at 10:57 Andrew Palumbo <[email protected]
<mailto:[email protected]>> wrote:
Hey Anand,
The h2o Module has been failing regularly in the Jenkins
build. It
seems to fail about every other build test. (i.e. it does
still pass).
Jenkins was recently moved to a new server around the same
time that we
started getting this. We also are running tests for 2
profiles now,
hadoop 1 and hadoop 2, which made me think that its possible
that they
are discovering each other and running as a 2 node cloud
(just a theory)?
I was thinking maybe we should have a different test context
for h2o
which does not call
H2O.joinOthers();
Not sure if this is the answer though. Any thoughts?
[32mDistributedDecompositionsSuite: [0m
04-17 17:31:31.521 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: -----
H2O started -----
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git branch: master
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git hash: f6be02b33d23e5be437bbd1b4055c0f1955e1083
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git describe: RELEASE-0.1.24-47-gf6be02b
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
project version: 0.1.25.99999
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
by: 'michal'
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
on: '2015-02-05 10:44:16'
04-17 17:31:31.578 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
availableProcessors: 16
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
heap totalMemory: 77.4 MB
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
heap maxMemory: 910.3 MB
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
version: Java 1.7.0_25 (from Oracle Corporation)
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: OS
version: Linux 3.13.0-36-lowlatency (i386)
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: docker0 (docker0),
fe80:0:0:0:5484:7aff:fefe:9799%4
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: docker0 (docker0), 172.17.42.1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: em1 (em1), fe80:0:0:0:7ae3:b5ff:fe17:8984%2
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: em1 (em1), 67.195.81.190
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: lo (lo), 127.0.0.1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Multiple local IPs detected:
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
/172.17.42.1 <http://172.17.42.1> /67.195.81.190
<http://67.195.81.190>
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Attempting to determine correct address...
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Using
/67.195.81.190 <http://67.195.81.190>
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Internal communication uses port: 54322
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Listening for HTTP and REST traffic
onhttp://67.195.81.190:54321/
04-17 <http://67.195.81.190:54321/04-17> 17:31:31.583
67.195.81.190:54321 <http://67.195.81.190:54321> 17293
#onsSuite INFO: H2O cloud name: 'mah2out' on
/67.195.81.190:54321 <http://67.195.81.190:54321>, discovery
address /229.95.69.48:58719 <http://229.95.69.48:58719>
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: If you
have trouble connecting, try SSH tunneling from your local
machine (e.g., via port 55555):
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: 1.
Open a terminal and run 'ssh -L
55555:localhost:[email protected]
<mailto:55555%3Alocalhost%[email protected]>'
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: 2.
Point your browser tohttp://localhost:55555
04-17 17:31:31.682 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Log
dir: '/tmp/h2o-jenkins/h2ologs'
04-17 17:31:31.682 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Cur
dir:<https://builds.apache.org/job/Mahout-Quality/ws/h2o>
04-17 17:31:31.813 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Cloud
of size 1 formed [/67.195.81.190:54321
<http://67.195.81.190:54321>]
04-17 17:31:31.993 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -21
04-17 17:31:36.009 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -17
04-17 17:31:37.016 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -16
04-17 17:31:38.023 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -15
04-17 17:31:41.038 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -12
04-17 17:31:42.043 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -11
04-17 17:31:43.048 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -10
04-17 17:31:46.061 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -7
04-17 17:31:47.067 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -6
04-17 17:31:48.071 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -5
04-17 17:31:51.081 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -2
04-17 17:31:52.105 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -1
04-17 17:31:56.118 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: Out of
Memory and no swap space left from /67.195.81.188:54321
<http://67.195.81.188:54321>
04-17 17:31:56.118 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv FATAL: Exiting.
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Skipping Apache Mahout
[INFO] This project has been banned from the build due to
previous failures.
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Skipping Mahout Build Tools
[INFO] This project has been banned from the build due to
previous failures.
[INFO]
------------------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Mahout Build Tools ................................
SUCCESS [1.989s]
[INFO] Apache Mahout .....................................
SUCCESS [0.206s]
[INFO] Mahout Math .......................................
SUCCESS [1:14.339s]
[INFO] Mahout HDFS .......................................
SUCCESS [3.378s]
[INFO] Mahout Map-Reduce .................................
SUCCESS [12:50.243s]
[INFO] Mahout Integration ................................
SUCCESS [54.793s]
[INFO] Mahout Examples ...................................
SUCCESS [22.309s]
[INFO] Mahout Math Scala bindings ........................
SUCCESS [1:36.708s]
[INFO] Mahout H2O backend ................................
FAILURE [43.843s]
[INFO] Mahout Spark bindings .............................
SKIPPED
[INFO] Mahout Spark bindings shell .......................
SKIPPED
[INFO] Mahout Release Package ............................
SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 17:50.229s
[INFO] Finished at: Fri Apr 17 17:31:56 UTC 2015
[INFO] Final Memory: 57M/438M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.scalatest:scalatest-maven-plugin:1.0:test (test) on
project mahout-h2o_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run
Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full
debug logging.
[ERROR]
[ERROR] For more information about the errors and possible
solutions, please read the following articles:
[ERROR] [Help
1]http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the
build with the command
[ERROR] mvn <goals> -rf :mahout-h2o_2.10
Build step 'Invoke top-level Maven targets' marked build as
failure
[PMD] Skipping publisher since build result is FAILURE
[TASKS] Skipping publisher since build result is FAILURE
