spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite
Repository: spark
Updated Branches:
refs/heads/master b293afc42 - 85cf06368
[SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running
FlumeStreamSuite
When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows.
sbt.ForkMain$ForkError: The code passed to eventually never returned
normally. Attempted 52 times over 10.094849836 seconds. Last failure message:
Error connecting to localhost/127.0.0.1:23456.
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
This error is caused by check-then-act logic when it find free-port .
/** Find a free port */
private def findFreePort(): Int = {
Utils.startServiceOnPort(23456, (trialPort: Int) = {
val socket = new ServerSocket(trialPort)
socket.close()
(null, trialPort)
}, conf)._2
}
Removing the check-then-act is not easy but we can reduce the chance of having
the error by choosing random value for initial port instead of 23456.
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #4337 from sarutak/SPARK-5559 and squashes the following commits:
16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort
c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark
into SPARK-5559
1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
33357e3 [Kousuke Saruta] Changed findFreePort method in MQTTStreamSuite and
FlumeStreamSuite so that it can choose valid random port
a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from
23456 to random value
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/85cf0636
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/85cf0636
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/85cf0636
Branch: refs/heads/master
Commit: 85cf0636825d1997d64d0bdc04618f29b7222da1
Parents: b293afc
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Authored: Tue Mar 24 16:13:25 2015 +
Committer: Sean Owen so...@cloudera.com
Committed: Tue Mar 24 16:20:52 2015 +
--
core/src/main/scala/org/apache/spark/util/Utils.scala | 4
.../org/apache/spark/streaming/flume/FlumeStreamSuite.scala | 5 +++--
.../scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala | 4 +++-
3 files changed, 10 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/85cf0636/core/src/main/scala/org/apache/spark/util/Utils.scala
--
diff --git
spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite
Repository: spark
Updated Branches:
refs/heads/branch-1.3 e5451432e - 8722369c2
[SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running
FlumeStreamSuite
When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows.
sbt.ForkMain$ForkError: The code passed to eventually never returned
normally. Attempted 52 times over 10.094849836 seconds. Last failure message:
Error connecting to localhost/127.0.0.1:23456.
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
This error is caused by check-then-act logic when it find free-port .
/** Find a free port */
private def findFreePort(): Int = {
Utils.startServiceOnPort(23456, (trialPort: Int) = {
val socket = new ServerSocket(trialPort)
socket.close()
(null, trialPort)
}, conf)._2
}
Removing the check-then-act is not easy but we can reduce the chance of having
the error by choosing random value for initial port instead of 23456.
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #4337 from sarutak/SPARK-5559 and squashes the following commits:
16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort
c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark
into SPARK-5559
1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
33357e3 [Kousuke Saruta] Changed findFreePort method in MQTTStreamSuite and
FlumeStreamSuite so that it can choose valid random port
a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from
23456 to random value
(cherry picked from commit 85cf0636825d1997d64d0bdc04618f29b7222da1)
Signed-off-by: Sean Owen so...@cloudera.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8722369c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8722369c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8722369c
Branch: refs/heads/branch-1.3
Commit: 8722369c24991d050e8b2f55c271eeab06005fe7
Parents: e545143
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Authored: Tue Mar 24 16:13:25 2015 +
Committer: Sean Owen so...@cloudera.com
Committed: Tue Mar 24 16:22:07 2015 +
--
core/src/main/scala/org/apache/spark/util/Utils.scala | 4
.../org/apache/spark/streaming/flume/FlumeStreamSuite.scala | 5 +++--
.../scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala | 4 +++-
3 files changed, 10 insertions(+), 3 deletions(-)
--
spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite
Repository: spark
Updated Branches:
refs/heads/branch-1.2 e080cc3e5 - 8ef69957f
[SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running
FlumeStreamSuite
When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows.
sbt.ForkMain$ForkError: The code passed to eventually never returned
normally. Attempted 52 times over 10.094849836 seconds. Last failure message:
Error connecting to localhost/127.0.0.1:23456.
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307)
at
org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116)
at
org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66)
at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
This error is caused by check-then-act logic when it find free-port .
/** Find a free port */
private def findFreePort(): Int = {
Utils.startServiceOnPort(23456, (trialPort: Int) = {
val socket = new ServerSocket(trialPort)
socket.close()
(null, trialPort)
}, conf)._2
}
Removing the check-then-act is not easy but we can reduce the chance of having
the error by choosing random value for initial port instead of 23456.
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #4337 from sarutak/SPARK-5559 and squashes the following commits:
16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort
c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark
into SPARK-5559
1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
33357e3 [Kousuke Saruta] Changed findFreePort method in MQTTStreamSuite and
FlumeStreamSuite so that it can choose valid random port
a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark
into SPARK-5559
8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from
23456 to random value
(cherry picked from commit 85cf0636825d1997d64d0bdc04618f29b7222da1)
Signed-off-by: Sean Owen so...@cloudera.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8ef69957
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8ef69957
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8ef69957
Branch: refs/heads/branch-1.2
Commit: 8ef69957fb3735045f3b2ea9d54913936377ac62
Parents: e080cc3
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Authored: Tue Mar 24 16:13:25 2015 +
Committer: Sean Owen so...@cloudera.com
Committed: Tue Mar 24 16:22:22 2015 +
--
core/src/main/scala/org/apache/spark/util/Utils.scala | 4
.../org/apache/spark/streaming/flume/FlumeStreamSuite.scala | 5 +++--
.../scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala | 4 +++-
3 files changed, 10 insertions(+), 3 deletions(-)
--
