Here you can find an experimental branch of Spark for Scala 2.10.
https://github.com/MartinWeindel/incubator-spark/tree/0.9_Scala-2.10.3
I have also updated Akka to version 2.1.4.
The branch compiles with both sbt and mvn, but there are a few tests
which are failing and even more worse producing deadlocks.
Also there are a lot of warnings, most related to usage of
ClassManifest, which should be replaced with ClassTag.
But I don't think it is a good idea to fix these warnings at the moment,
as this would make merging with the master branch harder.
I would like to know about the official road map for supporting Scala 2.10.
Does it make sense to investigate the test problems in more details on
my experimental branch?
Best regards,
Martin
P.S.: Below are the failing tests (probably not complete because of the
deadlocks)
DriverSuite:
- driver should exit after finishing *** FAILED ***
TestFailedDueToTimeoutException was thrown during property
evaluation. (DriverSuite.scala:36)
Message: The code passed to failAfter did not complete within 30 seconds.
Location: (DriverSuite.scala:37)
Occurred at table row 0 (zero based, not counting headings), which
had values (
master = local
)
UISuite:
- jetty port increases under contention *** FAILED ***
java.net.BindException: Die Adresse wird bereits verwendet
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:286)
...
AccumulatorSuite:
- add value to collection accumulators *** FAILED ***
org.apache.spark.SparkException: Job failed: Task not serializable:
java.io.NotSerializableException: org.scalatest.Engine
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:762)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:760)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:555)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:502)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:360)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:440)
at
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:148)
...
- localValue readable in tasks *** FAILED ***
org.apache.spark.SparkException: Job failed: Task not serializable:
java.io.NotSerializableException: org.scalatest.Engine
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:762)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:760)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:555)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:502)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:360)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:440)
at
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:148)
...
ShuffleNettySuite:
*deadlock* on "shuffle serializer"
FileServerSuite:
*deadlock* on "Distributing files on a standalone cluster"
