Hmm, the Java listener was added in 1.3, so I think it will work for my needs.
Might be worth it to make it clear in the SparkListener documentation that people should avoid using it directly. Or follow Reynold's suggestion. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell <pwend...@gmail.com> wrote: > One related note here is that we have a Java version of this that is > an abstract class - in the doc it says that it exists more or less to > allow for binary compatibility (it says it's for Java users, but > really Scala could use this also): > > > https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 > > I think it might be reasonable that the Scala trait provides only > source compatibitly and the Java class provides binary compatibility. > > - Patrick > > On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin <van...@cloudera.com> > wrote: > > Hey all, > > > > Just noticed this when some of our tests started to fail. SPARK-4072 > added a > > new method to the "SparkListener" trait, and even though it has a default > > implementation, it doesn't seem like that applies retroactively. > > > > Namely, if you have an existing, compiled app that has an implementation > of > > SparkListener, that app won't work on 1.5 without a recompile. You'll get > > something like this: > > > > java.lang.AbstractMethodError > > at > > > org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) > > at > > > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > > at > > > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > > at > org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) > > at > > > org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) > > at > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) > > at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) > > at > > > org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) > > > > > > Now I know that "SparkListener" is marked as @DeveloperApi, but is this > > something we should care about? Seems like adding methods to traits is > just > > as backwards-incompatible as adding new methods to Java interfaces. > > > > > > -- > > Marcelo > -- Marcelo
