Or, alternatively, the bus could catch that error and ignore / log it, instead of stopping the context...
On Wed, Jul 15, 2015 at 12:20 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hmm, the Java listener was added in 1.3, so I think it will work for my > needs. > > Might be worth it to make it clear in the SparkListener documentation that > people should avoid using it directly. Or follow Reynold's suggestion. > > On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell <pwend...@gmail.com> > wrote: > >> One related note here is that we have a Java version of this that is >> an abstract class - in the doc it says that it exists more or less to >> allow for binary compatibility (it says it's for Java users, but >> really Scala could use this also): >> >> >> https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 >> >> I think it might be reasonable that the Scala trait provides only >> source compatibitly and the Java class provides binary compatibility. >> >> - Patrick >> >> On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin <van...@cloudera.com> >> wrote: >> > Hey all, >> > >> > Just noticed this when some of our tests started to fail. SPARK-4072 >> added a >> > new method to the "SparkListener" trait, and even though it has a >> default >> > implementation, it doesn't seem like that applies retroactively. >> > >> > Namely, if you have an existing, compiled app that has an >> implementation of >> > SparkListener, that app won't work on 1.5 without a recompile. You'll >> get >> > something like this: >> > >> > java.lang.AbstractMethodError >> > at >> > >> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) >> > at >> > >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> > at >> > >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> > at >> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) >> > at >> > >> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) >> > at >> > >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) >> > at >> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) >> > at >> > >> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) >> > >> > >> > Now I know that "SparkListener" is marked as @DeveloperApi, but is this >> > something we should care about? Seems like adding methods to traits is >> just >> > as backwards-incompatible as adding new methods to Java interfaces. >> > >> > >> > -- >> > Marcelo >> > > > > -- > Marcelo > -- Marcelo
