Re: Error in invoking a custom StandaloneRecoveryModeFactory in java env (Spark v1.3.0)
Hi Josh, I tried using the spark 1.4.0 upgrade. here is the class I'm trying to use package org.wso2.carbon.analytics.spark.core.util.master import akka.serialization.Serialization import org.apache.spark.SparkConf import org.apache.spark.deploy.master._ class AnalyticsRecoveryModeFactoryScala(conf: SparkConf, serializer: Serialization) extends StandaloneRecoveryModeFactory(conf, serializer) { override def createPersistenceEngine(): PersistenceEngine = new AnalyticsPersistenceEngine(conf, serializer) override def createLeaderElectionAgent(master: LeaderElectable): LeaderElectionAgent = new AnalyticsLeaderElectionAgent(master) } object AnalyticsRecoveryModeFactoryScala { } when I invoke this factory from the master, I get a similar error as before [2015-07-05 17:06:55,384] ERROR {akka.actor.OneForOneStrategy} - org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.(org.apache.spark.SparkConf, akka.serialization.Serialization) akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.NoSuchMethodException: org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.(org.apache.spark.SparkConf, akka.serialization.Serialization) at java.lang.Class.getConstructor0(Class.java:2810) at java.lang.Class.getConstructor(Class.java:1718) at org.apache.spark.deploy.master.Master.preStart(Master.scala:168) at akka.actor.Actor$class.aroundPreStart(Actor.scala:470) at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52) at akka.actor.ActorCell.create(ActorCell.scala:580) ... 9 more what could be the reason for this? rgds On Thu, Jun 25, 2015 at 11:42 AM, Niranda Perera wrote: > thanks Josh. > > this looks very similar to my problem. > > On Thu, Jun 25, 2015 at 11:32 AM, Josh Rosen wrote: > >> This sounds like https://issues.apache.org/jira/browse/SPARK-7436, which >> has been fixed in Spark 1.4+ and in branch-1.3 (for Spark 1.3.2). >> >> On Wed, Jun 24, 2015 at 10:57 PM, Niranda Perera < >> niranda.per...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'm trying to implement a custom StandaloneRecoveryModeFactory in the >>> Java environment. Pls find the implementation here. [1] . I'm new to Scala, >>> hence I'm trying to use Java environment as much as possible. >>> >>> when I start a master with spark.deploy.recoveryMode.factory property to >>> be "CUSTOM", I encounter a NoSuchMethodException for my custom class's >>> constructor. >>> it has the following constructor. >>> >>> public AnalyticsStandaloneRecoveryModeFactory(SparkConf conf, >>> Serialization serializer) >>> >>> but from the Master, it looks for a constructor for, >>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.(org.apache.spark.SparkConf, >>> akka.serialization.Serialization$) >>> >>> I see in the Spark source code for Master, that it uses reflection to >>> get the custom recovery mode factory class. >>> >>> case "CUSTOM" => >>> val clazz = >>> Class.forName(conf.get("spark.deploy.recoveryMode.factory")) >>> val factory = clazz.getConstructor(conf.getClass, >>> Serialization.getClass) >>> .newInstance(conf, SerializationExtension(context.system)) >>> .asInstanceOf[StandaloneRecoveryModeFactory] >>> (factory.createPersistenceEngine(), >>> factory.createLeaderElectionAgent(this)) >>> >>> here, Serialization.getClass returns a akka.serialization.Serialization$ >>> object, where as my custom class's constructor >>> accepts akka.serialization.Serialization object. >>> >>> so I would like to know, >>> 1. if this happens because I'm using this in the Java environment? >>> 2. what is the workaround to this? >>> >>> thanks >>> >>> Please find the full stack trace of the error below. >>> >>> [2015-06-25 10:59:01,095] ERROR {akka.actor.OneForOneStrategy} - >>> >>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.(org.apache.spark.SparkConf, >>> akka.serialization.Serialization$) >>> akka.actor.ActorInitializationException: exception during creation >>> at akka.actor.ActorInitializationException$.apply(Actor.scala:164) >>> at akka.actor.ActorCell.create(ActorCell.scala:59
Re: Error in invoking a custom StandaloneRecoveryModeFactory in java env (Spark v1.3.0)
Hi, Sorry this was a class loading issue at my side. Sorted it out. Sorry if I caused any inconvenience Rgds Niranda Perera +94 71 554 8430 On Jul 5, 2015 17:08, "Niranda Perera" wrote: > Hi Josh, > > I tried using the spark 1.4.0 upgrade. > > here is the class I'm trying to use > > package org.wso2.carbon.analytics.spark.core.util.master > > import akka.serialization.Serialization > import org.apache.spark.SparkConf > import org.apache.spark.deploy.master._ > > class AnalyticsRecoveryModeFactoryScala(conf: SparkConf, serializer: > Serialization) > extends StandaloneRecoveryModeFactory(conf, serializer) { > > override def createPersistenceEngine(): PersistenceEngine = new > AnalyticsPersistenceEngine(conf, serializer) > > override def createLeaderElectionAgent(master: LeaderElectable): > LeaderElectionAgent = new > AnalyticsLeaderElectionAgent(master) > } > > object AnalyticsRecoveryModeFactoryScala { > > } > > when I invoke this factory from the master, I get a similar error as > before > > [2015-07-05 17:06:55,384] ERROR {akka.actor.OneForOneStrategy} - > > org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.(org.apache.spark.SparkConf, > akka.serialization.Serialization) > akka.actor.ActorInitializationException: exception during creation > at akka.actor.ActorInitializationException$.apply(Actor.scala:164) > at akka.actor.ActorCell.create(ActorCell.scala:596) > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.NoSuchMethodException: > org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.(org.apache.spark.SparkConf, > akka.serialization.Serialization) > at java.lang.Class.getConstructor0(Class.java:2810) > at java.lang.Class.getConstructor(Class.java:1718) > at org.apache.spark.deploy.master.Master.preStart(Master.scala:168) > at akka.actor.Actor$class.aroundPreStart(Actor.scala:470) > at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52) > at akka.actor.ActorCell.create(ActorCell.scala:580) > ... 9 more > > > what could be the reason for this? > > rgds > > On Thu, Jun 25, 2015 at 11:42 AM, Niranda Perera > wrote: > >> thanks Josh. >> >> this looks very similar to my problem. >> >> On Thu, Jun 25, 2015 at 11:32 AM, Josh Rosen >> wrote: >> >>> This sounds like https://issues.apache.org/jira/browse/SPARK-7436, >>> which has been fixed in Spark 1.4+ and in branch-1.3 (for Spark 1.3.2). >>> >>> On Wed, Jun 24, 2015 at 10:57 PM, Niranda Perera < >>> niranda.per...@gmail.com> wrote: >>> Hi all, I'm trying to implement a custom StandaloneRecoveryModeFactory in the Java environment. Pls find the implementation here. [1] . I'm new to Scala, hence I'm trying to use Java environment as much as possible. when I start a master with spark.deploy.recoveryMode.factory property to be "CUSTOM", I encounter a NoSuchMethodException for my custom class's constructor. it has the following constructor. public AnalyticsStandaloneRecoveryModeFactory(SparkConf conf, Serialization serializer) but from the Master, it looks for a constructor for, org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.(org.apache.spark.SparkConf, akka.serialization.Serialization$) I see in the Spark source code for Master, that it uses reflection to get the custom recovery mode factory class. case "CUSTOM" => val clazz = Class.forName(conf.get("spark.deploy.recoveryMode.factory")) val factory = clazz.getConstructor(conf.getClass, Serialization.getClass) .newInstance(conf, SerializationExtension(context.system)) .asInstanceOf[StandaloneRecoveryModeFactory] (factory.createPersistenceEngine(), factory.createLeaderElectionAgent(this)) here, Serialization.getClass returns a akka.serialization.Serialization$ object, where as my custom class's constructor accepts akka.serialization.Serialization object. so I would like to know, 1. if this happens because I'm using this in the Java environment? 2. what is the workaround to this? thanks Please find the full stack trace of the error below. [2015-06-25 10:59:01,095] ERROR {akka.actor.OneForOne