Hi Josh,
I tried using the spark 1.4.0 upgrade.
here is the class I'm trying to use
package org.wso2.carbon.analytics.spark.core.util.master
import akka.serialization.Serialization
import org.apache.spark.SparkConf
import org.apache.spark.deploy.master._
class AnalyticsRecoveryModeFactoryScala(conf: SparkConf, serializer:
Serialization)
extends StandaloneRecoveryModeFactory(conf, serializer) {
override def createPersistenceEngine(): PersistenceEngine = new
AnalyticsPersistenceEngine(conf, serializer)
override def createLeaderElectionAgent(master: LeaderElectable):
LeaderElectionAgent = new
AnalyticsLeaderElectionAgent(master)
}
object AnalyticsRecoveryModeFactoryScala {
}
when I invoke this factory from the master, I get a similar error as before
[2015-07-05 17:06:55,384] ERROR {akka.actor.OneForOneStrategy} -
org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
akka.serialization.Serialization)
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NoSuchMethodException:
org.wso2.carbon.analytics.spark.core.util.master.AnalyticsRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
akka.serialization.Serialization)
at java.lang.Class.getConstructor0(Class.java:2810)
at java.lang.Class.getConstructor(Class.java:1718)
at org.apache.spark.deploy.master.Master.preStart(Master.scala:168)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52)
at akka.actor.ActorCell.create(ActorCell.scala:580)
... 9 more
what could be the reason for this?
rgds
On Thu, Jun 25, 2015 at 11:42 AM, Niranda Perera <niranda.per...@gmail.com>
wrote:
> thanks Josh.
>
> this looks very similar to my problem.
>
> On Thu, Jun 25, 2015 at 11:32 AM, Josh Rosen <rosenvi...@gmail.com> wrote:
>
>> This sounds like https://issues.apache.org/jira/browse/SPARK-7436, which
>> has been fixed in Spark 1.4+ and in branch-1.3 (for Spark 1.3.2).
>>
>> On Wed, Jun 24, 2015 at 10:57 PM, Niranda Perera <
>> niranda.per...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to implement a custom StandaloneRecoveryModeFactory in the
>>> Java environment. Pls find the implementation here. [1] . I'm new to Scala,
>>> hence I'm trying to use Java environment as much as possible.
>>>
>>> when I start a master with spark.deploy.recoveryMode.factory property to
>>> be "CUSTOM", I encounter a NoSuchMethodException for my custom class's
>>> constructor.
>>> it has the following constructor.
>>>
>>> public AnalyticsStandaloneRecoveryModeFactory(SparkConf conf,
>>> Serialization serializer)
>>>
>>> but from the Master, it looks for a constructor for,
>>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>>> akka.serialization.Serialization$)
>>>
>>> I see in the Spark source code for Master, that it uses reflection to
>>> get the custom recovery mode factory class.
>>>
>>> case "CUSTOM" =>
>>> val clazz =
>>> Class.forName(conf.get("spark.deploy.recoveryMode.factory"))
>>> val factory = clazz.getConstructor(conf.getClass,
>>> Serialization.getClass)
>>> .newInstance(conf, SerializationExtension(context.system))
>>> .asInstanceOf[StandaloneRecoveryModeFactory]
>>> (factory.createPersistenceEngine(),
>>> factory.createLeaderElectionAgent(this))
>>>
>>> here, Serialization.getClass returns a akka.serialization.Serialization$
>>> object, where as my custom class's constructor
>>> accepts akka.serialization.Serialization object.
>>>
>>> so I would like to know,
>>> 1. if this happens because I'm using this in the Java environment?
>>> 2. what is the workaround to this?
>>>
>>> thanks
>>>
>>> Please find the full stack trace of the error below.
>>>
>>> [2015-06-25 10:59:01,095] ERROR {akka.actor.OneForOneStrategy} -
>>>
>>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>>> akka.serialization.Serialization$)
>>> akka.actor.ActorInitializationException: exception during creation
>>> at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
>>> at akka.actor.ActorCell.create(ActorCell.scala:596)
>>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
>>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>>> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>>> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>>> at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>> at
>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> Caused by: java.lang.NoSuchMethodException:
>>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>>> akka.serialization.Serialization$)
>>> at java.lang.Class.getConstructor0(Class.java:2810)
>>> at java.lang.Class.getConstructor(Class.java:1718)
>>> at org.apache.spark.deploy.master.Master.preStart(Master.scala:165)
>>> at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
>>> at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52)
>>> at akka.actor.ActorCell.create(ActorCell.scala:580)
>>> ... 9 more
>>>
>>>
>>>
>>> [1]
>>> https://github.com/nirandaperera/carbon-analytics/blob/spark_master_persistance/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/util/master/AnalyticsStandaloneRecoveryModeFactory.java
>>> <https://github.com/nirandaperera/carbon-analytics/blob/spark_master_persistance/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/util/master/AnalyticsStandaloneRecoveryModeFactory.java>
>>>
>>> --
>>> Niranda
>>> @n1r44 <https://twitter.com/N1R44>
>>> https://pythagoreanscript.wordpress.com/
>>>
>>
>>
>
>
> --
> Niranda
> @n1r44 <https://twitter.com/N1R44>
> https://pythagoreanscript.wordpress.com/
>
--
Niranda
@n1r44 <https://twitter.com/N1R44>
https://pythagoreanscript.wordpress.com/
