Problem still persists.
1. I changed boot procedure to wait until cluster starts. it is done by:
class ClusterWaiter(p: Promise[Boolean]) extends Actor {
override def receive = {
case _ =>
p.complete(Success(true))
context.stop(self)
}
}
def startCluster() = {
//eagerly init cluster dispatcher, the one used to start cluster daemon
system.dispatchers.lookup("cluster-dispatcher")
val p = Promise[Boolean]
val waiter = system.actorOf(actor.Props(classOf[ClusterWaiter], p),
"cluster-waiter")
// sendCurrentClusterState requires internal cluster actor to start
Cluster(system).sendCurrentClusterState(waiter)
//wait for some signal from cluster
Await.ready(p.future, 60 seconds)
}
In that time, nothing else takes CPU or IO, app just waits for cluster to
start
2. I changed actor creation timeout to 120 seconds
3. Still I get exceptions on startup:
12:15:43.025 [sgActors-akka.actor.default-dispatcher-3] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
12:17:43.105 [main] ERROR n.l.h.p.HTTPProvider: Failed to Boot! Your
application may not run properly
2 minutes of idle waiting for cluster start.
12:17:43.186 [sgActors-akka.actor.default-dispatcher-4] ERROR
Cluster(akka://sgActors): Failed to startup Cluster. You can try to
increase 'akka.actor.creation-timeout'.
java.util.concurrent.TimeoutException: Futures timed out after [120000
milliseconds]
at
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
~[scala-library-2.11.7.jar:na]
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
~[scala-library-2.11.7.jar:na]
at
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
~[scala-library-2.11.7.jar:na]
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
~[scala-library-2.11.7.jar:na]
at scala.concurrent.Await$.result(package.scala:190)
~[scala-library-2.11.7.jar:na]
at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:172)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster.<init>(Cluster.scala:171)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.apply(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
~[akka-remote_2.11-2.3.12.jar:na]
at
akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
~[akka-actor_2.11-2.3.12.jar:na]
at
akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
~[akka-actor_2.11-2.3.12.jar:na]
Caused by: java.lang.NullPointerException: null
at akka.cluster.Cluster.shutdown(Cluster.scala:358)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:176)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster.<init>(Cluster.scala:171)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.apply(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
~[akka-remote_2.11-2.3.12.jar:na]
at
akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
~[akka-actor_2.11-2.3.12.jar:na]
at
akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
~[akka-actor_2.11-2.3.12.jar:na]
and then
12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
12:17:43.223 [sgActors-akka.actor.default-dispatcher-20] ERROR
a.a.OneForOneStrategy: exception during creation
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:166)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.create(ActorCell.scala:596)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
~[akka-actor_2.11-2.3.12.jar:na]
at
akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
~[akka-actor_2.11-2.3.12.jar:na]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
[akka-actor_2.11-2.3.12.jar:na]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[scala-library-2.11.7.jar:na]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[scala-library-2.11.7.jar:na]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[scala-library-2.11.7.jar:na]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[scala-library-2.11.7.jar:na]
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[na:1.8.0_60]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[na:1.8.0_60]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[na:1.8.0_60]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
~[na:1.8.0_60]
at akka.util.Reflect$.instantiate(Reflect.scala:66)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.Props.newActor(Props.scala:252)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.create(ActorCell.scala:578)
~[akka-actor_2.11-2.3.12.jar:na]
... 9 common frames omitted
Caused by: akka.actor.InvalidActorNameException: actor name [cluster] is
not unique!
at
akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130)
~[akka-actor_2.11-2.3.12.jar:na]
at
akka.actor.dungeon.Children$class.reserveChild(Children.scala:76)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.reserveChild(ActorCell.scala:369)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.dungeon.Children$class.makeChild(Children.scala:201)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.dungeon.Children$class.attachChild(Children.scala:41)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorCell.attachChild(ActorCell.scala:369)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ActorSystemImpl.systemActorOf(ActorSystem.scala:551)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.cluster.Cluster.<init>(Cluster.scala:162)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
~[akka-cluster_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
~[akka-actor_2.11-2.3.12.jar:na]
at akka.cluster.Cluster$.apply(Cluster.scala:37)
~[akka-cluster_2.11-2.3.12.jar:na]
at
akka.cluster.ClusterMetricsCollector.<init>(ClusterMetricsCollector.scala:47)
~[akka-cluster_2.11-2.3.12.jar:na]
... 18 common frames omitted
It seems that
private[cluster] def shutdown(): Unit = {
if (_isTerminated.compareAndSet(false, true)) {
logInfo("Shutting down...")
system.stop(clusterDaemons)
readView.close()
closeScheduler()
clusterJmx foreach { _.unregisterMBean() }
logInfo("Successfully shut down")
}
}
fails on readView.close due to NPE during cluster shutdown
Usually the whole scenario works within milliseconds, but from time to time
it fails, regardles of the actor.creation-timeout setting
On the other hand, [cluster] name is not unique suggests that the [cluster]
actor was already created
other log entries:
12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
12:17:43.188 [sgActors-akka.actor.default-dispatcher-4] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
Cluster(akka://sgActors): Cluster Node
[akka.tcp://sgact...@app1.groupl.es:2552] - Starting up...
suggests that Cluster extension is starting several times, from different
threads. Maybe it is a root cause of the problem? Maybe there is a race in
extension startup / cluster startup ?
W dniu czwartek, 6 sierpnia 2015 11:59:39 UTC+2 użytkownik Marek Żebrowski
napisał:
>
> I did not change the default dispatcher. I hope that I have all heavy task
> run by other dispatchers - I'll double check it.
>
> W dniu środa, 5 sierpnia 2015 22:57:05 UTC+2 użytkownik Patrik Nordwall
> napisał:
>>
>> Have you changed the default configuration of the default-dispatcher, or
>> do you do any blocking (long running) tasks on the default-dispatcher?
>> /Patrik
>>
>> On Mon, Aug 3, 2015 at 10:42 AM, Marek Żebrowski <marek.z...@gmail.com>
>> wrote:
>>
>>> That failure leads to another failure: actor name not unique
>>>
>>> 08:11:39.893 [sgActors-akka.actor.default-dispatcher-17] ERROR
>>>> a.c.ClusterCoreSupervisor: actor name [cluster] is not unique!
>>>> akka.actor.InvalidActorNameException: actor name [cluster] is not
>>>> unique!
>>>>
>>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>>
>> Patrik Nordwall
>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>> Twitter: @patriknw
>>
>>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
