On Wed, Aug 26, 2015 at 3:50 PM, Marek Żebrowski <marek.zebrow...@gmail.com>
wrote:
> Problem still persists.
> 1. I changed boot procedure to wait until cluster starts. it is done by:
>
> class ClusterWaiter(p: Promise[Boolean]) extends Actor {
> override def receive = {
> case _ =>
> p.complete(Success(true))
> context.stop(self)
> }
> }
>
> def startCluster() = {
> //eagerly init cluster dispatcher, the one used to start cluster daemon
> system.dispatchers.lookup("cluster-dispatcher")
> val p = Promise[Boolean]
> val waiter = system.actorOf(actor.Props(classOf[ClusterWaiter], p),
> "cluster-waiter")
> // sendCurrentClusterState requires internal cluster actor to start
> Cluster(system).sendCurrentClusterState(waiter)
> //wait for some signal from cluster
> Await.ready(p.future, 60 seconds)
> }
>
>
> In that time, nothing else takes CPU or IO, app just waits for cluster to
> start
>
> 2. I changed actor creation timeout to 120 seconds
>
> 3. Still I get exceptions on startup:
>
> 12:15:43.025 [sgActors-akka.actor.default-dispatcher-3] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.105 [main] ERROR n.l.h.p.HTTPProvider: Failed to Boot! Your
> application may not run properly
>
> 2 minutes of idle waiting for cluster start.
>
>
> 12:17:43.186 [sgActors-akka.actor.default-dispatcher-4] ERROR
> Cluster(akka://sgActors): Failed to startup Cluster. You can try to
> increase 'akka.actor.creation-timeout'.
> java.util.concurrent.TimeoutException: Futures timed out after [120000
> milliseconds]
> at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> ~[scala-library-2.11.7.jar:na]
> at scala.concurrent.Await$.result(package.scala:190)
> ~[scala-library-2.11.7.jar:na]
> at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:172)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:171)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
> ~[akka-remote_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
> ~[akka-actor_2.11-2.3.12.jar:na]
>
>
That is interesting. I have tried to reproduce but I can't. As you can see
from above stack trace the cluster extension is initialized from the thread
that creates the ActorSystem. The ActorSystem.apply method has not even
returned yet. So your code for `startCluster` has not been invoked yet,
unless you invoke that from some other thread (but that sounds unlikely
since you access system from there.
Can you provide a small test case or sample app that I can use to reproduce
the issue. It's enough if it only fails sporadically.
Everything below is irrelevant. That is just side effects of that the first
initialization failed. We have improved that part in 2.4.0-RC1.
/Patrik
>
> Caused by: java.lang.NullPointerException: null
> at akka.cluster.Cluster.shutdown(Cluster.scala:358)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:176)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:171)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
> ~[akka-remote_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
> ~[akka-actor_2.11-2.3.12.jar:na]
>
> and then
>
> 12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.223 [sgActors-akka.actor.default-dispatcher-20] ERROR
> a.a.OneForOneStrategy: exception during creation
> akka.actor.ActorInitializationException: exception during creation
> at akka.actor.ActorInitializationException$.apply(Actor.scala:166)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.create(ActorCell.scala:596)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> [akka-actor_2.11-2.3.12.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> [scala-library-2.11.7.jar:na]
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.8.0_60]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> ~[na:1.8.0_60]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[na:1.8.0_60]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> ~[na:1.8.0_60]
> at akka.util.Reflect$.instantiate(Reflect.scala:66)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.Props.newActor(Props.scala:252)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.newActor(ActorCell.scala:552)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.create(ActorCell.scala:578)
> ~[akka-actor_2.11-2.3.12.jar:na]
> ... 9 common frames omitted
> Caused by: akka.actor.InvalidActorNameException: actor name [cluster] is
> not unique!
> at
> akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.dungeon.Children$class.reserveChild(Children.scala:76)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.reserveChild(ActorCell.scala:369)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.dungeon.Children$class.makeChild(Children.scala:201)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.dungeon.Children$class.attachChild(Children.scala:41)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.attachChild(ActorCell.scala:369)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.systemActorOf(ActorSystem.scala:551)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:162)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterMetricsCollector.<init>(ClusterMetricsCollector.scala:47)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> ... 18 common frames omitted
>
> It seems that
>
> private[cluster] def shutdown(): Unit = {
> if (_isTerminated.compareAndSet(false, true)) {
> logInfo("Shutting down...")
>
> system.stop(clusterDaemons)
> readView.close()
>
> closeScheduler()
>
> clusterJmx foreach { _.unregisterMBean() }
>
> logInfo("Successfully shut down")
> }
> }
>
>
> fails on readView.close due to NPE during cluster shutdown
>
> Usually the whole scenario works within milliseconds, but from time to
> time it fails, regardles of the actor.creation-timeout setting
>
> On the other hand, [cluster] name is not unique suggests that the
> [cluster] actor was already created
>
> other log entries:
>
> 12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.188 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
>
>
> suggests that Cluster extension is starting several times, from different
> threads. Maybe it is a root cause of the problem? Maybe there is a race in
> extension startup / cluster startup ?
>
>
> W dniu czwartek, 6 sierpnia 2015 11:59:39 UTC+2 użytkownik Marek Żebrowski
> napisał:
>>
>> I did not change the default dispatcher. I hope that I have all heavy
>> task run by other dispatchers - I'll double check it.
>>
>> W dniu środa, 5 sierpnia 2015 22:57:05 UTC+2 użytkownik Patrik Nordwall
>> napisał:
>>>
>>> Have you changed the default configuration of the default-dispatcher, or
>>> do you do any blocking (long running) tasks on the default-dispatcher?
>>> /Patrik
>>>
>>> On Mon, Aug 3, 2015 at 10:42 AM, Marek Żebrowski <marek.z...@gmail.com>
>>> wrote:
>>>
>>>> That failure leads to another failure: actor name not unique
>>>>
>>>> 08:11:39.893 [sgActors-akka.actor.default-dispatcher-17] ERROR
>>>>> a.c.ClusterCoreSupervisor: actor name [cluster] is not unique!
>>>>> akka.actor.InvalidActorNameException: actor name [cluster] is not
>>>>> unique!
>>>>>
>>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to akka-user+...@googlegroups.com.
>>>> To post to this group, send email to akka...@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Patrik Nordwall
>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>>> Twitter: @patriknw
>>>
>>> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
Patrik Nordwall
Typesafe <http://typesafe.com/> - Reactive apps on the JVM
Twitter: @patriknw
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
