Last guess about parall creation of extension does not make sense - it
happens after initial failure.
Maybe there is a way to cleanly re-create ActorSystem.
I'm only worried about that NPE on cluster shutdown. it might not leave
system in a stable state, as closeScheduler()will not be called
W dniu środa, 26 sierpnia 2015 15:50:30 UTC+2 użytkownik Marek Żebrowski
napisał:
>
> Problem still persists.
> 1. I changed boot procedure to wait until cluster starts. it is done by:
>
> class ClusterWaiter(p: Promise[Boolean]) extends Actor {
> override def receive = {
> case _ =>
> p.complete(Success(true))
> context.stop(self)
> }
> }
>
> def startCluster() = {
> //eagerly init cluster dispatcher, the one used to start cluster daemon
> system.dispatchers.lookup("cluster-dispatcher")
> val p = Promise[Boolean]
> val waiter = system.actorOf(actor.Props(classOf[ClusterWaiter], p),
> "cluster-waiter")
> // sendCurrentClusterState requires internal cluster actor to start
> Cluster(system).sendCurrentClusterState(waiter)
> //wait for some signal from cluster
> Await.ready(p.future, 60 seconds)
> }
>
>
> In that time, nothing else takes CPU or IO, app just waits for cluster to
> start
>
> 2. I changed actor creation timeout to 120 seconds
>
> 3. Still I get exceptions on startup:
>
> 12:15:43.025 [sgActors-akka.actor.default-dispatcher-3] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.105 [main] ERROR n.l.h.p.HTTPProvider: Failed to Boot! Your
> application may not run properly
>
> 2 minutes of idle waiting for cluster start.
>
>
> 12:17:43.186 [sgActors-akka.actor.default-dispatcher-4] ERROR
> Cluster(akka://sgActors): Failed to startup Cluster. You can try to
> increase 'akka.actor.creation-timeout'.
> java.util.concurrent.TimeoutException: Futures timed out after [120000
> milliseconds]
> at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> ~[scala-library-2.11.7.jar:na]
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>
> ~[scala-library-2.11.7.jar:na]
> at scala.concurrent.Await$.result(package.scala:190)
> ~[scala-library-2.11.7.jar:na]
> at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:172)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:171)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
>
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
> ~[akka-remote_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
> ~[akka-actor_2.11-2.3.12.jar:na]
>
>
> Caused by: java.lang.NullPointerException: null
> at akka.cluster.Cluster.shutdown(Cluster.scala:358)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.liftedTree1$1(Cluster.scala:176)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:171)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.createRemoteWatcher(ClusterActorRefProvider.scala:66)
>
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:186)
> ~[akka-remote_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:620)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
> ~[akka-actor_2.11-2.3.12.jar:na]
>
> and then
>
> 12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.223 [sgActors-akka.actor.default-dispatcher-20] ERROR
> a.a.OneForOneStrategy: exception during creation
> akka.actor.ActorInitializationException: exception during creation
> at akka.actor.ActorInitializationException$.apply(Actor.scala:166)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.create(ActorCell.scala:596)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>
> [akka-actor_2.11-2.3.12.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> [scala-library-2.11.7.jar:na]
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> [scala-library-2.11.7.jar:na]
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.8.0_60]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> ~[na:1.8.0_60]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> ~[na:1.8.0_60]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> ~[na:1.8.0_60]
> at akka.util.Reflect$.instantiate(Reflect.scala:66)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.Props.newActor(Props.scala:252)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.newActor(ActorCell.scala:552)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.create(ActorCell.scala:578)
> ~[akka-actor_2.11-2.3.12.jar:na]
> ... 9 common frames omitted
> Caused by: akka.actor.InvalidActorNameException: actor name [cluster] is
> not unique!
> at
> akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130)
>
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.dungeon.Children$class.reserveChild(Children.scala:76)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.reserveChild(ActorCell.scala:369)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.dungeon.Children$class.makeChild(Children.scala:201)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at
> akka.actor.dungeon.Children$class.attachChild(Children.scala:41)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorCell.attachChild(ActorCell.scala:369)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ActorSystemImpl.systemActorOf(ActorSystem.scala:551)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster.<init>(Cluster.scala:162)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:42)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.createExtension(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:713)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> ~[akka-actor_2.11-2.3.12.jar:na]
> at akka.cluster.Cluster$.apply(Cluster.scala:37)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> at
> akka.cluster.ClusterMetricsCollector.<init>(ClusterMetricsCollector.scala:47)
> ~[akka-cluster_2.11-2.3.12.jar:na]
> ... 18 common frames omitted
>
> It seems that
>
> private[cluster] def shutdown(): Unit = {
> if (_isTerminated.compareAndSet(false, true)) {
> logInfo("Shutting down...")
>
> system.stop(clusterDaemons)
> readView.close()
>
> closeScheduler()
>
> clusterJmx foreach { _.unregisterMBean() }
>
> logInfo("Successfully shut down")
> }
> }
>
>
> fails on readView.close due to NPE during cluster shutdown
>
> Usually the whole scenario works within milliseconds, but from time to
> time it fails, regardles of the actor.creation-timeout setting
>
> On the other hand, [cluster] name is not unique suggests that the
> [cluster] actor was already created
>
> other log entries:
>
> 12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.187 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.188 [sgActors-akka.actor.default-dispatcher-4] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
> 12:17:43.201 [sgActors-akka.actor.default-dispatcher-20] INFO
> Cluster(akka://sgActors): Cluster Node [akka.tcp://
> sgact...@app1.groupl.es:2552] - Starting up...
>
>
> suggests that Cluster extension is starting several times, from different
> threads. Maybe it is a root cause of the problem? Maybe there is a race in
> extension startup / cluster startup ?
>
>
> W dniu czwartek, 6 sierpnia 2015 11:59:39 UTC+2 użytkownik Marek Żebrowski
> napisał:
>>
>> I did not change the default dispatcher. I hope that I have all heavy
>> task run by other dispatchers - I'll double check it.
>>
>> W dniu środa, 5 sierpnia 2015 22:57:05 UTC+2 użytkownik Patrik Nordwall
>> napisał:
>>>
>>> Have you changed the default configuration of the default-dispatcher, or
>>> do you do any blocking (long running) tasks on the default-dispatcher?
>>> /Patrik
>>>
>>> On Mon, Aug 3, 2015 at 10:42 AM, Marek Żebrowski <marek.z...@gmail.com>
>>> wrote:
>>>
>>>> That failure leads to another failure: actor name not unique
>>>>
>>>> 08:11:39.893 [sgActors-akka.actor.default-dispatcher-17] ERROR
>>>>> a.c.ClusterCoreSupervisor: actor name [cluster] is not unique!
>>>>> akka.actor.InvalidActorNameException: actor name [cluster] is not
>>>>> unique!
>>>>>
>>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ:
>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>> >>>>>>>>>> Search the archives:
>>>> https://groups.google.com/group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to akka-user+...@googlegroups.com.
>>>> To post to this group, send email to akka...@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Patrik Nordwall
>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM
>>> Twitter: @patriknw
>>>
>>>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
