Hi,
During one of my runs of bring up and down the interfaces of cluster
members, I hit the following NPE after all 3 instances were isolated
twice. Let me know if you need any more info besides the log below.
2017-04-05 19:08:51,860 | WARN | lt-dispatcher-28 |
ConcurrentDOMDataBroker | 193 -
org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 |
Tx: DOM-32 Error during phase CAN_COMMIT, starting Abort
org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException:
Shard member-2-shard-default-operational currently has no leader. Try again
later.
at
org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.createNoShardLeaderException(ShardManager.java:723)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.onShardNotInitializedTimeout(ShardManager.java:537)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardManager.java:216)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:29)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[181:com.typesafe.akka.persistence:2.4.7]
at
org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[175:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
2017-04-05 19:08:51,863 | WARN | tor-ComputeTimer |
GenericTransactionUtils | 301 - com.infinera.sdn.utils.transaction
- 0.1.0.SNAPSHOT | Transaction for add of object State [_cpuInfo=CpuInfo
[_processorCount=6, _usage=0.48, augmentation=[]], _memInfo=MemInfo
[_memFree=138797056,
_memTotal=12302811136, augmentation=[]], _status=class
org.opendaylight.yang.gen.v1.urn.infinera.system.compute.rev160510.Running,
augmentation=[]] failed with error {}
2017-04-05 19:09:14,056 | INFO | lt-dispatcher-35 |
kka://opendaylight-cluster-data) | 176 - com.typesafe.akka.slf4j - 2.4.7 |
Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.12:2550] -
Leader is moving node [akka.tcp://opendaylight-cluster-data@172.17.0.11:2550]
to [Up]
2017-04-05 19:09:14,057 | INFO | lt-dispatcher-35 | ShardManager
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | shard-manager-operational: Received MemberUp: memberName:
MemberName{name=member-1}, address: akka.tcp:
//opendaylight-cluster-data@172.17.0.11:2550
2017-04-05 19:09:14,057 | INFO | lt-dispatcher-35 | ShardInformation
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | updatePeerAddress for peer
member-1-shard-default-operational with address akka.tcp://opendaylight-clus
ter-data@172.17.0.11:2550/user/shardmanager-operational/member-1-shard-default-operational
2017-04-05 19:09:14,057 | INFO | lt-dispatcher-35 | ShardInformation
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | updatePeerAddress for peer
member-1-shard-entity-ownership-operational with address akka.tcp://opendayl
ight-cluster-data@172.17.0.11:2550/user/shardmanager-operational/member-1-shard-entity-ownership-operational
2017-04-05 19:09:14,058 | INFO | lt-dispatcher-18 | ShardManager
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | shard-manager-config: Received MemberUp: memberName:
MemberName{name=member-1}, address: akka.tcp://ope
ndaylight-cluster-data@172.17.0.11:2550
2017-04-05 19:09:14,058 | INFO | lt-dispatcher-18 | ShardInformation
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | updatePeerAddress for peer member-1-shard-default-config
with address akka.tcp://opendaylight-cluster-d
ata@172.17.0.11:2550/user/shardmanager-config/member-1-shard-default-config
2017-04-05 19:09:14,068 | INFO | lt-dispatcher-18 | ShardManager
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | shard-manager-config: All Shards are ready - data store
config is ready, available count is 0
2017-04-05 19:09:14,068 | INFO | lt-dispatcher-18 | Shard
| 188 - org.opendaylight.controller.sal-akka-raft -
1.4.2.Boron-SR2 | Peer address for peer member-1-shard-default-config set
to akka.tcp://opendaylight-cluster-data@172.17.0.11:2550/us
er/shardmanager-config/member-1-shard-default-config
2017-04-05 19:09:14,063 | INFO | lt-dispatcher-28 | EntityOwnershipShard
| 188 - org.opendaylight.controller.sal-akka-raft -
1.4.2.Boron-SR2 | Peer address for peer
member-1-shard-entity-ownership-operational set to
akka.tcp://opendaylight-cluster-data@172.1
7.0.11:2550/user/shardmanager-operational/member-1-shard-entity-ownership-operational
2017-04-05 19:09:14,070 | INFO | lt-dispatcher-33 | Shard
| 188 - org.opendaylight.controller.sal-akka-raft -
1.4.2.Boron-SR2 | Peer address for peer member-1-shard-default-operational
set to akka.tcp://opendaylight-cluster-data@172.17.0.11:25
50/user/shardmanager-operational/member-1-shard-default-operational
22017-04-05 19:11:31,513 | WARN | lt-dispatcher-17 | OneForOneStrategy
| 176 - com.typesafe.akka.slf4j - 2.4.7 | null
2017-04-05 19:11:31,514 | WARN | lt-dispatcher-18 | ShardManager
| 193 - org.opendaylight.controller.sal-distributed-datastore -
1.4.2.Boron-SR2 | Supervisor Strategy caught unexpected exception - resuming
java.lang.NullPointerException
at
org.opendaylight.controller.cluster.raft.SnapshotManager$AbstractSnapshotState.doTrimLog(SnapshotManager.java:211)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.SnapshotManager$Idle.trimLog(SnapshotManager.java:293)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.SnapshotManager.trimLog(SnapshotManager.java:91)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.AbstractRaftActorBehavior.performSnapshotWithoutCapture(AbstractRaftActorBehavior.java:470)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.purgeInMemoryLog(AbstractLeader.java:400)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.handleAppendEntriesReply(AbstractLeader.java:368)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.AbstractRaftActorBehavior.handleMessage(AbstractRaftActorBehavior.java:404)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.handleMessage(AbstractLeader.java:457)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.behaviors.PreLeader.handleMessage(PreLeader.java:49)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActor.possiblyHandleBehaviorMessage(RaftActor.java:302)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:290)[188:org.opendaylight.controller.sal-akka-raft:1.4.2.Boron-SR2]
at
org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:29)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[181:com.typesafe.akka.persistence:2.4.7]
at
org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.Actor$class.aroundReceive(Actor.scala:484)[175:com.typesafe.akka.actor:2.4.7]
at
akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[181:com.typesafe.akka.persistence:2.4.7]
at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[175:com.typesafe.akka.actor:2.4.7]
at
akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com.typesafe.akka.actor:2.4.7]
at
akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com.typesafe.akka.actor:2.4.7]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
