Hi all, It seems like there is a massive 3node cluster regression in carbon, since approximately 21:0PM January 5th . We can see that many 3node CSIT fails across projects.
After a quick look Koby and I have found what we assume is the faulty zip. We looked at the last successful openflowplugin-3 node run: https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/379/console.log.gz which used distribution: 0.6.0-20170105.205720-2879.zip the next run failed, with the same error we are seeing now in all runs (at least the few we checked): https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/380/console.log.gz which used distribution: 0.6.0-20170105.235121-2883.zip we were able to narrow it down to a single distribution that was the first to fail: https://logs.opendaylight.org/releng/jenkins092/netvirt-legacy-csit-3node-clustering-only-carbon/144/console.log.gz 0.6.0-20170105.222635-2880.zip We would appreciate your help with narrowing down the faulty commit/s triggering the 0.6.0-20170105.222635-2880.zip distribution. The first error we saw in the logs is : 2017-01-06 05:25:30,158 | ERROR | lt-dispatcher-12 | ClusterActorRefProvider | 199 - com.typesafe.akka.slf4j - 2.4.16 | No root guardian at [akka.tcp://opendaylight-cluster-data@10.29.13.119:2550] java.lang.IllegalArgumentException: Wrong protocol of [akka.tcp://opendaylight-cluster-data@10.29.13.119:2550/], expected [akka] at akka.remote.RemoteActorRef.<init>(RemoteActorRefProvider.scala:504)[211:com.typesafe.akka.remote:2.4.16] at akka.remote.RemoteActorRefProvider.rootGuardianAt(RemoteActorRefProvider.scala:367)[211:com.typesafe.akka.remote:2.4.16] at akka.actor.ActorRefFactory$class.actorSelection(ActorRefProvider.scala:356)[198:com.typesafe.akka.actor:2.4.16] at akka.actor.ActorCell.actorSelection(ActorCell.scala:374)[198:com.typesafe.akka.actor:2.4.16] at akka.cluster.JoinSeedNodeProcess$$anonfun$receive$4$$anonfun$applyOrElse$1.applyOrElse(ClusterDaemon.scala:1271)[209:com.typesafe.akka.cluster:2.4.16] at akka.cluster.JoinSeedNodeProcess$$anonfun$receive$4$$anonfun$applyOrElse$1.applyOrElse(ClusterDaemon.scala:1270)[209:com.typesafe.akka.cluster:2.4.16] at scala.PartialFunction$$anonfun$runWith$1.apply(PartialFunction.scala:141)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.PartialFunction$$anonfun$runWith$1.apply(PartialFunction.scala:140)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.Iterator$class.foreach(Iterator.scala:893)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.AbstractIterable.foreach(Iterable.scala:54)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.TraversableLike$class.collect(TraversableLike.scala:271)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.collection.AbstractTraversable.collect(Traversable.scala:104)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at akka.cluster.JoinSeedNodeProcess$$anonfun$receive$4.applyOrElse(ClusterDaemon.scala:1270)[209:com.typesafe.akka.cluster:2.4.16] at akka.actor.Actor$class.aroundReceive(Actor.scala:496)[198:com.typesafe.akka.actor:2.4.16] at akka.cluster.JoinSeedNodeProcess.aroundReceive(ClusterDaemon.scala:1254)[209:com.typesafe.akka.cluster:2.4.16] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[198:com.typesafe.akka.actor:2.4.16] at akka.actor.ActorCell.invoke(ActorCell.scala:495)[198:com.typesafe.akka.actor:2.4.16] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[198:com.typesafe.akka.actor:2.4.16] at akka.dispatch.Mailbox.run(Mailbox.scala:224)[198:com.typesafe.akka.actor:2.4.16] at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[198:com.typesafe.akka.actor:2.4.16] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[193:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8] thanks, Koby & Ravit.
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
