I am trying to upgrade our system to 2.5.3 and play 2.6 and when we do so
the cluster nodes can no longer connect to each other. We get logs like
this:
Jul 07 16:25:38 jp-6.mycompany.com application-9001.log: 2017-07-07
21:25:38 +0000 - [WARN] - [ReliableDeliverySupervisor]
akka.tcp://play-akka@127.0.0.1:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fplay-akka%4010.8.1.142%3A2551-1
- Association with remote system [akka.tcp://play-akka@10.8.1.142:2551]
has failed, address is now gated for [5000] ms. Reason: [Association failed
with [akka.tcp://play-akka@10.8.1.142:2551]] Caused by: [Connection
refused: /10.8.1.142:2551]
Jul 07 16:25:42 jp-6.mycompany.com application-9001.log: 2017-07-07
21:25:42 +0000 - [WARN] - [JoinSeedNodeProcess]
akka.tcp://play-akka@127.0.0.1:2551/system/cluster/core/daemon/joinSeedNodeProcess-1
- Couldn't join seed nodes after [317] attempts, will try again.
seed-nodes=[akka.tcp://play-akka@10.8.1.249:2551,
akka.tcp://play-akka@10.8.1.142:2551]
Jul 07 16:25:42 jp-6.mycompany.com application-9001.log: 2017-07-07
21:25:42 +0000 - [WARN] - [NettyTransport] NettyTransport(akka://play-akka)
- *Remote connection to [null] failed* with java.net.ConnectException:
Connection refused: /10.8.1.142:2551
This goes on until the VM actually dies. I am at a loss to explain why this
happens only with the upgrade when we changed very little other code. All
of this is happening inside the Amazon AWS cluster when we find the seed
nodes with code like the following:
final String awsEnabled = System.getenv(GlobalConstants.AWS_ENABLED);
final String environment = System.getenv(GlobalConstants.AWS_ENVIRONMENT_NAME);
final String seedNodes = System.getenv(GlobalConstants.TCP_MEMBERS);
final String protocol = "akka.tcp";
final String system = actorSystem.name();
final int port =
Optional.ofNullable(System.getProperty("akka.cluster.port")).map(Integer::parseInt).orElse(2551);
if (awsEnabled != null && awsEnabled.equalsIgnoreCase("true")) {
final AutoScalingGroup group = AwsUtils.getAutoScalingClient()
.describeAutoScalingGroups(new DescribeAutoScalingGroupsRequest()
.withAutoScalingGroupNames(environment)).getAutoScalingGroups().get(0);
return group.getInstances().stream()
.map(i -> Address.apply(protocol, system,
SystemStartupActor.findInstanceFromId(i.getInstanceId()).getPrivateIpAddress(),
port)).collect(Collectors.toList());
You an see that this is pretty standard code and nothing has changed other
than the upgrade. The code above has been working fine for ages.
Can anyone clue me in as to what may be going on here?
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
