I see the same issue with 2.3.14.
On Tuesday, March 22, 2016 at 2:00:15 PM UTC-4, Guido Medina wrote:
>
> To eliminate noise please update to 2.3.14 which from 2.3.11 has some
> cluster fixes, there are also several fixes on Scala 2.11.8 (not related)
>
> I don't know, I just have the custom of keeping my libs up to date.
>
> HTH,
>
> Guido.
>
> On Tuesday, March 22, 2016 at 5:34:23 PM UTC, Benjamin Black wrote:
>>
>> Hello,
>>
>> I'm trying to understand the cause of nodes being quarantined and
>> possible solutions to fixing it. I'm using akka 2.3.11. On the quarantined
>> node I see this logging:
>>
>> 2:45:44.204 ERROR [geyser-akka.remote.default-remote-dispatcher-6]
>> a.r.EndpointWriter - AssociationError [akka.tcp://
>> geyser@172.16.120.174:7000] <- [akka.tcp://geyser@172.17.100.105:7000]:
>> Error [Invalid address: akka.tcp://geyser@172.17.100.105:7000] [
>> akka.remote.InvalidAssociation: Invalid address: akka.tcp://
>> geyser@172.17.100.105:7000
>> Caused by: akka.remote.transport.Transport$InvalidAssociationException:
>> The remote system has quarantined this system. No further associations to
>> the remote system are possible until this system is restarted.
>> ]
>> 12:45:44.205 WARN [geyser-akka.remote.default-remote-dispatcher-25]
>> Remoting - Tried to associate with unreachable remote address [akka.tcp://
>> geyser@172.17.100.105:7000]. Address is now gated for 5000 ms, all
>> messages to this address will be delivered to dead letters. Reason: [The
>> remote system has quarantined this system. No further associations to the
>> remote system are possible until this system is restarted.]
>>
>> And on the node that cause the box to be quarantined I see this logging:
>>
>> 12:45:44.194 WARN [geyser-akka.remote.default-remote-dispatcher-6]
>> Remoting - Association to [akka.tcp://geyser@172.16.120.174:7000] having
>> UID [-450748474] is irrecoverably failed. UID is now quarantined and all
>> messages to this UID will be delivered to dead letters. Remote actorsystem
>> must be restarted to recover from this situation.
>> 12:45:44.202 WARN [geyser-akka.remote.default-remote-dispatcher-7]
>> a.r.EndpointWriter - AssociationError [akka.tcp://
>> geyser@172.17.100.105:7000] -> [akka.tcp://geyser@172.16.120.174:7000]:
>> Error [Invalid address: akka.tcp://geyser@172.16.120.174:7000] [
>> akka.remote.InvalidAssociation: Invalid address: akka.tcp://
>> geyser@172.16.120.174:7000
>> Caused by: akka.remote.transport.Transport$InvalidAssociationException:
>> The remote system has a UID that has been quarantined. Association aborted.
>> ]
>> 12:45:44.203 WARN [geyser-akka.remote.default-remote-dispatcher-7]
>> Remoting - Tried to associate with unreachable remote address [akka.tcp://
>> geyser@172.16.120.174:7000]. Address is now gated for 5000 ms, all
>> messages to this address will be delivered to dead letters. Reason: [The
>> remote system has a UID that has been quarantined. Association aborted.]
>> 12:45:44.221 ERROR [geyser-akka.remote.default-remote-dispatcher-7]
>> Remoting - Association to [akka.tcp://geyser@172.16.120.174:7000] with
>> UID [-450748474] irrecoverably failed. Quarantining address.
>> java.lang.IllegalStateException: Error encountered while processing
>> system message acknowledgement buffer: [-1 {}] ack: ACK[6, {}]
>> at
>> akka.remote.ReliableDeliverySupervisor$$anonfun$receive$1.applyOrElse(Endpoint.scala:288)
>>
>> ~[geyser.jar:1.1.17-SNAPSHOT]
>> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
>> ~[geyser.jar:1.1.17-SNAPSHOT]
>> Caused by: java.lang.IllegalArgumentException: Highest SEQ so far was -1
>> but cumulative ACK is 6
>> at
>> akka.remote.AckedSendBuffer.acknowledge(AckedDelivery.scala:103)
>> ~[geyser.jar:1.1.17-SNAPSHOT]
>> at
>> akka.remote.ReliableDeliverySupervisor$$anonfun$receive$1.applyOrElse(Endpoint.scala:284)
>>
>> ~[geyser.jar:1.1.17-SNAPSHOT]
>> ... 11 common frames omitted
>> 12:45:44.221 WARN [geyser-akka.remote.default-remote-dispatcher-7]
>> Remoting - Association to [akka.tcp://geyser@172.16.120.174:7000] having
>> UID [-450748474] is irrecoverably failed. UID is now quarantined and all
>> messages to this UID will be delivered to dead letters. Remote actorsystem
>> must be restarted to recover from this situation.
>>
>> Quite a bit of data can be passed between the nodes ~200 Mb/sec and maybe
>> the system is hitting a capacity issue although I don't see any issue with
>> CPU or memory. I noticed that the default-remote-dispatcher only has two
>> threads. Are these threads being used to send the data? And if so should I
>> try increase the thread count? Are there any other settings I could play
>> with of things I can look for in the logs that might highlight what is
>> wrong?
>>
>> Thanks,
>> Ben
>>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
