Hi -
I'm being bitten by a client/server application where multiple clients
subscribe for updates from a server where, upon a long GC pause, the
clients are being quarantined. Here is some client logs for an attempt to
rediscover the server actor using "server ? Identify", which times out. I
can see that this is because the client has quarantined the server.
24-Oct-2014 11:30:28:310: [(akka)Remoting - WARNING] [31]: Tried to
associate with unreachable remote address
[akka.tcp://gekkoremot...@loypcore4.intra.gsacapital.com:35411]. Address is
now gated for 5000 ms, all messages to this address will be delivered to
dead letters. Reason: The remote system has quarantined this system. No
further associations to the remote system are possible until this system is
restarted.
I can guess (it's not a guess, the times line up perfectly) in the server
when the original disconnect happened:
1.611: [Full GC (Metadata GC Threshold) 87M->14M(52M), 0.1731620 secs]
7.622: [Full GC (Metadata GC Threshold) 175M->85M(220M), 0.4527592 secs]
8658.035: [Full GC (Metadata GC Threshold) 3638M->419M(4192M), 1.8916884
secs]
244257.679: [Full GC (Allocation Failure) 6516M->2622M(14G), 9.2884735
secs]
*391857.856: [Full GC (Allocation Failure) 7390M->3758M(12G), 13.7533193
secs]*
the server's akka logs state the following happendd at this time (the
server quarantines my client):
[WARN] [10/24/2014 10:57:22.416]
[gekkoRemoting-akka.remote.default-remote-dispatcher-7]
[akka.tcp://gekkoremot...@loypcore4.intra.gsacapital.com:35411/system/remote-watcher]
Detected unreachable: [akka.tcp://papagui@10.210.50.92:60091]
[WARN] [10/24/2014 10:57:22.416]
[gekkoRemoting-akka.remote.default-remote-dispatcher-11015] [Remoting]
Association to [akka.tcp://papagui@10.210.50.92:60091] having UID
[1196983173] is irrecoverably failed. UID is now quarantined and all
messages to this UID will be delivered to dead letters. Remote actorsystem
must be restarted to recover from this situation.
My question is "how on earth do I code against this?" - currently I have a
"canary" which creates the client connection. The client connection picks
up the Terminated message from the server and stops itself; the canary
picks this Termination up and spools up (after a pause for a few minutes) a
new client to attempt to connect to the server. Except the new client
cannot connect to the server because it has been quarantined. How is my
client supposed to know this? There's no "you've been quarantined"
callback, just a timeout looking up a server. Do I need to just assume that
the failure to lookup a server might indicate a quarantine?
here's the server's remote configuration:
remote {
log-remote-lifecycle-events = on
retry-gate-closed-for = 5 s
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
maximum-frame-size = 100 MiB
}
watch-failure-detector {
acceptable-heartbeat-pause = 20 s
heartbeat-interval = 5 s
}
transport-failure-detector {
acceptable-heartbeat-pause = 10 s
heartbeat-interval = 3 s
}
}
here's the client's remote configuration:
remote {
log-remote-lifecycle-events = on
gate-invalid-addresses-for = 5 s
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
port = 0
maximum-frame-size = 100 MiB
}
watch-failure-detector {
acceptable-heartbeat-pause = 20 s
heartbeat-interval = 5 s
}
transport-failure-detector {
acceptable-heartbeat-pause = 12 s
heartbeat-interval = 3 s
}
}
Chris
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
