[
https://issues.apache.org/jira/browse/CASSANDRA-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912464#comment-13912464
]
sankalp kohli commented on CASSANDRA-6747:
------------------------------------------
While going through the code of MS, I have found another problem.
When we send a message through sendRR in MS, we add its callback to an expiring
map with a timeout of RPC timeout in most cases.
When the timeout is hit if the response did not come, the map will remove the
callback from the map.
So all late arriving messages will not be able to run the callback.
The problem with this is that in ActiveRepairService.prepareForRepair(as given
in JIRA description), waiting for 1 hour does not make sense since it will
never get the response back once RPC time has passed.
Also snapshot phase in repair will hang forever.
There might be more places where this is a problem.
> MessagingService should handle failures on remote nodes.
> --------------------------------------------------------
>
> Key: CASSANDRA-6747
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6747
> Project: Cassandra
> Issue Type: Improvement
> Reporter: sankalp kohli
> Priority: Minor
> Labels: Core
>
> While going through the code of MessagingService, I discovered that we don't
> handle callbacks on failure very well. If a Verb Handler on the remote
> machine throws an exception, it goes right through uncaught exception
> handler. The machine which triggered the message will keep waiting and will
> timeout. On timeout, it will so some stuff hard coded in the MS like hints
> and add to Latency. There is no way in IAsyncCallback to specify that to do
> on timeouts and also on failures.
> Here are some examples which I found will help if we enhance this system to
> also propagate failures back. So IAsyncCallback will have methods like
> onFailure.
> 1) From ActiveRepairService.prepareForRepair
> IAsyncCallback callback = new IAsyncCallback()
> {
> @Override
> public void response(MessageIn msg)
> {
> prepareLatch.countDown();
> }
> @Override
> public boolean isLatencyForSnitch()
> {
> return false;
> }
> };
> List<UUID> cfIds = new ArrayList<>(columnFamilyStores.size());
> for (ColumnFamilyStore cfs : columnFamilyStores)
> cfIds.add(cfs.metadata.cfId);
> for(InetAddress neighbour : endpoints)
> {
> PrepareMessage message = new PrepareMessage(parentRepairSession,
> cfIds, ranges);
> MessageOut<RepairMessage> msg = message.createMessage();
> MessagingService.instance().sendRR(msg, neighbour, callback);
> }
> try
> {
> prepareLatch.await(1, TimeUnit.HOURS);
> }
> catch (InterruptedException e)
> {
> parentRepairSessions.remove(parentRepairSession);
> throw new RuntimeException("Did not get replies from all
> endpoints.", e);
> }
> 2) During snapshot phase in repair, if SnapshotVerbHandler throws an
> exception, we will wait forever.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
