[
https://issues.apache.org/jira/browse/CASSANDRA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504005#comment-14504005
]
Benedict commented on CASSANDRA-8789:
-------------------------------------
I don't doubt there are problems with this, but I'm not sure they're
significantly worse under the new scheme than the old... Currently messages are
split along the following boundaries:
REQUEST_RESPONSE,
INTERNAL_RESPONSE,
GOSSIP,
READ,
MUTATION,
COUNTER_MUTATION,
ANTI_ENTROPY,
MIGRATION,
MISC,
TRACING,
READ_REPAIR;
READ_RESPONSE is half of the problem messages you highlighted, and in many
workloads likely significantly more of a problem than mutations (since with
clustering data they have the potential to deliver much larger payloads), and
they currently operate on the same channel as gossip. The main difference is
that you won't see them on a pure stress write workload; a mixed workload you
would. So if this is a potentially serious problem, it is likely already being
exhibited. I should make clear that I'm not disputing there's a problem - this
seems very clearly something we want to avoid. But I don't think we have made
matters _worse_ with this ticket (though the profile has perhaps changed).
Introducing "extra" channels that are managed via NIO for whom we have no
throughput requirements, only latency, seems like a potential solution to this.
Or a priority queue and a capped send buffer size (capped low for slow WAN
connections, for instance). I would quite like to see us abstract
MessagingService so that not only the transport can be pluggable, but it can be
different per end-point (e.g. cross-dc), and per message type. I think all of
these endeavours are orthogonal to this ticket, though, and deserve their own.
> OutboundTcpConnectionPool should route messages to sockets by size not type
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-8789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8789
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
> Fix For: 3.0
>
> Attachments: 8789.diff
>
>
> I was looking at this trying to understand what messages flow over which
> connection.
> For reads the request goes out over the command connection and the response
> comes back over the ack connection.
> For writes the request goes out over the command connection and the response
> comes back over the command connection.
> Reads get a dedicated socket for responses. Mutation commands and responses
> both travel over the same socket along with read requests.
> Sockets are used uni-directional so there are actually four sockets in play
> and four threads at each node (2 inbounded, 2 outbound).
> CASSANDRA-488 doesn't leave a record of what the impact of this change was.
> If someone remembers what situations were made better it would be good to
> know.
> I am not clear on when/how this is helpful. The consumer side shouldn't be
> blocking so the only head of line blocking issue is the time it takes to
> transfer data over the wire.
> If message size is the cause of blocking issues then the current design mixes
> small messages and large messages on the same connection retaining the head
> of line blocking.
> Read requests share the same connection as write requests (which are large),
> and write acknowledgments (which are small) share the same connections as
> write requests. The only winner is read acknowledgements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
