[
https://issues.apache.org/jira/browse/CASSANDRA-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998310#comment-13998310
]
Jason Brown edited comment on CASSANDRA-4718 at 5/15/14 12:18 AM:
------------------------------------------------------------------
[~enigmacurry] How many threads are you running thrift with? If you aren't
setting it explicitly, (iirc) it gets set to the number of processors, which is
far below what anything sane should run with. For our machines, I've been using
512 for writes, and 128 for reads (mirroring what we run with in prod, which is
same hardware as the machines I'm testing on, more or less). I think this may
explain we we do not see the vast discrepancy between thrift and native
protocol ops/second - native protocol defaults to 128 threads.
Also, are you using sync or hsha for thrift?
was (Author: jasobrown):
[~enigmacurry] How many threads are you running thrift with? If you aren't
setting it explicitly, (iirc) it gets set to the number of processors, which is
far below what anything sane should run with. For our machines, I've been using
512 for writes, and 128 for reads (mirroring what we run with in prod, which is
same hardware as the machines I'm testing on, more or less). I think this may
explain we we do not see the vast discrepancy between thrift and native
protocol ops/second - native protocol default to 128 threads.
Also, are you using sync or hsha for thrift?
> More-efficient ExecutorService for improved throughput
> ------------------------------------------------------
>
> Key: CASSANDRA-4718
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4718
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Benedict
> Priority: Minor
> Labels: performance
> Fix For: 2.1.0
>
> Attachments: 4718-v1.patch, PerThreadQueue.java, aws.svg,
> aws_read.svg, backpressure-stress.out.txt, baq vs trunk.png,
> belliotsmith_branches-stress.out.txt, jason_read.svg, jason_read_latency.svg,
> jason_write.svg, op costs of various queues.ods, stress op rate with various
> queues.ods, v1-stress.out
>
>
> Currently all our execution stages dequeue tasks one at a time. This can
> result in contention between producers and consumers (although we do our best
> to minimize this by using LinkedBlockingQueue).
> One approach to mitigating this would be to make consumer threads do more
> work in "bulk" instead of just one task per dequeue. (Producer threads tend
> to be single-task oriented by nature, so I don't see an equivalent
> opportunity there.)
> BlockingQueue has a drainTo(collection, int) method that would be perfect for
> this. However, no ExecutorService in the jdk supports using drainTo, nor
> could I google one.
> What I would like to do here is create just such a beast and wire it into (at
> least) the write and read stages. (Other possible candidates for such an
> optimization, such as the CommitLog and OutboundTCPConnection, are not
> ExecutorService-based and will need to be one-offs.)
> AbstractExecutorService may be useful. The implementations of
> ICommitLogExecutorService may also be useful. (Despite the name these are not
> actual ExecutorServices, although they share the most important properties of
> one.)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
