Following up from our earlier post... We have continued to do exhaustive testing and measuring of the numerous hardware and configuration variables here. What we have uncovered is that on identical hardware (including the configuration we run in production), something between versions 2.0.17 and 2.1.13 introduced this write timeout for our workload. We still aren't any closer to identifying the what or why, but it is easily reproduced using our workload when we bump to the 2.1.x release line.
At the moment we are going to focus on hardening this new hardware configuration using the 2.0.17 release and roll it out internally to some of our production rings. We also want to bisect the 2.1.x release line to find if there was a particular point release that introduced the timeout. If anyone has suggestions for particular changes to look out for we'd be happy to focus a test on that earlier. Thanks, Mike On Wed, Feb 10, 2016 at 2:51 PM, Mike Heffner <m...@librato.com> wrote: > Hi all, > > We've recently embarked on a project to update our Cassandra > infrastructure running on EC2. We are long time users of 2.0.x and are > testing out a move to version 2.2.5 running on VPC with EBS. Our test setup > is a 3 node, RF=3 cluster supporting a small write load (mirror of our > staging load). > > We are writing at QUORUM and while p95's look good compared to our staging > 2.0.x cluster, we are seeing frequent write operations that time out at the > max write_request_timeout_in_ms (10 seconds). CPU across the cluster is < > 10% and EBS write load is < 100 IOPS. Cassandra is running with the Oracle > JDK 8u60 and we're using G1GC and any GC pauses are less than 500ms. > > We run on c4.2xl instances with GP2 EBS attached storage for data and > commitlog directories. The nodes are using EC2 enhanced networking and have > the latest Intel network driver module. We are running on HVM instances > using Ubuntu 14.04.2. > > Our schema is 5 tables, all with COMPACT STORAGE. Each table is similar to > the definition here: https://gist.github.com/mheffner/4d80f6b53ccaa24cc20a > > This is our cassandra.yaml: > https://gist.github.com/mheffner/fea80e6e939dd483f94f#file-cassandra-yaml > > Like I mentioned we use 8u60 with G1GC and have used many of the GC > settings in Al Tobey's tuning guide. This is our upstart config with JVM > and other CPU settings: > https://gist.github.com/mheffner/dc44613620b25c4fa46d > > We've used several of the sysctl settings from Al's guide as well: > https://gist.github.com/mheffner/ea40d58f58a517028152 > > Our client application is able to write using either Thrift batches using > Asytanax driver or CQL async INSERT's using the Datastax Java driver. > > For testing against Thrift (our legacy infra uses this) we write batches > of anywhere from 6 to 1500 rows at a time. Our p99 for batch execution is > around 45ms but our maximum (p100) sits less than 150ms except when it > periodically spikes to the full 10seconds. > > Testing the same write path using CQL writes instead demonstrates similar > behavior. Low p99s except for periodic full timeouts. We enabled tracing > for several operations but were unable to get a trace that completed > successfully -- Cassandra started logging many messages as: > > INFO [ScheduledTasks:1] - MessagingService.java:946 - _TRACE messages > were dropped in last 5000 ms: 52499 for internal timeout and 0 for cross > node timeout > > And all the traces contained rows with a "null" source_elapsed row: > https://gist.githubusercontent.com/mheffner/1d68a70449bd6688a010/raw/0327d7d3d94c3a93af02b64212e3b7e7d8f2911b/trace.out > > > We've exhausted as many configuration option permutations that we can > think of. This cluster does not appear to be under any significant load and > latencies seem to largely fall in two bands: low normal or max timeout. > This seems to imply that something is getting stuck and timing out at the > max write timeout. > > Any suggestions on what to look for? We had debug enabled for awhile but > we didn't see any msg that pointed to something obvious. Happy to provide > any more information that may help. > > We are pretty much at the point of sprinkling debug around the code to > track down what could be blocking. > > > Thanks, > > Mike > > -- > > Mike Heffner <m...@librato.com> > Librato, Inc. > > -- Mike Heffner <m...@librato.com> Librato, Inc.
