Ariel Weisberg created CASSANDRA-10695: ------------------------------------------
Summary: Thrift HSHA appears to not work in 2.1+ with high client thread counts in stress Key: CASSANDRA-10695 URL: https://issues.apache.org/jira/browse/CASSANDRA-10695 Project: Cassandra Issue Type: Bug Reporter: Ariel Weisberg In 2.2+ my stress runs make almost no progress. In 2.1 they make some progress but this assertion is still in the log. Things are generally pretty unstable. I was able to benchmark with the sync server at 500 threads without issue. Where I ran into trouble was attempting to benchmark with 2000 threads. The sync server crapped out due to thread counts and I didn't bother trying to fiddle with it I switched to HSHA which worked at 500, but fails at 2000. My test configuration is an OS X laptop running 2.2 stress (sometimes trunk, didn't seem to matter) and a quad core Linux desktop. I can reproduce this the other way around as well. Stress command was {code} cassandra-stress write n=19000000 -rate threads=2000 -mode thrift -node 192.168.1.3 {code} I will attach the YAML. Error was {code} java.lang.RuntimeException: java.lang.AssertionError: Invoke called in invalid state: READY_TO_WRITE at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45) ~[disruptor-3.0.1.jar:na] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:126) ~[disruptor-3.0.1.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60] Caused by: java.lang.AssertionError: Invoke called in invalid state: READY_TO_WRITE at com.thinkaurelius.thrift.Message.invoke(Message.java:306) ~[thrift-server-0.3.7.jar:na] at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90) ~[thrift-server-0.3.7.jar:na] at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695) ~[thrift-server-0.3.7.jar:na] at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689) ~[thrift-server-0.3.7.jar:na] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112) ~[disruptor-3.0.1.jar:na] ... 3 common frames omitted {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)