Robert Stupp created CASSANDRA-9923: ---------------------------------------
Summary: stress against counters hangs Key: CASSANDRA-9923 URL: https://issues.apache.org/jira/browse/CASSANDRA-9923 Project: Cassandra Issue Type: Bug Reporter: Robert Stupp (Sorry for the vague description) I tried some cstar tests against counter columns. But all these tests against 2.1 and 2.2 ended (hang) during with the following output: {noformat} Created keyspaces. Sleeping 3s for propagation. Sleeping 2s... Warming up COUNTER_WRITE with 150000 iterations... Running COUNTER_WRITE with 300 threads for 15000000 iteration type, total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb total, 98073, 98054, 98054, 98054, 3.1, 1.7, 8.9, 23.2, 89.9, 107.7, 1.0, 0.00000, 0, 0, 0, 0, 0, 0 total, 188586, 72492, 72492, 72492, 4.1, 1.5, 10.0, 61.4, 202.8, 214.7, 2.2, 0.13101, 0, 3, 564, 564, 6, 3447 total, 363880, 137986, 137986, 137986, 2.2, 1.4, 4.1, 9.6, 207.1, 253.3, 3.5, 0.18684, 0, 0, 0, 0, 0, 0 total, 460122, 105062, 105062, 105062, 2.8, 1.4, 4.6, 14.7, 225.6, 236.2, 4.4, 0.13969, 0, 1, 214, 214, 0, 1199 total, 600625, 111453, 111453, 111453, 2.7, 1.4, 3.8, 10.4, 231.5, 241.6, 5.7, 0.11366, 0, 2, 442, 442, 1, 2389 total, 745680, 149583, 149583, 149583, 2.0, 1.4, 3.6, 6.7, 155.8, 159.7, 6.7, 0.11318, 0, 0, 0, 0, 0, 0 total, 828453, 63632, 63632, 63632, 4.7, 1.4, 4.8, 261.9, 274.5, 279.3, 8.0, 0.12645, 0, 3, 782, 782, 1, 3542 total, 1009560, 172429, 172429, 172429, 1.7, 1.4, 3.7, 6.1, 16.2, 29.7, 9.0, 0.11629, 0, 0, 0, 0, 0, 0 total, 1062409, 53860, 53860, 53860, 5.5, 1.3, 8.6, 270.3, 293.4, 324.3, 10.0, 0.12738, 0, 2, 542, 542, 7, 2354 total, 1186672, 96540, 96540, 96540, 3.1, 1.5, 5.9, 14.5, 266.4, 277.6, 11.3, 0.11451, 0, 1, 260, 260, 0, 1183 {noformat} ... {noformat} total, 4977251, 238, 238, 238, 0.7, 0.6, 0.7, 1.3, 3.4, 158.5, 352.3, 0.11749, 0, 0, 0, 0, 0, 0 total, 4979839, 214, 214, 214, 0.6, 0.6, 0.7, 1.3, 2.5, 2.8, 364.4, 0.11761, 0, 0, 0, 0, 0, 0 total, 4981729, 191, 191, 191, 0.6, 0.6, 0.7, 1.3, 3.2, 3.3, 374.3, 0.11774, 0, 0, 0, 0, 0, 0 total, 4983362, 167, 167, 167, 0.8, 0.7, 1.8, 2.7, 3.9, 5.8, 384.0, 0.11787, 0, 0, 0, 0, 0, 0 total, 4985171, 153, 153, 153, 0.7, 0.6, 1.2, 1.4, 2.0, 3.3, 395.9, 0.11799, 0, 0, 0, 0, 0, 0 total, 4986684, 137, 137, 137, 0.7, 0.6, 0.8, 1.3, 2.0, 2.0, 406.9, 0.11812, 0, 0, 0, 0, 0, 0 total, 4988410, 121, 121, 121, 0.7, 0.7, 0.8, 1.3, 2.0, 2.8, 421.1, 0.11824, 0, 0, 0, 0, 0, 0 total, 4990216, 99, 99, 99, 0.7, 0.7, 0.8, 1.4, 2.6, 2.8, 439.5, 0.11836, 0, 0, 0, 0, 0, 0 total, 4991765, 81, 81, 81, 0.8, 0.7, 0.8, 1.4, 30.1, 81.6, 458.7, 0.11848, 0, 1, 159, 159, 0, 1179 total, 4993731, 67, 67, 67, 0.7, 0.7, 0.8, 1.4, 3.2, 3.2, 488.1, 0.11860, 0, 0, 0, 0, 0, 0 total, 4996565, 45, 45, 45, 0.9, 0.7, 0.9, 1.5, 84.7, 218.3, 551.5, 0.11872, 0, 1, 248, 248, 0, 1180 java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck. Check GC/Heap size at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98) at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156) at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37) at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104) at java.lang.Thread.run(Thread.java:745) {noformat} I don't know whether it's a stress or a counter issue - but wanted to record it for further investigation. [Console log file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console] cstar test steps: # counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300 # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300 # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50 -- This message was sent by Atlassian JIRA (v6.3.4#6332)