Robert Stupp created CASSANDRA-9923:
---------------------------------------
Summary: stress against counters hangs
Key: CASSANDRA-9923
URL: https://issues.apache.org/jira/browse/CASSANDRA-9923
Project: Cassandra
Issue Type: Bug
Reporter: Robert Stupp
(Sorry for the vague description)
I tried some cstar tests against counter columns. But all these tests against
2.1 and 2.2 ended (hang) during with the following output:
{noformat}
Created keyspaces. Sleeping 3s for propagation.
Sleeping 2s...
Warming up COUNTER_WRITE with 150000 iterations...
Running COUNTER_WRITE with 300 threads for 15000000 iteration
type, total ops, op/s, pk/s, row/s, mean, med, .95,
.99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv
ms, mb
total, 98073, 98054, 98054, 98054, 3.1, 1.7, 8.9,
23.2, 89.9, 107.7, 1.0, 0.00000, 0, 0, 0, 0,
0, 0
total, 188586, 72492, 72492, 72492, 4.1, 1.5, 10.0,
61.4, 202.8, 214.7, 2.2, 0.13101, 0, 3, 564, 564,
6, 3447
total, 363880, 137986, 137986, 137986, 2.2, 1.4, 4.1,
9.6, 207.1, 253.3, 3.5, 0.18684, 0, 0, 0, 0,
0, 0
total, 460122, 105062, 105062, 105062, 2.8, 1.4, 4.6,
14.7, 225.6, 236.2, 4.4, 0.13969, 0, 1, 214, 214,
0, 1199
total, 600625, 111453, 111453, 111453, 2.7, 1.4, 3.8,
10.4, 231.5, 241.6, 5.7, 0.11366, 0, 2, 442, 442,
1, 2389
total, 745680, 149583, 149583, 149583, 2.0, 1.4, 3.6,
6.7, 155.8, 159.7, 6.7, 0.11318, 0, 0, 0, 0,
0, 0
total, 828453, 63632, 63632, 63632, 4.7, 1.4, 4.8,
261.9, 274.5, 279.3, 8.0, 0.12645, 0, 3, 782, 782,
1, 3542
total, 1009560, 172429, 172429, 172429, 1.7, 1.4, 3.7,
6.1, 16.2, 29.7, 9.0, 0.11629, 0, 0, 0, 0,
0, 0
total, 1062409, 53860, 53860, 53860, 5.5, 1.3, 8.6,
270.3, 293.4, 324.3, 10.0, 0.12738, 0, 2, 542, 542,
7, 2354
total, 1186672, 96540, 96540, 96540, 3.1, 1.5, 5.9,
14.5, 266.4, 277.6, 11.3, 0.11451, 0, 1, 260, 260,
0, 1183
{noformat}
...
{noformat}
total, 4977251, 238, 238, 238, 0.7, 0.6, 0.7,
1.3, 3.4, 158.5, 352.3, 0.11749, 0, 0, 0, 0,
0, 0
total, 4979839, 214, 214, 214, 0.6, 0.6, 0.7,
1.3, 2.5, 2.8, 364.4, 0.11761, 0, 0, 0, 0,
0, 0
total, 4981729, 191, 191, 191, 0.6, 0.6, 0.7,
1.3, 3.2, 3.3, 374.3, 0.11774, 0, 0, 0, 0,
0, 0
total, 4983362, 167, 167, 167, 0.8, 0.7, 1.8,
2.7, 3.9, 5.8, 384.0, 0.11787, 0, 0, 0, 0,
0, 0
total, 4985171, 153, 153, 153, 0.7, 0.6, 1.2,
1.4, 2.0, 3.3, 395.9, 0.11799, 0, 0, 0, 0,
0, 0
total, 4986684, 137, 137, 137, 0.7, 0.6, 0.8,
1.3, 2.0, 2.0, 406.9, 0.11812, 0, 0, 0, 0,
0, 0
total, 4988410, 121, 121, 121, 0.7, 0.7, 0.8,
1.3, 2.0, 2.8, 421.1, 0.11824, 0, 0, 0, 0,
0, 0
total, 4990216, 99, 99, 99, 0.7, 0.7, 0.8,
1.4, 2.6, 2.8, 439.5, 0.11836, 0, 0, 0, 0,
0, 0
total, 4991765, 81, 81, 81, 0.8, 0.7, 0.8,
1.4, 30.1, 81.6, 458.7, 0.11848, 0, 1, 159, 159,
0, 1179
total, 4993731, 67, 67, 67, 0.7, 0.7, 0.8,
1.4, 3.2, 3.2, 488.1, 0.11860, 0, 0, 0, 0,
0, 0
total, 4996565, 45, 45, 45, 0.9, 0.7, 0.9,
1.5, 84.7, 218.3, 551.5, 0.11872, 0, 1, 248, 248,
0, 1180
java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
got stuck. Check GC/Heap size
at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98)
at
org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
at
org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37)
at
org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
at java.lang.Thread.run(Thread.java:745)
{noformat}
I don't know whether it's a stress or a counter issue - but wanted to record it
for further investigation.
[Console log
file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console]
cstar test steps:
# counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300
# counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300
# counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
