[
https://issues.apache.org/jira/browse/CASSANDRA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064180#comment-15064180
]
T Jake Luciani commented on CASSANDRA-9923:
-------------------------------------------
This is a known error. Mainly stress is running low on heap. We can add an
env variable for stress shell script to change the max heap, or maybe just set
mx to something high [~snazy]
> stress write and counter_write hangs
> ------------------------------------
>
> Key: CASSANDRA-9923
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9923
> Project: Cassandra
> Issue Type: Bug
> Reporter: Robert Stupp
> Assignee: T Jake Luciani
> Labels: stress
>
> (Sorry for the vague description)
> I tried some cstar tests against counter columns. But all these tests against
> 2.1 and 2.2 ended (hang) during with the following output:
> {noformat}
> Created keyspaces. Sleeping 3s for propagation.
> Sleeping 2s...
> Warming up COUNTER_WRITE with 150000 iterations...
> Running COUNTER_WRITE with 300 threads for 15000000 iteration
> type, total ops, op/s, pk/s, row/s, mean, med, .95,
> .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms,
> sdv ms, mb
> total, 98073, 98054, 98054, 98054, 3.1, 1.7, 8.9,
> 23.2, 89.9, 107.7, 1.0, 0.00000, 0, 0, 0, 0,
> 0, 0
> total, 188586, 72492, 72492, 72492, 4.1, 1.5, 10.0,
> 61.4, 202.8, 214.7, 2.2, 0.13101, 0, 3, 564, 564,
> 6, 3447
> total, 363880, 137986, 137986, 137986, 2.2, 1.4, 4.1,
> 9.6, 207.1, 253.3, 3.5, 0.18684, 0, 0, 0, 0,
> 0, 0
> total, 460122, 105062, 105062, 105062, 2.8, 1.4, 4.6,
> 14.7, 225.6, 236.2, 4.4, 0.13969, 0, 1, 214, 214,
> 0, 1199
> total, 600625, 111453, 111453, 111453, 2.7, 1.4, 3.8,
> 10.4, 231.5, 241.6, 5.7, 0.11366, 0, 2, 442, 442,
> 1, 2389
> total, 745680, 149583, 149583, 149583, 2.0, 1.4, 3.6,
> 6.7, 155.8, 159.7, 6.7, 0.11318, 0, 0, 0, 0,
> 0, 0
> total, 828453, 63632, 63632, 63632, 4.7, 1.4, 4.8,
> 261.9, 274.5, 279.3, 8.0, 0.12645, 0, 3, 782, 782,
> 1, 3542
> total, 1009560, 172429, 172429, 172429, 1.7, 1.4, 3.7,
> 6.1, 16.2, 29.7, 9.0, 0.11629, 0, 0, 0, 0,
> 0, 0
> total, 1062409, 53860, 53860, 53860, 5.5, 1.3, 8.6,
> 270.3, 293.4, 324.3, 10.0, 0.12738, 0, 2, 542, 542,
> 7, 2354
> total, 1186672, 96540, 96540, 96540, 3.1, 1.5, 5.9,
> 14.5, 266.4, 277.6, 11.3, 0.11451, 0, 1, 260, 260,
> 0, 1183
> {noformat}
> ...
> {noformat}
> total, 4977251, 238, 238, 238, 0.7, 0.6, 0.7,
> 1.3, 3.4, 158.5, 352.3, 0.11749, 0, 0, 0, 0,
> 0, 0
> total, 4979839, 214, 214, 214, 0.6, 0.6, 0.7,
> 1.3, 2.5, 2.8, 364.4, 0.11761, 0, 0, 0, 0,
> 0, 0
> total, 4981729, 191, 191, 191, 0.6, 0.6, 0.7,
> 1.3, 3.2, 3.3, 374.3, 0.11774, 0, 0, 0, 0,
> 0, 0
> total, 4983362, 167, 167, 167, 0.8, 0.7, 1.8,
> 2.7, 3.9, 5.8, 384.0, 0.11787, 0, 0, 0, 0,
> 0, 0
> total, 4985171, 153, 153, 153, 0.7, 0.6, 1.2,
> 1.4, 2.0, 3.3, 395.9, 0.11799, 0, 0, 0, 0,
> 0, 0
> total, 4986684, 137, 137, 137, 0.7, 0.6, 0.8,
> 1.3, 2.0, 2.0, 406.9, 0.11812, 0, 0, 0, 0,
> 0, 0
> total, 4988410, 121, 121, 121, 0.7, 0.7, 0.8,
> 1.3, 2.0, 2.8, 421.1, 0.11824, 0, 0, 0, 0,
> 0, 0
> total, 4990216, 99, 99, 99, 0.7, 0.7, 0.8,
> 1.4, 2.6, 2.8, 439.5, 0.11836, 0, 0, 0, 0,
> 0, 0
> total, 4991765, 81, 81, 81, 0.8, 0.7, 0.8,
> 1.4, 30.1, 81.6, 458.7, 0.11848, 0, 1, 159, 159,
> 0, 1179
> total, 4993731, 67, 67, 67, 0.7, 0.7, 0.8,
> 1.4, 3.2, 3.2, 488.1, 0.11860, 0, 0, 0, 0,
> 0, 0
> total, 4996565, 45, 45, 45, 0.9, 0.7, 0.9,
> 1.5, 84.7, 218.3, 551.5, 0.11872, 0, 1, 248, 248,
> 0, 1180
> java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
> got stuck. Check GC/Heap size
> at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98)
> at
> org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
> at
> org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37)
> at
> org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I don't know whether it's a stress or a counter issue - but wanted to record
> it for further investigation.
> [Console log
> file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console]
> cstar test steps:
> # counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
