[ https://issues.apache.org/jira/browse/CASSANDRA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284625#comment-15284625 ]
Himansh verma commented on CASSANDRA-9923: ------------------------------------------ I am getting the following exception while running load on my table. If thread count exceeds 20 only then it shows the following exception. In case of less threads its working fine. I also tried to increase the HEAP size by giving -Xmx3G in cassandra-stress shell file java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck. Check GC/Heap size at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98) at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:157) at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:38) at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:105) at java.lang.Thread.run(Unknown Source) I am using the below command to run my stress : cassandra-stress user profile=myTable.yaml ops\(insert=5, get1=500 , get2=500 , update1=50 , update2=50 , delete1=5\) cl=QUORUM no-warmup duration=7200s -rate threads=24 -log file=myTableReport.txt -node xxx.xxx.xxx.xxx -mode thrift user=xxx password=xxx Here is the yaml file. ----------------------- keyspace: loadtest # The CQL for creating a keyspace (optional if it already exists) keyspace_definition: | CREATE KEYSPACE loadtest WITH replication = {'class':'NetworkTopologyStrategy','load':3}; # Table name table: myTable # The CQL for creating a table you wish to stress (optional if it already exists) table_definition: | CREATE TABLE myTable ( col1 text PRIMARY KEY, col2 text, col3 text, col4 text, col5 text ) WITH bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class':'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'} AND compression = {'sstable_compression':'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; columnspec: - name: col1 size: gaussian(36..36) population: uniform(1..10M) - name: col2 size: gaussian(1..320) - name: col3 size: gaussian(1..320) - name: col4 size: gaussian(1..15) - name: col5 size: gaussian(1..15) ### Batch Ratio Distribution Specifications ### insert: partitions: fixed(100) batchtype: UNLOGGED # Unlogged batches # # A list of queries you wish to run against the schema # queries: get1: cql: select * from myTable where col1 = ? fields: samerow get2: cql: select col1, col2, col3 from myTable where col1 = ? fields: samerow update1: cql: update myTable set col3 = ? where col1 = ? fields: samerow update2: cql: update myTable set col2 = ? , col3 = ? , col4 = ? , col5 = ? where col1 = ? fields: samerow delete1: cql: delete from myTable where col1 = ? fields: samerow ---------------------------------------- currently I'm running it on 2 vCPU machine. Is it Ok? or How much vCPU machine is required ?? > stress write and counter_write hangs > ------------------------------------ > > Key: CASSANDRA-9923 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9923 > Project: Cassandra > Issue Type: Bug > Components: Tools > Reporter: Robert Stupp > Assignee: T Jake Luciani > Priority: Minor > Labels: stress > > (Sorry for the vague description) > I tried some cstar tests against counter columns. But all these tests against > 2.1 and 2.2 ended (hang) during with the following output: > {noformat} > Created keyspaces. Sleeping 3s for propagation. > Sleeping 2s... > Warming up COUNTER_WRITE with 150000 iterations... > Running COUNTER_WRITE with 300 threads for 15000000 iteration > type, total ops, op/s, pk/s, row/s, mean, med, .95, > .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, > sdv ms, mb > total, 98073, 98054, 98054, 98054, 3.1, 1.7, 8.9, > 23.2, 89.9, 107.7, 1.0, 0.00000, 0, 0, 0, 0, > 0, 0 > total, 188586, 72492, 72492, 72492, 4.1, 1.5, 10.0, > 61.4, 202.8, 214.7, 2.2, 0.13101, 0, 3, 564, 564, > 6, 3447 > total, 363880, 137986, 137986, 137986, 2.2, 1.4, 4.1, > 9.6, 207.1, 253.3, 3.5, 0.18684, 0, 0, 0, 0, > 0, 0 > total, 460122, 105062, 105062, 105062, 2.8, 1.4, 4.6, > 14.7, 225.6, 236.2, 4.4, 0.13969, 0, 1, 214, 214, > 0, 1199 > total, 600625, 111453, 111453, 111453, 2.7, 1.4, 3.8, > 10.4, 231.5, 241.6, 5.7, 0.11366, 0, 2, 442, 442, > 1, 2389 > total, 745680, 149583, 149583, 149583, 2.0, 1.4, 3.6, > 6.7, 155.8, 159.7, 6.7, 0.11318, 0, 0, 0, 0, > 0, 0 > total, 828453, 63632, 63632, 63632, 4.7, 1.4, 4.8, > 261.9, 274.5, 279.3, 8.0, 0.12645, 0, 3, 782, 782, > 1, 3542 > total, 1009560, 172429, 172429, 172429, 1.7, 1.4, 3.7, > 6.1, 16.2, 29.7, 9.0, 0.11629, 0, 0, 0, 0, > 0, 0 > total, 1062409, 53860, 53860, 53860, 5.5, 1.3, 8.6, > 270.3, 293.4, 324.3, 10.0, 0.12738, 0, 2, 542, 542, > 7, 2354 > total, 1186672, 96540, 96540, 96540, 3.1, 1.5, 5.9, > 14.5, 266.4, 277.6, 11.3, 0.11451, 0, 1, 260, 260, > 0, 1183 > {noformat} > ... > {noformat} > total, 4977251, 238, 238, 238, 0.7, 0.6, 0.7, > 1.3, 3.4, 158.5, 352.3, 0.11749, 0, 0, 0, 0, > 0, 0 > total, 4979839, 214, 214, 214, 0.6, 0.6, 0.7, > 1.3, 2.5, 2.8, 364.4, 0.11761, 0, 0, 0, 0, > 0, 0 > total, 4981729, 191, 191, 191, 0.6, 0.6, 0.7, > 1.3, 3.2, 3.3, 374.3, 0.11774, 0, 0, 0, 0, > 0, 0 > total, 4983362, 167, 167, 167, 0.8, 0.7, 1.8, > 2.7, 3.9, 5.8, 384.0, 0.11787, 0, 0, 0, 0, > 0, 0 > total, 4985171, 153, 153, 153, 0.7, 0.6, 1.2, > 1.4, 2.0, 3.3, 395.9, 0.11799, 0, 0, 0, 0, > 0, 0 > total, 4986684, 137, 137, 137, 0.7, 0.6, 0.8, > 1.3, 2.0, 2.0, 406.9, 0.11812, 0, 0, 0, 0, > 0, 0 > total, 4988410, 121, 121, 121, 0.7, 0.7, 0.8, > 1.3, 2.0, 2.8, 421.1, 0.11824, 0, 0, 0, 0, > 0, 0 > total, 4990216, 99, 99, 99, 0.7, 0.7, 0.8, > 1.4, 2.6, 2.8, 439.5, 0.11836, 0, 0, 0, 0, > 0, 0 > total, 4991765, 81, 81, 81, 0.8, 0.7, 0.8, > 1.4, 30.1, 81.6, 458.7, 0.11848, 0, 1, 159, 159, > 0, 1179 > total, 4993731, 67, 67, 67, 0.7, 0.7, 0.8, > 1.4, 3.2, 3.2, 488.1, 0.11860, 0, 0, 0, 0, > 0, 0 > total, 4996565, 45, 45, 45, 0.9, 0.7, 0.9, > 1.5, 84.7, 218.3, 551.5, 0.11872, 0, 1, 248, 248, > 0, 1180 > java.lang.RuntimeException: Timed out waiting for a timer thread - seems one > got stuck. Check GC/Heap size > at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98) > at > org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156) > at > org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37) > at > org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I don't know whether it's a stress or a counter issue - but wanted to record > it for further investigation. > [Console log > file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console] > cstar test steps: > # counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300 > # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300 > # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50 -- This message was sent by Atlassian JIRA (v6.3.4#6332)