[ https://issues.apache.org/jira/browse/CASSANDRA-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dikang Gu reopened CASSANDRA-8681: ---------------------------------- Reproduced In: 3.9, 2.2.0 beta 1 (was: 2.2.0 beta 1) I want to re-open this issue, because I constantly hit this when benchmarking 3.9 branch. command I use: {code} /tools/bin/cassandra-stress user n=1000000000 profile=/root/cassandra_stress.yaml ops\(insert=2,singlepost=1\) cl=ONE -node 2401:db00:21:2121:face:0:13:0 -rate threads=100 -log file=~/cass-stress-1b.log {code} error: {code} total, 123245370, 137904, 119461, 119461, 0.7, 0.5, 1.9, 3.5, 6.6, 21.4, 980.2, 0.13322, 0, 0, 0, 0, 0, 0 com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write) com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write) insert, 82257602, 79373, 79373, 79373, 0.7, 0.4, 1.7, 3.3, 18.2, 133.2, 981.3, 0.13313, 0, 0, 0, 0, 0, 0 singlepost, 41114666, 40220, 24092, 24092, 1.0, 0.7, 2.1, 4.0, 129.1, 131.9, 981.3, 0.13313, 0, 0, 0, 0, 0, 0 total, 123372268, 119590, 103462, 103462, 0.8, 0.5, 1.9, 3.6, 18.7, 133.2, 981.3, 0.13313, 0, 0, 0, 0, 0, 0 com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write) com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write) com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency ONE (1 responses were required but only 0 replica responded, 1 failed) java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.util.NoSuchElementException java.io.IOException: Operation x10 on key(s) [|=VO oq3# H"gz`_U%A'qqp3uH{C(ux-1^6>9=PYq]M:+]: Error executing: (NoSuchElementException) at org.apache.cassandra.stress.Operation.error(Operation.java:138) at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116) at org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:156) at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:434) FAILURE java.lang.RuntimeException: Failed to execute stress action at org.apache.cassandra.stress.StressAction.run(StressAction.java:86) at org.apache.cassandra.stress.Stress.run(Stress.java:133) at org.apache.cassandra.stress.Stress.main(Stress.java:61) [root@cassandrarocksdbtest1-prn <lxc> /data/apache-cassandra-3.9 ] # {code} > cassandra-stress fails after single ReadTimeoutException > -------------------------------------------------------- > > Key: CASSANDRA-8681 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8681 > Project: Cassandra > Issue Type: Bug > Components: Tools > Reporter: Robert Stupp > Assignee: T Jake Luciani > Labels: stress > > After a single exception is propagated from the Java Driver, cassandra-stress > prints out some {{NoSuchElementException}}s and stops working. > {noformat} > Running [singlepost, timeline] with 36 threads 2 minutes > total ops , adj row/s, op/s, pk/s, row/s, mean, med, .95, > .99, .999, max, time, stderr, gc: #, max ms, sum ms, sdv > ms, mb > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (timeout while waiting for repair of > inconsistent replica) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (timeout while waiting for repair of > inconsistent replica) > java.util.NoSuchElementException > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > java.util.NoSuchElementException > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > java.util.NoSuchElementException > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > java.util.NoSuchElementException > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded) > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (timeout while waiting for repair of > inconsistent replica) > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.util.NoSuchElementException > java.io.IOException: Operation x10 on key(s) [Z$Vq"zj\Aw}KG2>9$TY(!">e) > > m#$D1x??]D?2S]: Error executing: (NoSuchElementException) > java.util.NoSuchElementException > java.util.NoSuchElementException > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > java.util.NoSuchElementException > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)java.util.NoSuchElementException > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159)java.util.NoSuchElementException > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [ot^XJz*C{ > HYUg+"' j^36` > KX aj0n;]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [\!d,*&M D4 *^%=c MPZBZ+%W=2J > > Ic~s1/[Ba?3}V]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > f\llVm8IMYExceptX"RCSD}:Hx~0-n x&Q on key(s) [3+`&BP0yu,sdQ!s+CYw]ILLPk > v)^IJ<z`>SU- > `]: Error executing: > (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [KDIWY$$D?.+(y2iT="toY > 5\.^ > _dV]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > [(>V 1]: Error executing: > (NoSuchElementException)4f}n-K|.ly6q&5-/QLgu~2V{fmw5OJ;+!(v' <T?#A$ > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) > [$:YcLz5^?(v$aQR7b~JEjd,|]X8R><6u@P(zCTS"d0Z]: Error executing: > (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [6-1YKka#HB2^o.g{9Mvq({ > I9-I^!5S?jhNA>h#A434]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [/aW4D-aS > lv2)fa_0I+4$ > > C&]}WedM^B9D<v.\]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > java.io.IOException: Operation x10 on key(s) [lk/! > py% > u"@$%wO4)2q]#j+1kE+0 > x.8lvA}"R6}f_]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > $ e}*#D*T&*j Ction: Operation x10 on key(s) [ > T6{DlsMm' u.z]: Error executing: (NoSuchElementException) > at org.apache.cassandra.stress.Operation.error(Operation.java:211) > at > org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183) > at > org.apache.cassandra.stress.operations.userdefined.SchemaQuery.run(SchemaQuery.java:159) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:296) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)