[
https://issues.apache.org/jira/browse/CASSANDRA-13871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Cranford updated CASSANDRA-13871:
----------------------------------------
Description:
o.a.c.stress.Operation will retry queries a configurable number of times. When
the "user" command is invoked the o.a.c.stress.operations.userdefined
SchemaInsert and SchemaQuery operations are used.
When SchemaInsert and SchemaQuery are retried (eg after a Read/WriteTimeout
exception), they advance the PartitionIterator used to generate the keys to
insert/query (SchemaInsert.java:85 SchemaQuery.java:129) This means each retry
will use a different set of keys.
The predefined set of operations avoid this problem by packaging up the
arguments to bind to the query into the RunOp object so that retrying the
operation results in exactly the same query with the same arguments being run.
This problem was introduced by CASSANDRA-7964. Prior to CASSANDRA-7964 the
PartitionIterator (Partition.RowIterator before the change) was reinitialized
prior to each query retry, thus generating the same set of keys each time.
This problem is reported rather confusingly. The only error that shows up in a
log file (specified with -log file=foo.log) is the unhelpful
{noformat}
java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
(NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:136)
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
at
org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
at
org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
{noformat}
Standard error is only slightly more helpful, displaying the ignorable initial
read/write error, and confusing java.util.NoSuchElementException lines (caused
by PartitionIterator exhaustion) followed by the above IOException with stack
trace, eg
{noformat}
com.datastax.drive.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query....
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
(NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:136)
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
at
org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
at
org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
{noformat}
was:
o.a.c.stress.Operation will retry queries a configurable number of times. When
the "user" command is invoked the o.a.c.stress.operations.userdefined
SchemaInsert and SchemaQuery operations are used.
When SchemaInsert and SchemaQuery are retried (eg after a Read/WriteTimeout
exception), they advance the PartitionIterator used to generate the keys to
insert/query (SchemaInsert.java:85 SchemaQuery.java:129) This means each retry
will use a different set of keys.
The predefined set of operations avoid this problem by packaging up the
arguments to bind to the query into the RunOp object so that retrying the
operation results in exactly the same query with the same arguments being run.
This problem was introduced by CASSANDRA-7964. Prior to CASSANDRA-7964 the
PartitionIterator (Partition.RowIterator before the change) was reinitialized
prior to each query retry, thus generating the same set of keys each time.
This problem is reported rather confusingly. The only error that shows up in a
log file (specified with -log file=foo.log) is the unhelpful
{{{
java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
(NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:136)
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
at
org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
at
org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
}}}
Standard error is only slightly more helpful, displaying the ignorable initial
read/write error, and confusing java.util.NoSuchElementException lines (caused
by PartitionIterator exhaustion) followed by the above IOException with stack
trace, eg
{{{
com.datastax.drive.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query....
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.util.NoSuchElementException
java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
(NoSuchElementException)
at org.apache.cassandra.stress.Operation.error(Operation.java:136)
at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
at
org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
at
org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
}}}
> cassandra-stress user command misbehaves when retrying operations
> -----------------------------------------------------------------
>
> Key: CASSANDRA-13871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13871
> Project: Cassandra
> Issue Type: Bug
> Reporter: Daniel Cranford
>
> o.a.c.stress.Operation will retry queries a configurable number of times.
> When the "user" command is invoked the o.a.c.stress.operations.userdefined
> SchemaInsert and SchemaQuery operations are used.
> When SchemaInsert and SchemaQuery are retried (eg after a Read/WriteTimeout
> exception), they advance the PartitionIterator used to generate the keys to
> insert/query (SchemaInsert.java:85 SchemaQuery.java:129) This means each
> retry will use a different set of keys.
> The predefined set of operations avoid this problem by packaging up the
> arguments to bind to the query into the RunOp object so that retrying the
> operation results in exactly the same query with the same arguments being run.
> This problem was introduced by CASSANDRA-7964. Prior to CASSANDRA-7964 the
> PartitionIterator (Partition.RowIterator before the change) was reinitialized
> prior to each query retry, thus generating the same set of keys each time.
> This problem is reported rather confusingly. The only error that shows up in
> a log file (specified with -log file=foo.log) is the unhelpful
> {noformat}
> java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
> (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:136)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
> at
> org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
> {noformat}
> Standard error is only slightly more helpful, displaying the ignorable
> initial read/write error, and confusing java.util.NoSuchElementException
> lines (caused by PartitionIterator exhaustion) followed by the above
> IOException with stack trace, eg
> {noformat}
> com.datastax.drive.core.exceptions.ReadTimeoutException: Cassandra timeout
> during read query....
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.util.NoSuchElementException
> java.io.IOException Operation x10 on key(s) [foobarkey]: Error executing:
> (NoSuchElementException)
> at org.apache.cassandra.stress.Operation.error(Operation.java:136)
> at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:114)
> at
> org.apache.cassandra.stress.userdefined.SchemaQuery.run(SchemaQuery.java:158)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:459)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
