[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018958#comment-14018958 ] Benedict commented on CASSANDRA-7245: - Might be worthwhile clarifying the comment on only releasing the prefix of mutations we'd actually despatched. Otherwise LGTM +1 > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > 7245v4.txt, 7245v5.txt, 7245v6.txt, netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018731#comment-14018731 ] Benedict commented on CASSANDRA-7245: - bq. MS.addCallback also retains the mutation, this will be released by either the timeoutReporter, or in the ResponseVerbHander Unfortunately there's still a race condition there, as these can be released by the ExpiringMap reaper (or by the responses arriving in the interval) before the exception block runs, so that they've been reallocated by the time we submit them as hints > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > 7245v4.txt, 7245v5.txt, netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018428#comment-14018428 ] T Jake Luciani commented on CASSANDRA-7245: --- bq. In SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. MS.addCallback also retains the mutation, this will be released by either the timeoutReporter, or in the ResponseVerbHander You are right on the other two points, I got rid of the ThreadLocal and moved the frame to QueryState only. and I fixed the DroppableRunnable so release always gets called. Attached as v5 > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > 7245v4.txt, netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018257#comment-14018257 ] Benedict commented on CASSANDRA-7245: - [~jasobrown] I got this one if you've got better stuff to do :-) [~tjke]: * I think the change you made doesn't actually totally eliminate the bug: since it extends DroppableRunnable, the finally block may never be run * I think it would be good in SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. It might be nice, however, to have the response handler release() the reference once we receive enough responses from our replicas, so that we don't keep all of the references if we're just waiting for one mutation * Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store it directly in a field in QueryState? Nit: * WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser since it's always init'd empty Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection with a simple array. We know the range is < 32K (<.1K for older clients), and each index is accessed by a single thread at any given time, so we'd just need to be able to atomically swap in larger arrays if we wanted dynamic sizing. Otherwise LGTM > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > 7245v4.txt, netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017249#comment-14017249 ] Jason Brown commented on CASSANDRA-7245: [~tjake] v4 looks good for my tests. [~benedict] looks like you've been playing the at home game, as well. Do you want to handle the review? Else, I can do it (not a problem) > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > 7245v4.txt, netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016926#comment-14016926 ] Benedict commented on CASSANDRA-7245: - I think there's a bug if there is a counter write sent to an owning coordinator: SP.applyCounterMutationOnCoordinator does not take retain a reference, but it gets executed asynchronously on the countermutation executor service. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, > netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015976#comment-14015976
]
Jason Brown commented on CASSANDRA-7245:
[~tjake] can you rebase? the v3 patch didn't apply cleanly:
{code}
$ git apply ../patches/reviews/7245v3.txt
error: patch failed: src/java/org/apache/cassandra/service/StorageProxy.java:780
error: src/java/org/apache/cassandra/service/StorageProxy.java: patch does not
apply
error: patch failed: src/java/org/apache/cassandra/transport/Frame.java:199
error: src/java/org/apache/cassandra/transport/Frame.java: patch does not apply
{code}
> Out-of-Order keys with stress + CQL3
>
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Pavel Yaskevich
>Assignee: T Jake Luciani
> Fix For: 2.1.0
>
> Attachments: 7245-v2.txt, 7245.txt, 7245v3.txt,
> netty-all-4.0.19.Final.jar
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718
> and found following problem almost in every SSTable generated (~200 GB of
> data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1
> and 6.
> Once I ran 'sstablekeys' on the generated SSTable files I got following
> exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985,
> *30303039443745364242*) > DecoratedKey(-1767746583617597213,
> *30303039443437454333*)
> 0a30303033343933
> 3734441388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581,
> *3734441388343933*) > DecoratedKey(-7565486415339257200,
> *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180,
> *30303033354133423742*) > DecoratedKey(-7838239767410066684,
> *30303033354145344534*)
> 30303034313442354137
> 3034313635363334
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462,
> *3034313635363334*) > DecoratedKey(-9106177395653818217,
> *3030303431444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540,
> *30303035373044334631*) > DecoratedKey(-4296696226469000945,
> *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196,
> *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690,
> *30303041333943303232*)
> 3030303332314144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635,
> *30303033323346343932*) > DecoratedKey(5233296131921119414,
> *53d83e0012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193,
> *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663,
> *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 444236451388
> 1388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of
> out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015969#comment-14015969 ] Jason Brown commented on CASSANDRA-7245: will give it a shot this afternoon - thanks! > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt, 7245v3.txt, > netty-all-4.0.19.Final.jar > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014880#comment-14014880 ] T Jake Luciani commented on CASSANDRA-7245: --- After much experimentation I've found one of the issues here: Our version of netty throws away buffers once they have been read. This issue was fixed in the latest netty so buffer with refcounts aren't cannibalized https://github.com/netty/netty/issues/2327 I've tested a with 4.0.19 release and the token corruption is now gone. I am however seeing some kind of race condition which I think may be netty related as well. It seems like the same Message is being dispatched to multiple threads sometimes and causing refcount exceptions. I've added logging to capture the history when this happens and I'm trying to track this down. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1.0 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012349#comment-14012349 ] T Jake Luciani commented on CASSANDRA-7245: --- Ok, I pushed Jason's fix > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012339#comment-14012339 ] T Jake Luciani commented on CASSANDRA-7245: --- Both accomplish the same goal. since you've already tested your patch [~jasobrown] let's go with that (ninja?). I'll continue working on this in the meantime. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012324#comment-14012324 ] Jason Brown commented on CASSANDRA-7245: I'm reasonably OK with the idea of shipping either my fix or [~tjake]'s pool suggestion as a temporary relief. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012260#comment-14012260 ] Sylvain Lebresne commented on CASSANDRA-7245: - Since it's time to go ahead with 2.1 RC1, and unless somewhat has a fix he is 100% confident with now, can we commit Jason's copy changes above (or change the allocator) for now? We can then take the time to fix this correctly without copy for 2.1.1 (or 2.1.0 final, I don't really mind if we test properly). > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012074#comment-14012074 ] Pavel Yaskevich commented on CASSANDRA-7245: Perfect, that's what I wanted to hear :) > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012067#comment-14012067
]
Jason Brown commented on CASSANDRA-7245:
I took the 2.1 branch made the following minor change to CBUtil.readValue():
current:
{code}
public static ByteBuffer readValue(ByteBuf cb)
{
int length = cb.readInt();
if (length < 0)
return null;
ByteBuf slice = cb.readSlice(length);
if (slice.nioBufferCount() > 0)
return slice.nioBuffer();
else
return ByteBuffer.wrap(readRawBytes(cb));
}
{code}
slight change to always copy the netty buffer:
{code}
public static ByteBuffer readValue(ByteBuf cb)
{
int length = cb.readInt();
if (length < 0)
return null;
ByteBuf slice = cb.readSlice(length);
return ByteBuffer.wrap(readRawBytes(slice));
}
{code}
and ran stress again. It has been going for greater than four hours, and no
sign of the reported corruption (keys out of order exception) on the compaction
merge. Thus, I think it's reasonable to assume that data corruption is sneaking
in due to the wrapping/reuse of the netty buffer.
> Out-of-Order keys with stress + CQL3
>
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Pavel Yaskevich
>Assignee: T Jake Luciani
> Fix For: 2.1 rc1
>
> Attachments: 7245-v2.txt, 7245.txt
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718
> and found following problem almost in every SSTable generated (~200 GB of
> data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1
> and 6.
> Once I ran 'sstablekeys' on the generated SSTable files I got following
> exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985,
> *30303039443745364242*) > DecoratedKey(-1767746583617597213,
> *30303039443437454333*)
> 0a30303033343933
> 3734441388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581,
> *3734441388343933*) > DecoratedKey(-7565486415339257200,
> *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180,
> *30303033354133423742*) > DecoratedKey(-7838239767410066684,
> *30303033354145344534*)
> 30303034313442354137
> 3034313635363334
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462,
> *3034313635363334*) > DecoratedKey(-9106177395653818217,
> *3030303431444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540,
> *30303035373044334631*) > DecoratedKey(-4296696226469000945,
> *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196,
> *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690,
> *30303041333943303232*)
> 3030303332314144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635,
> *30303033323346343932*) > DecoratedKey(5233296131921119414,
> *53d83e0012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193,
> *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663,
> *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 444236451388
> 1388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of
> out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011960#comment-14011960 ] Pavel Yaskevich commented on CASSANDRA-7245: That's fine what we really want to test is if slicing of ByteBuf is what creates an issue or is there something else. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011927#comment-14011927 ] T Jake Luciani commented on CASSANDRA-7245: --- You can also work around the problem by changing the allocator in CBUtil to UnpooledByteBufAllocator(false) > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011880#comment-14011880 ] Pavel Yaskevich commented on CASSANDRA-7245: Ah, so that's the piece I'm missing, we check actual bytes iff tokens are equal. We are going to do a test when instead of ByteBuf.slice we are always going to copy the data to the byte[], let's see if that runs into the same problem. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011854#comment-14011854 ] Benedict commented on CASSANDRA-7245: - Correct on all the above :-) > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011850#comment-14011850 ] Jason Brown commented on CASSANDRA-7245: [~benedict] Just to clarify a few thing: Is the sorting within Memtable.put(), due to adding the entry to the ConcurrentSkipListMap? bq. assert order at flush SSTableWriter.beforeAppend(), correct? bq. whereas we write the raw key only still SSTableWriter? > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011835#comment-14011835 ] Benedict commented on CASSANDRA-7245: - Except that we sort in the Memtable/CLSM (and assert order at flush) by Token which is computed much earlier, whereas we write the raw key only. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011825#comment-14011825 ] Pavel Yaskevich commented on CASSANDRA-7245: I still not sure if Netty is a direct cause of this problem and this is why - in Memtable.put(DecoratedKey, ...) we clone key and cf, after that while Memtable is being flushed SSTableWriter.beforeAppend(DecoratedKey) does check if the last written key greater or equals to current key as there are no errors reported on the flush that most luckily leaves us with the situation when corruption (if it is a corruption) is happening after SSTableWriter.beforeAppend(DecoratedKey) check. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011749#comment-14011749 ] Benedict commented on CASSANDRA-7245: - There's a race in your patch where a response can come back from another machine while the expiringmap is timing it out (and writing a hint), causing a ref can be decremented twice. Can fix this by only calling the postExpireHook if remove() actually removes the item. It's a pretty narrow race, though, so not convinced it's the cause of such easy reproduction. Is it also possible Netty is tidying a reference for us if e.g. a connection gets disconnected? Should we take a single reference to guard our native-transport-request thread's participation in the write operation? Again, not convinced it's causing these problems, but it's a concern for border cases. FYI, it's not necessary for sourceFrame to be volatile, as it's always guarded by either an executor service or concurrenthashmap access > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011738#comment-14011738 ] Jason Brown commented on CASSANDRA-7245: [~tjake] well, i see less IllegalReferenceCountExceptions, but still getting the keys out of order exception. I can get the cluster into that state within < 5 minutes using the same stress command i inserted earlier in this ticket. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > Attachments: 7245-v2.txt, 7245.txt > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011299#comment-14011299
]
Jason Brown commented on CASSANDRA-7245:
Hmm, unfortunately, this patch makes things worse for me :(. I still am seeing
the decorated key exception on compaction (including here since we haven'y so
far in the ticket):
{code}ERROR [CompactionExecutor:4] 2014-05-28 16:42:36,909
CassandraDaemon.java:166 - Exception in thread
Thread[CompactionExecutor:4,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2628246975040818544,
30303030373145304638) >= current key DecoratedKey(-7730803788015025148,
30303030364642324138) writing into
/u/sdd/cassandra-jasobrown/data/Keyspace1/Standard1-48037880e68611e3865a919228363515/Keyspace1-Standard1-tmp-ka-22-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:196)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:109)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:177)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:64)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:235)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_13]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
~[na:1.7.0_13]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
~[na:1.7.0_13]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_13]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_13]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
{code}
But now I also ref counting exceptions:
{code}ERROR [Native-Transport-Requests:176] 2014-05-28 16:37:54,838
ErrorMessage.java:218 - Unexpected exception during request
io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
at
io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:64)
~[netty-all-4.0.17.Final.jar:4.0.17.Final]
at org.apache.cassandra.transport.Frame.retain(Frame.java:60)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.retain(Mutation.java:114)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:584)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:650)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:812)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:122)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:703)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:467)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:537)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:503)
~[apache-cassandra-2.1.0-beta2-SNAPSHOT.jar:2.1.0-beta2-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(Mo
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011012#comment-14011012 ] Benedict commented on CASSANDRA-7245: - Well, 5663 depends on 6861, so it explicitly releases references (as 6861 did), so it's not exactly a clean test > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010772#comment-14010772 ] Pavel Yaskevich commented on CASSANDRA-7245: Ref counting seems an obvious culprit to me but [~jasobrown] is reporting that CASSANDRA-5663 has the same problem with CASSANDRA-6861 reverted. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010738#comment-14010738
]
T Jake Luciani commented on CASSANDRA-7245:
---
Good news is I can reproduce now with the following dtest:
{code}
import time
from dtest import Tester, debug
from assertions import assert_unavailable
from tools import (since)
#Run with MAX_HEAP_SIZE=150M HEAP_NEWSIZE=100M
class TestNettyTransport(Tester):
@since('2.1')
def reused_bytebuffers_test(self):
debug("Creating a ring")
cluster = self.cluster
cluster.set_configuration_options(values={'concurrent_writes' : 2},
batch_commitlog=True)
cluster.populate(2).start()
[node1, node2] = cluster.nodelist()
cluster.start()
node1.stress(['write', 'n=10', '-schema',
'replication(factor=2)','-rate', 'threads=191','-mode','native','cql3',
'prepared','-col', 'n=fixed(21)', 'size=exp(11..42)'])
node1.stress(['read', 'n=10', '-mode','native','cql3', 'prepared'])
{code}
I've confirmed the issue is the pooled buffer allocator reclaiming the buffers
too soon. Working on patching the Mutations to keep references to the netty
frame.
> Out-of-Order keys with stress + CQL3
>
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Pavel Yaskevich
>Assignee: T Jake Luciani
> Fix For: 2.1 rc1
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718
> and found following problem almost in every SSTable generated (~200 GB of
> data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1
> and 6.
> Once I ran 'sstablekeys' on the generated SSTable files I got following
> exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985,
> *30303039443745364242*) > DecoratedKey(-1767746583617597213,
> *30303039443437454333*)
> 0a30303033343933
> 3734441388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581,
> *3734441388343933*) > DecoratedKey(-7565486415339257200,
> *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180,
> *30303033354133423742*) > DecoratedKey(-7838239767410066684,
> *30303033354145344534*)
> 30303034313442354137
> 3034313635363334
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462,
> *3034313635363334*) > DecoratedKey(-9106177395653818217,
> *3030303431444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540,
> *30303035373044334631*) > DecoratedKey(-4296696226469000945,
> *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196,
> *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690,
> *30303041333943303232*)
> 3030303332314144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635,
> *30303033323346343932*) > DecoratedKey(5233296131921119414,
> *53d83e0012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193,
> *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663,
> *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 444236451388
> 1388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of
> out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010389#comment-14010389 ] Benedict commented on CASSANDRA-7245: - To answer your question [~xedin], with RF>1 it's possible for the surmised bug to be hit without timeouts - if the write stage finishes applying its mutation before the OTC has forwarded the data to the next node, and we're running with CL.ONE (stress default), then the other node can get the corrupted data. If the other node was in the middle of a flush/GC, the OTC could easily be backed up causing this bug to present very commonly. I believe Jason hit this bug much more easily with RF=2, so this would explain everything. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010355#comment-14010355 ] Jason Brown commented on CASSANDRA-7245: seeing the corruption with 6861 and without 5663. So, in short, the presence of either patch causes the corruption. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010331#comment-14010331 ] Jason Brown commented on CASSANDRA-7245: Err, hold on - just ran a test with 6881 and no 5663, and saw the corruption (again, rather quickly). Let me try one more time, but with 6861 and without 5663. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010321#comment-14010321 ] T Jake Luciani commented on CASSANDRA-7245: --- I'm quite sure the issue is 6861 so let me post a patch for you to test > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010295#comment-14010295 ] Jason Brown commented on CASSANDRA-7245: Will do, but requires a bit of finesse as 5663 depends on some of the changes introduced in 6861. Just want to ensure those aren't very interesting changes. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010285#comment-14010285 ] Pavel Yaskevich commented on CASSANDRA-7245: It still leaves us with either 6861 or 5663, can you please generate with only 6861 reverted to rule out 5663? > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[
https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010271#comment-14010271
]
Jason Brown commented on CASSANDRA-7245:
Reverted CASSANDRA-6861 as well as CASSANDRA-5663, and was unable to reproduce
the keys out of order bug. I used a rather beefy, three node cluster and used
this stress command:
{code}cassandra-stress write n=6 -schema replication\(factor\=2\) -key
populate=1..6 -rate threads=42 -mode native prepared cql3 -port
native=9043 thrift=9161 -col n=fixed\(21\) size=exp\(11..42\){code}
> Out-of-Order keys with stress + CQL3
>
>
> Key: CASSANDRA-7245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Pavel Yaskevich
>Assignee: T Jake Luciani
> Fix For: 2.1 rc1
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718
> and found following problem almost in every SSTable generated (~200 GB of
> data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1
> and 6.
> Once I ran 'sstablekeys' on the generated SSTable files I got following
> exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985,
> *30303039443745364242*) > DecoratedKey(-1767746583617597213,
> *30303039443437454333*)
> 0a30303033343933
> 3734441388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581,
> *3734441388343933*) > DecoratedKey(-7565486415339257200,
> *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180,
> *30303033354133423742*) > DecoratedKey(-7838239767410066684,
> *30303033354145344534*)
> 30303034313442354137
> 3034313635363334
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462,
> *3034313635363334*) > DecoratedKey(-9106177395653818217,
> *3030303431444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540,
> *30303035373044334631*) > DecoratedKey(-4296696226469000945,
> *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196,
> *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690,
> *30303041333943303232*)
> 3030303332314144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635,
> *30303033323346343932*) > DecoratedKey(5233296131921119414,
> *53d83e0012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193,
> *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663,
> *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 444236451388
> 1388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of
> out-of-order keys is much larger than that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008777#comment-14008777 ] Jonathan Ellis commented on CASSANDRA-7245: --- [bq] [~jasobrown] Can you please generate the same amount of data as you did before but with CASSANDRA-6861 reverted so we can test shared buffer + ref counting theory? Good idea. (And if it still reproduces w/o 6861, can you check 2.0?) > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich >Assignee: T Jake Luciani > Fix For: 2.1 rc1 > > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004252#comment-14004252 ] Benedict commented on CASSANDRA-7245: - Whilst it isn't conclusive, I have been able to generate the same errors inserting large data sets with cql3 over native transport, but not over thrift. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004124#comment-14004124 ] Pavel Yaskevich commented on CASSANDRA-7245: [~jasobrown] Can you please generate the same amount of data as you did before but with CASSANDRA-6861 reverted so we can test shared buffer + ref counting theory? > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999517#comment-13999517 ] Pavel Yaskevich commented on CASSANDRA-7245: Jason told me that they idea was that it could only happen when there is a operation timeout but I'm not sure what almost every of 862 SSTables has such keys when we only got 22 timeouts and there is no batching or something, also it's not obvious why normal keys are getting sorted out of order too, also if the data is corrupted why the key is of correct size?... > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999509#comment-13999509 ] T Jake Luciani commented on CASSANDRA-7245: --- We were talking on the IRC and we suspect this is caused by CASSANDRA-6861 We need to push the ref counting of pooled buffers into the mutations in case they live longer than the request. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > There is also a problem with sorting of normal looking keys: > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > And sometimes there is no problem at all: > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999588#comment-13999588 ] Pavel Yaskevich commented on CASSANDRA-7245: [~brandon.williams] It's using native, so no thrift involvement what so ever. [~tjake] We will try to do that tomorrow. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999572#comment-13999572 ] Brandon Williams commented on CASSANDRA-7245: - Any chance you're using HSHA? CASSANDRA-6285 which is still not yet committed (I have a reason for that) could be a factor in that case. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999536#comment-13999536 ] T Jake Luciani commented on CASSANDRA-7245: --- Well an easy check would be roll back 6861 locally and see if the issue goes away. > Out-of-Order keys with stress + CQL3 > > > Key: CASSANDRA-7245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Pavel Yaskevich > > We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 > and found following problem almost in every SSTable generated (~200 GB of > data and 821 SSTables). > We set up they keys to be 10 bytes in size (default) and population between 1 > and 6. > Once I ran 'sstablekeys' on the generated SSTable files I got following > exceptions: > _There is a problem with sorting of normal looking keys:_ > 30303039443538353645 > 30303039443745364242 > java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, > *30303039443745364242*) > DecoratedKey(-1767746583617597213, > *30303039443437454333*) > 0a30303033343933 > 3734441388343933 > java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, > *3734441388343933*) > DecoratedKey(-7565486415339257200, > *30303033344639443137*) > 30303033354244363031 > 30303033354133423742 > java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, > *30303033354133423742*) > DecoratedKey(-7838239767410066684, > *30303033354145344534*) > 30303034313442354137 > 3034313635363334 > java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, > *3034313635363334*) > DecoratedKey(-9106177395653818217, > *3030303431444238*) > 30303035373044373435 > 30303035373044334631 > java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, > *30303035373044334631*) > DecoratedKey(-4296696226469000945, > *30303035373132364138*) > _And completely different ones:_ > 30303041333745373543 > 7cd045c59a90d7587d8d > java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, > *7cd045c59a90d7587d8d*) > DecoratedKey(-5146766422778260690, > *30303041333943303232*) > 3030303332314144 > 30303033323346343932 > java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, > *30303033323346343932*) > DecoratedKey(5233296131921119414, > *53d83e0012287e03*) > 30303034314531374431 > 3806734b256c27e41ec2 > java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, > *3806734b256c27e41ec2*) > DecoratedKey(-8072288379146044663, > *30303034314136413343*) > _And sometimes there is no problem at all:_ > 30303033353144463637 > 002a31b3b31a1c2f > 5d616dd38211ebb5d6ec > 444236451388 > 1388138844463744 > 30303033353143394343 > It's worth to mention that we have got 22 timeout exceptions but number of > out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
