[jira] [Commented] (CASSANDRA-11340) Heavy read activity on system_auth tables can cause apparent livelock
[
https://issues.apache.org/jira/browse/CASSANDRA-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192164#comment-15192164
]
Jeff Jirsa commented on CASSANDRA-11340:
Speculative retry is apparently not required to see this behavior. Will attempt
to disable read repair on the canary seeing this behavior to see if that helps.
> Heavy read activity on system_auth tables can cause apparent livelock
> -
>
> Key: CASSANDRA-11340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11340
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Jirsa
>Assignee: Aleksey Yeschenko
>
> Reproduced in at least 2.1.9.
> It appears possible for queries against system_auth tables to trigger
> speculative retry, which causes auth to block on traffic going off node. In
> some cases, it appears possible for threads to become deadlocked, causing
> load on the nodes to increase sharply. This happens even in clusters with RF
> of system_auth == N, as all requests being served locally puts the bar for
> 99% SR pretty low.
> Incomplete stack trace below, but we haven't yet figured out what exactly is
> blocking:
> {code}
> Thread 82291: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> -
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(long)
> @bci=28, line=307 (Compiled frame)
> - org.apache.cassandra.utils.concurrent.SimpleCondition.await(long,
> java.util.concurrent.TimeUnit) @bci=76, line=63 (Compiled frame)
> - org.apache.cassandra.service.ReadCallback.await(long,
> java.util.concurrent.TimeUnit) @bci=25, line=92 (Compiled frame)
> -
> org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.maybeTryAdditionalReplicas()
> @bci=39, line=281 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.fetchRows(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel) @bci=175, line=1338 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.readRegular(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel) @bci=9, line=1274 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.read(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel,
> org.apache.cassandra.service.ClientState) @bci=57, line=1199 (Compiled frame)
> -
> org.apache.cassandra.cql3.statements.SelectStatement.execute(org.apache.cassandra.service.pager.Pageable,
> org.apache.cassandra.cql3.QueryOptions, int, long,
> org.apache.cassandra.service.QueryState) @bci=35, line=272 (Compiled frame)
> -
> org.apache.cassandra.cql3.statements.SelectStatement.execute(org.apache.cassandra.service.QueryState,
> org.apache.cassandra.cql3.QueryOptions) @bci=105, line=224 (Compiled frame)
> - org.apache.cassandra.auth.Auth.selectUser(java.lang.String) @bci=27,
> line=265 (Compiled frame)
> - org.apache.cassandra.auth.Auth.isExistingUser(java.lang.String) @bci=1,
> line=86 (Compiled frame)
> -
> org.apache.cassandra.service.ClientState.login(org.apache.cassandra.auth.AuthenticatedUser)
> @bci=11, line=206 (Compiled frame)
> -
> org.apache.cassandra.transport.messages.AuthResponse.execute(org.apache.cassandra.service.QueryState)
> @bci=58, line=82 (Compiled frame)
> -
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(io.netty.channel.ChannelHandlerContext,
> org.apache.cassandra.transport.Message$Request) @bci=75, line=439 (Compiled
> frame)
> -
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(io.netty.channel.ChannelHandlerContext,
> java.lang.Object) @bci=6, line=335 (Compiled frame)
> -
> io.netty.channel.SimpleChannelInboundHandler.channelRead(io.netty.channel.ChannelHandlerContext,
> java.lang.Object) @bci=17, line=105 (Compiled frame)
> -
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(java.lang.Object)
> @bci=9, line=333 (Compiled frame)
> -
> io.netty.channel.AbstractChannelHandlerContext.access$700(io.netty.channel.AbstractChannelHandlerContext,
> java.lang.Object) @bci=2, line=32 (Compiled frame)
> - io.netty.channel.AbstractChannelHandlerContext$8.run() @bci=8, line=324
> (Compiled frame)
> - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
> (Compiled frame)
> -
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run()
> @bci=5, line=164 (Compiled frame)
> - org.apache.cassandra.concurrent.SEPWorker.run() @bci=87, line=105
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {code}
> In a cluster with many connected clients (potentially thousands), a
> reconnection flood (for example, resta
[jira] [Updated] (CASSANDRA-11340) Heavy read activity on system_auth tables can cause apparent livelock
[
https://issues.apache.org/jira/browse/CASSANDRA-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-11340:
---
Summary: Heavy read activity on system_auth tables can cause apparent
livelock (was: Speculative retry on system_auth tables can cause deadlock)
> Heavy read activity on system_auth tables can cause apparent livelock
> -
>
> Key: CASSANDRA-11340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11340
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Jirsa
>Assignee: Aleksey Yeschenko
>
> Reproduced in at least 2.1.9.
> It appears possible for queries against system_auth tables to trigger
> speculative retry, which causes auth to block on traffic going off node. In
> some cases, it appears possible for threads to become deadlocked, causing
> load on the nodes to increase sharply. This happens even in clusters with RF
> of system_auth == N, as all requests being served locally puts the bar for
> 99% SR pretty low.
> Incomplete stack trace below, but we haven't yet figured out what exactly is
> blocking:
> {code}
> Thread 82291: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338
> (Compiled frame)
> -
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(long)
> @bci=28, line=307 (Compiled frame)
> - org.apache.cassandra.utils.concurrent.SimpleCondition.await(long,
> java.util.concurrent.TimeUnit) @bci=76, line=63 (Compiled frame)
> - org.apache.cassandra.service.ReadCallback.await(long,
> java.util.concurrent.TimeUnit) @bci=25, line=92 (Compiled frame)
> -
> org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.maybeTryAdditionalReplicas()
> @bci=39, line=281 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.fetchRows(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel) @bci=175, line=1338 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.readRegular(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel) @bci=9, line=1274 (Compiled frame)
> - org.apache.cassandra.service.StorageProxy.read(java.util.List,
> org.apache.cassandra.db.ConsistencyLevel,
> org.apache.cassandra.service.ClientState) @bci=57, line=1199 (Compiled frame)
> -
> org.apache.cassandra.cql3.statements.SelectStatement.execute(org.apache.cassandra.service.pager.Pageable,
> org.apache.cassandra.cql3.QueryOptions, int, long,
> org.apache.cassandra.service.QueryState) @bci=35, line=272 (Compiled frame)
> -
> org.apache.cassandra.cql3.statements.SelectStatement.execute(org.apache.cassandra.service.QueryState,
> org.apache.cassandra.cql3.QueryOptions) @bci=105, line=224 (Compiled frame)
> - org.apache.cassandra.auth.Auth.selectUser(java.lang.String) @bci=27,
> line=265 (Compiled frame)
> - org.apache.cassandra.auth.Auth.isExistingUser(java.lang.String) @bci=1,
> line=86 (Compiled frame)
> -
> org.apache.cassandra.service.ClientState.login(org.apache.cassandra.auth.AuthenticatedUser)
> @bci=11, line=206 (Compiled frame)
> -
> org.apache.cassandra.transport.messages.AuthResponse.execute(org.apache.cassandra.service.QueryState)
> @bci=58, line=82 (Compiled frame)
> -
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(io.netty.channel.ChannelHandlerContext,
> org.apache.cassandra.transport.Message$Request) @bci=75, line=439 (Compiled
> frame)
> -
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(io.netty.channel.ChannelHandlerContext,
> java.lang.Object) @bci=6, line=335 (Compiled frame)
> -
> io.netty.channel.SimpleChannelInboundHandler.channelRead(io.netty.channel.ChannelHandlerContext,
> java.lang.Object) @bci=17, line=105 (Compiled frame)
> -
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(java.lang.Object)
> @bci=9, line=333 (Compiled frame)
> -
> io.netty.channel.AbstractChannelHandlerContext.access$700(io.netty.channel.AbstractChannelHandlerContext,
> java.lang.Object) @bci=2, line=32 (Compiled frame)
> - io.netty.channel.AbstractChannelHandlerContext$8.run() @bci=8, line=324
> (Compiled frame)
> - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
> (Compiled frame)
> -
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run()
> @bci=5, line=164 (Compiled frame)
> - org.apache.cassandra.concurrent.SEPWorker.run() @bci=87, line=105
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {code}
> In a cluster with many connected clients (potentially thousands), a
> reconnection flood (for example, restarting all at once) is likely to trigger
> this bug. However, it is un
[jira] [Created] (CASSANDRA-11348) Compaction Filter in Cassandra
Dikang Gu created CASSANDRA-11348: - Summary: Compaction Filter in Cassandra Key: CASSANDRA-11348 URL: https://issues.apache.org/jira/browse/CASSANDRA-11348 Project: Cassandra Issue Type: New Feature Components: Compaction Reporter: Dikang Gu Priority: Minor Fix For: 3.x RocksDB has the feature called "Compaction Filter" to allow application to modify/delete a key-value during the background compaction. https://github.com/facebook/rocksdb/blob/v4.1/include/rocksdb/options.h#L201-L226 It could be valuable to implement this feature in C* as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10735) Support netty openssl (netty-tcnative) for client encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191824#comment-15191824
]
Andy Tolbert edited comment on CASSANDRA-10735 at 3/12/16 9:36 PM:
---
I went ahead and updated my
[branch|https://github.com/tolbertam/cassandra/tree/CASSANDRA-10735] today to
rebase it off of trunk and add a commit that enables using netty openssl in
cassandra stress, example usage:
{code}
tools/bin/cassandra-stress write n=1000 -rate threads=250 -transport
ssl-openssl ssl-private-key=client.pem ssl-cert=client.crt
truststore=client.truststore truststore-password=cassandra
{code}
The performance characteristics were pretty much exactly the same as I had
observed previously, with exception to the no ssl run which had a few pauses
(didn't read too much into it as I was focusing on the SSL configs).
There are a few caveats / things I wanted to mention:
# The cipher suite format for openssl is different than the format jdk ssl
expects. Netty appropriately does translation when you provide the ciphers to
convert the jdk-style cipher names into what openssl expects, but there is no
way to convert it going back. {{SSLServerSocket.getSupportedCipherSuites()}}
appears to return the open ssl style. This makes things tricky as
{{SSLFactory.filterCipherSuites}} compares cipher's on the socket to those in
our configuration, which currently use the JDK format. I think this becomes a
non-issue if [~spo...@gmail.com]'s [CASSANDRA-10508] is implemented. You can
currently work around it by providing openssl cipher suites in your
cassandra.yaml under client_encryption_options (see
[this|https://gist.github.com/tolbertam/87267fa5708d998b57ee] for a list).
# One thing that is interesting to me is that netty's openssl builder will
allow you to use a truststore for establishing trust, but not a keystore for
your private keys. It looks like it does some work extracting items from the
truststore into X509 certs. Because of this we need separate options for
private key & cert (for openssl) and keystore (for jdk). I wonder if something
similar can be done for keys in netty, will pursue this.
# The transport encryption options do not let you override the keystore and
keystore password used for jdk ssl, it just uses defaults (conf/.keystore and
cassandra for password). Would be good to add those as extra options, will add
commit for that.
# With the version of netty we are using here (4.0.33) it seems the most recent
version of netty-tcnative that works is
[1.1.33.Fork11|http://central.maven.org/maven2/io/netty/netty-tcnative/1.1.33.Fork11/].
I ran into problems using Fork14 on the client side
([issue|https://github.com/netty/netty-tcnative/issues/124]), but it seems to
work well on the server side. For openssl to work, download the appropriate
platform jar and dump it in cassandra's lib directory.
# We could consider including the netty-tcnative jars in the lib directory as
part of the distribution. The only problem would be where/if we have multiple
jars that could match the same platform. For example, in Fork11 there are
separate jars for fedora x64 and linux x64 and the presence of both in the
classpath can be a problem.
Attached [^netty-ssl-trunk.tgz] which includes stress logs, example
keystores/certs/pems/etc., and the html benchmark report (doesn't work in
chrome for some reason).
!sslbench12-03.png!
(Red: no encryption, Orange: openssl on both ends, Purple: openssl client, jdk
server, Green: jdk client jdk server, Blue: jdk client, openssl server
was (Author: andrew.tolbert):
I went ahead and updated my
[branch|https://github.com/tolbertam/cassandra/tree/CASSANDRA-10735] today to
rebase it off of trunk and add a commit that enables using netty openssl in
cassandra stress, example usage:
{code}
tools/bin/cassandra-stress write n=1000 -rate threads=250 -transport
ssl-openssl ssl-private-key=client.pem ssl-cert=client.crt
truststore=client.truststore truststore-password=cassandra
{code}
The performance characteristics were pretty much exactly the same as I had
observed previously, with exception to the no ssl run which had a few pauses
(didn't read too much into it as I was focusing on the SSL configs).
There are a few caveats / things I wanted to mention:
# The cipher suite format for openssl is different than the format jdk ssl
expects. Netty appropriately does translation when you provide the ciphers to
convert the jdk-style cipher names into what openssl expects, but there is no
way to convert it going back. {{SSLServerSocket.getSupportedCipherSuites()}}
appears to return the open ssl style. This makes things tricky as
{{SSLFactory.filterCipherSuites}} compares cipher's on the socket to those in
our configuration, which currently use the JDK format. I think this becomes a
non-issue if [
[jira] [Commented] (CASSANDRA-10735) Support netty openssl (netty-tcnative) for client encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191836#comment-15191836
]
Andy Tolbert commented on CASSANDRA-10735:
--
{quote}
This has been addressed in [CASSANDRA-9325]
{quote}
That looks perfect, would definitely be nice to have :)
> Support netty openssl (netty-tcnative) for client encryption
>
>
> Key: CASSANDRA-10735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10735
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
> Fix For: 3.x
>
> Attachments: netty-ssl-trunk.tgz, nettyssl-bench.tgz,
> nettysslbench.png, nettysslbench_small.png, sslbench12-03.png
>
>
> The java-driver recently added support for using netty openssl via
> [netty-tcnative|http://netty.io/wiki/forked-tomcat-native.html] in
> [JAVA-841|https://datastax-oss.atlassian.net/browse/JAVA-841], this shows a
> very measured improvement (numbers incoming on that ticket). It seems
> likely that this can offer improvement if implemented C* side as well.
> Since netty-tcnative has platform specific requirements, this should not be
> made the default, but rather be an option that one can use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10735) Support netty openssl (netty-tcnative) for client encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191830#comment-15191830
]
Stefan Podkowinski commented on CASSANDRA-10735:
{quote}
The transport encryption options do not let you override the keystore and
keystore password used for jdk ssl, it just uses defaults (conf/.keystore and
cassandra for password).
{quote}
This has been addressed in CASSANDRA-9325
> Support netty openssl (netty-tcnative) for client encryption
>
>
> Key: CASSANDRA-10735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10735
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
> Fix For: 3.x
>
> Attachments: netty-ssl-trunk.tgz, nettyssl-bench.tgz,
> nettysslbench.png, nettysslbench_small.png, sslbench12-03.png
>
>
> The java-driver recently added support for using netty openssl via
> [netty-tcnative|http://netty.io/wiki/forked-tomcat-native.html] in
> [JAVA-841|https://datastax-oss.atlassian.net/browse/JAVA-841], this shows a
> very measured improvement (numbers incoming on that ticket). It seems
> likely that this can offer improvement if implemented C* side as well.
> Since netty-tcnative has platform specific requirements, this should not be
> made the default, but rather be an option that one can use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10735) Support netty openssl (netty-tcnative) for client encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Tolbert updated CASSANDRA-10735:
-
Attachment: sslbench12-03.png
netty-ssl-trunk.tgz
I went ahead and updated my
[branch|https://github.com/tolbertam/cassandra/tree/CASSANDRA-10735] today to
rebase it off of trunk and add a commit that enables using netty openssl in
cassandra stress, example usage:
{code}
tools/bin/cassandra-stress write n=1000 -rate threads=250 -transport
ssl-openssl ssl-private-key=client.pem ssl-cert=client.crt
truststore=client.truststore truststore-password=cassandra
{code}
The performance characteristics were pretty much exactly the same as I had
observed previously, with exception to the no ssl run which had a few pauses
(didn't read too much into it as I was focusing on the SSL configs).
There are a few caveats / things I wanted to mention:
# The cipher suite format for openssl is different than the format jdk ssl
expects. Netty appropriately does translation when you provide the ciphers to
convert the jdk-style cipher names into what openssl expects, but there is no
way to convert it going back. {{SSLServerSocket.getSupportedCipherSuites()}}
appears to return the open ssl style. This makes things tricky as
{{SSLFactory.filterCipherSuites}} compares cipher's on the socket to those in
our configuration, which currently use the JDK format. I think this becomes a
non-issue if [~spo...@gmail.com]'s [CASSANDRA-10508] is implemented. You can
currently work around it by providing openssl cipher suites in your
cassandra.yaml under client_encryption_options (see
[this|https://gist.github.com/tolbertam/87267fa5708d998b57ee] for a list).
# One thing that is interesting to me is that netty's openssl builder will
allow you to use a truststore for establishing trust, but not a keystore for
your private keys. It looks like it does some work extracting items from the
truststore into X509 certs. Because of this we need separate options for
private key & cert (for openssl) and keystore (for jdk). I wonder if something
similar can be done for keys in netty, will pursue this.
# The transport encryption options do not let you override the keystore and
keystore password used for jdk ssl, it just uses defaults (conf/.keystore and
cassandra for password). Would be good to add those as extra options, will add
commit for that.
# With the version of netty we are using here (4.0.33) it seems the most recent
version of netty-tcnative that works is
[1.1.33.Fork11|http://central.maven.org/maven2/io/netty/netty-tcnative/1.1.33.Fork11/].
I ran into problems using Fork14 on the client side
([issue|https://github.com/netty/netty-tcnative/issues/124]), but it seems to
work well on the server side. For openssl to work, download the appropriate
platform jar and dump it in cassandra's lib directory.
# We could consider including the netty-tcnative jars in the lib directory as
part of the distribution. The only problem would be where/if we have multiple
jars that could match the same platform. For example, in Fork11 there are
separate jars for fedora x64 and linux x64 and the presence of both in the
classpath can be a problem.
Attached [^netty-ssl-trunk.tgz] which includes stress logs, example
keystores/certs/pems/etc., and the html benchmark report (doesn't work in
chrome for some reason).
!sslbench12-03.png|thumbnail!
(Red: no encryption, Orange: openssl on both ends, Purple: openssl client, jdk
server, Green: jdk client jdk server, Blue: jdk client, openssl server
> Support netty openssl (netty-tcnative) for client encryption
>
>
> Key: CASSANDRA-10735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10735
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Andy Tolbert
>Assignee: Aleksey Yeschenko
> Fix For: 3.x
>
> Attachments: netty-ssl-trunk.tgz, nettyssl-bench.tgz,
> nettysslbench.png, nettysslbench_small.png, sslbench12-03.png
>
>
> The java-driver recently added support for using netty openssl via
> [netty-tcnative|http://netty.io/wiki/forked-tomcat-native.html] in
> [JAVA-841|https://datastax-oss.atlassian.net/browse/JAVA-841], this shows a
> very measured improvement (numbers incoming on that ticket). It seems
> likely that this can offer improvement if implemented C* side as well.
> Since netty-tcnative has platform specific requirements, this should not be
> made the default, but rather be an option that one can use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11341) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_2_1_HEAD_UpTo_2_2.whole_list_conditional_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191797#comment-15191797
]
Russ Hatch commented on CASSANDRA-11341:
ok, I can now repro this locally (by adding a lot of repetitions within the
test code). It appears like there may be a very short window in which after a
list is replaced when it can be returned as Null. At this point i'm not certain
if this is a consistency issue (test problem), a driver issue, or a cassandra
bug
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_2_1_HEAD_UpTo_2_2.whole_list_conditional_test
> ---
>
> Key: CASSANDRA-11341
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11341
> Project: Cassandra
> Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/22/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_2_1_HEAD_UpTo_2_2/whole_list_conditional_test
> Failed on CassCI build upgrade_tests-all #22
> There's only one flap in the history currently. This was the failure:
> {code}
> Expected [[0, ['foo', 'bar', 'foobar']]] from SELECT * FROM tlist, but got
> [[0, None]]
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-SF2dOV
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: upgrading node1 to 2.2.5
> dtest: DEBUG: Querying upgraded node
> - >> end captured logging << -
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 253, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 4294, in whole_list_conditional_test
> check_applies("l != null AND l IN (['foo', 'bar', 'foobar'])")
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 4282, in check_applies
> assert_one(cursor, "SELECT * FROM %s" % (table,), [0, ['foo', 'bar',
> 'foobar']])
> File "/home/automaton/cassandra-dtest/assertions.py", line 50, in assert_one
> assert list_res == [expected], "Expected %s from %s, but got %s" %
> ([expected], query, list_res)
> "Expected [[0, ['foo', 'bar', 'foobar']]] from SELECT * FROM tlist, but got
> [[0, None]]\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /mnt/tmp/dtest-SF2dOV\ndtest: DEBUG: Custom init_config not found. Setting
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold':
> 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\ndtest: DEBUG: upgrading node1 to 2.2.5\ndtest: DEBUG: Querying
> upgraded node\n- >> end captured logging <<
> -"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
