Database not getting updated
Dear community, Sometimes I've noticed that the changes done in the database, even when done successfully are not reflected e.g. I added/deleted a row via cqlsh/Datastax C++ driver, the command/API was successful, yet when I fetched the contents of the table, the row that I had added/deleted, was missing/still present. Has anybody, noticed something similar before? I've had this observation on the latest 3.11.2(earlier I had observed this on 3.0.14). There were 2 nodes in my cluster, replication factor was 2 and read/write consistency was also 2. Thanks and regards, Vishal Sharma "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Re: Database not getting updated
This usually happens when you’ve got weird timestamps or a delete in the future - check your clocks, make sure your timestamp resolution uses the same units in all applications (likely microseconds), make sure you’re running NTP everywhere -- Jeff Jirsa > On May 11, 2018, at 5:37 PM, > wrote: > > Dear community, > > Sometimes I’ve noticed that the changes done in the database, even when done > successfully are not reflected e.g. I added/deleted a row via cqlsh/Datastax > C++ driver, the command/API was successful, yet when I fetched the contents > of the table, the row that I had added/deleted, was missing/still present. > Has anybody, noticed something similar before? > I’ve had this observation on the latest 3.11.2(earlier I had observed this on > 3.0.14). There were 2 nodes in my cluster, replication factor was 2 and > read/write consistency was also 2. > > Thanks and regards, > Vishal Sharma > > "Confidentiality Warning: This message and any attachments are intended only > for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments from > your system. > > Virus Warning: Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment."
RE: Database not getting updated
Who’s awesome? Jeff Jirsa is awesome! ☺ Thanks and regards, Vishal Sharma From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Friday, May 11, 2018 5:51 PM To: user@cassandra.apache.org Subject: Re: Database not getting updated This usually happens when you’ve got weird timestamps or a delete in the future - check your clocks, make sure your timestamp resolution uses the same units in all applications (likely microseconds), make sure you’re running NTP everywhere -- Jeff Jirsa On May 11, 2018, at 5:37 PM, mailto:vishal1.sha...@ril.com>> mailto:vishal1.sha...@ril.com>> wrote: Dear community, Sometimes I’ve noticed that the changes done in the database, even when done successfully are not reflected e.g. I added/deleted a row via cqlsh/Datastax C++ driver, the command/API was successful, yet when I fetched the contents of the table, the row that I had added/deleted, was missing/still present. Has anybody, noticed something similar before? I’ve had this observation on the latest 3.11.2(earlier I had observed this on 3.0.14). There were 2 nodes in my cluster, replication factor was 2 and read/write consistency was also 2. Thanks and regards, Vishal Sharma "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Re: Cassandra upgrade from 2.1 to 3.0
After further analyzing the data. I see some pattern. The rows which were
updated in last 2-3 weeks, the column which were not part of this update have
the null values.
Has anyone encountered this issue during the upgrade?
Thank you,
On Thursday, 10 May 2018, 19:49:50 GMT-7, kooljava2
wrote:
Hello Jeff,
2.1.19 to 3.0.15.
Thank you.
On Thursday, 10 May 2018, 17:43:58 GMT-7, Jeff Jirsa
wrote:
Which minor version of 3.0
-- Jeff Jirsa
On May 11, 2018, at 2:54 AM, kooljava2 wrote:
Hello,
Upgraded Cassandra 2.1 to 3.0. We see certain data in few columns being set to
"null". These null columns were created during the row creation time.
After looking at the data see a pattern where update was done on these rows.
Rows which were updated has data but rows which were not part of the update are
set to null.
created_on | created_by | id
-+-+-
null | null |
12345
sstabledump:-
WARN 20:47:38,741 Small cdc volume detected at /var/lib/cassandra/cdc_raw;
setting cdc_total_space_in_mb to 1278. You can override this in cassandra.yaml
[
{
"partition" : {
"key" : [ "12345" ],
"position" : 5155159
},
"rows" : [
{
"type" : "row",
"position" : 5168738,
"deletion_info" : { "marked_deleted" : "2018-03-28T20:38:08.05Z",
"local_delete_time" : "2018-03-28T20:38:08Z" },
"cells" : [
{ "name" : "doc_type", "value" : false, "tstamp" :
"2018-03-28T20:38:08.060Z" },
{ "name" : "industry", "deletion_info" : { "local_delete_time" :
"2018-03-28T20:38:08Z" },
"tstamp" : "2018-03-28T20:38:08.060Z"
},
{ "name" : "last_modified_by", "value" : "12345", "tstamp" :
"2018-03-28T20:38:08.060Z" },
{ "name" : "last_modified_date", "value" : "2018-03-28
20:38:08.059Z", "tstamp" : "2018-03-28T20:38:08.060Z" },
{ "name" : "locale", "deletion_info" : { "local_delete_time" :
"2018-03-28T20:38:08Z" },
"tstamp" : "2018-03-28T20:38:08.060Z"
},
{ "name" : "postal_code", "deletion_info" : { "local_delete_time" :
"2018-03-28T20:38:08Z" },
"tstamp" : "2018-03-28T20:38:08.060Z"
},
{ "name" : "ticket", "deletion_info" : { "marked_deleted" :
"2018-03-28T20:38:08.05Z", "local_delete_time" : "2018-03-28T20:38:08Z" } },
{ "name" : "ticket", "path" : [ "TEMP_DATA" ], "value" :
"{\"name\":\"TEMP_DATA\",\"ticket\":\"a42638dae8350e889f2603be1427ac6f5dec5e486d4db164a76bf80820cdf68d635cff5e7d555e6d4eabb9b5b82597b68bec0fcd735fcca\",\"lastRenewedDate\":\"2018-03-28T20:38:08Z\"}",
"tstamp" : "2018-03-28T20:38:08.060Z" },
{ "name" : "ticket", "path" : [ "TEMP_TEMP2" ], "value" :
"{\"name\":\"TEMP_TEMP2\",\"ticket\":\"a4263b7350d1f2683\",\"lastRenewedDate\":\"2018-03-28T20:38:07Z\"}",
"tstamp" : "2018-03-28T20:38:08.060Z" },
{ "name" : "ppstatus_pf", "deletion_info" : { "marked_deleted" :
"2018-03-28T20:38:08.05Z", "local_delete_time" : "2018-03-28T20:38:08Z" } },
{ "name" : "ppstatus_pers", "deletion_info" : { "marked_deleted" :
"2018-03-28T20:38:08.05Z", "local_delete_time" : "2018-03-28T20:38:08Z" } }
]
}
]
}
]WARN 20:47:41,325 Small cdc volume detected at /var/lib/cassandra/cdc_raw;
setting cdc_total_space_in_mb to 1278. You can override this in cassandra.yaml
[
{
"partition" : {
"key" : [ "12345" ],
"position" : 18743072
},
"rows" : [
{
"type" : "row",
"position" : 18751808,
"liveness_info" : { "tstamp" : "2017-10-25T10:22:41.612Z" },
"cells" : [
{ "name" : "created_by", "value" : "12345" },
{ "name" : "created_on", "value" : "2017-10-25 10:22:41.637Z" },
{ "name" : "doc_type", "value" : false, "tstamp" :
"2017-10-25T10:22:42.487Z" },
{ "name" : "last_modified_by", "value" : "12345", "tstamp" :
"2017-10-25T10:22:42.487Z" },
{ "name" : "last_modified_date", "value" : "2017-11-10
00:09:52.668Z", "tstamp" : "2017-11-10T00:09:52.668Z" },
{ "name" : "per_type", "value" : "user" },
{ "name" : "lists", "path" : [ "cn.cncn.bpnp" ], "value" :
"[\"::accid:ab\",\"::accid:e1\",\"::accid:d2\",\"::accid:d3\",\"::accid:f3\",\"::accid:g3\",\"::accid:f4\",\"::accid:9c486ae5-00b2-3c63-af70-cff2950c4181\"]",
"tstamp" : "2017-10-25T10:22:42.782Z" },
{ "name" : "ticket", "path" : [ "TEMP_TEMP2" ], "value" :
"{\"name\":\"TEMP_TEMP2\",\"ticket\":\"a4263820be49c3a222e0248532bcefc80c773194a804057561a97382e595b51f36bb46b8675589fc89dea4a5c0ceb944d63861b39d63c0067161e84c79328077c650df33530c7625857444711dc4b1051638123694ba6e9e29b1f906663f3\",\"lastRenewedDate\":\"2017-11-10T00:09:52Z\"}",
"tstamp" : "2017-11-10T00:09:52.668Z" }
Error after 3.1.0 to 3.11.2 upgrade
HI All, Seen below stack trace messages , in errorlog one day after upgrade. one of the blogs said this might be due to old drivers, but not sure on it. FYI : INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / 10.152.205.150 ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10] at org.apa
Re: Error after 3.1.0 to 3.11.2 upgrade
That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to > /10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.10.jar:3.10] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_51] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.10.jar:3.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > Caused by: com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] >
Re: Error after 3.1.0 to 3.11.2 upgrade
Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is repeatedly coming , whats solution for this? WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize # for ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) ~[apache-cassandra-3.11.2.jar:3.11.2] On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: > That looks like Cassandra 3.10 not 3.11.2 > > It’s also just the auth cache failing to refresh - if it’s transient it’s > probably not a big deal. If it continues then there may be an issue with > the cache refresher. > > -- > Jeff Jirsa > > > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / > 10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: > Cannot achieve consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) > ~[apache-cassandra-3.10.jar:3.10] > at org.apache.cassandra.service.ClientState. > checkPermissionOnResourceChain(ClientState.java:352) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassand
Re: Error after 3.1.0 to 3.11.2 upgrade
The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated isn’t great, but is the application impacted? -- Jeff Jirsa > On May 12, 2018, at 6:17 AM, Abdul Patel wrote: > > Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is > repeatedly coming , whats solution for this? > > WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 > CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize # cas_sar_dit> for > ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) > ~[apache-cassandra-3.11.2.jar:3.11.2] > >> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: >> That looks like Cassandra 3.10 not 3.11.2 >> >> It’s also just the auth cache failing to refresh - if it’s transient it’s >> probably not a big deal. If it continues then there may be an issue with the >> cache refresher. >> >> -- >> Jeff Jirsa >> >> >>> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >>> >>> HI All, >>> >>> Seen below stack trace messages , in errorlog one day after upgrade. >>> one of the blogs said this might be due to old drivers, but not sure on it. >>> >>> FYI : >>> >>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>> OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 >>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to >>> /10.152.205.150 >>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve >>> consistency level LOCAL_ONE >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(Authenti
Re: Error after 3.1.0 to 3.11.2 upgrade
No applicatiom isnt impacted ..no complains .. Also its an 4 node cluster in lower non production and all are on same version. On Friday, May 11, 2018, Jeff Jirsa wrote: > The read is timing out - is the cluster healthy? Is it fully upgraded or > mixed versions? Repeated isn’t great, but is the application impacted? > > -- > Jeff Jirsa > > > On May 12, 2018, at 6:17 AM, Abdul Patel wrote: > > Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this > is repeatedly coming , whats solution for this? > > WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 > CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize > # for > ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: > Operation timed out - received only 0 responses. > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.service.ClientState. > checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.cql3.statements.SelectStatement. > checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar: > 3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) > ~[apache-cassandra-3.11.2.jar:3.11.2] > > On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: > >> That looks like Cassandra 3.10 not 3.11.2 >> >> It’s also just the auth cache failing to refresh - if it’s transient it’s >> probably not a big deal. If it continues then there may be an issue with >> the cache refresher. >> >> -- >> Jeff Jirsa >> >> >> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >> >> HI All, >> >> Seen below stack trace messages , in errorlog one day after upgrade. >> one of the blogs said this might be due to old drivers, but not sure on >> it. >> >> FYI : >> >> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >> OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 >> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / >> 10.152.205.150 >> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >> ErrorMessage.java:384 - Unexpected exception during request >> com.google.common.util.concurrent.UncheckedExecutionException: >> com.google.common.util.concurrent.UncheckedExecutionException: >> java.lang.RuntimeException: >> org.apache.cassandra.exceptions.UnavailableException: >> Cannot achieve consistency level LOCAL_ONE >> at >> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >> ~[guava-18.0.jar:na] >> at >> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >> ~[guava-18.0.jar:na] >> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >> ~[apache-cassandra-3.10.jar:3.10]
Re: Error after 3.1.0 to 3.11.2 upgrade
Hi, Did you check replication strategy and amounts of replicas of system_auth keyspace? Hannu > Abdul Patel kirjoitti 12.5.2018 kello 5.21: > > No applicatiom isnt impacted ..no complains .. > Also its an 4 node cluster in lower non production and all are on same > version. > >> On Friday, May 11, 2018, Jeff Jirsa wrote: >> The read is timing out - is the cluster healthy? Is it fully upgraded or >> mixed versions? Repeated isn’t great, but is the application impacted? >> >> -- >> Jeff Jirsa >> >> >>> On May 12, 2018, at 6:17 AM, Abdul Patel wrote: >>> >>> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this >>> is repeatedly coming , whats solution for this? >>> >>> WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 >>> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize >>> # for >>> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - >>> received only 0 responses. >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on > it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to > /10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at > com.googl
