Apologies. The entire error stack:
ERROR [SharedPool-Worker-5] 2016-09-20 11:23:20,039 ErrorMessage.java:251 -
Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
~[guava-16.0.jar:na]
at
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72)
~[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.service.ClientState.authorize(ClientState.java:353)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:195)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.14.jar:2.1.14]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[na:1.7.0_79]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
~[guava-16.0.jar:na]
... 23 common frames omitted
On Tue, Sep 20, 2016 at 11:22 AM, George Sigletos <sigle...@textkernel.nl>
wrote:
> This appears in the system log:
>
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 2 responses.
> at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at
> org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at com.google.common.cache.LocalCache$LoadingValueReference.
> loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.
> lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
> ~[guava-16.0.jar:na]
>
>
> On Tue, Sep 20, 2016 at 11:12 AM, George Sigletos <sigle...@textkernel.nl>
> wrote:
>
>> I am also getting the same error:
>> cqlsh <my host> -u cassandra -p cassandra
>>
>> Connection error: ('Unable to connect to any servers', {'<my host>':
>> OperationTimedOut('errors=Timed out creating connection (5 seconds),
>> last_host=None',)})
>>
>> But it is not consistent. Sometimes I manage to connect. It is random.
>> Using 2.1.14
>>
>> On Tue, Jun 14, 2016 at 4:29 AM, joseph gao <gaojf.bok...@gmail.com>
>> wrote:
>>
>>> hi, Patric, [image: 内嵌图片 1],
>>> netstat -lepunt looks like above
>>>
>>> 2016-05-27 23:16 GMT+08:00 Patrick McFadin <pmcfa...@gmail.com>:
>>>
>>>> Can you do a netstat -lepunt and show the output? If Cassandra is
>>>> running you aren't trying to connect to the ip/port it's bound to.
>>>>
>>>> Patrick
>>>>
>>>>
>>>> On Monday, May 23, 2016, joseph gao <gaojf.bok...@gmail.com> wrote:
>>>>
>>>>> I used to think it's firewall/network issues too. So I make ufw to be
>>>>> inactive. I really don't what's the reason.
>>>>>
>>>>> 2016-05-09 19:01 GMT+08:00 kurt Greaves <k...@instaclustr.com>:
>>>>>
>>>>>> Don't be fooled, despite saying tcp6 and :::*, it still listens on
>>>>>> IPv4. As far as I'm aware this happens on all 2.1 Cassandra nodes, and
>>>>>> may
>>>>>> just be an oddity of netstat. It would be unrelated to your connection
>>>>>> timeout issues, that's most likely related to firewall/network issues.
>>>>>>
>>>>>> On 9 May 2016 at 09:59, joseph gao <gaojf.bok...@gmail.com> wrote:
>>>>>>
>>>>>>> It doesn't work ,still using ipv6 [image: 内嵌图片 1]
>>>>>>>
>>>>>>> And I already set [image: 内嵌图片 2]
>>>>>>>
>>>>>>> Now I'm using 4.1.1 using 9160 port instead of 5.x.x。
>>>>>>>
>>>>>>> Hopefully this could be resolved, Thanks!
>>>>>>>
>>>>>>> 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com>:
>>>>>>>
>>>>>>>> Hi Joseph,
>>>>>>>>
>>>>>>>> why cassandra using tcp6 for 9042 port like :
>>>>>>>>> tcp6 0 0 0.0.0.0:9042 :::*
>>>>>>>>> LISTEN
>>>>>>>>>
>>>>>>>>
>>>>>>>> if I remember correctly, in 2.1 and higher, cqlsh uses native
>>>>>>>> transport, port 9042 (instead of thrift port 9160) and your clients
>>>>>>>> (if
>>>>>>>> any) are also probably using native transport (port 9042). So yes, this
>>>>>>>> could be an issue indeed.
>>>>>>>>
>>>>>>>> You should have something like:
>>>>>>>>
>>>>>>>> tcp 0 0 1.2.3.4:9042 :::*
>>>>>>>> LISTEN
>>>>>>>>
>>>>>>>> You are using IPv6 and no rpc address. Try setting it to the listen
>>>>>>>> address and using IPv4.
>>>>>>>>
>>>>>>>> C*heers,
>>>>>>>>
>>>>>>>> -----------------------
>>>>>>>>
>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>
>>>>>>>> France
>>>>>>>>
>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>
>>>>>>>> http://www.thelastpickle.com
>>>>>>>>
>>>>>>>> 2016-03-30 6:09 GMT+02:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>
>>>>>>>>> why cassandra using tcp6 for 9042 port like :
>>>>>>>>> tcp6 0 0 0.0.0.0:9042 :::*
>>>>>>>>> LISTEN
>>>>>>>>> would this be the problem
>>>>>>>>>
>>>>>>>>> 2016-03-30 11:34 GMT+08:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> still have not fixed it . cqlsh: error: no such option:
>>>>>>>>>> --connect-timeout
>>>>>>>>>> cqlsh version 5.0.1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi Joseph.
>>>>>>>>>>>
>>>>>>>>>>> As I can't reproduce here, I believe you are having network
>>>>>>>>>>> issue of some kind.
>>>>>>>>>>>
>>>>>>>>>>> MacBook-Pro:~ alain$ cqlsh --version
>>>>>>>>>>> cqlsh 5.0.1
>>>>>>>>>>> MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
>>>>>>>>>>> --connect-timeout=5 --request-timeout=10
>>>>>>>>>>> system_traces system
>>>>>>>>>>> MacBook-Pro:~ alain$
>>>>>>>>>>>
>>>>>>>>>>> It's been a few days, did you manage to fix it ?
>>>>>>>>>>>
>>>>>>>>>>> C*heers,
>>>>>>>>>>> -----------------------
>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>> France
>>>>>>>>>>>
>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>
>>>>>>>>>>> 2016-03-21 9:59 GMT+01:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> cqlsh version 5.0.1. nodetool tpstats looks good, log looks
>>>>>>>>>>>> good. And I used specified port 9042. And it immediately returns
>>>>>>>>>>>> fail (less
>>>>>>>>>>>> than 3 seconds). By the way where should I use
>>>>>>>>>>>> '--connect-timeout', cqlsh
>>>>>>>>>>>> seems don't have such parameters.
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ <arodr...@gmail.com>
>>>>>>>>>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>>> Is the node fully healthy or rejecting some requests ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> What are the outputs for "grep -i "ERROR"
>>>>>>>>>>>>> /var/log/cassandra/system.log" and "nodetool tpstats"?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any error? Any pending / blocked or dropped messages?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also did you try using distinct ports (9160 for thrift, 9042
>>>>>>>>>>>>> for native) - out of curiosity, not sure this will help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is your version of cqlsh "cqlsh --version" ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> doesn't work most times. But some time it just work fine
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you fill like this is due to a timeout (query being too
>>>>>>>>>>>>> big, cluster being to busy)? Try setting this higher:
>>>>>>>>>>>>>
>>>>>>>>>>>>> --connect-timeout=CONNECT_TIMEOUT
>>>>>>>>>>>>>
>>>>>>>>>>>>> Specify the connection timeout in
>>>>>>>>>>>>> seconds (default: 5 seconds).
>>>>>>>>>>>>>
>>>>>>>>>>>>> --request-timeout=REQUEST_TIMEOUT
>>>>>>>>>>>>>
>>>>>>>>>>>>> Specify the default request timeout in
>>>>>>>>>>>>> seconds (default: 10 seconds).
>>>>>>>>>>>>>
>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>> -----------------------
>>>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>>>> France
>>>>>>>>>>>>>
>>>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-03-18 4:49 GMT+01:00 joseph gao <gaojf.bok...@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Of course yes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2016-03-17 22:35 GMT+08:00 Vishwas Gupta <
>>>>>>>>>>>>>> vishwas.gu...@snapdeal.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Have you started the Cassandra service?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sh cassandra
>>>>>>>>>>>>>>> On 17-Mar-2016 7:59 pm, "Alain RODRIGUEZ" <
>>>>>>>>>>>>>>> arodr...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, did you try with the address of the node rather than
>>>>>>>>>>>>>>>> 127.0.0.1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is the transport protocol used by cqlsh (not sure if it is
>>>>>>>>>>>>>>>> thrift or binary - native in 2.1) active ? What is the
>>>>>>>>>>>>>>>> "nodetool info"
>>>>>>>>>>>>>>>> output ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>>>> -----------------------
>>>>>>>>>>>>>>>> Alain Rodriguez - al...@thelastpickle.com
>>>>>>>>>>>>>>>> France
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The Last Pickle - Apache Cassandra Consulting
>>>>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2016-03-17 14:26 GMT+01:00 joseph gao <
>>>>>>>>>>>>>>>> gaojf.bok...@gmail.com>:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> hi, all
>>>>>>>>>>>>>>>>> cassandra version 2.1.7
>>>>>>>>>>>>>>>>> When I use cqlsh to connect cassandra, something is wrong
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Connection error: ( Unable to connect to any servers',
>>>>>>>>>>>>>>>>> {'127.0.0.1': OperationTimedOut('errors=None,
>>>>>>>>>>>>>>>>> last_host=None,)})
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This happens lots of times, but sometime it works just
>>>>>>>>>>>>>>>>> fine. Anybody knows why?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> ------
>>>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> ------
>>>>>>>>>>>> Joseph Gao
>>>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>>>> QQ: 409343351
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> ------
>>>>>>>>>> Joseph Gao
>>>>>>>>>> PhoneNum:15210513582
>>>>>>>>>> QQ: 409343351
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ------
>>>>>>>>> Joseph Gao
>>>>>>>>> PhoneNum:15210513582
>>>>>>>>> QQ: 409343351
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ------
>>>>>>> Joseph Gao
>>>>>>> PhoneNum:15210513582
>>>>>>> QQ: 409343351
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kurt Greaves
>>>>>> k...@instaclustr.com
>>>>>> www.instaclustr.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ------
>>>>> Joseph Gao
>>>>> PhoneNum:15210513582
>>>>> QQ: 409343351
>>>>>
>>>>
>>>
>>>
>>> --
>>> ------
>>> Joseph Gao
>>> PhoneNum:15210513582
>>> QQ: 409343351
>>>
>>
>>
>
