http://issues.apache.org/jira/browse/CASSANDRA-44
On Thu, Aug 20, 2009 at 4:37 PM, Jun Rao<jun...@almaden.ibm.com> wrote: > You should be aware that Cassandra doesn't support changing column families > without reloading the data. > > Jun > IBM Almaden Research Center > K55/B1, 650 Harry Road, San Jose, CA 95120-6099 > > jun...@almaden.ibm.com > > > Phillip Michalak ---08/20/2009 04:16:44 PM---I can't reproduce this after > cleaning out the data/ directories (and commit logs) and restarting t > > > From: > Phillip Michalak <phil.micha...@digitalreasoning.com> > To: > cassandra-user@incubator.apache.org > Date: > 08/20/2009 04:16 PM > Subject: > Re: quorum read timeout > ________________________________ > > > I can't reproduce this after cleaning out the data/ directories (and commit > logs) and restarting the cluster. I suspect that I had gotten to an > inconsistent state as I was playing around with the keyspace column > families. > > I've updated the bug report with a comment to this effect, but have left it > open for someone with more experience to close it out. > > Cheers, > Phil > > On Aug 19, 2009, at 5:14 PM, Phillip Michalak wrote: > > Sure thing. Filed as https://issues.apache.org/jira/browse/CASSANDRA-381 > > Thanks, > Phil > > On Aug 19, 2009, at 4:54 PM, Jonathan Ellis wrote: > > Looks like a bug in TcpConnectionManager. Can you file a ticket? > > thanks, > > -Jonathan > > On Wed, Aug 19, 2009 at 2:49 PM, Phillip > Michalak<phil.micha...@digitalreasoning.com> wrote: > > It's cassandra-0.4-beta1. > > Thanks! > Phil > > On Aug 19, 2009, at 4:43 PM, Jonathan Ellis wrote: > > Is this 0.3 or 0.4/trunk? > > On Wed, Aug 19, 2009 at 2:36 PM, Phillip > Michalak<phil.micha...@digitalreasoning.com> wrote: > > I'm running three Cassandra nodes in virtual machines. > During a 'get' operation from Cassandra-remote directed at one of these > nodes, I'm receiving the following output > > vad...@vadmin:~/cassandra$ interface/gen-py/cassandra/Cassandra-remote -h > 192.168.133.130:9160 get 'MockElementLibrary' '0401318uuuuruepwdcznr' > "ColumnPath('strings', None, 'id')" 2 > /usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:58: > DeprecationWarning: BaseException.message has been deprecated as of > Python > 2.6 > self.message = message > /usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:99: > DeprecationWarning: BaseException.message has been deprecated as of > Python > 2.6 > self.message = iprot.readString(); > Traceback (most recent call last): > File "interface/gen-py/cassandra/Cassandra-remote", line 93, in <module> > pp.pprint(client.get(args[0],args[1],eval(args[2]),eval(args[3]),)) > File > > "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py", > line 182, in get > return self.recv_get() > File > > "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py", > line 201, in recv_get > raise x > > thrift.Thrift.TApplicationException/usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:76: > DeprecationWarning: BaseException.message has been deprecated as of > Python > 2.6 > if self.message: > /usr/local/lib/python2.6/dist-packages/thrift/Thrift.py:77: > DeprecationWarning: BaseException.message has been deprecated as of > Python > 2.6 > return self.message > : Internal error processing get > > The same 'get' operation from Cassandra-remote directed at another of > these > nodes, yields 'normal' output > vad...@vadmin:~/cassandra$ interface/gen-py/cassandra/Cassandra-remote -h > 192.168.133.129:9160 get 'MockElementLibrary' '0401318uuuuruepwdcznr' > "ColumnPath('strings', None, 'id')" 2 > Traceback (most recent call last): > File "interface/gen-py/cassandra/Cassandra-remote", line 93, in <module> > pp.pprint(client.get(args[0],args[1],eval(args[2]),eval(args[3]),)) > File > > "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py", > line 182, in get > return self.recv_get() > File > > "/home/vadmin/cassandra-0.4.0-beta1/interface/gen-py/cassandra/Cassandra.py", > line 210, in recv_get > raise result.nfe > ttypes.NotFoundException: NotFoundException() > Furthermore, querying the same column for (some) other keys is successful > when no matter which node it is directed at. > Looking at the log for the node that produced the error from the query > above: > > DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618 CassandraServer.java > (line > 221) get > DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,618 StorageProxy.java (line > 420) strongread reading data for > SliceByNamesReadCommand(table='MockElementLibrary', > key='0401318uuuuruepwdcznr', > columnParent='QueryPath(columnFamilyName='strings', > superColumnName='null', > columnName='null')', columns=[id,]) from 38...@null > DEBUG [pool-1-thread-22] 2009-08-19 16:54:57,619 StorageProxy.java (line > 427) strongread reading digest for > SliceByNamesReadCommand(table='MockElementLibrary', > key='0401318uuuuruepwdcznr', > columnParent='QueryPath(columnFamilyName='strings', > superColumnName='null', > columnName='null')', columns=[id,]) from 38...@192.168.133.129:7000 > WARN [MESSAGE-SERIALIZER-POOL:4] 2009-08-19 16:54:57,619 > MessageSerializationTask.java (line 81) Exception was generated at : > 08/19/2009 16:54:57 on thread MESSAGE-SERIALIZER-POOL:4 > java.lang.NullPointerException > at > org.apache.cassandra.net.TcpConnection.<init>(TcpConnection.java:83) > at > > org.apache.cassandra.net.TcpConnectionManager.getConnection(TcpConnectionManager.java:64) > at > > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:306) > at > > org.apache.cassandra.net.MessageSerializationTask.run(MessageSerializationTask.java:66) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > DEBUG [RESPONSE-STAGE:4] 2009-08-19 16:54:57,622 ResponseVerbHandler.java > (line 38) Processing response on a callback > from 65b1e352-a0a3-1a7f-138b-9bea3e1d7...@192.168.133.129:7000 > ERROR [pool-1-thread-22] 2009-08-19 16:55:02,619 Cassandra.java (line > 608) > Internal error processing get > java.lang.RuntimeException: java.util.concurrent.TimeoutException: > Operation > timed out - received only 1 responses from 192.168.133.129:7000 . > at > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100) > at > > org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:226) > at > > org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:602) > at > > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:560) > at > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.util.concurrent.TimeoutException: Operation timed out - > received only 1 responses from 192.168.133.129:7000 . > at > > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:86) > at > > org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:435) > at > > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:330) > at > > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92) > ... 7 more > > It appears to me that there is a timeout during the QuorumResponseHandler > processing, stemming from a NullPointerException that occurs as part of > the > read process. I suspect that this NullPointerException has something to > do > with the second DEBUG [pool-1-thread-22] comment regarding strongread ... > from 38...@null. > Does anyone know why this might be happening? > Thanks for any insight, > Phil > > > > -- Evan Weaver
