> i'm not sure if using framed transport is an option with hector. http://hector-client.github.com/hector//source/content/API/core/0.8.0-2/me/prettyprint/cassandra/service/CassandraHostConfigurator.html#setUseThriftFramedTransport(boolean)
> what should i be in the logs looking for to find the cause of these dropped > reads? These look like transport errors. If something is happening on the server side it will be logged at ERROR level. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 25/05/2012, at 1:00 PM, Deno Vichas wrote: > i'm not sure if using framed transport is an option with hector. > > what should i be in the logs looking for to find the cause of these dropped > reads? > > thanks, > > On 5/24/2012 3:04 AM, aaron morton wrote: >> >> Dropped read messages occur when the node could not process a read task >> within rpc_timeout. It generally means the cluster has been overwhelmed at >> some point: too many requests, to much GC, compaction hurting, etc. >> >> Check the server side logs for errors but I doubt it is related to the call >> stack below. >> >> Have you confirmed that you are using framed transport on the client? >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 24/05/2012, at 5:52 PM, Deno Vichas wrote: >> >>> i've notice the my nodes seem to have a large (?, not really sure what >>> acceptable numbers are) read dropped count from tpstats. >>> could they be related? >>> >>> On 5/23/2012 2:55 AM, aaron morton wrote: >>>> >>>> No sure but >>>> >>>>> at >>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>> >>>> Looks like the client is not using framed transport. The server defaults >>>> to framed. >>>> >>>> Cheers >>>> >>>> ----------------- >>>> Aaron Morton >>>> Freelance Developer >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>> >>>> On 23/05/2012, at 5:35 AM, Deno Vichas wrote: >>>> >>>>> could somebody clue me in to the cause of this exception? i see these >>>>> randomly. >>>>> >>>>> AnalyzerService-2 2012-05-22 13:28:00,385 :: WARN >>>>> cassandra.connection.HConnectionManager - Exception: >>>>> me.prettyprint.hector.api.exceptions.HectorTransportException: >>>>> org.apache.thrift.transport.TTransportException >>>>> at >>>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:39) >>>>> at >>>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:851) >>>>> at >>>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:840) >>>>> at >>>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:99) >>>>> at >>>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:243) >>>>> at >>>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131) >>>>> at >>>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceServiceImpl.java:857) >>>>> at >>>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:57) >>>>> at >>>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:52) >>>>> at >>>>> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) >>>>> at >>>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85) >>>>> at >>>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColumnQuery.java:51) >>>>> at >>>>> com.stocktouch.dao.StockDaoImpl.getHistorical(StockDaoImpl.java:365) >>>>> at >>>>> com.stocktouch.dao.StockDaoImpl.getHistoricalQuote(StockDaoImpl.java:433) >>>>> at >>>>> com.stocktouch.service.StockHistoryServiceImpl.getHistoricalQuote(StockHistoryServiceImpl.java:480) >>>>> at >>>>> com.stocktouch.service.AnalyzerServiceImpl.getClose(AnalyzerServiceImpl.java:180) >>>>> at >>>>> com.stocktouch.service.AnalyzerServiceImpl.calcClosingPrices(AnalyzerServiceImpl.java:90) >>>>> at >>>>> com.stocktouch.service.AnalyzerServiceImpl.nightlyRollup(AnalyzerServiceImpl.java:66) >>>>> at >>>>> com.stocktouch.service.AnalyzerServiceImpl$2.run(AnalyzerServiceImpl.java:55) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> Caused by: org.apache.thrift.transport.TTransportException >>>>> at >>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>> at >>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>> at >>>>> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >>>>> at >>>>> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) >>>>> at >>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >>>>> at >>>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >>>>> at >>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_get(Cassandra.java:509) >>>>> at >>>>> org.apache.cassandra.thrift.Cassandra$Client.get(Cassandra.java:492) >>>>> at >>>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:846) >>>>> ... 20 more >>>>> >>>>> >>>>> thanks, >>>>> deno >>>>> >>>>> >>>> >>> >> >
