Hi,
I have 7 nodes in cluster version 2.1.11 replication 3, 4 with this problem
above, and 3 is ok.
Anyone know what the hell is happening with these 4 nodes
[default@prod] list counterCF ;
Using default limit of 100
Using default cell limit of 100
null
TimedOutException()
at
Hi All,
We are getting TimedOutException() when inserting data into Cassandra, it
was working fine for a few months, but suddenly got this problem. I have
increase rpc_timout_in_ms to 3, but it still timed out in 30 secs.
I turned on debug, I saw many of this error in the log
DEBUG [pool-2
We are using Cassandra 1.1.0 with an older Pelops version, but I don't
think that in itself is a problem here.
I am getting this exception:
TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7660)
Look at the nodetool tpstats when you get the TimedOutException, to work out
which nodes are backing up with pending messages. Then try to identify why.
Check the server logs for GC, and the CPU and IO usage.
Somehow the cluster is getting overwhelmed and cannot respond. Either the
clients
On Fri, Jun 1, 2012 at 9:39 AM, Oleg Dulin wrote:
>
> Is my understanding correct that this is where cassandra is telling us it
> can't accomplish something within that timeout value -- as opposed to
> network timeout ? Where is it set ?
>
That's correct. Basically, the coordinator sees that a
Tyler Hobbs wrote:
> On Fri, Jun 1, 2012 at 9:39 AM, Oleg Dulin wrote:
>
> Is my understanding correct that this is where cassandra is telling us it
> can't accomplish something within that timeout value -- as opposed to
> network timeout ? Where is it set ?
>
> That's correct. Basically, the
> Is the node we are connecting to try to proxy requests ? Wouldn't our
> configuration ensure all nodes have replicas ?
It can still time out even when reading locally. (The thread running the query
is waiting on the read thread).
Look in the server side logs to see if there are any errors. If
tasks is:
java.io.IOException: TimedOutException(acknowledged_by:0)
at
org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
Caused by: TimedOutException(acknowledged_by:0)
at
org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result
n reading at a CL of local_quorum:
1:44:14.176 [main] ERROR m.p.c.connection.HConnectionManager - Could not
fullfill request on this host CassandraClient
11:44:14.182 [main] ERROR m.p.c.connection.HConnectionManager - Exception:
me.prettyprint.hector.api.e
aise
the TimedOutException.
> You options are:
>
> * allocate more memory
> * ease back on the stress test.
> * work as a CL QUORUM so that one node failing does result in the error.
>
> see also http://wiki.apache.org/cassandra/FAQ#slows_down_afte
Hi,
I can see some strange behaviour on my test cluster and in production.
Both running cassandra 0.8.10. Strange is that when I compile my
mapreduce job against cassandra-all 0.8.7 everything is ok, but if I
use higher version I get quite a lots of TimedOutException.
java.lang.RuntimeException
xception I get when reading at a CL of local_quorum:
>
> 1:44:14.176 [main] ERROR m.p.c.connection.HConnectionManager - Could not
> fullfill request on this host CassandraClient
> 11:44:14.182 [main] ERROR m.p.c.connection.HConnectionManager - Exception:
> me.prettyprint.hect
any read CL other
than LOCAL_QUORUM.
At time 15:58 I performed a write with CL = ONE, a subsequent read with CL =
LOCAL_QUORUM and received a TimedOutException.
System.log and cassandra.log are from server dlc05-sea to corresponds with the
hector error message "Closing client CassandraC
Trying out stress.py on AWS EC2 environment (4 Large instances. Each
of 2-cores and 7.5GB RAM. All in the same region/zone.)
python stress.py -o insert -d
10.253.203.224,10.220.203.48,10.220.17.84,10.124.89.81 -l 2 -e ALL -t
10 -n 500 -S 100 -k
(I want to try with column size of about 1MB. I
Some more info, running the cluster with debug enabled shows this error:
DEBUG 14:23:05,749 Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTranspor
There are two more similar exceptions:
DEBUG 14:23:12,817 Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transpor
see some strange behaviour on my test cluster and in production.
> Both running cassandra 0.8.10. Strange is that when I compile my
> mapreduce job against cassandra-all 0.8.7 everything is ok, but if I
> use higher version I get quite a lots of TimedOutException.
>
> ja
Hi Jeremy,
I've seen the page and tried the values but to no help.
Here goes tcpdump of one failed TCP connection:
15:06:20.231421 IP 10.0.18.87.9160 > 10.0.18.87.39396: Flags [P.], seq
137891735:137904068, ack 708, win 282, options [nop,nop,TS val
4119981636 ecr 4119981636], length 12333
15:06:
By chance are you in EC2?
On Feb 24, 2012, at 8:33 AM, Patrik Modesto wrote:
> Hi Jeremy,
>
> I've seen the page and tried the values but to no help.
>
> Here goes tcpdump of one failed TCP connection:
>
> 15:06:20.231421 IP 10.0.18.87.9160 > 10.0.18.87.39396: Flags [P.], seq
> 137891735:13790
No, we have our own servers, no cloud service.
P.
On Feb 24, 2012 5:57 PM, "Jeremy Hanna" wrote:
> By chance are you in EC2?
>
> On Feb 24, 2012, at 8:33 AM, Patrik Modesto wrote:
>
> > Hi Jeremy,
> >
> > I've seen the page and tried the values but to no help.
> >
> > Here goes tcpdump of one fa
Did you see the notes here?
http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
On Sat, Feb 25, 2012 at 2:04 AM, Patrik Modesto
wrote:
> No, we have our own servers, no cloud service.
>
> P.
>
> On Feb 24, 2012 5:57 PM, "Jeremy Hanna" wrote:
>>
>> By chance are you in EC2?
>>
>> On F
On Sun, Feb 26, 2012 at 04:25, Edward Capriolo wrote:
> Did you see the notes here?
I'm not sure what do you mean by the notes?
I'm using the mapred.* settings suggested there:
mapred.max.tracker.failures
20
mapred.map.max.attempts
20
What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms
? Have you tried reducing the first and/or increasing the second ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/02/2012, at 8:02 PM, Patrik Modesto wrote
Hi aaron,
this is our current settings:
cassandra.range.batch.size
1024
cassandra.input.split.size
16384
rpc_timeout_in_ms: 3
Regards,
P.
On Mon, Feb 27, 2012 at 21:54, aaron morton wrote:
> What settings do you have for
Have you tried lowering the batch size and increasing the time out? Even just
to get it to work.
If you get a TimedOutException it means CL number of servers did not respond in
time.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 28
I'll alter these settings and will let you know.
Regards,
P.
On Tue, Feb 28, 2012 at 09:23, aaron morton wrote:
> Have you tried lowering the batch size and increasing the time out? Even
> just to get it to work.
>
> If you get a TimedOutException it means CL number of server
ards,
> P.
>
> On Tue, Feb 28, 2012 at 09:23, aaron morton wrote:
>> Have you tried lowering the batch size and increasing the time out? Even
>> just to get it to work.
>>
>> If you get a TimedOutException it means CL number of servers did not respond
>> in tim
>
> On Tue, Feb 28, 2012 at 12:29, Patrik Modesto
> wrote:
> > I'll alter these settings and will let you know.
> >
> > Regards,
> > P.
> >
> > On Tue, Feb 28, 2012 at 09:23, aaron morton
> wrote:
> >> Have you tried lowering the
> wrote:
>> I'll alter these settings and will let you know.
>>
>> Regards,
>> P.
>>
>> On Tue, Feb 28, 2012 at 09:23, aaron morton wrote:
>>> Have you tried lowering the batch size and increasing the time out? Even
>>> just to g
alter these settings and will let you know.
>> >
>> > Regards,
>> > P.
>> >
>> > On Tue, Feb 28, 2012 at 09:23, aaron morton
>> > wrote:
>> >> Have you tried lowering the batch size and increasing the time out?
>> >> Even
>
gt; >> task_201202281457_2027_m_000108 9364.17%
> >>
> >> None of this happens with cassandra-all 0.8.7.
> >>
> >> Regards,
> >> P.
> >>
> >>
> >>
> >> On Tue, Feb 28, 2012 at 12:29, Patrik Modesto >
>
t; Task Complete
>> >> task_201202281457_2027_m_41 9076.81%
>> >> task_201202281457_2027_m_73 9639.04%
>> >> task_201202281457_2027_m_000105 10538.60%
>> >> task_201202281457_2027_m_000108 9364.17%
>> >>
>> >> None of this hap
gt;>> >> 0.8.10: 595
>>> >>
>>> >> Task Complete
>>> >> task_201202281457_2027_m_000041 9076.81%
>>> >> task_201202281457_2027_m_73 9639.04%
>>> >> task_201202281457_2027_m_000105
t;
>> On Tue, Feb 28, 2012 at 12:29, Patrik Modesto
>> wrote:
>>> I'll alter these settings and will let you know.
>>>
>>> Regards,
>>> P.
>>>
>>> On Tue, Feb 28, 2012 at 09:23, aaron morton wrote:
>>>> Have
t; P.
> >>
> >>
> >>
> >> On Tue, Feb 28, 2012 at 12:29, Patrik Modesto
> wrote:
> >>> I'll alter these settings and will let you know.
> >>>
> >>> Regards,
> >>> P.
> >>>
> &g
; task_201202281457_2027_m_000105 10538.60%
>> >> task_201202281457_2027_m_000108 9364.17%
>> >>
>> >> None of this happens with cassandra-all 0.8.7.
>> >>
>> >> Regards,
>> >> P.
>> >>
>> &
; task_201202281457_2027_m_41 9076.81%
> >> >> task_201202281457_2027_m_73 9639.04%
> >> >> task_201202281457_2027_m_000105 10538.60%
> >> >> task_201202281457_2027_m_000108 9364.17%
> >> >>
> >> >&
s the job creates:
>> >> >> 0.8.7: 4680
>> >> >> 0.8.10: 595
>> >> >>
>> >> >> Task Complete
>> >> >> task_201202281457_2027_m_41 9076.81%
>> >> >> task_201202281457_2027_m_
Hi all,
I have been experiencing the unavailableException and TimedOutException on a
3-node cassandra cluster
during a multiGetSliceQuery with 1000 columns. Since there are many keys
involved in the query, I divided
them into groups of 5000 rows and process each group individually in a for
loop
a know issue or am I doing something wrong in
implementation.
---error
10/04/29 13:47:37 INFO mapred.JobClient: Task Id :
attempt_201004291109_0024_m_00_1, Status : FAILED
java.lang.RuntimeException: TimedOutException()
at
org.apache.cassandra.hadoop.Colu
me I'll get it all in one email. :-)
At time 15:55 I performed a write with CL = ONE, a subsequent read with CL =
ONE and received correct results. Similar results occur with any read CL other
than LOCAL_QUORUM.
At time 15:58 I performed a write with CL = ONE, a subsequent read with CL =
LOCA
2011/3/8 A J
> Trying out stress.py on AWS EC2 environment (4 Large instances. Each
> of 2-cores and 7.5GB RAM. All in the same region/zone.)
>
> python stress.py -o insert -d
> 10.253.203.224,10.220.203.48,10.220.17.84,10.124.89.81 -l 2 -e ALL -t
> 10 -n 500 -S 100 -k
>
> (I want to try wit
Is this a client side time out or a server side one? What does the error stack
look like ?
Also check the server side logs for errors. The thrift API will raise a timeout
when less the CL level of nodes return in rpc_timeout.
Good luck
Aaron
On 9/03/2011, at 7:37 AM, ruslan usifov wrote:
>
ssandra/Cassandra.py",
line 899, in recv_batch_mutate
raise result.te
TimedOutException: TimedOutException()
Process Inserter-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "stress.py
Cool, so it's a server side because
- in the client side stack the thrift code is raising the error
- server side log has this DEBUG 22:29:10,318 ... timed out
The TimedOutException is raised when the number of replicas required by your CL
have not returned inside the timespan specifi
5000 rows in a mutli get is way, way, way (did I say way ? ) to many.
Whenever you get a TimedOutException check the tp stats on the nodes, you will
normally see a high pending count. Every row get get turns into an message in a
TP. So if you ask for 5k rows you flood the TP with 5k messages
Hi Aaron,
thanks for the suggestion. It works again after I cut back the # of rows.
On Wed, Oct 5, 2011 at 1:43 PM, aaron morton wrote:
> 5000 rows in a mutli get is way, way, way (did I say way ? ) to many.
>
> Whenever you get a TimedOutException check the tp stats on the nodes, y
red.JobClient: Task Id :
> attempt_201004291109_0024_m_00_1, Status : FAILED
> java.lang.RuntimeException: TimedOutException()
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:165)
> at
> org.apache.cassandra.h
without anyproblems at all.
> >
> > Do you thinki it's a know issue or am I doing something wrong in
> > implementation.
> >
> > ---error
> > 10/04/29 13:47:37 INFO mapred.JobClient: Task Id :
> > attempt_201004291109_0024_m_00
Hi,
I get a weird issue with cassandra 1.2.13. As written in the subject, a query
executed by class CqlPagingRecordReader raises a TimedOutException exception in
Java but I don't have any error when I use it with cqlsh. What's the difference
between those 2 ways ? Does cqlsh b
Hi
My system is 4 nodes 64 bit cassandra cluster, 6G big per node,default
configuration (which means 1/3 heap for memtable), replicate number 3,
write all, read one.
When I run stress load testing, I got this TimedOutException, and some
operation failed, and all traffic hang for a while.
And
, a query
> executed by class CqlPagingRecordReader raises a TimedOutException exception
> in Java but I don't have any error when I use it with cqlsh. What's the
> difference between those 2 ways ? Does cqlsh bypass some configuration
> compared to Java ?
>
>
-
> Cyril SCETBON
>
> On 13 Feb 2014, at 17:22, Cyril Scetbon wrote:
>
> > Hi,
> >
> > I get a weird issue with cassandra 1.2.13. As written in the subject, a
> query executed by class CqlPagingRecordReader raises a TimedOutException
> exception in Java but I
On 13 Feb 2014, at 17:22, Cyril Scetbon wrote:
>
> > Hi,
> >
> > I get a weird issue with cassandra 1.2.13. As written in the subject, a
> > query executed by class CqlPagingRecordReader raises a TimedOutException
> > exception in Java but I don't have any er
the query. Whatever cql3 query I
>> do, I always get the same exception. If someone sees something ...
>> --
>> Cyril SCETBON
>>
>> On 13 Feb 2014, at 17:22, Cyril Scetbon wrote:
>>
>> > Hi,
>> >
>> > I get a weird issue with cassandra 1.2
The cluster is running into GC problems and this is slowing it down under the
stress test. When it slows down one or more of the nodes is failing to perform
the write within rpc_timeout . This causes the coordinator of the write to
raise the TimedOutException.
You options are:
* allocate
56 matches
Mail list logo
