her:26] 2019-03-28 01:49:25,217
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint
/10.10.2.55<http://10.10.2.55/>: db485ac6-8acd-4241-9e21-7a2b540459de
The node 10.10.2.55 is in DC-B, lets call this n
large amounts of hint files (~100 GB). In
>> the logs of this node I see lots of messages like the following:
>>
>> INFO [HintsDispatcher:26] 2019-03-28 01:49:25,217
>> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
>> db485ac6-8acd-4241-9e21-7a2b540459de-155
01:49:25,217
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint
/10.10.2.55<http://10.10.2.55/>: db485ac6-8acd-4241-9e21-7a2b540459de
The node 10.10.2.55 is in DC-B, lets call this node B1. There is no indication
whatsoeve
amounts of hint files (~100 GB). In
> the logs of this node I see lots of messages like the following:
>
> INFO [HintsDispatcher:26] 2019-03-28 01:49:25,217
> HintsDispatchExecutor.java:289 - Finished hinted handoff of file
> db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hi
:
INFO [HintsDispatcher:26] 2019-03-28 01:49:25,217
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
db485ac6-8acd-4241-9e21-7a2b540459de-1553419324363-1.hints to endpoint
/10.10.2.55: db485ac6-8acd-4241-9e21-7a2b540459de
The node 10.10.2.55 is in DC-B, lets call this node B1
Please find my comments inline.
From: kurt greaves
Reply-To: "user@cassandra.apache.org"
Date: Tuesday, August 7, 2018 at 1:20 AM
To: User
Subject: Re: Hinted Handoff
Does Cassandra TTL out the hints after max_hint_window_in_ms? From my
understanding, Cassandra only stops collec
ut can still keep replaying the hints if the node
> > > comes back again. Is this correct? Is there a way to TTL out hints?
> > >
> > > Thanks,
> > > Pratik
> > >
> > > From: Kyrylo Lebediev
> > > Reply-To: "user@cassandra.apache.org&quo
keep replaying the hints if the node
> comes back again. Is this correct? Is there a way to TTL out hints?
>
>
>
> Thanks,
>
> Pratik
>
>
>
> *From: *Kyrylo Lebediev
> *Reply-To: *"user@cassandra.apache.org"
> *Date: *Monday, August 6, 2018 at 4:1
: Kyrylo Lebediev
Reply-To: "user@cassandra.apache.org"
Date: Monday, August 6, 2018 at 4:10 PM
To: "user@cassandra.apache.org"
Subject: Re: Hinted Handoff
Small gc_grace_seconds value lowers max allowed node downtime, which is 15
minutes in your case. After 15 minutes of
for gc_grace_seconds (lets say,
hours or a day) will you get visible read amplification / waste a lot of disk
space / issues with compactions?
Hinted handoff may be the reason in case hinted handoff window is longer than
gc_grace_seconds. To me it looks like hinted handoff window
(max_hint_window_in_ms
deleted data is resurrected (because of short lifecycle of our data). My guess
at the moment is that these data is resurrected due to hinted handoff.
Interesting point to note here is that data keeps resurrecting at periodic
intervals (like an hour) and then finally stops. Could this be caused
The coordinator will refuse to send writes/hints to a node if it has a
large backlog of hints (128 * #cores) already and the destination replica
is one of the nodes with hints destined to it.
It will still send writes to any "healthy" node (a node with no outstanding
hints).
The idea is to not
Refs :
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html
On Thu, Jul 12, 2018 at 7:46 PM Karthick V wrote:
> Hi everyone,
>
> If several nodes experience brief outages simultaneously, substantial
>> memory pressure can build up on the
Hi everyone,
If several nodes experience brief outages simultaneously, substantial
> memory pressure can build up on the coordinator.* The coordinator tracks
> how many hints it is currently writing, and if the number increases too
> much, the coordinator refuses writes and throws the *
>
they are hints.
On 12 April 2018 at 13:51, Jinhua Luo <luajit...@gmail.com> wrote:
> Hi All,
>
> In the doc:
> https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dml
> AboutDeletes.html
>
> It said "When an unresponsive node recovers, Cassandra uses hinted
Hi All,
In the doc:
https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlAboutDeletes.html
It said "When an unresponsive node recovers, Cassandra uses hinted
handoff to replay the database mutationsthe node missed while it was
down. Cassandra does not replay a mutation for a tombs
Hi,
Recently I Came across this line
Cassandra uses hinted handoff
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html>
> to replay the database mutations
> <https://docs.datastax.com/en/glossary/doc/glossary/gloss_mutation.html&
> while to have all other nodes be marked as up and this corresponds to nodes
> that haven't finished playing hints.
>
> We looked at the hinted handoff throttling, noticed it was still the
> default of 1024, so we tried to turn it off by setting it to zero. Reading
> the source, it look
We have a 96 node cluster running 3.11 with 256 vnodes each. We're running
a rolling restart. As we restart nodes, we notice that each node takes a
while to have all other nodes be marked as up and this corresponds to nodes
that haven't finished playing hints.
We looked at the hinted handoff
grow until I
run repair, which then takes a really long time because it is significantly out
of date. Is there some magic setting I cannot find in the documentation to
enable hinted handoff? I’m running 2.0.11. Any insights would be greatly
appreciated.
Thanks
Robert
doesn’t grow until I
run repair, which then takes a really long time because it is significantly out
of date. Is there some magic setting I cannot find in the documentation to
enable hinted handoff? I’m running 2.0.11. Any insights would be greatly
appreciated.
Thanks
Robert
long time because it is
significantly out of date. Is there some magic setting I cannot find in the
documentation to enable hinted handoff? I’m running 2.0.11. Any insights
would be greatly appreciated.
Thanks
Robert
out of date. Is there some magic setting I cannot find in the
documentation to enable hinted handoff? I’m running 2.0.11. Any insights
would be greatly appreciated.
Thanks
Robert
it is
significantly out of date. Is there some magic setting I cannot find in the
documentation to enable hinted handoff? I’m running 2.0.11. Any insights
would be greatly appreciated.
Thanks
Robert
(although again, glacially). However, I
noticed that hinted handoff doesn’t seem to be working. I added several million
records (with consistency=one), and nothing appeared in system.hints (du -hs
showed a few dozen K bytes), nor did I get any pending Hinted Handoff tasks in
the Thread Pool
After check the log and configuration, I found it caused by two reason.
1. GC grace seconds
I using hector client to connect cassandra, and the default value of GC
grace seconds for each column family is **Zero** ! So when hinted handoff
replay the temporary value, the tombstone on other two
, the hinted handoff model will write back the
data to this node. So there are lots of create, update, delete.
I don't know due to GC or compaction, the delete records on other two nodes
seems not work, and if using cassandra client to list the data (also
consistency quorum), the deleted row show again
Thanks for sharing, here is some more information…
1 - At first, one of my node came down 5 min and when it came back it get
flooded by Hinted Handoff so hard that it could not handle the real time
queries properly. I haven't find a way to prioritize app queries rather than
Hinted Handoff
back and that I removed
earlier (because of the heavy load induced by Hinted Handoff while coming
back, which induced a lot of latencies in our app. This node didn't manage
to come back after 10 minutes, I removed it).
So there I faced 3 bugs (or problems) :
1 - At first, one of my node came down 5
is that there is no way, afaik, to limit the speed of these
Hinted Handoff since it's not a stream like repair or bootstrap, no way
either to keep the node out of the ring during the time it is receiving
hints since hints and normal traffic both go through gossip protocol on
port 7000.
How to avoid this Hinted
Hi,
Using C*1.2.2 12 EC2 xLarge cluster.
When I restart a node, if it spend a few minutes down, when I bring it up,
all the cpu are blocked at 100%, even once compactions are disabled,
inducing a very big and intolerable latency in my app. I suspect Hinted
Handoff to be the cause
The point is that there is no way, afaik, to limit the speed of these
Hinted Handoff since it's not a stream like repair or bootstrap, no way
either to keep the node out of the ring during the time it is receiving
hints since hints and normal traffic both go through gossip protocol on
port 7000
On Sun, Apr 14, 2013 at 11:28 AM, aaron morton aa...@thelastpickle.com wrote:
If hints are being stored, doesn't that imply DOWN nodes, and why don't I
see that in the logs?
Hints are stored for two reasons. First if the node is down when the write
request starts, second if the node does not
request starts, second if the node does not reply to the coordinator before
rpc_timeout. If you are not seeing dropped write messages it may indicate
network issues between the nodes.
I'm seeing hinted handoff kick in on all our nodes during periods of
high activity,
Are you seeing log messages
I'm seeing hinted handoff kick in on all our nodes during periods of
high activity, but all the nodes seem to be up (according to the logs
and nodetool status). The pattern in the logs is something like this:
18:10:45 194 READ messages dropped in last 5000ms
18:11:10 Started hinted handoff
On Fri, Apr 12, 2013 at 1:12 PM, Dane Miller d...@optimalsocial.com wrote:
I'm seeing hinted handoff kick in on all our nodes during periods of
high activity, but all the nodes seem to be up (according to the logs
and nodetool status). The pattern in the logs is something like this:
18:10:45
It contains mutation (data) that is to be sent to proper endpoint.
M.
W dniu 25.03.2013 20:15, Kanwar Sangha pisze:
Hi - Quick question. Do hints contain the actual data or the data is read from
the SStables and then sent to the other node when it comes up ?
Thanks,
Kanwar
Hi - Quick question. Do hints contain the actual data or the data is read from
the SStables and then sent to the other node when it comes up ?
Thanks,
Kanwar
have enough hints being created that its significantly
impacting your heap I have a feeling things are going to get out of sync
very quickly.
On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote:
Why do you think disabling hinted handoff will improve memory usage?
Thanks.
-Wei
Sent
wz1...@yahoo.com wrote:
Why do you think disabling hinted handoff will improve memory usage?
Thanks.
-Wei
Sent from my Samsung smartphone on ATT
Original message
Subject: Re: hinted handoff disabling trade-offs
From: Michael Kjellman mkjell...@barracuda.com
To: user
nodes, does
this still hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream
from say 100 nodes...
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: 06 March 2013 23:47
To: user@cassandra.apache.org
Subject: Re: Hinted handoff
Check the IO utilisation using iostat
You
will take roughly 2.2hrs assume we stream from say
100 nodes...
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: 06 March 2013 23:47
To: user@cassandra.apache.org
Subject: Re: Hinted handoff
Check the IO utilisation using iostat
You *really* should not need to make HH run faster, if you do
and virtual nodes, does this
still hold ? 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say
100 nodes...
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: 06 March 2013 23:47
To: user@cassandra.apache.org
Subject: Re: Hinted handoff
Check the IO utilisation using iostat
You
Hi - Is there a way to increase the hinted handoff throughput ? I am seeing
around 8Mb/s (bits).
Thanks,
Kanwar
Got the param. thanks
From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 13:50
To: user@cassandra.apache.org
Subject: Hinted handoff
Hi - Is there a way to increase the hinted handoff throughput ? I am seeing
around 8Mb/s (bits).
Thanks,
Kanwar
After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It
still does not go above 25Mb/s. Is there a limitation ?
From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user@cassandra.apache.org
Subject: RE: Hinted handoff
Got the param. thanks
2013 15:11
To: user@cassandra.apache.org
Subject: RE: Hinted handoff
After trying to bump up the hinted_handoff_throttle_in_kb to 1G/b per sec, It
still does not go above 25Mb/s. Is there a limitation ?
From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013 14:41
To: user
March 2013 15:11
To: user@cassandra.apache.org
Subject: RE: Hinted handoff
After trying to bump up the “hinted_handoff_throttle_in_kb” to 1G/b per sec,
It still does not go above 25Mb/s. Is there a limitation ?
From: Kanwar Sangha [mailto:kan...@mavenir.com]
Sent: 06 March 2013
quickly.
On Mar 4, 2013, at 9:17 PM, Wz1975 wz1...@yahoo.com wrote:
Why do you think disabling hinted handoff will improve memory usage?
Thanks.
-Wei
Sent from my Samsung smartphone on ATT
Original message
Subject: Re: hinted handoff disabling trade-offs
I am looking to get a second opinion about disabling hinted-handoffs. I
have an application that can tolerate a fair amount of inconsistency
(advertising domain), and so I'm weighting the pros and cons of hinted
handoffs. I'm running Cassandra 1.0, looking to upgrade to 1.1 soon.
Pros of
Repair is slow.
On Mar 4, 2013, at 8:07 PM, Matt Kap matvey1...@gmail.com wrote:
I am looking to get a second opinion about disabling hinted-handoffs. I
have an application that can tolerate a fair amount of inconsistency
(advertising domain), and so I'm weighting the pros and cons of hinted
Why do you think disabling hinted handoff will improve memory usage?
Thanks.
-Wei
Sent from my Samsung smartphone on ATT
Original message
Subject: Re: hinted handoff disabling trade-offs
From: Michael Kjellman mkjell...@barracuda.com
To: user@cassandra.apache.org user
Also, if you have enough hints being created that its significantly impacting
your heap I have a feeling things are going to get out of sync very quickly.
On Mar 4, 2013, at 9:17 PM, Wz1975
wz1...@yahoo.commailto:wz1...@yahoo.com wrote:
Why do you think disabling hinted handoff will improve
On 2012-11-24, at 10:37 AM, Chuan-Heng Hsiao hsiao.chuanh...@gmail.com wrote:
However, I continue seeing the following in /var/log/cassandra/system.log
INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token
Hi Mina,
Thank you very much for replying.
My machines are running Oracle java 1.7.09.
so that might be the issue. (My machines are running
Oracle java 1.7.09 from the beginning.)
I cleaned up the specific empty CF again, and the
hinted handoff of 0 rows showed up again, and
the scrub system
in /var/log/cassandra/system.log
INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
HintedHandOffManager.java (line 392
in /var/log/cassandra/system.log
INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
HintedHandOffManager.java (line 392
) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /192.168.0.10
every ten mins.
(It seems fall into some infinite loop
the following in /var/log/cassandra/system.log
INFO [HintedHandoff:1] 2012-11-24 22:58:28,088
HintedHandOffManager.java (line 296) Started hinted handoff for token:
27949589543905115548813332729343195104 with IP: /192.168.0.10
INFO [HintedHandoff:1] 2012-11-24 22:58:28,089
every night. It's a
low traffic cluster to test cassandra, if that matters.
.vegard,
- Original Message -
From: user@cassandra.apache.org
To:
Cc:
Sent:Sun, 11 Nov 2012 13:16:01 -0600
Subject:Re: Hinted Handoff runs every ten minutes
How many hint sstables are there? What does sstable2json
How many hint sstables are there? What does sstable2json show?
On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner m...@librato.com wrote:
Is there a ticket open for this for 1.1.6?
We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
0 row hinted handoff every 10 minutes. N
Is there a ticket open for this for 1.1.6?
We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
while that node hints to another node.
On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget p...@fantasista.no
]
Sent: Monday, October 29, 2012 3:46 PM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff?
you
the column family:
[default@system] truncate HintsColumnFamily;
HintsColumnFamily truncated.
-Original Message-
From: Radim Kolar [mailto:h...@filez.com]
Sent: Monday, October 29, 2012 3:46 PM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23
-
From: user@cassandra.apache.org
To:
Cc:
Sent:Mon, 29 Oct 2012 23:45:30 +0100
Subject:Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff
-
From: user@cassandra.apache.org
To:
Cc:
Sent:Mon, 29 Oct 2012 23:45:30 +0100
Subject:Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff
With both data centers functional, the test takes just a few minutes to run,
with one data center down, 15x the amount of time.
Could you provide the numbers, it's easier to get a feel for how the throughput
is dropping. Does latency reported by nodetool cf stats change ?
I'm also interested
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff?
Steve
-Original Message-
From: Brandon Williams [mailto:dri...@gmail.com]
Sent: Wednesday, October 24, 2012 4:56 AM
To: user@cassandra.apache.org
Subject: Re: Hinted
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in
cassandra-cli to check
the hints from JConsole by using HintedHadOffManager MBean.
Thanks.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-stop-hinted-handoff-tp7583060p7583086.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive
On Sun, Oct 21, 2012 at 6:44 PM, aaron morton aa...@thelastpickle.com wrote:
I *think* this may be ghost rows which have not being compacted.
You would be correct in the case of 1.0.8:
https://issues.apache.org/jira/browse/CASSANDRA-3955
-Brandon
Is there a walk around other than upgrade?
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
ta...@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Wed, Oct 24, 2012 at 1:56 PM, Brandon Williams dri...@gmail.com wrote:
I'm testing various scenarios in a multi data center configuration. The setup
is 10 Cassandra 1.1.5 nodes configured into two data centers, 5 nodes in each
DC (RF DC1:3,DC2:3, write consistency LOCAL_QUORUM). I have a synthetic random
data generator that I can run, and each run adds roughly
Thanks.
I thought it had been addressed so before but couldn't find the ticket.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/10/2012, at 12:56 AM, Brandon Williams dri...@gmail.com wrote:
On Sun, Oct 21, 2012 at 6:44 PM, aaron
Hints store the columns, row key, KS name and CF id(s) for each mutation to
each node. Where an executed mutation will store the most recent columns
collated with others under the same row key. So depending on the type of
mutation hints will take up more space.
The worse case would be lots of
://www.thelastpickle.com
On 19/10/2012, at 7:16 AM, David Daeschler david.daesch...@gmail.com
wrote:
Hi Steve,
Also confirming this. After having a node go down on Cassandra 1.0.8
there seems to be hinted handoff between two of our 4 nodes every 10
minutes. Our setup also shows 0 rows. It does
wrote:
Hi Steve,
Also confirming this. After having a node go down on Cassandra 1.0.8
there seems to be hinted handoff between two of our 4 nodes every 10
minutes. Our setup also shows 0 rows. It does not appear to have any
effect on the operation of the ring, just fills up the log files
just fine.
I stopped the load generation, and got distracted with some other
project/problem. A few days later, I noticed something strange on one of the
nodes. On this node hinted handoff starts every ten minutes, and while it seems
to finish without any errors, it will be started again in ten
Hi Steve,
Also confirming this. After having a node go down on Cassandra 1.0.8
there seems to be hinted handoff between two of our 4 nodes every 10
minutes. Our setup also shows 0 rows. It does not appear to have any
effect on the operation of the ring, just fills up the log files.
- David
if you don't use Write Level ANY, It will be automactically turned off.
Just use Write Level One.
2012/10/9 Manu Zhang owenzhang1...@gmail.com
Hi, all
I tried out the client_only example as another local node 127.0.0.2 and
then it went down. Now the node (127.0.0.1) started hinted handoff
...@gmail.com
Hi, all
I tried out the client_only example as another local node 127.0.0.2 and
then it went down. Now the node (127.0.0.1) started hinted handoff to
iteself. How to stop that?
Thanks!
-21 09:49:56,757 HintedHandOffManager.java
(line 294) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 09:49:56,775 HintedHandOffManager.java
(line 382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
INFO
Hi,
We are running a small test cluster and recently installed Cassandra 1.1 and
started with a new clean database. We keep seeing these messages in the log
on just one of our nodes:
INFO [HintedHandoff:1] 2012-05-21 09:49:56,757 HintedHandOffManager.java (line
294) Started hinted handoff
/cassandradb/data/system/LocationInfo-hc-52-Data.db,]. 844 to 438
(~51% of original) bytes for 4 keys at 0.003941MB/s. Time: 106ms.
2012-03-13 13:13:42,410 INFO [HintedHandOffManager] Started hinted handoff
for token: 59868989542208531803879358296598929701 with IP: /10.1.161.67
2012-03-13 13:13
Hi,
I have been running Cassandra 1.0.7 and in the log file I see the log saying
Finished hinted handoff of 0 rows to endpoint /{ipaddress}
The above issue can be reproduced by the following steps,
1. Start a cluster with 2 node, suppose node1 and node2
2. Create a keyspace with rf=2, create
. Restart node2
When the node2 is restarted, node1 sends the hints to the node2 and from the
log I see that 100 rows are sent. But, after that in the interval of
approximately 10 mins, Cassandra logs Finished hinted handoff of 0 rows to
the endpoint ..
When I do the list hintscolumnfamily from
(line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704) Enqueuing flush of
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2
ops)
INFO [FlushWriter:31] 2012-02-23 20:05:50,049
running Cassandra 1.0.7 and in the log file I see the log
saying
Finished hinted handoff of 0 rows to endpoint /{ipaddress}
The above issue can be reproduced by the following steps,
1. Start a cluster with 2 node, suppose node1 and node2
2. Create a keyspace with rf=2, create column family
3
it started pushing the hints.
Here is the log:
INFO [GossipStage:1] 2012-02-23 20:05:32,516 StorageService.java (line
988) Node /192.0.2.1 state jump to normal
INFO [HintedHandoff:1] 2012-02-23 20:05:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP
:32,516 StorageService.java (line
988) Node /192.0.2.1 state jump to normal
INFO [HintedHandoff:1] 2012-02-23 20:05:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704
:49,766
HintedHandOffManager.java (line 296) Started hinted handoff for token: 1
with IP: /192.0.2.1
INFO [HintedHandoff:1] 2012-02-23 20:05:50,048 ColumnFamilyStore.java
(line 704) Enqueuing flush of
Memtable-HintsColumnFamily@1352140719(205/1639 serialized/live bytes, 2
ops)
INFO [FlushWriter
Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.
I've done some debugging in
You're right, good catch.
Do you mind opening a ticket on jira
(https://issues.apache.org/jira/browse/CASSANDRA)?
--
Sylvain
On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
fredrik.l.stigb...@sitevision.se wrote:
Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node
Yes, I'll do that.
/Fredrik
Sylvain Lebresne skrev 2011-12-01 11:10:
You're right, good catch.
Do you mind opening a ticket on jira
(https://issues.apache.org/jira/browse/CASSANDRA)?
--
Sylvain
On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
fredrik.l.stigb...@sitevision.se wrote:
Hi,
Sorry for not checking source to see if things have changed but i just
remembered an issue I have forgotten to make jira for.
In old days, nodes would periodically try to deliver queues.
However, this was at some stage changed so it only deliver if a node is being
marked up.
However, you can
Nope, that's a separate issue.
https://issues.apache.org/jira/browse/CASSANDRA-3554
On Thu, Dec 1, 2011 at 5:59 PM, Terje Marthinussen
tmarthinus...@gmail.com wrote:
Sorry for not checking source to see if things have changed but i just
remembered an issue I have forgotten to make jira for.
If you are still hunting 1.0 assert during hintedhandoff, i can
reproduce it anytime. both nodes are 1.0.2. And just hint, we are using
super column families.
INFO [HintedHandoff:1] 2011-11-08 21:33:54,118
HintedHandOffManager.java (line 268) Started hinted handoff for token: 0
with IP
super column
families.
INFO [HintedHandoff:1] 2011-11-08 21:33:54,118 HintedHandOffManager.java
(line 268) Started hinted handoff for token: 0 with IP: /***.104.18
ERROR [HintedHandoff:1] 2011-11-08 21:33:54,257 AbstractCassandraDaemon.java
(line 133) Fatal exception in thread Thread
mine issue has different stacktrace
You're right. https://issues.apache.org/jira/browse/CASSANDRA-3466
On Tue, Nov 8, 2011 at 3:14 PM, Radim Kolar h...@sendmail.cz wrote:
mine issue has different stacktrace
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
1 - 100 of 154 matches
Mail list logo
