Re: Monitoring move progress

2011-12-22 Thread aaron morton
 My expectation is that after the compactions (which the project wiki refers 
 to as anti-compactions), I would start to see outbound streaming activity 
 in netstats.  
That is very old information. 
The compactions you saw may have been the result of flushing to disk before 
starting the move. 
You may also see them after the streaming as completed and the new SSTables are 
compacted with the older ones. 


What is nodetool ring saying ? 

The last thing that logs before the streaming starts is Sleeping {} ms before 
start streaming/fetching ranges.

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 22/12/2011, at 5:07 PM, Ethan Rowe wrote:

 I've got some nodes in a moving state in a cluster (the nodes to which they 
 stream shouldn't overlap), and I'm finding it difficult to determine if 
 they're actually doing anything related to the move at this point, or if 
 they're stuck in the state and not actually doing anything.
 
 In each case, I issued the move command per usual.
 
 The log shows information about the move when it begins, showing the correct 
 token change that I would expect in each case.
 
 Compactions took place on each moving node, which can be viewed through 
 nodetool compactionstats or through the CompactionManager in JMX.
 
 But eventually the compactions stopped, apart from various ongoing secondary 
 index rebuilds and consequent related index compactions.  Yet I see no stream 
 transfers via netstats.  My expectation is that after the compactions (which 
 the project wiki refers to as anti-compactions), I would start to see 
 outbound streaming activity in netstats.  Yet I do not.
 
 I don't see any errors listed in the logs on the moving servers since the 
 moves began.
 
 Using cassandra 1.0.5.  ByteOrderedPartitioner.
 
 Any suggestions on how to determine what's going on?
 
 Thanks in advance.
 - Ethan



Monitoring move progress

2011-12-21 Thread Ethan Rowe
I've got some nodes in a moving state in a cluster (the nodes to which
they stream shouldn't overlap), and I'm finding it difficult to determine
if they're actually doing anything related to the move at this point, or if
they're stuck in the state and not actually doing anything.

In each case, I issued the move command per usual.

The log shows information about the move when it begins, showing the
correct token change that I would expect in each case.

Compactions took place on each moving node, which can be viewed through
nodetool compactionstats or through the CompactionManager in JMX.

But eventually the compactions stopped, apart from various ongoing
secondary index rebuilds and consequent related index compactions.  Yet I
see no stream transfers via netstats.  My expectation is that after the
compactions (which the project wiki refers to as anti-compactions), I
would start to see outbound streaming activity in netstats.  Yet I do not.

I don't see any errors listed in the logs on the moving servers since the
moves began.

Using cassandra 1.0.5.  ByteOrderedPartitioner.

Any suggestions on how to determine what's going on?

Thanks in advance.
- Ethan