Re: Waiting for riak to handoff ?
For archiving purposes, running the code in the below link resolved Marcel's issue. On Mon, Aug 25, 2014 at 9:55 AM, Luke Bakken wrote: > Hi Marcel, > > I discussed this issue with a colleague and he agrees that running the > code you found in these release notes should fix this issue: > > https://github.com/basho/riak/blob/1.0.2-release/RELEASE-NOTES.org#ownership-handoff-stall > > Somehow the final phase of handoff was interrupted and the ring > changes did not gossip. Let us know if this resolves your issue. > > -- > Luke Bakken > CSE > lbak...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Waiting for riak to handoff ?
Hi Marcel - Could you run the following commands to see if it restarts handoff? riak-admin transfer-limit 0 riak-admin transfer-limit 8 -- Luke Bakken CSE lbak...@basho.com On Fri, Aug 22, 2014 at 12:29 AM, Marcel Koopman wrote: > Hello, we upgraded to 1.4.10 on all nodes. Please see below: > > riak-admin transfers > > 'riak@newnode5 waiting to handoff 11 partitions > > 'riak@newnode4' waiting to handoff 11 partitions > > 'riak@newnode3' waiting to handoff 11 partitions > > 'riak@newnode2' waiting to handoff 11 partitions > > 'riak@newnode1' waiting to handoff 11 partitions > > 'riak@oldnode2' waiting to handoff 11 partitions > > 'riak@oldnode3' waiting to handoff 11 partitions ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Waiting for riak to handoff ?
Hello, we upgraded to 1.4.10 on all nodes. Please see below:
riak-admin transfers
'riak@newnode5 waiting to handoff 11 partitions
'riak@newnode4' waiting to handoff 11 partitions
'riak@newnode3' waiting to handoff 11 partitions
'riak@newnode2' waiting to handoff 11 partitions
'riak@newnode1' waiting to handoff 11 partitions
'riak@oldnode2' waiting to handoff 11 partitions
'riak@oldnode3' waiting to handoff 11 partitions
Active Transfers:
$ riak-admin transfer-limit
=== Transfer Limit
LimitNode
---
2'riak@oldnode3'
2'riak@oldnode2'
2'riak@oldnode1'
2'riak@newnode1'
2'riak@newnode2'
2'riak@newnode3'
2'riak@newnode4'
2'riak@newnode5'
---
Note: You can change transfer limits with 'riak-admin transfer_limit
'
and 'riak-admin transfer_limit '
$ riak-admin status
1-minute stats for 'riak@newnode1'
---
riak_kv_stat_ts : 1408692262
vnode_gets : 0
vnode_gets_total : 374554
vnode_puts : 0
vnode_puts_total : 11795
vnode_index_refreshes : 0
vnode_index_refreshes_total : 0
vnode_index_reads : 0
vnode_index_reads_total : 683394
vnode_index_writes : 0
vnode_index_writes_total : 11715
vnode_index_writes_postings : 0
vnode_index_writes_postings_total : 10449
vnode_index_deletes : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings : 0
vnode_index_deletes_postings_total : 675
node_gets : 0
node_gets_total : 117806
node_get_fsm_siblings_mean : 0
node_get_fsm_siblings_median : 0
node_get_fsm_siblings_95 : 0
node_get_fsm_siblings_99 : 0
node_get_fsm_siblings_100 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_time_mean : 0
node_get_fsm_time_median : 0
node_get_fsm_time_95 : 0
node_get_fsm_time_99 : 0
node_get_fsm_time_100 : 0
node_puts : 0
node_puts_total : 4575
node_put_fsm_time_mean : 0
node_put_fsm_time_median : 0
node_put_fsm_time_95 : 0
node_put_fsm_time_99 : 0
node_put_fsm_time_100 : 0
read_repairs : 0
read_repairs_total : 129
coord_redirs_total : 1427
executing_mappers : 0
precommit_fail : 0
postcommit_fail : 0
index_fsm_create : 0
index_fsm_create_error : 0
index_fsm_active : 0
list_fsm_create : 0
list_fsm_create_error : 0
list_fsm_active : 0
pbc_active : 56
pbc_connects : 60
pbc_connects_total : 163594
node_get_fsm_active : 0
node_get_fsm_active_60s : 10
node_get_fsm_in_rate : 0
node_get_fsm_out_rate : 0
node_get_fsm_rejected : 0
node_get_fsm_rejected_60s : 0
node_get_fsm_rejected_total : 0
node_put_fsm_active : 0
node_put_fsm_active_60s : 0
node_put_fsm_in_rate : 0
node_put_fsm_out_rate : 0
node_put_fsm_rejected : 0
node_put_fsm_rejected_60s : 0
node_put_fsm_rejected_total : 0
read_repairs_primary_outofdate_one : 0
read_repairs_primary_outofdate_count : 104
read_repairs_primary_notfound_one : 0
read_repairs_primary_notfound_count : 61
read_repairs_fallback_notfound_one : 0
read_repairs_fallback_notfound_count : 40
leveldb_read_block_error : 0
riak_pipe_stat_ts : 1408692262
pipeline_active : 0
pipeline_create_count : 0
pipeline_create_one : 0
pipeline_create_error_count : 0
pipeline_create_error_one : 0
cpu_nprocs : 263
cpu_avg1 : 20
cpu_avg5 : 23
cpu_avg15 : 33
mem_total : 8374890496
mem_allocated : 7878340608
disk : [{"/",5766260,42},
{"/dev",4080488,1},
{"/run",1635724,1},
{"/run/lock",5120,0},
{"/run/shm",4089300,0},
{"/var/log",1922060,4},
{"/data",258030980,69}]
nodename : 'riak@ *oldnode2*'
connected_nodes :
['newnode1','newnode2','newnode3','newnode4',newnode5','oldnode1','oldnode3',
'riak@oldnode2']
sys_driver_version : <<"2.0">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 2
sys_otp_release : <<"R15B01">>
sys_process_count : 1175
sys_smp_support : true
sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:2:2] [async-threads:64] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 64
sys_wordsize : 8
ring_members :
['newnode1','newnode2','newnode3','newnode4',newnode5','oldnode1','oldnode3',
'riak@oldnode2']
ring_num_partitions : 64
ring_ownership : <<"[{'riak@oldnode1',16},\n {'riak@oldnode2',32},\n
{'riak@oldnode3',16}]">>
ring_creation_size : 64
storage_backend : riak_kv_eleveldb_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.4.10-0-g73c43c3">>
cluster_info_version : <<"1.2.4">>
riak_search_version : <<"1.4.10-0-g6e548e7">>
merge_index_version : <<"
Re: Waiting for riak to handoff ?
Hi Marcel - What version of Riak are you using? What is the output of "riak-admin transfer-limit" ? -- Luke Bakken CSE lbak...@basho.com On Wed, Aug 20, 2014 at 8:02 AM, Marcel Koopman wrote: > Hello, > > It seems that adding new nodes does not have any effect. > > Why it Riak waiting for transfers? When does it start handoff the new nodes? > > > riak-admin member-status > > = Membership > == > > Status RingPendingNode > > --- > > valid 25.0% 34.4%'riak@oldnode1' > > valid 25.0% 32.8%'riak@oldnode2' > > valid 50.0% 32.8%'riak@oldnode3 > > valid 0.0% 0.0%'riak@newnode1' > > valid 0.0% 0.0%'riak@newnode2' > > valid 0.0% 0.0%'riak@newnode3' > > valid 0.0% 0.0%'riak@newnode4' > > valid 0.0% 0.0%'riak@newnode5' > > --- > > Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 > > > riak-admin transfers > > 'riak@newnode5' waiting to handoff 11 partitions > > 'riak@newnode4' waiting to handoff 11 partitions > > 'riak@newnode3' waiting to handoff 11 partitions > > 'riak@newnode2' waiting to handoff 11 partitions > > 'riak@newnode1' waiting to handoff 11 partitions > > 'riak@oldnode1' waiting to handoff 11 partitions > > 'riak@oldnode2' waiting to handoff 11 partitions > > Active Transfers: > > (nothing here) > > > > > > -- > Met vriendelijke groet, > > Marcel Koopman > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Waiting for riak to handoff ?
Here is some additional information, it seems the ownership handoff is waiting, see below: riak-admin ring-status == Claimant === Claimant: 'riak@oldnode1' Status: up Ring Ready: true == Ownership Handoff == Owner: riak@oldnode2 Next Owner: riak@oldnode3 Index: 68507889249886074290797726533575766546371837952 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 342539446249430371453988632667878832731859189760 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 616571003248974668617179538802181898917346541568 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 890602560248518965780370444936484965102833893376 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 1164634117248063262943561351070788031288321245184 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 1438665674247607560106752257205091097473808596992 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] --- Owner: riak@oldnode2 Next Owner: riak@oldnode1 Index: 159851741583067506678528028578343455274867621888 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 433883298582611803841718934712646521460354973696 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 707914855582156101004909840846949587645842325504 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 981946412581700398168100746981252653831329677312 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] Index: 125597796958124469533129165311720016817029120 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode] --- == Unreachable Nodes == All nodes are up and reachable 2014-08-20 17:02 GMT+02:00 Marcel Koopman : > Hello, > > It seems that adding new nodes does not have any effect. > > Why it Riak waiting for transfers? When does it start handoff the new > nodes? > > > riak-admin member-status > > = Membership > == > > Status RingPendingNode > > > --- > > valid 25.0% 34.4%'riak@oldnode1' > > valid 25.0% 32.8%'riak@oldnode2' > > valid 50.0% 32.8%'riak@oldnode3 > > valid 0.0% 0.0%'riak@newnode1' > > valid 0.0% 0.0%'riak@newnode2' > > valid 0.0% 0.0%'riak@newnode3' > > valid 0.0% 0.0%'riak@newnode4' > > valid 0.0% 0.0%'riak@newnode5' > > > --- > > Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 > > riak-admin transfers > > 'riak@newnode5' waiting to handoff 11 partitions > > 'riak@newnode4' waiting to handoff 11 partitions > > 'riak@newnode3' waiting to handoff 11 partitions > > 'riak@newnode2' waiting to handoff 11 partitions > > 'riak@newnode1' waiting to handoff 11 partitions > > 'riak@oldnode1' waiting to handoff 11 partitions > > 'riak@oldnode2' waiting to handoff 11 partitions > > Active Transfers: > > (nothing here) > > > > > > -- > Met vriendelijke groet, > > Marcel Koopman > -- Met vriendelijke groet, Marcel Koopman ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Waiting for riak to handoff ?
Hello, It seems that adding new nodes does not have any effect. Why it Riak waiting for transfers? When does it start handoff the new nodes? riak-admin member-status = Membership == Status RingPendingNode --- valid 25.0% 34.4%'riak@oldnode1' valid 25.0% 32.8%'riak@oldnode2' valid 50.0% 32.8%'riak@oldnode3 valid 0.0% 0.0%'riak@newnode1' valid 0.0% 0.0%'riak@newnode2' valid 0.0% 0.0%'riak@newnode3' valid 0.0% 0.0%'riak@newnode4' valid 0.0% 0.0%'riak@newnode5' --- Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 riak-admin transfers 'riak@newnode5' waiting to handoff 11 partitions 'riak@newnode4' waiting to handoff 11 partitions 'riak@newnode3' waiting to handoff 11 partitions 'riak@newnode2' waiting to handoff 11 partitions 'riak@newnode1' waiting to handoff 11 partitions 'riak@oldnode1' waiting to handoff 11 partitions 'riak@oldnode2' waiting to handoff 11 partitions Active Transfers: (nothing here) -- Met vriendelijke groet, Marcel Koopman ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
