Hey Andrew, Would you mind attaching a `riak-debug` from that node? That will give us a full picture of the node as well as all the logs.
Thanks, Brian -- Brian Sparrow Developer Advocate Basho Technologies Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, November 27, 2013 at 1:47 AM, Andrew Tynefield wrote: > Hey guys, > > Sorry for the bump, but, I'm kind of at a loss on how to troubleshoot this > further. In addition to all the previously mentioned stuff, I have since left > the cluster on riak1, deleted that VM, completely reprovisioned it and the > exact same issues are occurring. Please let me know if there's anything else > I can provide to help anyone help me. > > Thanks, > Andrew > > > > On Sat, Nov 23, 2013 at 2:14 PM, Andrew Tynefield <atynefi...@gmail.com > (mailto:atynefi...@gmail.com)> wrote: > > Hey guys, > > > > I'm encountering a timeout when attempting a write using `riak-admin test`. > > > > > (02:01:08) [riak1] ~ $ time riak-admin test > > > > > > Failed to write test value: {error,timeout} > > > > > > real 1m1.561s > > > user 0m0.325s > > > > > > sys 0m0.103s > > > > > > > > > (02:02:39) [riak1] ~ $ > > > > > > > This is one node of a 4 node cluster. The other nodes all work perfectly > > fine when attempting this. What I also find very weird is that this node > > was the second node to be created, nodes created after it (riak{2,3}) > > complete their tests fine and do not appear to have any issues. > > > > vm.args and app.config are generated by puppet and are used with identical > > values (except for the node name) by all nodes. > > > > Values of note may be: > > {pb_backlog, 128}, > > > > {ring_creation_size, 256}, > > > > > > > > I've also tried leaving the cluster from this node, waiting for the ring to > > re-stabilize and show it successfully leaving. I then removed the entire > > /var/lib/riak/*, started riak back up, rejoined the cluster. (Prior to > > rejoining, I did a riak-admin test and it succeeded.) > > > > I've provided a tail of all the logs within /var/log/riak below immediately > > proceeding the execution of the riak-admin test command. > > > > > ==> /var/log/riak/console.log <== > > > 2013-11-23 13:45:53.597 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,anti_entropy} = enabled_v1 > > > 2013-11-23 13:45:53.798 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,handoff_data_encoding} = encode_raw > > > 2013-11-23 13:45:54.187 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,object_format} = v1 > > > 2013-11-23 13:45:54.405 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,secondary_index_version} = v2 > > > 2013-11-23 13:45:54.683 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,vclock_data_encoding} = encode_zlib > > > 2013-11-23 13:45:55.066 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_kv,crdt} = [pncounter] > > > 2013-11-23 13:45:56.141 [info] > > > <0.157.0>@riak_core_capability:process_capability_changes:530 New > > > capability: {riak_control,member_info_version} = v1 > > > 2013-11-23 13:45:56.277 [info] <0.7.0> Application riak_control started > > > on node 'r...@riak1.tyne.io (mailto:r...@riak1.tyne.io)' > > > 2013-11-23 13:45:56.277 [info] <0.7.0> Application erlydtl started on > > > node 'r...@riak1.tyne.io (mailto:r...@riak1.tyne.io)' > > > 2013-11-23 13:46:11.607 [info] <0.538.0>@riak_core:wait_for_service:464 > > > Wait complete for service riak_kv (14 seconds) > > > > > > ==> /var/log/riak/crash.log <== > > > 2013-11-23 02:01:43 =ERROR REPORT==== > > > Hintfile > > > '/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43 =ERROR REPORT==== > > > Hintfile > > > '/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43 =ERROR REPORT==== > > > Hintfile > > > '/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43 =ERROR REPORT==== > > > Hintfile > > > '/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43 =ERROR REPORT==== > > > Hintfile > > > '/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint' > > > invalid > > > > > > ==> /var/log/riak/error.log <== > > > 2013-11-23 02:01:43.869 [error] <0.2775.0> Hintfile > > > '/var/lib/riak/bitcask/1353030812685249967243255099038121389290843799552/15.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.869 [error] <0.2723.0> Hintfile > > > '/var/lib/riak/bitcask/1261686960352068534855524796993353700562348015616/34.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.869 [error] <0.2593.0> Hintfile > > > '/var/lib/riak/bitcask/1170343108018887102467794494948586011833852231680/27.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.869 [error] <0.2623.0> Hintfile > > > '/var/lib/riak/bitcask/1216015034185477818661659645970969856198100123648/37.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.870 [error] <0.2704.0> Hintfile > > > '/var/lib/riak/bitcask/1238850997268773176758592221482161778380224069632/33.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.870 [error] <0.2855.0> Hintfile > > > '/var/lib/riak/bitcask/1444374665018431399630985401082889078019339583488/33.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.870 [error] <0.2805.0> Hintfile > > > '/var/lib/riak/bitcask/1375866775768545325340187674549313311472967745536/29.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.870 [error] <0.2561.0> Hintfile > > > '/var/lib/riak/bitcask/1124671181852296386273929343926202167469604339712/42.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.870 [error] <0.2823.0> Hintfile > > > '/var/lib/riak/bitcask/1421538701935136041534052825571697155837215637504/30.bitcask.hint' > > > invalid > > > 2013-11-23 02:01:43.871 [error] <0.2759.0> Hintfile > > > '/var/lib/riak/bitcask/1330194849601954609146322523526929467108719853568/35.bitcask.hint' > > > invalid > > > > > > ==> /var/log/riak/run_erl.log <== > > > run_erl [2259] Sat Nov 23 13:45:49 2013 > > > Args before exec of shell: > > > run_erl [2259] Sat Nov 23 13:45:49 2013 > > > argv[0] = sh > > > run_erl [2259] Sat Nov 23 13:45:49 2013 > > > argv[1] = -c > > > run_erl [2259] Sat Nov 23 13:45:49 2013 > > > argv[2] = exec /usr/sbin/riak console > > > > > > > > Here's some information about the installation and cluster: > > > > > (02:04:36) [riak1] ~ $ riak version > > > 1.4.2 > > > > > > > > > (02:08:07) [riak1] ~ $ riak-admin ringready > > > TRUE All nodes agree on the ring ['r...@riak.tyne.io > > > (mailto:r...@riak.tyne.io)','r...@riak1.tyne.io > > > (mailto:r...@riak1.tyne.io)', > > > > > > 'r...@riak2.tyne.io > > > (mailto:r...@riak2.tyne.io)','r...@riak3.tyne.io > > > (mailto:r...@riak3.tyne.io)'] > > > > > > > > > (02:07:31) [riak1] ~ $ riak-admin member-status > > > ================================= Membership > > > ================================== > > > > > > Status Ring Pending Node > > > > > > ------------------------------------------------------------------------------- > > > > > > valid 25.0% -- 'r...@riak.tyne.io > > > (mailto:r...@riak.tyne.io)' > > > > > > valid 25.0% -- 'r...@riak1.tyne.io > > > (mailto:r...@riak1.tyne.io)' > > > > > > valid 25.0% -- 'r...@riak2.tyne.io > > > (mailto:r...@riak2.tyne.io)' > > > > > > valid 25.0% -- 'r...@riak3.tyne.io > > > (mailto:r...@riak3.tyne.io)' > > > > > > ------------------------------------------------------------------------------- > > > > > > Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 > > > > > > > > > Touch test: > > > > > (02:07:00) [riak1] ~ $ for dir in /var/lib/riak/*/; do touch $dir/test; > > > stat $dir/test; done > > > File: `/var/lib/riak/anti_entropy//test' > > > > > > Size: 0 Blocks: 0 IO Block: 4096 regular empty file > > > > > > Device: 802h/2050d Inode: 1704796 Links: 1 > > > > > > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > > > > > > Access: 2013-11-23 14:07:31.284244924 -0600 > > > > > > Modify: 2013-11-23 14:07:31.284244924 -0600 > > > > > > Change: 2013-11-23 14:07:31.284244924 -0600 > > > > > > File: `/var/lib/riak/bitcask//test' > > > > > > Size: 0 Blocks: 0 IO Block: 4096 regular empty file > > > > > > Device: 802h/2050d Inode: 1705089 Links: 1 > > > > > > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > > > > > > Access: 2013-11-23 14:07:31.310243834 -0600 > > > > > > Modify: 2013-11-23 14:07:31.310243834 -0600 > > > > > > Change: 2013-11-23 14:07:31.310243834 -0600 > > > > > > File: `/var/lib/riak/kv_vnode//test' > > > > > > Size: 0 Blocks: 0 IO Block: 4096 regular empty file > > > > > > Device: 802h/2050d Inode: 1705150 Links: 1 > > > > > > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > > > > > > Access: 2013-11-23 14:07:31.316243674 -0600 > > > > > > Modify: 2013-11-23 14:07:31.316243674 -0600 > > > > > > Change: 2013-11-23 14:07:31.316243674 -0600 > > > > > > File: `/var/lib/riak/leveldb//test' > > > > > > Size: 0 Blocks: 0 IO Block: 4096 regular empty file > > > > > > Device: 802h/2050d Inode: 1705153 Links: 1 > > > > > > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > > > > > > Access: 2013-11-23 14:07:31.322243576 -0600 > > > > > > Modify: 2013-11-23 14:07:31.322243576 -0600 > > > > > > Change: 2013-11-23 14:07:31.322243576 -0600 > > > > > > File: `/var/lib/riak/ring//test' > > > > > > Size: 0 Blocks: 0 IO Block: 4096 regular empty file > > > > > > Device: 802h/2050d Inode: 1705154 Links: 1 > > > > > > Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) > > > > > > Access: 2013-11-23 14:07:31.326243490 -0600 > > > > > > Modify: 2013-11-23 14:07:31.326243490 -0600 > > > > > > Change: 2013-11-23 14:07:31.326243490 -0600 > > > > > > > > > Any help would be greatly appreciated! > > > > Thanks, > > Andrew > > > > -- > > [Andrew Tynefield] > > > -- > [Andrew Tynefield] > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
