Hi Richard, Thanks for getting back. Yes the nodes which were not able to see each other in my case riak4 and riak6 showed increased latency. Other node latency was fine.Yes I will start running mtr[0] and will update if this thing happens again.
Thanks, Shrikar On Tue, Feb 12, 2013 at 7:54 AM, Richard Shaw <[email protected]> wrote: > Hi Shrikar, > > I'd like to confirm there isn't underlying network weirdness going on. > > You said in your mail that "During this time the latency increases to 1 > second or more than that." can you elaborate, was that from running ping > $node manually that you saw the latency increase? > > What network monitoring do you have in place on your network at the > moment? If nothing formal, leave mtr[0] running in a screen/tmux session > between your nodes then review the latency and loss. > > [0]http://www.bitwizard.nl/mtr/ > > Regards > > Richard > > On Tue, Feb 12, 2013 at 6:46 AM, Shrikar archak <[email protected]>wrote: > >> Sorry guys for the delay in response. >> >> To answer dimitry's question >> This is what I observed >> >> Actual nodes ping : Works ( I literally mean ping riak1 , ping riak2 ....) >> I didnt try riak ping .. But from each Riak node I tried riak-admin >> ring-status , this is were I saw the some nodes not reachable. >> >> Example >> from riak6(192.168.8.136) I tried the above command and it replied back >> saying riak4(192.168.8.134) was not reachable >> >> And the same command from riak4( 192.168.8.134) replied back saying riak6 >> not reachable.. So I assume the riak ping on those nodes would >> have returned with pong reply. >> >> To answer Mark's question .. That was a typo in the mail I gave the >> actual ip correctly. >> Thanks, >> Shrikar >> >> >> On Mon, Feb 11, 2013 at 3:45 PM, Mark Phillips <[email protected]> wrote: >> >>> Hi Shrikar, >>> >>> I suspect you've gotten to the bottom of this (as we haven't heard >>> from you in a few days), but upon re-reading this I noticed the >>> following: >>> >>> > >>> > >>> The following nodes are unreachable: ['[email protected]', >>> '[email protected]'] >>> >>> electic@riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134 >>> PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data. >>> ^C >>> --- 92.168.8.134 ping statistics --- >>> 1 packets transmitted, 0 received, 100% packet loss, time 0ms >>> > >>> > >>> >>> It looks like you're trying to ping "92.168.8.134", which is different >>> from the "192.168.8.134" address above. >>> >>> That may not solve the issue, but it might not be a bad place to start. >>> >>> Hope that helps. Let us know if you're still chasing this. >>> >>> Mark >>> >>> >>> >>> >>> On Mon, Feb 4, 2013 at 4:20 PM, Shrikar archak <[email protected]> >>> wrote: >>> > Hi All, >>> > We have a 6 node riak cluster running on riak 1.2.1 >>> > I am often seeing this issue where the nodes become unreachable >>> > when I run the riak-admin command >>> > >>> > electic@riak-6:/var/lib/riak/leveldb$ riak-admin ring-status >>> > Attempting to restart script through sudo -H -u riak >>> > ================================== Claimant >>> > =================================== >>> > Claimant: '[email protected]' >>> > Status: up >>> > Ring Ready: true >>> > >>> > ============================== Ownership Handoff >>> > ============================== >>> > No pending changes. >>> > >>> > ============================== Unreachable Nodes >>> > ============================== >>> > The following nodes are unreachable: ['[email protected]', >>> > '[email protected]'] >>> > >>> > electic@riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134 >>> > PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data. >>> > ^C >>> > --- 92.168.8.134 ping statistics --- >>> > 1 packets transmitted, 0 received, 100% packet loss, time 0ms >>> > >>> > electic@riak-6:/var/lib/riak/leveldb$ ping 192.168.8.134 >>> > PING 192.168.8.134 (192.168.8.134) 56(84) bytes of data. >>> > 64 bytes from 192.168.8.134: icmp_req=1 ttl=64 time=0.062 ms >>> > 64 bytes from 192.168.8.134: icmp_req=2 ttl=64 time=0.057 ms >>> > >>> > electic@riak-6:/var/lib/riak/leveldb$ ping 192.168.8.135 >>> > PING 192.168.8.135 (192.168.8.135) 56(84) bytes of data. >>> > 64 bytes from 192.168.8.135: icmp_req=1 ttl=64 time=0.065 ms >>> > 64 bytes from 192.168.8.135: icmp_req=2 ttl=64 time=0.057 ms >>> > >>> > Has anyone faced similar problems like these? >>> > >>> > During this time the latency increases to 1 second or more than that. >>> > >>> > Please let me know if there is a way to get around this problem. >>> > >>> > Thanks, >>> > Shrikar >>> > >>> > _______________________________________________ >>> > riak-users mailing list >>> > [email protected] >>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> > >>> >> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
