Sad to say your LOG files suggest the same bug as seen elsewhere and fixed by 
recent changes in the leveldb code.

The tougher issue is that the fixes are currently only available for our 2.0 
product series.  A backport would be non-trivial due to the number of places 
changed between 1.4 and 2.0 and the number of places the fix overlaps those 
changes.  The corrected code is tagged “2.0.9” in eleveldb and leveldb.

The only path readily available to you is to have your receiving cluster 
upgraded to 2.0 Riak CS and manually build/patch eleveldb to the 2.0.9 version. 
Then start your handoffs.   (eleveldb version 2.0.9 is not present in any 
shipping version of Riak … yet). 

I will write again if I can think of an easier solution.  But nothing is 
occurring to me or the team members I have queried.

Matthew

> On Oct 29, 2015, at 12:14 PM, Vladyslav Zakhozhai 
> <v.zakhoz...@smartweb.com.ua> wrote:
> 
> Hi,
> 
> Matthew thank for you answer. eleveldb LOGs are attached.
> Here is LOGs from 2 eleveldb nodes (eggeater was not restarted; what about 
> rattlesnake I'm not sure).
> 
> On Thu, Oct 29, 2015 at 5:24 PM Matthew Von-Maszewski <matth...@basho.com 
> <mailto:matth...@basho.com>> wrote:
> Hi,
> 
> There was a known eleveldb bug with handoff receiving that could cause a 
> timeout.  But it does not sound like bug fits your symptoms.  However, I am 
> willing to verify my diagnosis.  I would need you to gather the LOG files 
> from all vnodes on the RECEIVING side (or at least from the vnode that you 
> are attempting and is failing).
> 
> I will check it for the symptoms of the known bug.
> 
> Note:  the LOG files reset on each restart of Riak.  So you must gather the 
> LOG files right after the failure without restarting Riak.
> 
> Matthew
> 
> 
>> On Oct 29, 2015, at 11:11 AM, Vladyslav Zakhozhai 
>> <v.zakhoz...@smartweb.com.ua <mailto:v.zakhoz...@smartweb.com.ua>> wrote:
>> 
>> Hi,
>> 
>> I want to make small update. Jon your hint about problems on sender side is 
>> correct. As I've already told there problems with available resources on 
>> sender nodes. There are no enough available RAM which is a cause of 
>> swapiness and load on disks. Restarting of sender nodes helps me (at least 
>> temoprarily).
>> 
>> 
>> On Thu, Oct 29, 2015 at 12:19 PM Vladyslav Zakhozhai 
>> <v.zakhoz...@smartweb.com.ua <mailto:v.zakhoz...@smartweb.com.ua>> wrote:
>> Hi,
>> 
>> Average size of objects in Riak - 300 Kb. This objects are images. This data 
>> updates very very rearly (there almost no updates).
>> 
>> I have GC turned on and works:
>> root@python:~# riak-cs-gc status
>> There is no garbage collection in progress
>>   The current garbage collection interval is: 900
>>   The current garbage collection leeway time is: 86400
>>   Last run started at: 20151029T100600Z
>>   Next run scheduled for: 20151029T102100Z
>> 
>> Network misconfigurations were not detected. The result of your script shows 
>> correct info.
>> 
>> But I see that almost all nodes with bitcask suffers from low free memory 
>> and they swapped. I think that it can be an issue. But my question is, what 
>> workaround is for this problem.
>> 
>> I've wrote in my first post that I tuned handoff_timeout and 
>> handoff_receive_timeout (now this vaules are 300000 and 600000). But 
>> situation is the same.
>> 
>> 
>> On Tue, Oct 27, 2015 at 4:06 PM Jon Meredith <jmered...@basho.com 
>> <mailto:jmered...@basho.com>> wrote:
>> Hi,
>> 
>> Handoff problems without obvious disk issues can be due to the database 
>> containing large objects.  Do you frequently update objects in CS, and if so 
>> have you had garbage collection running?
>> 
>> The timeout is happening on the receiver side after not receiving any tcp 
>> data for handoff_receive_timeout *milli*seconds.  I know you said you 
>> increased it, but not how high.  I would bump that up to 300000 to give the 
>> sender a chance to read larger objects off disk.
>> 
>> To check if the sender is transmitting, on the source node you could run
>>   redbug:start("riak_core_handoff_sender:visit_item", [{arity, 
>> true},{print_file,"/tmp/visit_item.log"},{time, 3600000},{msgs, 1000000}]).
>> 
>> That file should fill fairly fast with an entry for every object the sender 
>> tries to transmit.
>> 
>> There's a long shot it could be network misconfiguration. Run this from the 
>> source node having problems 
>> 
>> rpc:multicall(erlang, apply, [fun() -> TargetNode = node(), [_Name,Host] = 
>> string:tokens(atom_to_list(TargetNode), "@"), {ok, Port} = 
>> riak_core_gen_server:call({riak_core_handoff_listener, TargetNode}, 
>> handoff_port), HandoffIP = riak_core_handoff_listener:get_handoff_ip(), 
>> TNHandoffIP = case HandoffIP of error -> Host; {ok, "0.0.0.0"} -> Host; {ok, 
>> Other} -> Other end, {node(), HandoffIP, TNHandoffIP, 
>> inet:gethostbyname(TNHandoffIP), Port} end, []]).
>> 
>> and it will print out a a list of remote nodes and IP addresses (and 
>> hopefully an empty list of failed nodes)
>> 
>> {[{'dev1@127.0.0.1 <mailto:dev1@127.0.0.1>',          <---- node name
>>   {ok,"0.0.0.0"},             <---- handoff ip address configured in 
>> app.config
>>   "127.0.0.1",                <---- hostname passed to socket open
>>   {ok,{hostent,"127.0.0.1",[],inet,4,[{127,0,0,1}]}}, <--- DNS entry for 
>> hostname
>>   10019}],                    <---- handoff port
>>  []} <--- empty list of errors
>> 
>> Good luck, Jon.
>> 
>> On Tue, Oct 27, 2015 at 3:55 AM Vladyslav Zakhozhai 
>> <v.zakhoz...@smartweb.com.ua <mailto:v.zakhoz...@smartweb.com.ua>> wrote:
>> Hi,
>> 
>> Jon thank you for the answer. During approval of my mail to this list I've 
>> troubleshoot my issue more deep. And yes, your are right. Neither {error, 
>> enotconn} nor max_concurrency is my problem.
>> 
>> I'm going to migrate my cluster entierly to eleveldb only, i.e. I need to 
>> refuse using bitcask. I have a talk with basho support and they said that it 
>> is tricky to tune bitcask on servers with 32 GB RAM (and I guess that it is 
>> not tricky, but it is impossible, because bitcask loads all keys in memory 
>> regardless of free available RAM). With LevelDB I have opportunity to tune 
>> using RAM on servers.
>> 
>> So I have 15 nodes with multibackend (bitcask for data and leveldb for 
>> metadata). 2 additional servers are without multibackend - only with 
>> leveldb. Now I'm not sure do I need still use mutibackend with levedb-only 
>> backend.
>> 
>> And my problem is (as I mentioned earlier) the following. On leveldb-only 
>> nodes I see handoffs timedout and no further progress.
>> 
>> On multibackend hosts I have configuration:
>> 
>> {riak_kv, [
>>        {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-1.5.0/ebin"]},
>>        {storage_backend, riak_cs_kv_multi_backend},
>>        {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>>        {multi_backend_default, be_default},
>>        {multi_backend, [
>>            {be_default, riak_kv_eleveldb_backend, [
>>                {max_open_files, 50},
>>                {data_root, "/var/lib/riak/leveldb"}
>>            ]},
>>            {be_blocks, riak_kv_bitcask_backend, [
>>                {data_root, "/var/lib/riak/bitcask"}
>>            ]}
>>        ]},
>> 
>> And for hosts with leveldb-only backend:
>> 
>> {riak_kv, [
>>             {storage_backend, riak_kv_eleveldb_backend},
>> ...
>> {eleveldb, [                                    
>>             {data_root, "/var/lib/riak/leveldb"}
>> (default values for leveldb)
>> 
>> In leveldb logs I see nothing that could help me (no errors in logs).
>> 
>> 
>> On Mon, Oct 26, 2015 at 3:57 PM Jon Meredith <jmered...@basho.com 
>> <mailto:jmered...@basho.com>> wrote:
>> Hi,
>> 
>> I suspect your {error,enotconn} messages are unrelated - that's likely to be 
>> caused by an HTTP client closing the connection while Riak looks up  some 
>> networking information about the requestor.
>> 
>> The max_concurrency message you are seeing is related to the handoff 
>> transfer limit - it should be labelled as informational. When a node has 
>> data to handoff it starts the handoff sender process and if there are either 
>> too many local handoff processes or too many on the remote side it exits 
>> with max_concurrency.  You could increase with riak-admin transfer-limit but 
>> that probably won't help if you're timing out.
>> 
>> As you're using the multi-backend you're transferring data from bitcask and 
>> leveldb.  The next place I would look is in the leveldb LOG files to see if 
>> there are any leveldb vnodes that are having problems that's preventing 
>> repair.
>> 
>> Jon
>> 
>> On Mon, Oct 26, 2015 at 7:15 AM Vladyslav Zakhozhai 
>> <v.zakhoz...@smartweb.com.ua <mailto:v.zakhoz...@smartweb.com.ua>> wrote:
>> Hello,
>> 
>> I have a problem with persistent timeouts during ownership handoffs. I've 
>> tried to surf over Internet and current mail list but no success.
>> 
>> I have Riak 1.4.12 cluster with 17 nodes. Almost all nodes use multibackend 
>> with bitcask and eleveldb as storage backends (we need multiple backend for 
>> Riak CS 1.5.0 integration).
>> 
>> Now I'm working to migrate Riak cluster to eleveldb as primary and only 
>> backend. For now I have 2 nodes with eleveldb backend in the same cluster.
>> 
>> During ownership handoff process I permanently see errors of timed out 
>> handoff receivers and sender.
>> 
>> Here is partial output of riak-admin transfers:
>> ...
>> transfer type: ownership_transfer
>> vnode type: riak_kv_vnode
>> partition: 331121464707782692405522344912282871640797216768
>> started: 2015-10-21 08:32:55 [46.66 min ago]
>> last update: no updates seen
>> total size: unknown
>> objects transferred: unknown
>> 
>>                            unknown                            
>> riak@taipan.pleiad.uaprom <mailto:riak@taipan.pleiad.uaprom>  =======>  
>> r...@eggeater.pleiad.uapr <mailto:r...@eggeater.pleiad.uapr>
>>                                      om                       
>>         |                                           |   0%    
>>                            unknown                            
>> 
>> transfer type: ownership_transfer
>> vnode type: riak_kv_vnode
>> partition: 336830455478606531929755488790080852186328203264
>> started: 2015-10-21 08:32:54 [46.68 min ago]
>> last update: no updates seen
>> total size: unknown
>> objects transferred: unknown
>> ...
>> 
>> Some of partition handoffs state never updates, some of them terminates 
>> after partial handoff objects and never starts again.
>> 
>> I see nothing in logs but following:
>> 
>> On receiver side:
>> 
>> 2015-10-21 11:33:55.131 [error] 
>> <0.25390.1266>@riak_core_handoff_receiver:handle_info:105 Handoff receiver 
>> for partition 331121464707782692405522344912282871640797216768 timed out 
>> after processing 0 objects.
>> 
>> On sender side:
>> 
>> 2015-10-21 11:01:58.879 [error] <0.13177.1401> CRASH REPORT Process 
>> <0.13177.1401> with 0 neighbours crashed with reason: no function clause 
>> matching webmachine_request:peer_from_peername({error,enotconn}, 
>> {webmachine_request,{wm_reqstate,#Port<0.50978116>,[],undefined,undefined,undefined,{wm_reqdata,...},...}})
>>  line 150
>> 2015-10-21 11:32:50.055 [error] <0.207.0> Supervisor 
>> riak_core_handoff_sender_sup had child riak_core_handoff_sender started with 
>> {riak_core_handoff_sender,start_link,undefined} at <0.22312.1090> exit with 
>> reason max_concurrency in context child_terminated
>> 
>> {error, enotconn} - seems to be network issue. But I have no any problems 
>> with network. All hosts resolve their neighbors correctly and /etc/hosts on 
>> each node are correct.
>> 
>> I've tried to increase handoff_timeout and handoff_receive_timeout. But no 
>> success.
>> 
>> Forcing handoff helped me but for short period of time:
>> 
>> rpc:multicall([node() | nodes()], riak_core_vnode_manager, force_handoffs, 
>> []).
>> 
>> I see progress of handoffs (riak-admin transfers) but then I see handoff 
>> timed out again.
>> 
>> A week ago I've joined 4 nodes with bitcask. And there was no such problems.
>> 
>> I'm confused a little bit and need to understand my next steps in 
>> troubleshooting this issue.
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
> 
> <eggeater-leveldb-logs-old.tar.gz><rattlesnake-leveldb-logs-old.tar.gz><rattlesnake-leveldb-logs.tar.gz><eggeater-leveldb-logs.tar.gz>

_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Reply via email to