On 2 Oct 2014, at 17:59, Igor Senderovich <[email protected]> wrote:
> There are no other errors in any of the logs at exactly the same time but
> there are periodic errors in error.log and console.log of the following form
> (and these occurred seconds before and after the crash):
>
>
> ** Reason for termination =
> **
> {{case_clause,"immediate"},[{riak_kv_vnode,do_delete,3,[{file,"src/riak_kv_vnode.erl"},{line,1321}]},{riak_core_vnode,vnode_command,3,[{file,"src/riak_core_vnode.erl"},{line,299}]},{gen_fsm,handle_m
> sg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
> 2014-10-02 12:07:57 =CRASH REPORT====
> crasher:
> initial call: poolboy:init/1
> pid: <0.30125.18>
> registered_name: []
> exception exit:
> {{{case_clause,"immediate"},[{riak_kv_vnode,do_delete,3,[{file,"src/riak_kv_vnode.erl"},{line,1321}]},{riak_core_vnode,vnode_command,3,
Can I see your config? Looks like you have delete_mode configured with the
string “immediate” rather than the atom ‘immediate’.
Cheers
Russell
>
>
> On Thu, Oct 2, 2014 at 12:20 PM, Dmitri Zagidulin <[email protected]>
> wrote:
> Thanks. Are there entries in any of the other logs? (like the crash dump).
>
> Can you also provide more info on the nodes themselves. What size AWS
> instances are you running? Is the delete timeout happening while load
> testing?
>
> On Thu, Oct 2, 2014 at 12:11 PM, Igor Senderovich
> <[email protected]> wrote:
> Thanks for your help, Dmitri,
>
> I get the following in error.log:
> 2014-10-02 12:05:45.037 [error] <0.6359.19> Webmachine error at path
> "/buckets/imc/keys/5134a18660494ea5553d2c90ef9eea2f" : "Service Unavailable"
>
> And no, there is no load balancer on our cluster.
> Thank you
>
>
> On Thu, Oct 2, 2014 at 11:52 AM, Dmitri Zagidulin <[email protected]>
> wrote:
> One other question - are you using a load balancer for your cluster (like
> HAProxy or the like). In which case, take a look at its logs, also.
>
> On Thu, Oct 2, 2014 at 11:51 AM, Dmitri Zagidulin <[email protected]>
> wrote:
> Igor,
> Can you look in the riak log directory, in the error.log (and console log and
> crash dump file) to see if there's any entries, around the time of the delete
> operation? And post them here?
>
>
>
> On Thu, Oct 2, 2014 at 11:45 AM, Igor Senderovich
> <[email protected]> wrote:
> Hi,
>
> I get a timeout when deleting a key, reproducible in about 1 in 10 times:
> $ curl -i -vvv
> http://myhost:8098/buckets/imc/keys/5134a18660494ea5553d2c90ef9eea2f
>
> * About to connect() to dp1.prod6.ec2.cmg.net port 8098
> * Trying 10.12.239.90... connected
> * Connected to dp1.prod6.ec2.cmg.net (10.12.239.90) port 8098
> > DELETE /buckets/imc/keys/5134a18660494ea5553d2c90ef9eea2f HTTP/1.1
> > User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
> > OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> > Host: dp1.prod6.ec2.cmg.net:8098
> > Accept: */*
> >
> < HTTP/1.1 503 Service Unavailable
> HTTP/1.1 503 Service Unavailable
> < Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
> Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
> < Date: Wed, 01 Oct 2014 16:11:41 GMT
> Date: Wed, 01 Oct 2014 16:11:41 GMT
> < Content-Type: text/plain
> Content-Type: text/plain
> < Content-Length: 18
> Content-Length: 18
>
> request timed out
> * Connection #0 to host dp1.prod6.ec2.cmg.net left intact
> * Closing connection #0
>
>
> This is on Riak 1.4 on a 5 node cluster with an n-value of 3.
> Thank you for your help
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
