Hi,
During a test I just performed on a small Riak 1.4 cluster setup on Azure,
I started seeing the Riak errors messages listed below after about 10
minutes.
The simple test was performed using lastest Jmeter running on two Azure
instances,
which also each runs haproxy and loadbalances the http/rest requests
between
the 4 Riak nodes.
Each Jmeter node was running 300 threads each. Up to around 10 minutes
I was seeing about 2400 PUT req/s. Then some 23 000+ requests failed with
errors
during a 5 minute period or so, and then the tests continued without errors
until I
stopped it some 5 minutes later.
The 'entries' bucket already contains some 20+ million entries from other
similar tests.
Q1: What could be the cause of these error messages ?
Q2: Am I overwhelming the "small" Riak instances with too many requests/s ?
Q3: And is there any remedy, besides maybe reducing the number of
requests/s ?
Thanks.
Christian
@NorSoulx (twitter)
#------------------------------------------------------------------------------
Azure images used for Riak:
CentOS release 6.3
Riak installation done according to this document:
* http://docs.basho.com/riak/latest/ops/building/installing/azure/
The Azure setup:
4 x Large(4core,7GB) Riak 1.4 cluster.
2 x Medium(2core,3.5GB) for JMeter and HaProxy nodes.
I know these are small instances,
but it's what I could squeeze in on 20 Compute Units (during the free
trial).
I'm testing with LevelDB and 5 x 2i (secondary indexes) in each JSON object.
Riak config is out-of-the-box, except for activating LevelDB backend,
and setting LevelDB cache_size to:
{cache_size, 268435456 } %% 256MB default cache size per-partition
%% (64 PARTITIONS/4 NODES =
16 PER NODE)
Each JSON object is dynamically generated by Jmeter, and looks something
like this:
( key used in this test is "<ts>:<uuid>" )
curl -v
http://localhost:8098/riak/entries/1374747941671:7f3e42b5-3477-434b-a43e-5ae34efac1ea
* About to connect() to localhost port 8098 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8098 (#0)
> GET /riak/entries/1374747941671:7f3e42b5-3477-434b-a43e-5ae34efac1ea
HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8098
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Riak-Vclock: a85hYGBgzGDKBVIc6UvPHwh8v29PBlMiYx4rg0Zl6mm+LAA=
< x-riak-index-ts_int: 1374747941671
< x-riak-index-src_int: 38910224580
< x-riak-index-sno_int: 10010
< x-riak-index-pid_int: 6945
< x-riak-index-pfix_int: 389
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
< Link: </riak/entries>; rel="up"
< Last-Modified: Thu, 25 Jul 2013 10:25:44 GMT
< ETag: "3FNLsWE4uekv2y3OuIIGfa"
< Date: Thu, 25 Jul 2013 11:21:43 GMT
< Content-Type: application/json
< Content-Length: 386
<
* Connection #0 to host localhost left intact
{ "id":"1374747941671:7f3e42b5-3477-434b-a43e-5ae34efac1ea",
"ts":"1374747941671", "src": "38910224580", "sId":"SYSTEM_011",
"cid":"1374747941671:7f3e42b5-3477-434b-a43e-5ae34efac1ea",
"mTyp":"MTYPE_015","pid":"6945","dst":{"sno":"10010","sNum":"4580","dTyp":"DTYPE_020","co":"MK","pFix":"389"},
"bInfo":{"price":39, "cur":"MKD", "iTxt":"INVOICE_011"},
"prop":{"bMod":"BM_MEDIA_001"} }
The error messages:
/var/riak/log/console.log
-------------------------
2013-07-25 10:51:42.500 [error] <0.7736.172> Webmachine error at path
"/riak/entries/1374749501619:d7832329-2596-40f2-a8d0-56c278c5c1d7" :
"Internal Server Error"
2013-07-25 10:51:42.589 [error] <0.708.177> Webmachine error at path
"/riak/entries/1374749501502:9ef09821-ba94-4462-b36e-486d8098a305" :
"Internal Server Error"
2013-07-25 10:51:43.148 [error] <0.21245.174> Webmachine error at path
"/riak/entries/1374749502638:891e855b-018a-4809-975e-e813d034f7a7" :
"Internal Server Error"
2013-07-25 10:51:43.407 [error] <0.4894.167> Webmachine error at path
"/riak/entries/1374749502988:089d2c2e-fcff-4e0e-8c56-891f891393cb" :
"Internal Server Error"
2013-07-25 11:05:09.627 [error] <0.10618.174> Webmachine error at path
"/riak/entries/1374750249401:543cc0b9-1c5a-46b3-8631-47cc6297606e" :
"Service Unavailable"
2013-07-25 11:05:09.696 [error] <0.21230.174> Webmachine error at path
"/riak/entries/1374750249278:ed744534-c528-4c40-9865-4729c2246a0e" :
"Service Unavailable"
2013-07-25 11:05:10.396 [error] <0.19147.174> Webmachine error at path
"/riak/entries/1374750250000:311c493e-797a-422e-9569-d40d1245787d" :
"Service Unavailable"
2013-07-25 11:05:13.326 [error] <0.21076.166> Webmachine error at path
"/riak/entries/1374750250096:3a263a33-43db-4ec2-b16b-ce31af919963" :
"Service Unavailable"
/var/riak/log/error.log
-----------------------
2013-07-25 10:51:42.500 [error] <0.7736.172> Webmachine error at path
"/riak/entries/1374749501619:d7832329-2596-40f2-a8d0-56c278c5c1d7" :
"Internal Server Error"
2013-07-25 10:51:42.589 [error] <0.708.177> Webmachine error at path
"/riak/entries/1374749501502:9ef09821-ba94-4462-b36e-486d8098a305" :
"Internal Server Error"
2013-07-25 10:51:43.148 [error] <0.21245.174> Webmachine error at path
"/riak/entries/1374749502638:891e855b-018a-4809-975e-e813d034f7a7" :
"Internal Server Error"
2013-07-25 10:51:43.407 [error] <0.4894.167> Webmachine error at path
"/riak/entries/1374749502988:089d2c2e-fcff-4e0e-8c56-891f891393cb" :
"Internal Server Error"
/var/riak/log/crash.log
-----------------------
2013-07-25 10:51:42 =ERROR REPORT====
webmachine error:
path="/riak/entries/1374749501619:d7832329-2596-40f2-a8d0-56c278c5c1d7"
"Internal Server Error"
2013-07-25 10:51:42 =ERROR REPORT====
webmachine error:
path="/riak/entries/1374749501502:9ef09821-ba94-4462-b36e-486d8098a305"
"Internal Server Error"
2013-07-25 10:51:43 =ERROR REPORT====
webmachine error:
path="/riak/entries/1374749502638:891e855b-018a-4809-975e-e813d034f7a7"
"Internal Server Error"
2013-07-25 10:51:43 =ERROR REPORT====
webmachine error:
path="/riak/entries/1374749502988:089d2c2e-fcff-4e0e-8c56-891f891393cb"
"Internal Server Error"
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
