First of all, thank you for your replies, Brian and Christian.
I rewrote the script in python:
#######################
import riak
import time
import sys
hostname = '10.1.1.221'
portnumber = 8087
#portnumber = 8098
if len(sys.argv) > 1:
hostname = sys.argv[1]
#client = riak.RiakClient(host=hostname,port=portnumber)
client =
riak.RiakClient(host=hostname,port=portnumber,transport_class=riak.RiakPbcTransport)
print hostname + ':' + str(portnumber)
# Choose the bucket to store data in.
bucket = client.bucket('nBucket')
data = 'X' * 10240
cnt = 0
last = time.time()
while 1 :
obj = bucket.new(None, data)
obj.store()
cnt += 1
if time.time() - last >= 1:
print cnt
last = time.time()
cnt = 0
#######################
This time I got better numbers just changing from PHP to Python.
Using HTTP in python, I got roughly 96 objs/sec (against 72 objs/sec in
php).
Using Protocol Buffers in python, I got roughly 175 objs/sec (not available
in php).
I also tried to run more than one process in python, with PB:
1 proc = 175 objs/sec
2 proc = 157 objs/sec (each)
3 proc = 124 objs/sec (each)
As we can see, using a new language and Protocol Buffers I can get much
better results. Unfortunately, I think those numbers are still too low
(considering that I'm using the Memory backend). Do you think we can get
better results?
And again, thank you for your support.
On Fri, Nov 2, 2012 at 5:30 PM, Christian Dahlqvist <christ...@whitenode.com
> wrote:
> On 02/11/2012 19:10, Uruka Dark wrote:
>
> Hi Dimitri,
> I don't know why, but I could not receive your reply. I saw it following
> the url of the mailing list.
>
> Any way, thank you for your reply.
>
> This is my PHP script:
>
> *<?php*
> * require_once "riak_lib/riak.php";*
> *
> *
> * if (isset($argv[1]))*
> * {*
> * define('HOST', $argv[1]);*
> * }*
> * else*
> * {*
> * define('HOST', '10.1.1.221');*
> * }*
> * define('PORT', 8098);*
> *
> *
> * $client = new RiakClient(HOST, PORT);*
> * $str = HOST . ":" . PORT;*
> * if ($client->isAlive())*
> * {*
> * echo "$str - ALIVE\n";*
> * }*
> * else*
> * {*
> * echo "$str - DEAD\n";*
> * die();*
> * }*
> *
> *
> * $bucket = $client->bucket('zbucket');*
> * $bucket->setNVal(2);*
> *
> *
> * $cnt = 0;*
> * $last = time();*
> * $data = '';*
> * for ($i=0; $i<10240; $i++)*
> * {*
> * $data .= 'A';*
> * }*
> *
> *
> * // main loop*
> * for ($i=0; true; $i++)*
> * {*
> * $obj = $bucket->newObject(NULL, $data . $i);*
> * $obj->store();*
> * $cnt++;*
> * if (time() - $last == 1)*
> * {*
> * echo "$cnt in $i\n";*
> * $cnt = 0;*
> * $last = time();*
> * }*
> * }*
> *?>*
>
>
> ---------------------------
>
> Uruka,
>
> What does your load testing script look like? If you post the code
> somewhere, I'll take a look to see if any obvious stumbling block comes to
> mind. I agree, that's way lower than you should be getting with Riak on
> that hardware.
>
> Dmitri
>
>
> On Fri, Nov 2, 2012 at 2:30 PM, Uruka Dark <urukad...@gmail.com> wrote:
>
>> As I told, maybe I'm trying to use Riak in an improperly way, but this
>> is a kind of requirement for me and I expected to be able to do it with
>> Riak. I'm looking for a "NoSQL" solution and after try out a few of them, I
>> think that I have two great solutions, Couchbase and Riak. To me, Riak
>> seems to be much more mature, and this is my primary option, but, as I
>> told, I can achieve some numbers with Couchbase that I can't with Riak.
>> Given the maturity of Riak, I'm pretty sure that this is a configuration
>> problem, but I need some help to find out how to solve it. If any one could
>> help me with it, I'll be thankful. I don't want to give up on Riak so fast.
>> Thank you.
>>
>> On Fri, Nov 2, 2012 at 9:46 AM, Uruka Dark <urukad...@gmail.com> wrote:
>>
>>> I understand.
>>> To eliminate any problems related to Bitcask, I changed to Memory
>>> backend and now I can store roughly 80 objs/sec. This speed can be achieved
>>> hitting just one of them.
>>> I tried to hit both of them at same time, and the speed drops to roughly
>>> 68 objs/sec (each).
>>> Do you have any suggestion about it?
>>>
>>> Thank you.
>>>
>>> On Fri, Nov 2, 2012 at 9:20 AM, Sebastian Cohnen <
>>> sebastian.coh...@gmail.com> wrote:
>>>
>>>> You should hit both servers and not just with a concurrency level of
>>>> 1.
>>>>
>>>> There are many more factors to consider, but these are highly
>>>> dependent on your actual problem (not just a simple benchmark). Just to
>>>> name a few: bitcask settings (
>>>> http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/Bitcask/#Tuning-Bitcask),
>>>> w-quorum, HTTP vs ProtoBuf, ring_creation_size, ...
>>>>
>>>> On 02.11.2012, at 13:15, Uruka Dark <urukad...@gmail.com> wrote:
>>>>
>>>> I'm hitting just one of them.
>>>> At the beginning I tried to use the default settings (n_val = 3), then
>>>> I started to create the bucket with n_val = 2. I've tested a lot of
>>>> combinations to w, but I could not see any substantial improvement.
>>>> If you have any suggestion, please, let me know. I can do any test.
>>>>
>>>> On Fri, Nov 2, 2012 at 9:04 AM, Sebastian Cohnen <
>>>> sebastian.coh...@gmail.com> wrote:
>>>>
>>>>> What level of concurrency are you using in your test setup? Are you
>>>>> hitting both servers with your test? What is your n_val and w?
>>>>>
>>>>>
>>>>> On 02.11.2012, at 03:42, Uruka Dark <urukad...@gmail.com> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > I'm new here and with Riak. If I do something wrong, please, let me
>>>>> know.
>>>>> >
>>>>> > I've made a Riak cluster with two identical machines: Intel core i3
>>>>> 2.3GHz 4GB RAM 1TB HD. They are connected by a gigabit ethernet network.
>>>>> Everything is working fine. I'm using a Bitcask backend.
>>>>> >
>>>>> > I've made a PHP script to do a performance test and find out how
>>>>> fast Riak can be with these settings. What my script is doing is: to store
>>>>> 10000 objects with a 10K long data (string of 10240 x 'A'), and
>>>>> calculating
>>>>> how many objects it stores per second.
>>>>> >
>>>>> > Right now, using Bitcask backend, it can store roughly 68 objects
>>>>> per second. It seems to be a small number to me, but I don't know too much
>>>>> about Riak. I've tested the same script on a Couchbase cluster, with the
>>>>> same settings, and it could store roughly 1000 objects per second.
>>>>> Obviously, on Couchbase test, the data is not sent to non-volatile media
>>>>> immediately. Data is kept in memory to acknowledge the reception as fast
>>>>> as
>>>>> possible, and is sent to non-volatile media in background. I want Riak to
>>>>> behave the same way to increase the "writing speed", but I don't know how
>>>>> to do it or if it is possible. May be I'm trying to do something
>>>>> completely
>>>>> out of the purpose of Riak.
>>>>> >
>>>>> > I've tested Riak with Memory backend too, but it achieved only 72
>>>>> objects per second. I expected it to work faster with Memory backend,
>>>>> cause
>>>>> there is no disk activity involved on it, but the final result is not that
>>>>> high.
>>>>> >
>>>>> > Again, I don't know if I'm trying to do something inappropriate. I
>>>>> think I'm missing something.
>>>>> > Is there any way to do it?
>>>>> >
>>>>> > If I could not make myself clear, please, let me know.
>>>>> > Thank you.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > riak-users mailing list
>>>>> > riak-users@lists.basho.com
>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> riak-users mailing
> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> Uruka,
>
> The PHP client currently only supports the HTTP interface (
> http://docs.basho.com/riak/1.2.0/references/Client-Libraries/), which is
> slower than using protocol buffers. I looked through the documentation for
> the PHP client (have not used it myself) trying to see if keep-alive is
> enabled by default, but could not find any information about this nor any
> way to set it on the client. I suppose it therefore might be possible that
> a new connection is established for every request, which would add a lot of
> overhead in your case as you seem to be performing all the inserts using a
> single thread. The numbers you are getting seem to be of the same magnitude
> as I recall getting when performing a large number of curl inserts through
> a simple script (not multithreaded) on a machine of similar spec a couple
> of months ago, which leads me to suspect that keep-alive might very well
> not be enabled.
>
> You should be able to increase the throughput significantly by splitting
> the data and running multiple copies of your script in parallel against
> both nodes at once.
>
> If you can rewrite the test in some other language so that you can take
> advantage of a protocol buffer client, I would expect you to get
> significantly better numbers, especially if using multiple
> processes/threads.
>
> Best regards,
>
> Christian
>
>
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
