50-100 ops/sec on a 5 Node cluster

Hello, Here is my setup of Riak 1.1.4 - 5 Nodes - ami-a29943cb (EC2 AMI) - m1.large (7.5GB Ram, 4 EC2 compute instances) - EBS storage - LevelDB backend - Python Client with Protobuf (on a separate machine) While loading data I am getting around 50 ops/sec. I tried running bas

50-100 ops/sec on a 5 Node cluster

Hello, Here is my setup of Riak 1.1.4 - 5 Nodes - ami-a29943cb (EC2 AMI) - m1.large (7.5GB Ram, 4 EC2 compute instances) - EBS storage - LevelDB backend - Python Client with Protobuf (on a separate machine) While loading data I am getting around 50 ops/sec. I tried running bas

Re: 50-100 ops/sec on a 5 Node cluster

a low number of ops/sec is usually an indicator that you're >> hitting a bottleneck on the I/O path -- typically due to random reads. >> LevelDB has a number of characteristics that can make this worse -- >> what happens if you use bitcask? >> >> D. >>

Re: 50-100 ops/sec on a 5 Node cluster

My client is multi threaded. Instead of haproxy I had ELB in from of my cluster On Tuesday, June 26, 2012, Yousuf Fauzan wrote: > I used ephemeral storage without any improvement. > > Also iotop and iostat did not show any saturation. > > After talking with Sean Cribbs on IRC, I

Throughput issue contd. On Joyend Riak Smartmachine

put was 200 writes/sec Maybe I am making a fundamental mistake somewhere. I tried the above two scripts on EC2 clusters too and still got the same performance. Please, someone help Thanks, Yousuf Fauzan ___ riak-users mailing list riak-users@lists.basho.com h

Re: Throughput issue contd. On Joyend Riak Smartmachine

setting up a cluster which would give such kind of throughput. Thanks, Yousuf On Wed, Jun 27, 2012 at 4:02 PM, Eric Anderson wrote: > On Jun 27, 2012, at 5:13 AM, Yousuf Fauzan wrote: > > Hi, > > I setup a 3 machine riak SM cluster. Each machine used 4GB Ram and riak > OpenSource

Re: Throughput issue contd. On Joyend Riak Smartmachine

I did use basho bench on my clusters. It should throughput of around 150 On Wed, Jun 27, 2012 at 4:24 PM, Russell Brown wrote: > > On 27 Jun 2012, at 11:50, Yousuf Fauzan wrote: > > Its not about the difference in throughput in the two approaches I took. > Rather, the issue i

Re: Throughput issue contd. On Joyend Riak Smartmachine

}]}. On Wed, Jun 27, 2012 at 4:37 PM, Russell Brown wrote: > > On 27 Jun 2012, at 12:05, Yousuf Fauzan wrote: > > I did use basho bench on my clusters. It should throughput of around 150 > > > Could you share the config you used, please? > > > On Wed, Jun 27, 2012 a

Re: Throughput issue contd. On Joyend Riak Smartmachine

So I changed concurrency to 10 and put all the IPs of the nodes in basho bench config. Throughput is now around 1500. On Wed, Jun 27, 2012 at 4:40 PM, Russell Brown wrote: > > On 27 Jun 2012, at 12:09, Yousuf Fauzan wrote: > > I used examples/riakc_pb.config > > {mode, max}.

Re: Throughput issue contd. On Joyend Riak Smartmachine

script? On Wed, Jun 27, 2012 at 5:10 PM, Russell Brown wrote: > > On 27 Jun 2012, at 12:36, Yousuf Fauzan wrote: > > So I changed concurrency to 10 and put all the IPs of the nodes in basho > bench config. > Throughput is now around 1500. > > > I guess you can now try

Re: Throughput issue contd. On Joyend Riak Smartmachine

Oh! I think that may be an issue with my code then. Let me make some changes and get back to you. On Wed, Jun 27, 2012 at 5:25 PM, Reid Draper wrote: > > On Jun 27, 2012, at 7:48 AM, Yousuf Fauzan wrote: > > This is great. > > I was loading data using Python. My code would sp

Re: Throughput issue contd. On Joyend Riak Smartmachine

after this, I am getting <100 writes/sec On Wed, Jun 27, 2012 at 5:35 PM, Yousuf Fauzan wrote: > Oh! I think that may be an issue with my code then. > > Let me make some changes and get back to you. > > > On Wed, Jun 27, 2012 at 5:25 PM, Reid Draper wrote: > >>

Re: Throughput issue contd. On Joyend Riak Smartmachine

I get this error when I increase the number of threads accessing a single connection. Something related to getting unexpected number of bytes on the connection. On Wednesday, June 27, 2012, Reid Draper wrote: > > On Jun 27, 2012, at 8:41 AM, Yousuf Fauzan wrote: > > So I created

Re: 50-100 ops/sec on a 5 Node cluster

for loading data (I am using small instance of EC2) Thanks all for your help. -- Yousuf Fauzan http://fauzism.com On Fri, Jun 29, 2012 at 10:25 PM, Mark Rose wrote: > There are other reasons to avoid using an ELB for this purpose I'd like to > add. > > ELB works by resolving a

Re: 50-100 ops/sec on a 5 Node cluster

for loading data (I am using small instance of EC2) Thanks all for your help. -- Yousuf Fauzan http://fauzism.com On Fri, Jun 29, 2012 at 10:25 PM, Mark Rose wrote: > There are other reasons to avoid using an ELB for this purpose I'd like to > add. > > ELB works by resolving a

Large buckets with Secondary Index

Hello, Record size ~ 600 bytes, Indexed on 3 fields For a new bucket, I am getting around 1-1.5K writes/second. However, when the bucket size gets large (15 Million records in my case) then the write speed drops 5-6 times. Is this an expected behavior or am I doing something wrong? -- Yousuf

Re: Large buckets with Secondary Index

pump data into a brand new bucket then I get better write performance. However, the write speed continues to be low on the older large bucket. -- Yousuf Fauzan http://fauzism.com On Sun, Jul 1, 2012 at 4:42 AM, Yousuf Fauzan wrote: > Hello, > > Record size ~ 600 bytes, Indexed on 3 field

Re: args format for map function --> not accepting json format

I am not sure what you are doing here, but could you try adding 3 \ instead of 1 \? On Tue, Jul 3, 2012 at 4:41 PM, Venki Yedidha wrote: > Hi All, >Can anyone help me how I can send the json data as an args > parameter to the map function... >My data is in this format: > >

Re: group by country

Group By will require a reduce function which does the following Input: [{Country1:[obj1, obj2..], Country2:[[obj3, obj4]}, {Country1:[[obj5, obj6], Country3:[[obj7, obj8]} Output: [{Country1:[obj1, obj2,obj5, obj6], Country2:[[obj3, obj4], Country3:[[obj7, obj8]}] -- Yousuf On Mon, Jul 9, 2012

Re: check if a bucket exists

You can list buckets. Riak http apI has method to do that. Check the documentation On Jul 9, 2012 6:37 PM, "Anand Hegde" wrote: > I am using the riak gem in ruby, is there any way i can check if a bucket > exists? (without listing all the keys of the bucket?) > > Regards, > Anand Hegde > ianand.i

Re: return[{value.key : data}] problem

What the reduce function that you are using? On Wed, Jul 11, 2012 at 4:02 PM, Venki Yedidha wrote: > Hi All, > >I am stuck with Riak group by using json object.. > > when I am returning some thing like this: > > return [value.key] or return[value.bucket].. > > It is going to reduce func

Re: return[{value.key : data}] problem

this. -- Yousuf On Wed, Jul 11, 2012 at 4:12 PM, Yousuf Fauzan wrote: > What the reduce function that you are using? > > On Wed, Jul 11, 2012 at 4:02 PM, Venki Yedidha < > venkatesh.yedi...@gmail.com> wrote: > >> Hi All, >> >>I am stuck with Riak grou

Re: How to store data

012 at 6:35 PM, Andrew Kondratovich < andrew.kondratov...@gmail.com> wrote: > Hello, Yousuf. > > Thanks for your reply. > > We have several millions of items. It's about 10 000 of unique 'from' > fields (about 1000 items for each). Usually, we need to get i

Re: How to store data

able >> frequency. >> >> (not that I know what Andrew is looking for, exactly) >> >> >> 2012/7/25 Yousuf Fauzan >> >>> Since 500 is not that big a number, I think you can run that many M/Rs >>> with each emitting only records having "

Re: Large buckets with Secondary Index

Unfortunately No. I am still using 1.1. However, once I push my current implementation out, I will look into 1.2. Will update you guys then. -- Yousuf http://fauzism.com On Tue, Jul 31, 2012 at 12:25 PM, Matthew Tovbin wrote: > Yousuf, > > Thanks for the update! Did you try to reproduce with

MR Timeout

,{riak_kv_js_manager,blocking_dispatch,4},{riak_kv_mrc_map,map_js,3},{riak_kv_mrc_map,process,3},{riak_pipe_vnode_worker,process_input,3},{riak_pipe_vnode_worker,wait_for_input,2},{gen_fsm,handle_msg,7},{proc_lib,init_p_do_apply,3}]\"}">> However, if I re-run the query a couple of

Re: Large buckets with Secondary Index

then storing it. Is there a way to append link to a key in a bucket without fetching it? If not, is it possible to add such a feature? -- Yousuf http://fauzism.com On Thu, Aug 2, 2012 at 9:06 PM, Yousuf Fauzan wrote: > Unfortunately No. > > I am still using 1.1. > > However,

Postcommit hook and creating a new object

Hello, I have written a post commit hook on one of my buckets such that every time data is stored, it creates a new object in another bucket and stores it. My problem is that I do not wish to overwrite existing object in this other bucket. The way I am achieving this right now is pretty crude, I

Postcommit hook and storing json object

Hello, In a postcommit hook, I am storing an object in another bucket where data is equal to Data = iolist_to_binary(mochijson2:encode({struct, [ {<<"status">>, list_to_binary(Status)}, {<<"recorddate">>, list_to_binary(Recorddate)}, {<<"memberlo

Reip in Riak 1.2 vs 1.1.4

Hello, My riak setup is on EC2 with EBS volumes storing data and ring state. This ensures that I can teardown the cluster and restart it with all the data intact. When I restart the cluster, I do a reip of all nodes on all nodes. This used to work with Riak 1.1.4 but ever since I shifted to 1.2,