What backend are you using? Bitcask (default) or leveldb? What is your N val (3 default)? Do you have search enabled? The only change you mentioned would result in a four fold decrease in virtual nodes per physical host. That would be interesting to see how it plays out. I would imagine there would be less on disk fragmentation with fewer vnodes per machine. Often in the end, the culprit is disk io. There are backend specific tweaks that may be helpful. Riak schedules compaction at different thresholds. Updating a bunch of records in short order may be triggering that. Just some thoughts. I'm sure the Basho folk will be able to offer more specific help.
Cheers, -Alexander Sicular @siculars On Jan 31, 2012, at 3:03 PM, Sarang Deshpande wrote: > Hi, > I am working on setting up a riak cluster as a KV storage for ~150Million > documents with average document size being 4K. We started using 6 nodes (each > node: 32GB, 8 quad cores) in a cluster with partition size of 256 (planning > to take it down to 64 as next experiment). While the lookup is fast enough, > **bulk** updating/loading the documents in this cluster with 6 clients and > 100 connections each is quite slow (takes hours to update ~30Million docs). > Is there a better cluster configuration that we should try to speed up the > process? > > Thanks, > ~Sarang > > > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
