Bryan, My job is to test and optimize leveldb. I do not have a test case with your size objects. Hence I asked the specifics.
I do not have any data at this time to help with your question. But I will add your parameters to my standard testing and see what I can find / optimize. I am not currently performing comparison work against bitcask and must leave those opinions to others. Matthew On Mar 6, 2013, at 8:15 PM, Bryan Hughes <br...@go-factory.net> wrote: > Hi Matthew, > > Right now we are storing images captured from mobile phones in Riak. So far > it works really well as the image sizes range from a few hundred K to a few > MB. But, as camera resolutions increase, so does the data size, plus, users > are actually interested in the higher resolution images if they decide to > print them. > > I am hoping to understand is the largest "chunk" size that a binary value > should have and whether or not leveldb would be better or worse than bitcask. > For example, if it is as Mark suggested where the optimal size would be > 10MB, that would mean that we would be chunking a 50MB binary object into 5 > concurrent writes. For us, this is actually a very good solution since our > platform does not actually operate on any of the data it is persisting and > can achieve the scale necessary. > > A little bit of background - we have developed a novel message queue > architecture/platform in Erlang that is optimized for mobile devices that is > currently in private beta (will be opening it up to a public beta around the > end of March). We provide an SDK that allows mobile developers access to > light weight services running on our platform that gives instant group > collaboration functionality ranging from group/device discovery, group > formation and management, to real-time chat (feature set is actually pretty > comprehensive). The SDK allows developers to drop our service into any > existing or new project with just a few lines of code, which means that the > binary data will be whatever the developer decides to push along the wire, > from images to audio to even video clips. > > If you are interested, here is a link to our docs: > http://developer.go-factory.net/ > > Also note, I understand that the larger the data size, the more on-the-wire > cost within the cluster, for example if the data size is 10MB, with n_val > of 3, that is 30MB on the wire for each chunk, and with 5 chunks, that comes > out to 150MB. For us that is less of an issue as we host our own service on > dedicated dual ported gigabit hardware co-located at two major data centers. > > Does this help? > > Cheers, > Bryan > > > On 3/6/13 4:14 PM, Matthew Von-Maszewski wrote: >> Just curious, what is the typical size and the overall range of sizes for >> your image data? >> >> Matthew >> >> On Mar 6, 2013, at 6:08 PM, Bryan Hughes <br...@go-factory.net> wrote: >> >>> Hi Everyone, >>> >>> I am building a new 5 node cluster with 1.3.0 and am transitioning from >>> Bitcask to LevelDB (or perhaps a Mulit with LevelDB being the main) which >>> is all well understood. My question is regarding image data, and other >>> large binary data: Is one better than the other in terms of the size of >>> binary data that can be stored, as well as performance of reads? I recall >>> Mark suggesting to limit binary data to 10MB. >>> >>> I am curious where this limitation comes from? >>> >>> Thanks, >>> Bryan >>> >>> -- >>> Bryan Hughes >>> Go Factory >>> http://www.go-factory.net >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > -- > Bryan Hughes > CTO and Founder / Go Factory > (415) 515-7916 > http://www.go-factory.net > > "Art is never finished, only abandoned. - Leonardo da Vinci" >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
