Hi Matthew,
Right now we are storing images captured from mobile phones in Riak. So
far it works really well as the image sizes range from a few hundred K
to a few MB. But, as camera resolutions increase, so does the data
size, plus, users are actually interested in the higher resolution
images if they decide to print them.
I am hoping to understand is the largest "chunk" size that a binary
value should have and whether or not leveldb would be better or worse
than bitcask. For example, if it is as Mark suggested where the optimal
size would be 10MB, that would mean that we would be chunking a 50MB
binary object into 5 concurrent writes. For us, this is actually a very
good solution since our platform does not actually operate on any of the
data it is persisting and can achieve the scale necessary.
A little bit of background - we have developed a novel message queue
architecture/platform in Erlang that is optimized for mobile devices
that is currently in private beta (will be opening it up to a public
beta around the end of March). We provide an SDK that allows mobile
developers access to light weight services running on our platform that
gives instant group collaboration functionality ranging from
group/device discovery, group formation and management, to real-time
chat (feature set is actually pretty comprehensive). The SDK allows
developers to drop our service into any existing or new project with
just a few lines of code, which means that the binary data will be
whatever the developer decides to push along the wire, from images to
audio to even video clips.
If you are interested, here is a link to our docs:
http://developer.go-factory.net/
Also note, I understand that the larger the data size, the more
on-the-wire cost within the cluster, for example if the data size is
10MB, with n_val of 3, that is 30MB on the wire for each chunk, and with
5 chunks, that comes out to 150MB. For us that is less of an issue as
we host our own service on dedicated dual ported gigabit hardware
co-located at two major data centers.
Does this help?
Cheers,
Bryan
On 3/6/13 4:14 PM, Matthew Von-Maszewski wrote:
Just curious, what is the typical size and the overall range of sizes
for your image data?
Matthew
On Mar 6, 2013, at 6:08 PM, Bryan Hughes <br...@go-factory.net
<mailto:br...@go-factory.net>> wrote:
Hi Everyone,
I am building a new 5 node cluster with 1.3.0 and am transitioning
from Bitcask to LevelDB (or perhaps a Mulit with LevelDB being the
main) which is all well understood. My question is regarding image
data, and other large binary data: Is one better than the other in
terms of the size of binary data that can be stored, as well as
performance of reads? I recall Mark suggesting to limit binary data
to 10MB.
I am curious where this limitation comes from?
Thanks,
Bryan
--
Bryan Hughes
*Go Factory*
http://www.go-factory.net
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Bryan Hughes
CTO and Founder / *Go Factory*
(415) 515-7916
http://www.go-factory.net
/"Art is never finished, only abandoned. - Leonardo da Vinci"/
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
