Related to the recent thread about "Lowlevel Access to RiakCS objects" I plan to implement an extension to a Riak Object (in the Golang driver at https://github.com/tpjg/goriakpbc) that will cover two use cases: 1) storing larger files directly in Riak (not Riak CS) 2) store growing files, e.g. log files, efficiently in Riak
I have some questions for the Riak community. First: am I about to re-invent a wheel and if so can someone please point me to an example implementation? If not I would like to know if there is more interest and maybe more use cases so we can develop a standard way to store these objects in Riak, allowing access from multiple programming languages. If there is, please read the proposal below and provide feedback. Store the meta-information about a “BigObject” in a regular Riak value. This object will be stored in <bucket> , <key>. The object will have the following meta tags: - segment_size - size of each segment in bytes - segment_count - the number of segments of the object - total_size - optional total size of the entire object (see below). The objects data would be stored in segments in <bucket>, <key-%06d>. The Content-Type would be the same for all objects. Depending on the use case total_size could be filled. Using the two use cases above as an example: 1) storing large files, e.g. video: use a large segment_size, e.g. 1Mb and store the total_size since the file will be static and the meta-information can be written at once 2) store growing files, e.g. daily log files: use a smaller segment_size, e.g. 10Kb-100Kb, do not store total_size as otherwise each “Append” operation would require two PUTs. If a segment grows beyond the segment_size update the meta-information K/V, otherwise only PUT the last segment again. In the extension to a client driver a BigObject would store some state information that keeps track of the meta-information K/V and the last segment to make the Append operation somewhat efficient - note though that each append is overwriting the last segment. Any input is highly appreciated. Kind regards, Timo
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
