Damien and Dmitri – Thanks for the guidance. This really helps me out. I see from the link Dmitri provided that a PB fetch object call does indeed support a request for returning only metadata. Unfortunately, I do not think this functionality is exposed in the Riak Python API RiakBucket.get method, according to the docs at http://basho.github.io/riak-python-client/bucket.html#bucket-objects . I looked in the Python Riak client source code in GitHub (bucket.py), and also did not see a HEAD implementation in ./transports/http/transport.py. Am I looking in the right place? If it were implemented, it would be a function of the ‘get' method on the bucket object, right?
I did test the http HEAD method (albeit with curl’s flawed –X HEAD implementation) and that did work. I can live with internal cluster bandwidth being used for a metadata request. Simply add more nodes, right? :) I just don’t want to send large objects back down to the client that aren’t being used. I understand the strategy of storing metadata for a given key in a different bucket using the same key. I’m trying to avoid turning my key-value store into a key-key-value-value store. There is an elegance to storing both the data and the metadata at the same time and in the same place via the same operation, so that is the preferred direction. From: "Damien Krotkine" <[email protected]> To: "Dmitri Zagidulin" <[email protected]> Cc: "Joe Olson" <[email protected]>, "riak-users" <[email protected]> Sent: Tuesday, December 8, 2015 12:35:56 AM Subject: Re: Two quick questions about X-RIak-Meta-* headers.... Hi Joe, First of all, what Dmitri says makes a lot of sense. From what I understand, you are trying to avoid wasting network bandwidth by transferring data where you only need the metadata of your keys. As Dmitri pointed out, if your replication factor is 3 (default), then Riak will internally query all the replica full *data* and metadata, then only return the metadata to the client. From a network perspective, you'll save network bandwidth *outside* of your cluster, but you'll use quite a lot of network bandwidth *inside* of your cluster. Maybe that's not an issue. But if it is, read on: 1/ first solution : as Dmitri suggested, the best approach is to decouple your metadata and data, if possible. If your key is "my_stuff", then have your data stored in the bucket "data" and the metadata stored in the bucket "metadata". So that you'd fetch "metadata/my_stuff", then fetch "data/my_stuff". This should make your life way easier, but the issue is that you loose the strong relationship between data and metadata. To try to mitigate this: - when writing your data, start by writing "data/my_stuff" with conservative parameters ( w=3 for instance ) then wait for the successful write before storing the metadata. So that when the metadata is there, there is a very high chance that the data is there as well. - when updating your data: ry to be smart, like marking the metadata as invalid or unavailable, while you change the data underneath then update the metadata - be fault tolerant on your application: if you fetch some metadata, but the data is missing, retry, or wait, or gracefully fallback. - be fault tolerant again: when fetching some data, have it contain a header or an id that must match with the metadata. If it doesn't match, you need to wait/retry/fallback - if you don't want/can't handle that on the client side, it's possible to enrich the Riak API and have Riak do the bookeeping itself. If you're interested, let me know. 2/ second solution: you don't want / can't separate metadata and data. In this case, you can try to reduce the network usage - first, reduce the internal network usage inside the cluster. When querying the metadata, if you're using the PB API, you can pass "n_val" as parameter of your request. If you pass n_val=1, then Riak will not query the 3 replicas to fetch the value, instead it'll fetch only one replica, saving a lot of internal bandwidth. - second, you can have your client query one of the primary node (where one of the replica is) for a given key. Coupled with passing n_val=1, Riak will not transfer anything on the internal network. you can check out https://github.com/dams/riak_preflists to find the primary nodes. Dmitri Zagidulin wrote: Hi Joe, 1. Yes, it's possible (with the HTTP HEAD request, or the client library equivalent (I'm pretty sure all the client libraries expose the 'return only the headers' part of the object fetch -- see the Optional Parameters head=true section of the PB API http://docs.basho.com/riak/latest/dev/references/protocol-buffers/fetch-object/ )). However, this is not going to be very helpful in your case. Behind the scenes, a HEAD request still requests all replicas of an object -- *full* replicas, including the value. It's just that the node coordinating the request drops the actual object value before returning the metadata/headers to the client. So, if you use this 'just give me the metadata' request, you're only saving on the cost of shipping the object value down the wire from the cluster to the client. But you're still incurring the cost of all 3 copies of the object (so, 3-4MB, in your case) being transferred over the network between the nodes, as a result of that HEAD request. 2. I don't know that there is a specific size limit on the object header values. However, this question is definitely a red flag -- it's very likely that you're trying to use the custom headers in a way that they weren't intended for. Can you describe in more detail what's your use case? (As in, what are you trying to store as metadata, and why would retrieving just the headers be useful to you.) Don't forget, that if you need to store a LOT of metadata on an object, and you can't do it within the object value itself (for example, when storing binary images), you can simply store a separate metadata object, in a different bucket, using the same key as the object. For example, if I'm storing my primary objects in the 'blobs' bucket, I can also store a JSON object with corresponding metadata in a 'blobs-meta' object, like so: /buckets/blobs/keys/blob123 --> binary object value /buckets/blobs-meta/keys/blob123 --> json metadata object The downsides of this setup is that you're now doing 2 writes for each object (one to the blobs bucket, and one to the meta bucket). But the benefits are considerable: - You can store arbitrarily large metadata object (you're not abusing object headers by stuffing large values into them) - Since the metadata object is likely to be much smaller than the object it's referring to, you can use the metadata object to check for an object's existence (or to get the actual headers that you care about) without the cost of requesting the full giant blob. Dmitri On Mon, Nov 30, 2015 at 12:18 PM, Joe Olson < [email protected] > wrote: BQ_BEGIN Two quick questions about X-RIak-Meta-* headers: 1. Is it possible to pull the headers for a key without pulling the key itself? The reason I am interested in this is because the values for our keys are in the 1.2-1.6 MB range, so the headers are a lot smaller in comparison. I know I can index the headers using Solr or 2i, but I am trying to involve the overhead of doing that. 2. What are the size limits on the headers values that are strings? As always, thanks in advance! _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com BQ_END
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
