Re: how to determine riak version
Thanks for the confirmation. -DB On 4/22/16 12:06 PM, Douglas Rohrer wrote: David: riak_kv is actually a (semi)independent library we incorporate (along with many others) into Riak, the product. There are times where riak_kv, the library, is actually not versioned when we release a new Riak, so those two version numbers are not necessarily going to always be the same. Doug Rohrer On Apr 22, 2016, at 2:28 PM, David Byron <dby...@dbyron.com> wrote: I'm pondering upgrading riak from 2.1.3 to 2.1.4 and got myself confused confirming that I really am running 2.1.3 at the moment. I installed riak from here: https://packagecloud.io/basho/riak/packages/ubuntu/trusty/riak_2.1.3-1_amd64.deb. and all of this looks promising: $ riak version 2.1.3 $ ls /usr/lib/riak/releases/ 2.1.3 RELEASES start_erl.data $ dpkg -l | grep riak ii riak 2.1.3-1 amd64Riak is a distributed data store but then there's also this: $ sudo riak-admin status | grep riak_kv_version riak_kv_version : <<"2.1.2-0-gf969bba">> I really wanted riak_kv_version to say 2.1.3-. I'm clearly paranoid, but can someone help me feel better about this? Thanks much. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
how to determine riak version
I'm pondering upgrading riak from 2.1.3 to 2.1.4 and got myself confused confirming that I really am running 2.1.3 at the moment. I installed riak from here: https://packagecloud.io/basho/riak/packages/ubuntu/trusty/riak_2.1.3-1_amd64.deb. and all of this looks promising: $ riak version 2.1.3 $ ls /usr/lib/riak/releases/ 2.1.3 RELEASES start_erl.data $ dpkg -l | grep riak ii riak 2.1.3-1 amd64Riak is a distributed data store but then there's also this: $ sudo riak-admin status | grep riak_kv_version riak_kv_version : <<"2.1.2-0-gf969bba">> I really wanted riak_kv_version to say 2.1.3-. I'm clearly paranoid, but can someone help me feel better about this? Thanks much. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: epmd listener port
Thanks to you both for the good info. I'd still love to hear from some basho folks about good ways to document this. -DB On 2/17/16 4:03 AM, Damien Krotkine wrote: There is an easier way to do it, by using the advanced config file (usually /etc/riak/advanced.config ) Here is an example, where I use advanced.config to add a vm_args configuration fields. So I suppose you could use the same syntax to add your -env option: [ {vm_args, [ {'-s my_app', ""}, {'-s my_other_app', ""} ]} ]. Note to Basho people: this is not very well documented (if at all), and I've seen many people ask about it (be it for vm options, or adding paths, etc). It might be useful to ad some documentation section about it. Cheers, dams Christopher Meiklejohn wrote: On Tue, Feb 16, 2016 at 3:24 PM, David Byron<dby...@dbyron.com> wrote: http://docs.basho.com/riak/latest/community/faqs/operations/#is-it-possible-to-change-the-port-on-which-the-erl says to add -env ERL_EPMD_PORT to vm.args, but I get the sense from http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ that the goal is to avoid vm.args in preference to riak.conf. My struggle is that I don't see a way to set the epmd port listener via riak.conf. Is there a way? If not, are the new/latest docs missing a mention of the correct location for vm.args? riak.conf is generated by the cuttlefish extension, which apparently does not have control for environment variables [1], so, I don't believe there is a way to do this out of the box. However, you could modify the schema files to add this cuttlefish transformation [2], if you want to, or you could call the Riak initialization commands with the EPMD environment variable set. [1] https://github.com/basho/cuttlefish/issues/135 [2] https://github.com/basho/riak/blob/develop/rel/files/riak.schema ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
epmd listener port
http://docs.basho.com/riak/latest/community/faqs/operations/#is-it-possible-to-change-the-port-on-which-the-erl says to add -env ERL_EPMD_PORT to vm.args, but I get the sense from http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/ that the goal is to avoid vm.args in preference to riak.conf. My struggle is that I don't see a way to set the epmd port listener via riak.conf. Is there a way? If not, are the new/latest docs missing a mention of the correct location for vm.args? Thanks for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: accessor for FetchDatatype's location?
Finally got around to this: https://github.com/basho/riak-java-client/pull/590 Thanks for the help. -DB On 1/20/16 7:58 AM, Alex Moore wrote: Hey David, We include/shade that jar in the final riak-client jar so we stopped shipping it separately. You'll want to clone riak_pb, and build this tag to install it locally for dev: https://github.com/basho/riak_pb/tree/java-2.1.1.0 You'll need protocol buffers 2.5.0 <https://github.com/google/protobuf/releases/tag/v2.5.0> to build, and you should be able to build/install the riak_pb lib with `mvn install`. If you use homebrew you can use this <http://stackoverflow.com/a/23760535> procedure to install the older protobuf lib (just swap protobuf241 for protobuf250). Thanks, Alex On Thu, Jan 14, 2016 at 2:22 PM, David Byron <dby...@dbyron.com <mailto:dby...@dbyron.com>> wrote: On 1/14/16 7:40 AM, Alex Moore wrote: > Hi David, > > It doesn't look like we expose that property anywhere, but it can > probably be chalked up to YAGNI when it was written. Go forth and > PR :) Excellent...except for this at the HEAD of develop (24e1404). $ mvn clean install [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Riak Client for Java 2.0.5-SNAPSHOT [INFO] [WARNING] The POM for com.basho.riak.protobuf:riak-pb:jar:2.1.1.0 is missing, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 0.285 s [INFO] Finished at: 2016-01-14T11:17:22-08:00 [INFO] Final Memory: 10M/309M [INFO] [ERROR] Failed to execute goal on project riak-client: Could not resolve dependencies for project com.basho.riak:riak-client:jar:2.0.5-SNAPSHOT: Failure to find com.basho.riak.protobuf:riak-pb:jar:2.1.1.0 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] The latest version I see at http://mvnrepository.com/artifact/com.basho.riak.protobuf/riak-pb is 2.0.0.16. When I change pom.xml to use that version I get truckloads of errors. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: accessor for FetchDatatype's location?
On 1/14/16 7:40 AM, Alex Moore wrote: > Hi David, > > It doesn't look like we expose that property anywhere, but it can > probably be chalked up to YAGNI when it was written. Go forth and > PR :) Excellent...except for this at the HEAD of develop (24e1404). $ mvn clean install [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Riak Client for Java 2.0.5-SNAPSHOT [INFO] [WARNING] The POM for com.basho.riak.protobuf:riak-pb:jar:2.1.1.0 is missing, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 0.285 s [INFO] Finished at: 2016-01-14T11:17:22-08:00 [INFO] Final Memory: 10M/309M [INFO] [ERROR] Failed to execute goal on project riak-client: Could not resolve dependencies for project com.basho.riak:riak-client:jar:2.0.5-SNAPSHOT: Failure to find com.basho.riak.protobuf:riak-pb:jar:2.1.1.0 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] The latest version I see at http://mvnrepository.com/artifact/com.basho.riak.protobuf/riak-pb is 2.0.0.16. When I change pom.xml to use that version I get truckloads of errors. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
accessor for FetchDatatype's location?
I'm looking for access to the location member of FetchDatatype (https://github.com/basho/riak-java-client/blob/develop/src/main/java/com/basho/riak/client/api/commands/datatypes/FetchDatatype.java#L37) to verify in a test that I'm using the right location. I'm happy to make a pull request if this seems like a reasonable way forward. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: cluster health check using riak-java-client
I'm still curious what people think here. As I stare at this longer, I'd like to be able to call RiakNode.checkHealth(), but it's private. HealthMonitorTask.run that only calls checkHealth some of the time, so without the ability to call it directly, I think I'm getting a stale notion of health in circumstances like I outlined below -- when the last operation was successful, but the node has since gone down. Thanks for your input. -DB On 12/2/15 10:25 PM, David Byron wrote: I'm implementing a health check for a service of mine that uses riak. I've seen this code from https://github.com/basho/riak-java-client/issues/456: RiakCluster cluster = clientInstance.getRiakCluster(); List nodes = cluster.getNodes(); for (RiakNode node : nodes) { State state = node.getNodeState(); } and it's great. From what I can tell, it depends on some background processing that keeps track of the state of the nodes. I did a quick test though, and if I run 'riak stop' from the command line and then this loop with no intervening operations, the nodes report RUNNING. Even after some time passes (more than three minutes), still RUNNING. However, if I run do run an intervening operation (some actual query of data for example) that fails, the nodes then report HEALTH_CHECKING. Then, after 'riak start', the nodes report RUNNING again. I suppose that's good. So, I'm trying to decide how to implement the health check. The above loop doesn't seem to be enough, but do I really need to do something like: final RiakFuture<Void, Void> future = cluster.execute(new PingOperation()); try { future.await(); future.get(); } catch (ExecutionException | InterruptedException e) { // bad } // good Maybe it's sufficient to only do this if all the nodes report RUNNING? I suppose there's always a small window in time where a node could report bad, but via a ping I'd learn it was up...so I'm torn. Any suggestions for whether pinging every time is correct, or there's something more efficient (and safe)? Thanks for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
cluster health check using riak-java-client
I'm implementing a health check for a service of mine that uses riak. I've seen this code from https://github.com/basho/riak-java-client/issues/456: RiakCluster cluster = clientInstance.getRiakCluster(); List nodes = cluster.getNodes(); for (RiakNode node : nodes) { State state = node.getNodeState(); } and it's great. From what I can tell, it depends on some background processing that keeps track of the state of the nodes. I did a quick test though, and if I run 'riak stop' from the command line and then this loop with no intervening operations, the nodes report RUNNING. Even after some time passes (more than three minutes), still RUNNING. However, if I run do run an intervening operation (some actual query of data for example) that fails, the nodes then report HEALTH_CHECKING. Then, after 'riak start', the nodes report RUNNING again. I suppose that's good. So, I'm trying to decide how to implement the health check. The above loop doesn't seem to be enough, but do I really need to do something like: final RiakFuturefuture = cluster.execute(new PingOperation()); try { future.await(); future.get(); } catch (ExecutionException | InterruptedException e) { // bad } // good Maybe it's sufficient to only do this if all the nodes report RUNNING? I suppose there's always a small window in time where a node could report bad, but via a ping I'd learn it was up...so I'm torn. Any suggestions for whether pinging every time is correct, or there's something more efficient (and safe)? Thanks for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
simpler implementation of RiakMap
As I stare at the code for RiakMap in the riak-java-client (https://github.com/basho/riak-java-client/blob/develop/src/main/java/com/basho/riak/client/core/query/crdt/types/RiakMap.java), I have the urge for a simpler implementation. If I'm reading the code right, it looks like it handles a value of each RiakDatatype for each key. As in, it's possible for the same key in a map to have all of a counter, map, flag, register and set value. If it turns out that the underlying data has only one of those, or perhaps that I only care about one of those, it feels like private final Mapentries = new HashMap (); could become private final Map entries = new HashMap (); What I'm getting at is that the code I'm writing to iterate over the elements in a map is doing way more than I'd like. For a map whose values are all registers, I've got something like: RiakMap myMap; for (BinaryValue key : myMap.view().keySet()) { RiakRegister register = myMap.getRegister(key); } which looks clean enough, but is busy under the covers. I'm really whining about the implementation of getRegister and friends: public RiakRegister getRegister(BinaryValue key) { if (entries.containsKey(key)) { for (RiakDatatype dt : entries.get(key)) { if (dt.isRegister()) { return dt.getAsRegister(); } } } return null; } Because I'm iterating I already know the key is one of the entries, so would you consider a patch with an unsafe* (or any other name) set of accessors that assumes the key is present? I could of course call view and implement my own method, but I can't see how to take out the loop without changing the underlying data structure for entries. If I'm iterating over lots and lots of keys (i.e. myMap.view().keySet() contains lots and lots of elements), it might actually be enough savings to notice. In addition though, I think it's simpler enough to be helpful when trying to read/understand the code. It is of course possible that the underlying data isn't what I expect, and contains other data types too. So a full implementation might need hints from the caller about whether to use the more general/existing data structure, or the simplified one...and then whether to fail in the face of data that doesn't fit in the simplified one, or silently ignore it...or ignore it but notify the caller that it's present...and probably other stuff I haven't considered yet. I'd love to hear what you think about all of this. Thanks much for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
handling failure in RiakCommand#execute
From my reading of RiakCommand#execute (https://github.com/basho/riak-java-client/blob/develop/src/main/java/com/basho/riak/client/api/RiakCommand.java#L85), I'm curious why there's no call to future.isSuccess() I can imagine that future.await() throws an exception for all possible failures, but then the docs for RiakFuture (https://github.com/basho/riak-java-client/blob/develop/src/main/java/com/basho/riak/client/core/RiakFuture.java#L60) say: The typical use pattern is to call await(), check isSuccess(), then call getNow() or cause(). Maybe all I'm looking for is a comment in RiakCommand#execute explaining why it's not the typical use pattern...but my paranoid self is nervous at the moment. Thanks for helping me understand. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: counter crdt in riak
Whoops...didn't mean to drop the mailing list off the recipient list. Pull request here: https://github.com/basho/basho_docs/pull/1865. -DB On 10/21/15 7:52 AM, David Byron wrote: On 10/20/15 11:56 PM, Russell Brown wrote: On 21 Oct 2015, at 04:56, David Byron <dby...@dbyron.com> wrote: Apologies if this is basic question, or one that's already been answered. I've done a fair amount of digging (e.g. http://thread.gmane.org/gmane.comp.db.riak.user/12808/focus=12816), but I'm still confused about the documentation regarding conflict resolution for riak's crdt counter. Specifically this: https://github.com/basho/basho_docs/blame/riak/2.1.1/source/languages/en/riak/theory/concepts/crdts.md#L238 (using the blame view to get a line number reference in markdown), which says: "Counters | Each actor keeps an independent count for increments and decrements; upon merge, the pairwise maximum of the counts for each actor will win (e.g. if one count for an actor holds 172 and the other holds 173, 173 will win upon merge)" This makes it sound to me like an increment from one of the actors gets dropped. Fingers crossed this isn't actually what happens. How so? I guess the implicit information missing from the statement is that each actor is unique, and acts serially. “An independent count for increments and decrements” means two integers, one for increments, one for decrements. The notion of something winning (and therefore also something losing) is I think the thing that confuses me. The way I think of it, everybody wins. It's almost like there aren't any conflicts to resolve. I think this is saying the same thing that the cdrt paper (section 3.1.2 State-based increment-only Counter (G-Counter) of https://hal.inria.fr/file/index/docid/88/filename/techreport.pdf) says -- that increments that happen during a partition eventually resolve correctly. I could use a more step-by-step illustration though -- pairwise maximum of the counts for each actor is a little dense. Imagine the counter as a vector of triples, {ActorName, Increment Count, Decrement Count} Imagine 2 nodes A, C as replicas. A increments it’s counter by 1 so the data looks like [{A, 1, 0}] C decrements it’s counter, so it’s data is [{C, 0, 1}] A sends it’s counter to C so C performs a Merge. It looks at each entry in the vector, and for each entry takes the maximum Increment and maximum Decrement and keeps them. If an Actor is not present in some vector, it’s counters are considered zero. That is the “pairwise maximum.” For any pair of vectors, take the maximum integers for any pair of entries (i.e. {A, 1, 0} compared to {A, 0, 0} = {A, max(1, 0), max(0, 0}) So now C has [{A, 1, 0}, {C, 0, 1}] Now C does an increment [{A, 1, 0}, {C, 1, 1}] and sends it’s counter to A A increments it’s counter by 1 [{A, 2, 0}] then decrements it by 1 [{A, 2, 1}] C sends it’s counter to A and the merge results in [{A, 2, 1}, {C, 1, 1}] Any actor will _always_ have the maximum values for their own entries. Thanks for the detailed explanation. Apologies for ugly formatting, but an example like this would make more sense to me. - actor a increments counter a {a: 1} b {} - actor b increments counter a {a: 1} b {b: 1} - a merges b's info a {a: 1, b: 1} b {b: 1} - b merges a's info a:{a: 1, b: 1} b {a: 1, b: 1} so eventually this all resolves such that a and b both increment the counter by 2, right? Is this how riak's crdt counter works? If so I'll breathe a sigh of relief and make a pull request against the docs if you like. Yes that’s how they work, but there are two integers per actor, one for increments and one for decrements. Please do make your pull request. Will do. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
counter crdt in riak
Apologies if this is basic question, or one that's already been answered. I've done a fair amount of digging (e.g. http://thread.gmane.org/gmane.comp.db.riak.user/12808/focus=12816), but I'm still confused about the documentation regarding conflict resolution for riak's crdt counter. Specifically this: https://github.com/basho/basho_docs/blame/riak/2.1.1/source/languages/en/riak/theory/concepts/crdts.md#L238 (using the blame view to get a line number reference in markdown), which says: "Counters | Each actor keeps an independent count for increments and decrements; upon merge, the pairwise maximum of the counts for each actor will win (e.g. if one count for an actor holds 172 and the other holds 173, 173 will win upon merge)" This makes it sound to me like an increment from one of the actors gets dropped. Fingers crossed this isn't actually what happens. I think this is saying the same thing that the cdrt paper (section 3.1.2 State-based increment-only Counter (G-Counter) of https://hal.inria.fr/file/index/docid/88/filename/techreport.pdf) says -- that increments that happen during a partition eventually resolve correctly. I could use a more step-by-step illustration though -- pairwise maximum of the counts for each actor is a little dense. Apologies for ugly formatting, but an example like this would make more sense to me. - actor a increments counter a {a: 1} b {} - actor b increments counter a {a: 1} b {b: 1} - a merges b's info a {a: 1, b: 1} b {b: 1} - b merges a's info a:{a: 1, b: 1} b {a: 1, b: 1} so eventually this all resolves such that a and b both increment the counter by 2, right? Is this how riak's crdt counter works? If so I'll breathe a sigh of relief and make a pull request against the docs if you like. Thanks much for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: counter crdt in riak
On 10/20/15 9:26 PM, xia...@xiaclo.net wrote: This gives a little more detail about each of the CRDT types: https://github.com/basho/riak/issues/354 Thanks much. That link is very helpful. Your assumptions are correct. Within Riak, an actor is a vnode, and counters behave as G-Counters. Keep in mind that Riak counters store both an increment and decrement for each actor, which allows decrements and negative counters as well while preserving the accuracy of G-Counters. I'd read in https://github.com/basho/riak/blob/develop/releasenotes/riak-1.4.md#pn-counters that riak's counters were PN...which I think is what you're saying...except the part about them behaving as G-Counters. Sorry if I'm a little slow on the uptake here, and thanks again for your help. -DB Hope this helps, Jason From: David Byron <dby...@dbyron.com <mailto:dby...@dbyron.com>> Date: Wed Oct 21 2015 15:03:04 GMT+1100 (AUS Eastern Summer Time) To: riak-users@lists.basho.com <riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>> Subject: counter crdt in riak Apologies if this is basic question, or one that's already been answered. I've done a fair amount of digging (e.g. http://thread.gmane.org/gmane.comp.db.riak.user/12808/focus=12816), but I'm still confused about the documentation regarding conflict resolution for riak's crdt counter. Specifically this: https://github.com/basho/basho_docs/blame/riak/2.1.1/source/languages/en/riak/theory/concepts/crdts.md#L238 (using the blame view to get a line number reference in markdown), which says: "Counters | Each actor keeps an independent count for increments and decrements; upon merge, the pairwise maximum of the counts for each actor will win (e.g. if one count for an actor holds 172 and the other holds 173, 173 will win upon merge)" This makes it sound to me like an increment from one of the actors gets dropped. Fingers crossed this isn't actually what happens. I think this is saying the same thing that the cdrt paper (section 3.1.2 State-based increment-only Counter (G-Counter) of https://hal.inria.fr/file/index/docid/88/filename/techreport.pdf) says -- that increments that happen during a partition eventually resolve correctly. I could use a more step-by-step illustration though -- pairwise maximum of the counts for each actor is a little dense. Apologies for ugly formatting, but an example like this would make more sense to me. - actor a increments counter a {a: 1} b {} - actor b increments counter a {a: 1} b {b: 1} - a merges b's info a {a: 1, b: 1} b {b: 1} - b merges a's info a:{a: 1, b: 1} b {a: 1, b: 1} so eventually this all resolves such that a and b both increment the counter by 2, right? Is this how riak's crdt counter works? If so I'll breathe a sigh of relief and make a pull request against the docs if you like. Thanks much for your help. -DB ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com