Re: 1.4.2: 'riak-admin reip' no longer works?
Huh. I thought we were supposed to use force-replace instead of reip? I remember having a discussion a few months back about the documentation and we cleared it up a little. Is reip the same as force-replace, or is it more...quick-and-dirty? Is it deprecated? -B. On Mon, Oct 21, 2013 at 11:33 AM, Dave Brady dbr...@weborama.com wrote: Thanks for clarifying it! This functionality has been *great* to create quick-and-dirty test rings. -- Dave Brady -- *From: *Jared Morrow ja...@basho.com *To: *Dave Brady dbr...@weborama.com *Cc: *Riak Users Mailing List riak-users@lists.basho.com *Sent: *Lundi 21 Octobre 2013 16:42:22 *Subject: *Re: 1.4.2: 'riak-admin reip' no longer works? We actually discussed this recently, my understanding of reip was clearly not complete, and Joe Caswell put me on the right track. 'reip' will continue to stick around. -Jared On Mon, Oct 21, 2013 at 8:23 AM, Dave Brady dbr...@weborama.com wrote: No problem, Jared! I'm happy it was a very simple fix. What concerns me is that the 'cluster' commands do not have an analogous feature: 'replace' does not do the same thing as 'reip'. Is it truly a foregone conclusion that 'reip' is going away? -- Dave Brady -- *From: *Jared Morrow ja...@basho.com *To: *Dave Brady dbr...@weborama.com *Cc: *Riak Users Mailing List riak-users@lists.basho.com *Sent: *Lundi 21 Octobre 2013 16:17:52 *Subject: *Re: 1.4.2: 'riak-admin reip' no longer works? This was a mistake on my part, if you s/up/down/ (node_down_check) on that line it will correctly ensure the node is down before running reip. Sorry about that, clearly 'reip' is not heavily used with the new cluster commands and it slipped under the radar for all these months. -Jared On Mon, Oct 21, 2013 at 7:48 AM, Dave Brady dbr...@weborama.com wrote: Thanks to Joe Caswell at Basho: just have to comment out line 382 (node_up_check) in the riak-admin script. -- Dave Brady - Original Message - From: Dave Brady dbr...@weborama.com To: riak-users@lists.basho.com Sent: Lundi 21 Octobre 2013 13:59:37 Subject: 1.4.2: 'riak-admin reip' no longer works? Hi everyone, So, I went to copy /var/lib/riak/* from one five-node cluster to another, so that we could experiment on the data. I did this fairly frequently in the past ( 1.4.0), and always used riak-admin reip old new. Now, when I try that command on our 1.4.2 ring, 'riak-admin' dies with Node is not running! This confuses me, since reip requires the ring to be down, but the riak-admin script is indeed expecting the local node to be running when using this option. I see on the wiki that it is mentioned that cluster replace will replace reip in the future. I hope not, since cluster replace does not provide the same functionality. Anyone run into this and find a fix? (Or I am missing something obvious?) Thanks! -- Dave Brady ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
2i Index Durability/Resilience
What happens to your 2i indexes if you do a write and one of the nodes you're trying to write to is down? http://docs.basho.com/riak/latest/dev/using/2i/ says: - When you want or need anti-entropy. Since 2i is just metadata on the KV object and the indexes reside on the same node, 2i piggybacks off of read-repair. But http://docs.basho.com/riak/latest/ops/running/recovery/repairing-indexes/says: Riak Secondary indexes (2i) currently have no form of anti-entropy (such as read-repair). Furthermore, for performance and load balancing reasons, 2i reads from 1 random node. This means that when a replica loss has occurred, inconsistent results may be returned. I am building a solution around 2i - so I just wanted to know if there was any way to clarify these points - how resilient are these indexes? Under what circumstances will they stop working (or return inconsistent results)? -B. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: allow_mult vs. 2i
I'm doing the POST/PUT to the bucket thing, where it makes a key for me. -B. On Sep 28, 2013, at 3:53 PM, Runar Jordahl runar.jord...@gmail.com wrote: Yes, you are correct. Setting allow_mult will not be needed with your setup. I then assume the clients are guaranteed to use unique key when creating the objects. Runar 2013/9/27 Brady Wetherington br...@bespincorp.com Since I'm doing a write-once, update-never environment - I don't see how allow_mult would help me otherwise? A new write will always be to a new key. There will never be an update. So if that's the case - no need for allow_mult. Does that sound right? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: allow_mult vs. 2i
Oh, I get what the siblings/allow_mult business is for, just wondering if I can use it off-label a little, and eventually do 'conflict resolution' which would make the results be much more reasonable. But it sounds like I shouldn't do that. That's totally fine. Since I'm doing a write-once, update-never environment - I don't see how allow_mult would help me otherwise? A new write will always be to a new key. There will never be an update. So if that's the case - no need for allow_mult. Does that sound right? -B. On Wed, Sep 25, 2013 at 6:30 PM, Jeremiah Peschka jeremiah.pesc...@gmail.com wrote: inline. --- Jeremiah Peschka - Founder, Brent Ozar Unlimited MCITP: SQL Server 2008, MVP Cloudera Certified Developer for Apache Hadoop On Wed, Sep 25, 2013 at 2:47 PM, Brady Wetherington br...@bespincorp.comwrote: I've built it a solid proof-of-concept system on leveldb, and use some 2i indexes in order to search for certain things - usually just for counts of things. I have two questions so far: First off, why is Bitcask the default? Is it just because it is faster? Or is it considered more 'stable' or something? Long ago, when bitcask was elected as the default, LevelDB was not a thing. Databases strive for stability and the principle of least surprise. Changing anything can potentially introduce performance regressions, stability problems, and any host of other undesirable and reputation destroying things. Changing the storage back end is high up on the list of things I'd never want to do in a database. Why do you think MySQL still defaults to MyISAM? Next, I've learned about the allow_mult feature you can set on buckets. I wonder if I should use this for my most heavily-used primary-purpose queries? Is there a limit to how many 'siblings' you can have for an entry? Is it inadvisable to do what I'm talking about? Would fetching all of the siblings end up being a disastrous nightmare or something? The upper limit will depend on the size of your objects. You don't want to have object sizes (including siblings) much beyond 6MB. You'll have a lot of network congestion. You certainly *could* have bigger object + sibling collections, but you'd want to beef up the network backend to something like 10GbE, 40GbE, or InfiniBand to deal with the increased gossip. Fetching all of your siblings is bad if you never resolve siblings since you'll have a lot of data. Allow_mult is typically turned on for production clusters. This is set off by default to help new users get a handle on Riak quickly without having to worry about siblings. Once you get the hang of how Riak behaves, turning on siblings is usually a good thing. Depending on resolution, it's probably best to read your data, resolve siblings, and send that garbage collected object back to Riak - even if you're performing a read only query. The new Riak DT features eliminate some of the worry about siblings by pushing the responsibility back down to Riak. Those features are only available if you're building from source, but hopefully Riak 2.0 will be out soon. I *assume* - and I could be wrong - that a 2i query would be slower than a fetch-of-siblings for a particular key - is that wrong? If I switch from using 2i indexes to using allow_mult and siblings, we'd be talking a few hundred thousand to low millions for a sibling-count. I do not think 'siblings' means what you think it means. A sibling would occur if two clients, A and B, read v1 of an object and then issue writes. Client A updates object and sets preferences to ['cat pictures', 'ham sandwiches'] Client B updates object and sets preferences to ['knitting with bacon'] With allow_mult enabled you'd have two versions of the object. These are siblings. If you're thinking of some kind of index created by your application, you could look at 2i vs using siblings to build a secondary index: http://basho.com/index-for-fun-and-for-profit/ Even when you're creating your own secondary index, you still want to perform garbage collection on the data you're storing in Riak. Thanks for making an excellent product! Can't wait to get this bad boy into production and really see what it can do! -B. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
allow_mult vs. 2i
I've built it a solid proof-of-concept system on leveldb, and use some 2i indexes in order to search for certain things - usually just for counts of things. I have two questions so far: First off, why is Bitcask the default? Is it just because it is faster? Or is it considered more 'stable' or something? Next, I've learned about the allow_mult feature you can set on buckets. I wonder if I should use this for my most heavily-used primary-purpose queries? Is there a limit to how many 'siblings' you can have for an entry? Is it inadvisable to do what I'm talking about? Would fetching all of the siblings end up being a disastrous nightmare or something? I *assume* - and I could be wrong - that a 2i query would be slower than a fetch-of-siblings for a particular key - is that wrong? If I switch from using 2i indexes to using allow_mult and siblings, we'd be talking a few hundred thousand to low millions for a sibling-count. Thanks for making an excellent product! Can't wait to get this bad boy into production and really see what it can do! -B. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: DELETE and vclocks
Well, I'm not sure, and I'm new to Riak too, but I *think* you'd need to set allow_multi on the bucket. -B. On Mon, Sep 23, 2013 at 4:01 AM, Gints gints.gaili...@gmail.com wrote: I did a PUT, and the vclock that came back was: a85hYGBgzGDKBVIcypz/fgZJFylnMCUy5rEyNOdtOc2XBQA= Then I read the object back, modified it, and PUT it again, providing the old vclock, and got a new vclock: a85hYGBgzGDKBVIcypz/fgZJFylnMCUy5bEyNOdtOc2XBQA= (Notice they are slightly different: Uy5rEy - Uy5bEy) Then, I did a DELETE, using the first vector clock, not the second, updated one. After that the object was gone, I could not find it in the database any more. Is this supposed to be like that - deleting an object providing a stale vclock actually deletes it? Or am I probably doing something wrong? Thanks, Gints -- View this message in context: http://riak-users.197444.n3.nabble.com/DELETE-and-vclocks-tp4029176.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re:
I've been working with Basho to get those documents updated - in short - I
have a pull request that makes using Riak with EIP's in Classic-EC2 much,
much easier. They have accepted it, but it has not been merged with the
mainline yet. I'm assuming you are *not* on VPC, is that correct?
If so, you should use the ec2 public DNS name as your node identifier
instead. For the IP you listed above, that's:
ec2-54-247-68-179.eu-west-1.compute.amazonaws.com
If you find that name ungainly - go head and make a CNAME that points to it
-
riak-1.yourcompany.com CNAME
ec2-54-247-68-179.eu-west-1.compute.amazonaws.com
Once you've done this, you can use that shortname (riak-1.yourcompany.com)
as your node identifier - and it will survive a node restart (so long as
you reassociate the EIP with the instance).
The reason *why* this works is that that public DNS name (
ec2-54-247-68-179.eu-west-1.compute.amazonaws.com) resolves to the _inside_
IP address of your node when you're inside of amazon. So your server can
figure out whether or not it's talking to itself.
-B.
On Mon, Sep 9, 2013 at 7:38 AM, David Montgomery
davidmontgom...@gmail.comwrote:
Why do I get errors restarting riak when i use aws EIP's?
I change the ip address per this doc.
http://docs.basho.com/riak/1.2.1/cookbooks/Basic-Cluster-Setup/ I am
using the latest version of riak for ubuntu 12.04
I change the 127.0.0.1 to the EIP. It should work. Yetriak will not
work. Any issues I am missing? How do I resolve?
console.log
2013-09-09 11:28:28.211 [info] 0.7.0 Application webmachine started on
node 'riak@54.247.68.179'
2013-09-09 11:28:28.211 [info] 0.7.0 Application basho_stats started on
node 'riak@54.247.68.179'
2013-09-09 11:28:28.229 [info] 0.7.0 Application bitcask started on node
'riak@54.247.68.179'
2013-09-09 11:28:29.385 [error] 0.172.0 CRASH REPORT Process 0.172.0
with 0 neighbours exited with reason: eaddrnotavail in gen_server:init_it/6
line 320
2013-09-09 11:28:29.385 [error] 0.138.0 Supervisor riak_core_sup had
child http_54.247.68.179:8098 started with
webmachine_mochiweb:start([{name,http_54.247.68.179:8098},{ip,54.247.68.179},{p$
2013-09-09 11:28:29.387 [info] 0.7.0 Application riak_core exited with
reason: {shutdown,{riak_core_app,start,[normal,[]]}}
error.log
013-09-09 11:08:13.109 [error] 0.138.0 Supervisor riak_core_sup had
child riak_core_capability started with riak_core_capability:start_link()
at 0.156.0 exit with reason no function clause match$
2013-09-09 11:08:13.110 [error] 0.136.0 CRASH REPORT Process 0.136.0
with 0 neighbours exited with reason: {{function_clause,[{orddict,fetch,['
riak@10.239.130.225',[{'riak@127.0.0.1',[{{riak_cont$
2013-09-09 11:08:34.956 [error] 0.156.0 gen_server riak_core_capability
terminated with reason: no function clause matching orddict:fetch('
riak@10.239.130.225', [{'riak@127.0.0.1',[{{riak_control,m$
2013-09-09 11:08:34.957 [error] 0.156.0 CRASH REPORT Process
riak_core_capability with 0 neighbours exited with reason: no function
clause matching orddict:fetch('riak@10.239.130.225', [{'riak@127.$
2013-09-09 11:08:34.957 [error] 0.140.0 Supervisor riak_core_sup had
child riak_core_capability started with riak_core_capability:start_link()
at 0.156.0 exit with reason no function clause match$
2013-09-09 11:08:34.958 [error] 0.138.0 CRASH REPORT Process 0.138.0
with 0 neighbours exited with reason: {{function_clause,[{orddict,fetch,['
riak@10.239.130.225',[{'riak@127.0.0.1',[{{riak_cont$
2013-09-09 11:10:56.863 [error] 0.154.0 gen_server riak_core_capability
terminated with reason: no function clause matching orddict:fetch('
riak@54.247.68.179', []) line 72
2013-09-09 11:10:56.864 [error] 0.154.0 CRASH REPORT Process
riak_core_capability with 0 neighbours exited with reason: no function
clause matching orddict:fetch('riak@54.247.68.179', []) line 72 i$
[ Read 32 lines ]
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak-CS , S3cmd and A single Node outage on a 3 node cluster
Oh, I thought the rule was if you have one more node than your n_val, then you can be sure each replica will be on a distinct node - is that not correct? Also with n of 2 I would wonder how you would handle r and w - my guess would be you *could* set them to 1 each, and just deal with possible consistency issues if you lost a node. But it would be 'weird' in that you wouldn't really have a quorum to make - it'd be whoever is up right now wins. And then - when the downed node comes back - you might spit back inconsistent answers until AAE fixes up your data. Read-repair would *not* end up fixing your data up, because the r value of 1 would be satisfied. Does that sound right? I'm asking not only for the original poster; but to make sure I understand what I've got here for myself too! -B. On Thu, Aug 22, 2013 at 1:41 PM, Kelly McLaughlin ke...@basho.com wrote: Idan, Actually in the case you described of using a 3 node Riak cluster with n_val of 2 the behavior you see makes perfect sense. When using only three nodes Riak does not guarantee that all replicas of an object will be on distinct physical nodes. So if you have one node down you can hit a case where both of the replicas of an object stored in Riak live on the downed node. This explains the reason you see occasional failure to retrieve an object with s3cmd, but most of the time it works just fine. It is for this reason we strongly recommend people to use at least 5 nodes in their production deployments. For testing, using at least use 4 nodes in the cluster will likely reduce the chances of this situation occurring versus using only 3. Also just be very cautious about reducing the n_val to 2. You will save on storage space doing that, but there is a trade-off to be made in availability and performance in doing so. Hope that helps shed some light. Kelly On August 22, 2013 at 10:12:06 AM, Idan Shinberg (idan.shinb...@idomoo.com) wrote: Thanks for the aid This is a testing environemt . no-one accesses it aside of me Prior to creating the objects via riak-cs , I set the cluster to n_val 2 for the default bucket props . On that empty cluster , i started creating objects . ( around 500) . Then I stopped and killed one of the nodes . Then the issues mentioned above were seen. still doesn't make any sense... Regards, Idan Shinberg System Architect Idomoo Ltd. Mob +972.54.562.2072 email idan.shinb...@idomoo.com web www.idomoo.com [image: Description: file:///Users/kelly/Library/Containers/it.bloop.airmail/Data/Library/Application Support/Airmail/General/Local/1377192732334581760/Attachments/fc66336e-e750-4c2d-b6e3-985d5a06b...@idomoo.co.il] ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Practical Riak cluster choices in AWS (number of nodes? AZ's?)
That *does* sound like an interesting way to do it. Kinda best-of-both-worlds, depending on your backup schemes and whatnot. I'm definitely curious to hear about how it works out for you. -B. On Tue, Aug 13, 2013 at 4:03 PM, Dave Martorana d...@flyclops.com wrote: An interesting hybrid that I'm coming around to seems to be using a Unix release - OmniOS has an AMI, for instance - and ZFS. With a large-enough store, I can run without EBS on my nodes, and have a single ZFS backup instance with a huge amount of slow-EBS storage for accepting ZFS snapshots. I'm still learning all the pieces, but luckily I have a company upstairs from me that does a very similar thing with 300TB and is willing to help me set up my ZFS backup infrastructure. Dave On Mon, Aug 12, 2013 at 10:00 PM, Brady Wetherington br...@bespincorp.com wrote: I will probably stick with EBS-store for now. I don't know how comfortable I can get with a replica that could disappear with simply an unintended reboot (one of my nodes just did that randomly today, for example). Sure, I would immediately start rebuilding it as soon as that were to happen, but we could be talking a pretty huge chunk of data that would have to get rebuilt out of the cluster. And that sounds scary. Even though, logically, I understand that it should not be. I will get there; I'm just a little cautious. As I learn Riak better and get more comfortable with it, maybe I would be able to start to move in a direction like that. And certainly as the performance characteristics of EBS-volumes start to bite me in the butt; that might force me to get comfortable with instance-store real quick. I would at least hope to be serving a decent-sized chunk of my data from memory, however. As for throwing my instances in one AZ - I don't feel comfortable with that either. I'll try out the way I'm saying and will report back - do I end up with crazy latencies all over the map, or does it seem to just work? We'll see. In the meantime, I still feel funny about breaking the rules on the 5-node cluster policy. Given my other choices as having been kinda nailed-down for now, what do you guys think of that? E.g. - should I take the risk of putting a 5th instance up in the same AZ as one of the others, or should I just be ok with having 4? Or should I do something weird like changing my 'n' value to be one fewer or something like that? (I think, as I understand it so far, I'm really liking n=3, w=2, r=2 - but I could change it if it made more sense with the topology I've selected.) -B. Date: Sun, 11 Aug 2013 18:57:11 -0600 From: Jared Morrow ja...@basho.com To: Jeremiah Peschka jeremiah.pesc...@gmail.com Cc: riak-users riak-users@lists.basho.com Subject: Re: Practical Riak cluster choices in AWS (number of nodes? AZ's?) Message-ID: cacusovelpu8yfcivykexm9ztkhq-kdnowk1afvpflcsip2h...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 +1 to what Jeremiah said, putting a 4 or 5 node cluster in each US West and US East using MDC between them would be the optimum solution. I'm also not buying consistent latencies between AZ's, but I've also not tested it personally in a production environment. We have many riak-users members on AWS, so hopefully more experienced people will chime in. If you haven't seen them already, here's what I have in my Riak on AWS bookmark folder: http://media.amazonwebservices.com/AWS_NoSQL_Riak.pdf http://docs.basho.com/riak/latest/ops/tuning/aws/ http://basho.com/riak-on-aws-deployment-options/ -Jared On Sun, Aug 11, 2013 at 6:11 PM, Jeremiah Peschka jeremiah.pesc...@gmail.com wrote: I'd be wary of using EBS backed nodes for Riak - with only a single ethernet connection, it wil be very easy to saturate the max of 1000mbps available in a single AWS NIC (unless you're using cluster compute instances). I'd be more worried about temporarily losing contact with a node through network saturation than through AZ failure, truthfully. The beauty of Riak is that a node can drop and you can replace it with minimal fuss. Use that to your advantage and make every node in the cluster disposable. As far as doubling up in one AZ goes - if you're worried about AZ failure, you should treat each AZ as a separate data center and design your failure scenarios accordingly. Yes, Amazon say you should put one Riak node in each AZ; I'm not buying that. With no guarantee around latency, and no control around between DCs, you need to be very careful how much of that latency you're willing to introduce into your application. Were I in your position, I'd stand up a 5 node cluster in US-WEST-2 and be done with it. I'd consider Riak EE for my HA/DR solution once the business decides that off-site HA/DR is something it wants/needs. --- Jeremiah Peschka - Founder, Brent Ozar Unlimited MCITP: SQL Server 2008, MVP Cloudera Certified
Re: Practical Riak cluster choices in AWS (number of nodes? AZ's?)
I will probably stick with EBS-store for now. I don't know how comfortable
I can get with a replica that could disappear with simply an unintended
reboot (one of my nodes just did that randomly today, for example). Sure, I
would immediately start rebuilding it as soon as that were to happen, but
we could be talking a pretty huge chunk of data that would have to get
rebuilt out of the cluster. And that sounds scary. Even though, logically,
I understand that it should not be.
I will get there; I'm just a little cautious. As I learn Riak better and
get more comfortable with it, maybe I would be able to start to move in a
direction like that. And certainly as the performance characteristics of
EBS-volumes start to bite me in the butt; that might force me to get
comfortable with instance-store real quick. I would at least hope to be
serving a decent-sized chunk of my data from memory, however.
As for throwing my instances in one AZ - I don't feel comfortable with that
either. I'll try out the way I'm saying and will report back - do I end up
with crazy latencies all over the map, or does it seem to just work?
We'll see.
In the meantime, I still feel funny about breaking the rules on the
5-node cluster policy. Given my other choices as having been kinda
nailed-down for now, what do you guys think of that?
E.g. - should I take the risk of putting a 5th instance up in the same AZ
as one of the others, or should I just be ok with having 4? Or should I
do something weird like changing my 'n' value to be one fewer or something
like that? (I think, as I understand it so far, I'm really liking n=3,
w=2, r=2 - but I could change it if it made more sense with the topology
I've selected.)
-B.
Date: Sun, 11 Aug 2013 18:57:11 -0600
From: Jared Morrow ja...@basho.com
To: Jeremiah Peschka jeremiah.pesc...@gmail.com
Cc: riak-users riak-users@lists.basho.com
Subject: Re: Practical Riak cluster choices in AWS (number of nodes?
AZ's?)
Message-ID:
cacusovelpu8yfcivykexm9ztkhq-kdnowk1afvpflcsip2h...@mail.gmail.com
Content-Type: text/plain; charset=iso-8859-1
+1 to what Jeremiah said, putting a 4 or 5 node cluster in each US West and
US East using MDC between them would be the optimum solution. I'm also not
buying consistent latencies between AZ's, but I've also not tested it
personally in a production environment. We have many riak-users members on
AWS, so hopefully more experienced people will chime in.
If you haven't seen them already, here's what I have in my Riak on AWS
bookmark folder:
http://media.amazonwebservices.com/AWS_NoSQL_Riak.pdf
http://docs.basho.com/riak/latest/ops/tuning/aws/
http://basho.com/riak-on-aws-deployment-options/
-Jared
On Sun, Aug 11, 2013 at 6:11 PM, Jeremiah Peschka
jeremiah.pesc...@gmail.com wrote:
I'd be wary of using EBS backed nodes for Riak - with only a single
ethernet connection, it wil be very easy to saturate the max of 1000mbps
available in a single AWS NIC (unless you're using cluster compute
instances). I'd be more worried about temporarily losing contact with a
node through network saturation than through AZ failure, truthfully.
The beauty of Riak is that a node can drop and you can replace it with
minimal fuss. Use that to your advantage and make every node in the
cluster
disposable.
As far as doubling up in one AZ goes - if you're worried about AZ
failure,
you should treat each AZ as a separate data center and design your
failure
scenarios accordingly. Yes, Amazon say you should put one Riak node in
each
AZ; I'm not buying that. With no guarantee around latency, and no control
around between DCs, you need to be very careful how much of that latency
you're willing to introduce into your application.
Were I in your position, I'd stand up a 5 node cluster in US-WEST-2 and
be
done with it. I'd consider Riak EE for my HA/DR solution once the
business
decides that off-site HA/DR is something it wants/needs.
---
Jeremiah Peschka - Founder, Brent Ozar Unlimited
MCITP: SQL Server 2008, MVP
Cloudera Certified Developer for Apache Hadoop
On Sun, Aug 11, 2013 at 1:52 PM, Brady Wetherington
br...@bespincorp.comwrote:
Hi all -
I have some questions about how I want my Riak stuff to work - I've
already asked these questions of some Basho people and gotten some
answers,
but thought I would toss it out into the wider world to see what you all
have to say, too:
First off - I know 5 instances is the magic number of instances to
have. If I understand the thinking here, it's that at the default
redundancy level ('n'?) of 3, it is most likely to start getting me some
scaling (e.g., performance just that of a single node), and yet also
have
redundancy; whereby I can lose one box and not start to take a
performance
hit.
My question is - I think I can only do 4 in a way that makes sense. I
only have 4 AZ's that I can use right now; AWS won't let me
Practical Riak cluster choices in AWS (number of nodes? AZ's?)
Hi all -
I have some questions about how I want my Riak stuff to work - I've already
asked these questions of some Basho people and gotten some answers, but
thought I would toss it out into the wider world to see what you all have
to say, too:
First off - I know 5 instances is the magic number of instances to have.
If I understand the thinking here, it's that at the default redundancy
level ('n'?) of 3, it is most likely to start getting me some scaling
(e.g., performance just that of a single node), and yet also have
redundancy; whereby I can lose one box and not start to take a performance
hit.
My question is - I think I can only do 4 in a way that makes sense. I only
have 4 AZ's that I can use right now; AWS won't let me boot instances in
1a. My concern is if I try to do 5, I will be doubling up in one AZ - and
in AWS you're almost as likely to lose an entire AZ as you are a single
instance. And so, if I have instances doubled-up in one AZ (let's say
us-east-1e), and then I lose 1e, I've now lost two instances. What are the
chances that all three of my replicas of some chunk of my data are on those
two instances? I know that it's not guaranteed that all replicas are on
separate nodes.
So is it better for me to ignore the recommendation of 5 nodes, and just do
4? Or to ignore the fact that I might be doubling-up in one AZ? Also,
another note. These are designed to be 'durable' nodes, so if one should go
down I would expect to bring it back up *with* its data - or, if I
couldn't, I would do a force-replace or replace and rebuild it from the
other replicas. I'm definitely not doing instance-store. So I don't know if
that mitigates my need for a full 5 nodes. I would also consider losing one
node to be degraded and would probably seek to fix that problem as soon
as possible, so I wouldn't expect to be in that situation for long. I would
probably tolerate a drop in performance during that time, too. (Not a
super-severe one, but 20-30 percent? Sure.)
What do you folks think?
-B.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
