I will probably stick with EBS-store for now. I don't know how comfortable I can get with a replica that could disappear with simply an unintended reboot (one of my nodes just did that randomly today, for example). Sure, I would immediately start rebuilding it as soon as that were to happen, but we could be talking a pretty huge chunk of data that would have to get rebuilt out of the cluster. And that sounds scary. Even though, logically, I understand that it should not be.
I will get there; I'm just a little cautious. As I learn Riak better and get more comfortable with it, maybe I would be able to start to move in a direction like that. And certainly as the performance characteristics of EBS-volumes start to bite me in the butt; that might force me to get comfortable with instance-store real quick. I would at least hope to be serving a decent-sized chunk of my data from memory, however. As for throwing my instances in one AZ - I don't feel comfortable with that either. I'll try out the way I'm saying and will report back - do I end up with crazy latencies all over the map, or does it seem to "just work?" We'll see. In the meantime, I still feel funny about "breaking the rules" on the 5-node cluster policy. Given my other choices as having been kinda nailed-down for now, what do you guys think of that? E.g. - should I take the risk of putting a 5th instance up in the same AZ as one of the others, or should I just "be ok" with having 4? Or should I do something weird like changing my 'n' value to be one fewer or something like that? (I think, as I understand it so far, I'm really liking "n=3, w=2, r=2" - but I could change it if it made more sense with the topology I've selected.) -B. Date: Sun, 11 Aug 2013 18:57:11 -0600 > From: Jared Morrow <ja...@basho.com> > To: Jeremiah Peschka <jeremiah.pesc...@gmail.com> > Cc: riak-users <riak-users@lists.basho.com> > Subject: Re: Practical Riak cluster choices in AWS (number of nodes? > AZ's?) > Message-ID: > < > cacusovelpu8yfcivykexm9ztkhq-kdnowk1afvpflcsip2h...@mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > +1 to what Jeremiah said, putting a 4 or 5 node cluster in each US West and > US East using MDC between them would be the optimum solution. I'm also not > buying consistent latencies between AZ's, but I've also not tested it > personally in a production environment. We have many riak-users members on > AWS, so hopefully more experienced people will chime in. > > If you haven't seen them already, here's what I have in my "Riak on AWS" > bookmark folder: > > http://media.amazonwebservices.com/AWS_NoSQL_Riak.pdf > http://docs.basho.com/riak/latest/ops/tuning/aws/ > http://basho.com/riak-on-aws-deployment-options/ > > -Jared > > > > > On Sun, Aug 11, 2013 at 6:11 PM, Jeremiah Peschka < > jeremiah.pesc...@gmail.com> wrote: > > > I'd be wary of using EBS backed nodes for Riak - with only a single > > ethernet connection, it wil be very easy to saturate the max of 1000mbps > > available in a single AWS NIC (unless you're using cluster compute > > instances). I'd be more worried about temporarily losing contact with a > > node through network saturation than through AZ failure, truthfully. > > > > The beauty of Riak is that a node can drop and you can replace it with > > minimal fuss. Use that to your advantage and make every node in the > cluster > > disposable. > > > > As far as doubling up in one AZ goes - if you're worried about AZ > failure, > > you should treat each AZ as a separate data center and design your > failure > > scenarios accordingly. Yes, Amazon say you should put one Riak node in > each > > AZ; I'm not buying that. With no guarantee around latency, and no control > > around between DCs, you need to be very careful how much of that latency > > you're willing to introduce into your application. > > > > Were I in your position, I'd stand up a 5 node cluster in US-WEST-2 and > be > > done with it. I'd consider Riak EE for my HA/DR solution once the > business > > decides that off-site HA/DR is something it wants/needs. > > > > > > --- > > Jeremiah Peschka - Founder, Brent Ozar Unlimited > > MCITP: SQL Server 2008, MVP > > Cloudera Certified Developer for Apache Hadoop > > > > > > On Sun, Aug 11, 2013 at 1:52 PM, Brady Wetherington < > br...@bespincorp.com>wrote: > > > >> Hi all - > >> > >> I have some questions about how I want my Riak stuff to work - I've > >> already asked these questions of some Basho people and gotten some > answers, > >> but thought I would toss it out into the wider world to see what you all > >> have to say, too: > >> > >> First off - I know 5 instances is the "magic number" of instances to > >> have. If I understand the thinking here, it's that at the default > >> redundancy level ('n'?) of 3, it is most likely to start getting me some > >> scaling (e.g., performance > just that of a single node), and yet also > have > >> redundancy; whereby I can lose one box and not start to take a > performance > >> hit. > >> > >> My question is - I think I can only do 4 in a way that makes sense. I > >> only have 4 AZ's that I can use right now; AWS won't let me boot > instances > >> in 1a. My concern is if I try to do 5, I will be "doubling up" in one > AZ - > >> and in AWS you're almost as likely to lose an entire AZ as you are a > single > >> instance. And so, if I have instances doubled-up in one AZ (let's say > >> us-east-1e), and then I lose 1e, I've now lost two instances. What are > the > >> chances that all three of my replicas of some chunk of my data are on > those > >> two instances? I know that it's not guaranteed that all replicas are on > >> separate nodes. > >> > >> So is it better for me to ignore the recommendation of 5 nodes, and just > >> do 4? Or to ignore the fact that I might be doubling-up in one AZ? Also, > >> another note. These are designed to be 'durable' nodes, so if one > should go > >> down I would expect to bring it back up *with* its data - or, if I > >> couldn't, I would do a force-replace or replace and rebuild it from the > >> other replicas. I'm definitely not doing instance-store. So I don't > know if > >> that mitigates my need for a full 5 nodes. I would also consider losing > one > >> node to be "degraded" and would probably seek to fix that problem as > soon > >> as possible, so I wouldn't expect to be in that situation for long. I > would > >> probably tolerate a drop in performance during that time, too. (Not a > >> super-severe one, but 20-30 percent? Sure.) > >> > >> What do you folks think? > >> > >> -B. > >> > >> > >> _______________________________________________ > >> riak-users mailing list > >> riak-users@lists.basho.com > >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >> > >> > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130811/d350b1f1/attachment-0001.html > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
