All, A little bit of updates after a long vacation, I'm currently creating automated test scripts that randomly create/delete/update domain/account/user objects in random regions to trigger the sync-up and full scans regularly. Once they are completed, I'll post it in the github also and submit the review requests for this implementation.
Let me know if you have any comments. Thanks Alex Ough On Wed, Dec 18, 2013 at 3:39 PM, Alex Ough <alex.o...@sungard.com> wrote: > All, > > I updated the wiki after some logic changes, so please review them, > especially "Full Scan", which is newly introduced. > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Domain-Account-User+Sync+Up+Among+Multiple+Regions > > And I implemented this functionality in Java and you can get the pull > request of it here. > (This does not include the 'full scan' yet and I'm currently working > on this to finalize.) > https://github.com/alexoughsg/Albatross/pull/1 > > Especially, I really want to have your review on the "Full Scan" logic > to confirm if it does not miss any cases. > Thanks for your interest and your feedback will be very helpful. > Alex Ough > > On Tue, Nov 12, 2013 at 6:00 PM, Alex Ough <alex.o...@sungard.com> wrote: > > Good point, Chiradeep, > > > > I'm not sure if you reviewed my design doc in the wiki, but my design is > to > > just skip any actions for target resources that already took place by any > > means. > > But the issue is when conflict actions in the same resources (like > create & > > delete the same users) are enqueued in reversed orders, which is > hopefully > > rare. > > > > And to support consistency in the AP system, I'd like to provide a full > sync > > up, which will sync up all data in all region servers by selecting a > region > > as a master and force its data to other regions. > > > > Let me know what you think. > > Thanks > > Alex Ough > > > > > > On Tue, Nov 12, 2013 at 1:22 PM, Chiradeep Vittal > > <chiradeep.vit...@citrix.com> wrote: > >> > >> Missed this one. In a single region, the CloudStack DB is the master for > >> most operations. If the infra is not in the state the DB says it should > >> be, generally the approach is to whack it and make it conform. For some > >> exceptions (live migration/related use cases are exceptions) the DB is > the > >> slave -- the point is that the inconsistency that inevitably arise in an > >> AP system need a conflict resolution system. In a single region, the > >> default is to assume that the MySQL DB is correct and handle other cases > >> carefully. > >> > >> In a multi-region case, there is no master: disable an account in one > >> region, and it may not propagate to the other regions for many > hours/days. > >> You could add a user in one region and then add the same user in a > >> different region and conflict before the sync happens. > >> > >> This is of course not a problem unique to CloudStack -- people pay mucho > >> dinero for Global AD/LDAP sync. I don't think this is a problem for > >> CloudStack core to solve: I support the event-based mechanism for those > >> who want this facility. > >> > >> Distributed systems are hard and research continues to try and make > >> building these systems easier, but there are very few solutions for > global > >> state synchronization (Google Spanner comes to mind) > >> > >> -- > >> Chiradeep > >> > >> > >> On 11/8/13 4:53 PM, "Chip Childers" <chip.child...@gmail.com> wrote: > >> > >> >We are already (generally) AP for most infra changes really. I'd use > that > >> >model. Eventual consistency is better in this scenario. > >> > > >> >> On Nov 8, 2013, at 6:49 PM, Chiradeep Vittal > >> >><chiradeep.vit...@citrix.com> wrote: > >> >> > >> >> I'd also like to highlight that it isn't a trivial problem. > >> >> Let's say there's 3 regions: this means there are 3 copies of the > user > >> >> database that are geographically separated by network links that fail > >> >> quite often (orders of magnitude more than intra-DC networks). > >> >> > >> >> Here we run into the consequences of the CAP theorem [1]. > >> >> We can either have a CP or AP system: either approach makes some > >> >>tradeoffs: > >> >> 1. If we run a AP system, then the challenge is to resolve > conflicting > >> >> updates > >> >> 2. If we run a CP system, then the challenge is to detect partitions > >> >> reliably and disallow updates during partitions. > >> >> > >> >> [1] http://en.wikipedia.org/wiki/CAP_theorem > >> >> > >> >>> On 11/7/13 11:58 AM, "Chip Childers" <chipchild...@apache.org> > wrote: > >> >>> > >> >>> On Thu, Nov 7, 2013 at 2:37 PM, Chiradeep Vittal > >> >>> <chiradeep.vit...@citrix.com> wrote: > >> >>>> It may be an admin burden, but it has to be optional. There are > other > >> >>>> ways > >> >>>> to achieve global sync (e.g., LDAP/AD/Oauth). > >> >>>> A lot of service providers who run cloudstack have their own user > >> >>>> database > >> >>>> / portal. In their implementations the CloudStack database is not > the > >> >>>> master source of user records, but a slave. > >> >>> > >> >>> +1 to it being optional. > >> >> > >> > >> > > >
