Time for another update.... The crowbar is (finally) out, it made some splash at OSCON, but in case you missed it, you can find it here: https://github.com/dellcloudedge/crowbar
In case you really missed it - crowbar provides a bare metal to a fully deployed glance, nova and/or swift install system. It will take care of PXE booting the systems and walking them through the paces (discovery, hw-install and configure, os-install and configuration). Once the base OS is ready for the set of machines to be installed, clusters of compute or storage services are deployed and prep-ed for use. You can take crowbar for a spin in a virutal environment or on bare metal hw. take it for a spin, drop us a note. (p.s. it subsumes (by a lot) the previous swift-only recipes from this thread, and includes some bug fixes) On Mon, May 23, 2011 at 1:41 PM, andi abes <andi.a...@gmail.com> wrote: > > > It took a while, but finally: > https://github.com/dellcloudedge/openstack-swift > > Jay, I've added a swift-proxy-acct and swift-proxy (without account > management). > > This cookbook is an advanced "leak" of for swift, soon to be followed with > a leak of a nova cookbook. The full crowbar that was mentioned is on its > way... > > To use these recipes (with default settings) you just need to pick your > storage nodes and 1 or more proxies. Then assign the appropriate roles > (swift-storage swift-proxy or swift-proxy-acct) using the chef ui or a knife > command. Choose one of the nodes and assign it the swift-node-compute. and > the swift cluster is built (because of async nature of multi-node > deployments, it might require a few chef-client runs while the ring files > are generated and pushed around. > > have a spin. eager to hear comments. > > > > > > > > > On Mon, May 2, 2011 at 11:36 AM, andi abes <andi.a...@gmail.com> wrote: > >> Jay, >> >> hmmm, interesting point about account management in the proxy. Guess >> you're suggesting that you have 2 flavors of a proxy server - one with >> account management enabled and one without? >> Is the main concern here security - you'd have more controls on the >> account management servers? Or is this about something else? >> >> About ring-compute: >> so there are 2 concerns I was thinking about with rings - a) make sure the >> ring information is consistent across all the nodes in the cluster, and b) >> try not to lose the ring info. >> >> The main driver to have only 1 ring compute node was a). the main concern >> being guaranteeing consistency of the ring data among all nodes without >> causing too strong coupling to the underlying mechanisms used to build the >> ring. >> For example - if 2 new rings are created independently, then the order in >> which disks are added to the ring should be consistent (assuming that the >> disk/partition allocation algorithm is sensitive to ordering). Which implies >> that the query to chef should always return data in exactly the same order. >> If also would require that the ring building (and mandate that it will >> never be changed) does not use any heuristics that are time or machine >> dependent (I _think_ that right now that is the case, but I would rather not >> depend on it). >> >> I was thinking that these restrictions can be avoided easily by making >> sure that only 1 node computes the ring. To make sure that b) (don't lose >> the ring) is addressed - the ring is copied around. >> If the ring compute node fails, then any other node can be used to seed a >> new compute ring without any loss. >> Does that make sense? >> >> >> Right now I'm using a snapshot deb package built from bzr266. Changing the >> source of the bits is pretty esay... (and installing the deb includes the >> utilities you mentioned) >> >> >> Re: load balancers: >> What you're proposing makes perfect sense. Chef is pretty modular. So the >> swift configuration recipe focuses on setting up swift - not the whole >> environment. It would make sense to deploy some load balancer, firewall >> appliance etc in an environment. However, these would be add-ons to the >> basic swift configuration. >> A simple way to achieve this would be to have a recipe that would query >> the chef server for all nodes which have the swift-proxy role, and add them >> as internal addresses for the load balancer of your choice. >> (e.g. : >> http://wiki.opscode.com/display/chef/Search#Search-FindNodeswithaRoleintheExpandedRunList >> ) >> >> >> a. >> >> >> >> On Sun, May 1, 2011 at 10:14 AM, Jay Payne <lett...@gmail.com> wrote: >> >>> Andi, >>> >>> This looks great. I do have some thoughts/questions. >>> >>> If you are using 1.3, do you have a separate role for the management >>> functionality in the proxy? It's not a good idea to have all your >>> proxy servers running in management mode (unless you only have one >>> proxy). >>> >>> Why only 1 ring-compute node? If that node is lost or unavailable do >>> you loose your ring-builder files? >>> >>> When I create an environment I always setup utilities like st, >>> get-nodes, stats-report, and a simple functional test script on a >>> server to help troubleshoot and manage the cluster(s). >>> >>> Are you using packages or eggs to deploy the swift code? If your >>> using packages, are you building them yourself or using the ones from >>> launchpad? >>> >>> If you have more than three proxy servers, do you plan on using load >>> balancers? >>> >>> >>> Thanks >>> --J >>> >>> >>> >>> >>> On Sun, May 1, 2011 at 8:37 AM, andi abes <andi.a...@gmail.com> wrote: >>> > Judd, >>> > Sorry, today I won't be around. I'd love to hear feedback and >>> suggestions >>> > on what I have so far ( I'm not 100% sure when I can make the fully >>> > available, but I'm hoping this is very soon). I'm running with swift >>> 1.3 on >>> > ubuntu 10.10. >>> > I'm using the environment pattern in chef - when nodes search for >>> their >>> > peers a predicate comparing the node[:swift][:config][:environment] to >>> the >>> > corresponding value on the prospective peer. A "default" value is >>> assigned >>> > to this by the default recipe's attributes, so if only 1 cluster is >>> present, >>> > all nodes are eligible. For example, when proxy recipe creates the >>> memcached >>> > list of addresses, it searches for all the other nodes with the >>> swift-proxy >>> > role assigned to it anded with the above. >>> > Is that what you meant about having a classifier? >>> > To the basic roles you've described (base, storage and proxy) I've >>> added 1: >>> > ring-compute. This role should be assigned to only 1 node on top of >>> either >>> > the storage or proxy. This server will recompute the rings whenever the >>> set >>> > of storage servers change (new disks or machines added or existing ones >>> > removed). It uses information on the affected machines (ip, disk and >>> zone >>> > assignment) to update the ring. Once the ring is updated, it is rsynced >>> to >>> > all the other nodes in the cluster. >>> > [ this is achieved with a LWRP as I mentioned earlier, which first >>> parses >>> > the current ring info, then compares it to the current set of disks. It >>> > notifies chef if any changes were made, so that the rsync actions can >>> be >>> > triggered] >>> > At this point, all disks are added to all rings. Though it should be >>> easy to >>> > make this conditional on an attribute on the node/disk or some >>> heuristic. >>> > The 'base role' is also a bit more extensive than your description, but >>> > needs a bit more work. The recipe uses a swift_disk LWRP to ensure the >>> > partition table matches the configuration. The LWRP accepts an array >>> > describing the desired partition table, and allows using a :remaining >>> token >>> > to indicate using what's left (should only be used on the last >>> partition). >>> > At this point it, the recipe is pretty hard coded. It assumes that >>> /dev/sdb >>> > is dedicated to storage. it just requires a hard coded single >>> partition, >>> > that uses the whole disk. If it's not present, or different, the LWRP >>> > creates a BSD label, a single partition and xfs filesystem. Once this >>> is >>> > done, the available disk is 'published' as node attributes. If the >>> current >>> > state of the system matches the desired state, nothing is modified. >>> > The proxy and storage roles, do as you'd expect. install the relevant >>> > packages (including memcached on the proxy using the opscode recipe) >>> and >>> > plunk in the server/cluster specific info into the relevant config file >>> > templates. >>> > What's totally not yet addressed: >>> > - starting services >>> > - prep-ing the authentication >>> > - injecting disk configuration. >>> > This cookbook can be used by it self, but it is more powerful when used >>> in >>> > conjunction with the crowbar proposal mechanism. crowbar basically >>> allows >>> > you to define the qualifications for a system to be assigned a given >>> role, >>> > edit the automatic role assignment and configuration of the cluster and >>> then >>> > materialize the cluster based on these values by driving chef. Part of >>> the >>> > planned crowbar capability is performing automatic disk/zone >>> allocation, >>> > based on network topology and connectivity. In the mean time, the >>> allocation >>> > is done when the storage node is created. >>> > Currently, a proposal can provide values for the following: >>> > "cluster_hash", "cluster_admin_pw", "replicas", and user/group to run >>> swift >>> > as. I'm really curious to hear what other configuration parameters >>> might be >>> > useful >>> > I'm really curious to hear some feedback about this. >>> > and sorry about the timing, I'm in the boston area and it's finally >>> nice >>> > around here... hence other weekend plans. >>> > >>> > >>> > On Thu, Apr 28, 2011 at 11:25 PM, Judd Maltin <j...@newgoliath.com> >>> wrote: >>> >> >>> >> Hey andi, >>> >> I'm psyched to collaborate on this. I'm a busy guy, but I'm >>> dedicating >>> >> Sunday to this. So if you have time Sunday, that would be best to >>> catch up >>> >> via IRC, IM or voice. >>> >> Having a node classifier of some sort is critical. >>> >> -Judd >>> >> >>> >> Judd Maltin >>> >> +1 917 882 1270 >>> >> Happiness is a straight line and a goal. -fn >>> >> On Apr 28, 2011, at 11:59, andi abes <andi.a...@gmail.com> wrote: >>> >> >>> >> Judd, >>> >> Ok. Here are some of the thoughts I've had (and have mostly working, >>> but >>> >> hitting some swift snags..) Maybe we can collaborate on this? >>> >> Since Chef promotes idempotent operations and cookbooks, I put some >>> effort >>> >> in making sure that changes are only made if they're >>> required, particularly >>> >> around destructive or expensive operations. >>> >> The 2 main cases are: >>> >> - partitioning disks, which is obviously destructive >>> >> - building / rebuilding the ring files - rebalancing the ring is >>> >> relatively expensive. >>> >> for both cases I've built a LWRP which reads the current state of >>> affairs, >>> >> and decides if and what are the required changes. For disks, I'm using >>> ' >>> >> 'parted', which produces machine friendly output. For ring files I'm >>> using >>> >> the output from ring-builder. >>> >> the LWRP are driven by recipes which inspect databag - very similar to >>> >> your approach. However, they also utilize inventory information about >>> >> available resources created by crowbar in chef during the initial >>> bring up >>> >> of the deployment. >>> >> (As a side note - Dell has announced plans to opensource most of >>> crowbar, >>> >> but there's legalese involved) >>> >> >>> >> I'd be more than happy to elaborate and collaborate on this ! >>> >> >>> >> a >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> On Thu, Apr 28, 2011 at 11:35 AM, Judd Maltin <j...@newgoliath.com> >>> wrote: >>> >>> >>> >>> Hi Andi, >>> >>> >>> >>> Indeed, the swift recipes hadn't been updated since mid 2010, so I >>> pushed >>> >>> forward with my own. >>> >>> >>> >>> Thanks! >>> >>> -judd >>> >>> >>> >>> On Thu, Apr 28, 2011 at 10:03 AM, andi abes <andi.a...@gmail.com> >>> wrote: >>> >>>> >>> >>>> Judd, >>> >>>> this is a great idea... actually so great, that some folks @Dell >>> and >>> >>>> OpsCode, me included, have been working on it. >>> >>>> Have a peek on >>> >>>> : >>> https://github.com/opscode/openstack-cookbooks/tree/master/cookbooks >>> >>>> This effort is also being included into Crowbar (take a peek >>> >>>> here: http://robhirschfeld.com/tag/crowbar/) which adds the steps >>> needed to >>> >>>> start with bare metal (rather than installed OS), then using chef to >>> get to >>> >>>> a working Open stack deployment. >>> >>>> (if you're at the design meeting, there are demos scheduled). >>> >>>> That said - I'm updating the swift cookbook, and hope to update >>> github >>> >>>> soon. >>> >>>> a. >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> On Wed, Apr 27, 2011 at 9:55 PM, Jay Payne <lett...@gmail.com> >>> wrote: >>> >>>>> >>> >>>>> Judd, >>> >>>>> >>> >>>>> I'm not that familiar with Chef (I'll do some research) but I have >>> a >>> >>>>> couple of questions and some thoughts: >>> >>>>> >>> >>>>> 1. Is this for a multi-server environment? >>> >>>>> 2. Are all your proxy nodes going to have >>> "allow_account_management = >>> >>>>> true" in the configs? It might be a good idea to have a second >>> proxy >>> >>>>> config for account management only >>> >>>>> 3. Have you looked at using swauth instead of auth? >>> >>>>> 4. Have you thought about an admin or client node that has >>> utilities >>> >>>>> on it like st and stats-report? >>> >>>>> 5. How where will you do on-going ring management or changes? >>> >>>>> 6. I would think about including some type of functional test at >>> the >>> >>>>> end of the deployment process to verify everything was created >>> >>>>> properly and that all nodes can communicate. >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> --J >>> >>>>> >>> >>>>> On Wed, Apr 27, 2011 at 6:18 PM, Judd Maltin <j...@newgoliath.com> >>> >>>>> wrote: >>> >>>>> > Hi Folks, >>> >>>>> > >>> >>>>> > I've been hacking away at creating an automated deployment system >>> for >>> >>>>> > Swift >>> >>>>> > using Chef. I'd like to drop a design idea on you folks (most of >>> >>>>> > which I've >>> >>>>> > already implemented) and get feedback from this esteemed group. >>> >>>>> > >>> >>>>> > My end goal is to have a "manifest" (apologies to Puppet) which >>> will >>> >>>>> > define >>> >>>>> > an entire swift cluster, deploy it automatically, and allow edits >>> to >>> >>>>> > the >>> >>>>> > ingredients to manage the cluster. In this case, a "manifest" is >>> a >>> >>>>> > combination of a chef databag describing the swift settings, and >>> a >>> >>>>> > spiceweasel infrastructure.yaml file describing the OS >>> configuration. >>> >>>>> > >>> >>>>> > Ingredients: >>> >>>>> > - swift cookbook with base, proxy and server recipes. proxy >>> nodes >>> >>>>> > also >>> >>>>> > (provisionally) contain auth services. storage nodes handle >>> object, >>> >>>>> > container and account services. >>> >>>>> > -- Base recipe handles common package install, OS user creation. >>> >>>>> > Sets up >>> >>>>> > keys. >>> >>>>> > -- Proxy recipe handles proxy nodes: network config, package >>> install, >>> >>>>> > memcache config, proxy and auth package config, user creation, >>> ring >>> >>>>> > management (including builder file backup), user management >>> >>>>> > -- Storage recipe handles storage nodes: network config, storage >>> >>>>> > device >>> >>>>> > config, package install, ring management. >>> >>>>> > >>> >>>>> > - chef databag that describes a swift cluster (eg: >>> >>>>> > mycluster_databag.json) >>> >>>>> > -- proxy config settings >>> >>>>> > -- memcached settings >>> >>>>> > -- settings for all rings and devices >>> >>>>> > -- basic user settings >>> >>>>> > -- account management >>> >>>>> > >>> >>>>> > - chef "spiceweasel" file that auto-vivifies the infrastructure: >>> (eg: >>> >>>>> > mycluster_infra.yaml) >>> >>>>> > -- uploads cookbooks >>> >>>>> > -- uploads roles >>> >>>>> > -- uploads the cluster's databag >>> >>>>> > -- kicks off node provisioning by requesting from infrastructure >>> API >>> >>>>> > (ec2 or >>> >>>>> > what have you) the following: >>> >>>>> > --- chef roles applied (role[swift:proxy] or role[swift:storage]) >>> >>>>> > --- server flavor >>> >>>>> > --- storage device configs >>> >>>>> > --- hostname >>> >>>>> > --- proxy and storage network details >>> >>>>> > >>> >>>>> > By calling this spiceweasel file, the infrastructure can leap >>> into >>> >>>>> > existence. >>> >>>>> > >>> >>>>> > I'm more or less done with all this stuff - and I'd really >>> appreciate >>> >>>>> > conceptual feedback before I take out all the non-sense code I >>> have >>> >>>>> > in the >>> >>>>> > files and publish. >>> >>>>> > >>> >>>>> > Many thanks! Happy spring, northern hemispherians! >>> >>>>> > -judd >>> >>>>> > >>> >>>>> > Judd Maltin >>> >>>>> > T: 917-882-1270 >>> >>>>> > F: 501-694-7809 >>> >>>>> > A loving heart is never wrong. >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > _______________________________________________ >>> >>>>> > Mailing list: https://launchpad.net/~openstack >>> >>>>> > Post to : openstack@lists.launchpad.net >>> >>>>> > Unsubscribe : https://launchpad.net/~openstack >>> >>>>> > More help : https://help.launchpad.net/ListHelp >>> >>>>> > >>> >>>>> > >>> >>>>> >>> >>>>> _______________________________________________ >>> >>>>> Mailing list: https://launchpad.net/~openstack >>> >>>>> Post to : openstack@lists.launchpad.net >>> >>>>> Unsubscribe : https://launchpad.net/~openstack >>> >>>>> More help : https://help.launchpad.net/ListHelp >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Judd Maltin >>> >>> T: 917-882-1270 >>> >>> F: 501-694-7809 >>> >>> A loving heart is never wrong. >>> >>> >>> >>> >>> >>> >>> >> >>> > >>> > >>> >> >> > >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp