Added global support for section position
Project: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/repo Commit: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/commit/cfa914de Tree: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/tree/cfa914de Diff: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/diff/cfa914de Branch: refs/heads/master Commit: cfa914ded11bb466e8bf57793d4fb811f767a049 Parents: 95eb8a4 Author: Duncan Godwin <duncan.god...@cloudsoftcorp.com> Authored: Thu Mar 3 13:46:09 2016 +0000 Committer: Duncan Godwin <duncan.god...@cloudsoftcorp.com> Committed: Mon Mar 7 16:54:43 2016 +0000 ---------------------------------------------------------------------- _plugins/page_structure.rb | 114 ++++++++- _plugins/site_structure.rb | 13 +- guide/ops/locations/1.1_cloud-setup.md | 86 ------- guide/ops/locations/1_clouds.md | 239 ------------------ .../2_inheritance-and-named-locations.md | 78 ------ guide/ops/locations/3.1_ssh-keys.md | 86 ------- guide/ops/locations/3_localhost.md | 31 --- guide/ops/locations/4_byon.md | 72 ------ guide/ops/locations/6_more-locations.md | 52 ---- guide/ops/locations/byon.md | 74 ++++++ guide/ops/locations/cloud-specific-setup.md | 88 +++++++ guide/ops/locations/clouds.md | 241 +++++++++++++++++++ guide/ops/locations/index.md | 2 +- .../inheritance-and-named-locations.md | 80 ++++++ guide/ops/locations/localhost.md | 33 +++ guide/ops/locations/more-locations.md | 17 +- guide/ops/locations/ssh-keys.md | 13 +- 17 files changed, 648 insertions(+), 671 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/_plugins/page_structure.rb ---------------------------------------------------------------------- diff --git a/_plugins/page_structure.rb b/_plugins/page_structure.rb index 9579397..2833be3 100644 --- a/_plugins/page_structure.rb +++ b/_plugins/page_structure.rb @@ -12,19 +12,112 @@ module PageStructureUtils class ChildPage def initialize(yaml, content) - @page_yaml=yaml - @page_content=content - end - def content() - @page_content - end - def yaml() - @page_yaml + @yaml=yaml + @content=content end + attr_accessor :yaml + attr_accessor :content def to_s # called with print / puts - "YAML : #{@page_yaml}, Content : #{@page_content}" + "YAML : #{@yaml}" #, Content : #{@content}" end + + ## + # Sort a list of children by their YAML containing section positions. Do this with Gem:Version + # + # + def self.sortBySectionPositions(yaml) + $major = "1" + $minor = 1 + # first check all the child pages are numbered, if not, number them in the order they are + yaml.each do |i| + if i.yaml['section_position'] == nil + i.yaml['section_position'] = $major+"."+$minor.to_s + $minor += 1 + else + # Store any major, start incrementing minor + $major = i.yaml['section_position'].to_s + $minor = 1 + end + end + + # return the comparison between the versions + yaml.sort{ |x,y| Gem::Version.new(x.yaml['section_position'].to_s) <=> Gem::Version.new(y.yaml['section_position'].to_s) } + end + ## + # Sorts a list of yaml children, if there's no numbering, use the YAML order to create a numbering + # + def self.sortYAMLSectionPositions(yaml) +# puts "a > "+yaml.to_s + hashArray = [] + $major = "1" + $minor = 1 + # first check all the child pages are numbered, if not, number them in the order they are + yaml.each do |i| + hash = {} + # if it's a string, convert it to a hash + if i.instance_of? String + hash = { "path" => i } + else + hash = i + end + if i['section_position'] == nil + hash['section_position'] = $major+"."+$minor.to_s + $minor += 1 + else + # Store any major, start incrementing minor + $major = i['section_position'].to_s + $minor = 1 + end + hashArray << hash + end + # return the comparison between the versions (NB: sort! for in-place sorting) + hashArray.sort!{ |x,y| Gem::Version.new(x['section_position'].to_s) <=> Gem::Version.new(y['section_position'].to_s) } +# puts "2 > "+hashArray.to_s + end + + ## + # This function looks at all the *.md files at the YAML in the headers and produces a list of children ordered by section_position + # + # + def self.parseChildYAMLFromParent(page) + # get the base directory of the current file + $baseFile = Dir.pwd+page['dir'] + # list all of the files in that directory + $listings = Dir[$baseFile+"/*.md"] + $allPages = [] + + for $listing in $listings + + # read the file + $fileContent = IO.read($listing) + # try and split of any YAML + $partitionedFileContent = $fileContent.split('---'); + # if there's potentially partitioned YAML try and parse it + if $partitionedFileContent.size > 2 + # try and parse the YAML + yamlContent = YAML.load($partitionedFileContent[1]) + # if we can, use it (section_type needs to be one of the allowed) + if yamlContent != nil && yamlContent != false && yamlContent['section_type'] != nil && yamlContent['section_type'] != "default" + + if yamlContent['section_type'] == nil + # if no section position has been specified, put it at the end + yamlContent['section_position'] = Integer::MAX + end + # if there's YAML, check it has the section_position tag and put it into child pages + ($allPages ||= []) << ChildPage.new(yamlContent, $partitionedFileContent[2]) + end + end + end + $allPages = sortBySectionPositions($allPages) + # return the combined content + $allPages + end + + ## + # This function looks in a parent folder for all files in the format <Number>_<title>.md + # + # def self.parseChildPagesFromParent(page) # get the base directory of the current file $baseFile = Dir.pwd+page['dir'] @@ -75,7 +168,8 @@ module PageStructureUtils @text = text.strip end def render(context) - $childPages = ChildPage.parseChildPagesFromParent(context['page']) +# $childPages = ChildPage.parseChildPagesFromParent(context['page']) + $childPages = ChildPage.parseChildYAMLFromParent(context['page']) $content = "" for $childPage in $childPages #append the content http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/_plugins/site_structure.rb ---------------------------------------------------------------------- diff --git a/_plugins/site_structure.rb b/_plugins/site_structure.rb index 88331b8..5f2d42b 100644 --- a/_plugins/site_structure.rb +++ b/_plugins/site_structure.rb @@ -227,8 +227,8 @@ module SiteStructure # see: page_structure.rb # # require show_inline_children to be set to true to show in the menu - if page['show_inline_children'] == true - $childPages = PageStructureUtils::ChildPage.parseChildPagesFromParent(page) + if page['check_directory_for_children'] == true + $childPages = PageStructureUtils::ChildPage.parseChildYAMLFromParent(page) # add the child pages in before the site structure has been created for $childPage in $childPages if $childPage.yaml() != nil && $childPage.yaml() != false @@ -241,6 +241,14 @@ module SiteStructure end end + ## + # This sorts child pages by the YAML property section_position. This uses a versioning format so 1.1.0 > 1.0.4 + # Any sections missing numbers will use their current position to determine their order + # + if data['children'] + data['children'] = PageStructureUtils::ChildPage.sortYAMLSectionPositions(data['children']) + end + data['path'] = page.path if item['href_path'] href_page = find_page_with_path_absolute_or_relative_to(site, render_liquid(site, page, item['href_path']), parent, structure_processed_pages) @@ -329,6 +337,7 @@ module SiteStructure end if (data['children']) + data['menu'] = [] puts "children of #{data['path']} - #{data['children']}" if @@verbose data['children'].each do |child| http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/1.1_cloud-setup.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/1.1_cloud-setup.md b/guide/ops/locations/1.1_cloud-setup.md deleted file mode 100644 index 1f60b5b..0000000 --- a/guide/ops/locations/1.1_cloud-setup.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -section: Cloud Specific Setup ---- - -### Cloud Specific Setup - -To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity" and -"credential" in Brooklyn terminology. - -For private clouds (and for some clouds being targeted using a standard API), the "endpoint" -must also be specified, which is the cloud's URL. - -The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring -different clouds. - - -#### AWS - -##### Credentials - -AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential -respectively. - -These keys are the way for any programmatic mechanism to access the AWS API. - -To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws) -and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html). - -An example of the expected format is shown below: - - brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST - brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm - - -##### Tidying up after jclouds - -Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see -https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group -being deleted until there are no VMs using it. However, there is eventual consistency for -recording which VMs still reference those security groups. After deleting the VM, it can sometimes -take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but -does not block for longer. - -There is utility written by Cloudsoft for deleting these unused resources: -http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds. - - -#### Google Compute Engine - -##### Credentials - -Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a private key -as the credential. - -To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google). - -An example of the expected format is shown below (note the credential is one long line, -with `\n` to represent the new line characters): - - brooklyn.location.jclouds.google-compute-engine.identity=123456789...@developer.gserviceaccount.com - brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY----- - - -##### Quotas - -GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas). - -It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en). - - -##### Networks - -GCE accounts often have a limit to the number of networks that can be created. One work around -is to manually create a network with the required open ports, and to refer to that named network -in Brooklyn's location configuration. - -To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1). - -For example, for dev/demo purposes an "everything" network could be created that opens all ports. - -| Name | everything | -| Description | opens all tcp ports | -| Source IP Ranges | 0.0.0.0/0 | -| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 | - - http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/1_clouds.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/1_clouds.md b/guide/ops/locations/1_clouds.md deleted file mode 100644 index 13112cd..0000000 --- a/guide/ops/locations/1_clouds.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -section: Clouds ---- - -### Clouds - -For most cloud provisioning tasks, Brooklyn uses -<a href="http://jclouds.org";>Apache jclouds</a>. -The identifiers for some of the most commonly used jclouds-supported clouds are -(or [see the full list](http://jclouds.apache.org/reference/providers/)): - -* `jclouds:aws-ec2:<region>`: Amazon EC2, where `:<region>` might be `us-east-1` or `eu-west-1` (or omitted) -* `jclouds:softlayer:<region>`: IBM Softlayer, where `:<region>` might be `dal05` or `ams01` (or omitted) -* `jclouds:google-compute-engine`: Google Compute Engine -* `jclouds:openstack-nova:<endpoint>`: OpenStack, where `:<endpoint>` is the access URL (required) -* `jclouds:cloudstack:<endpoint>`: Apache CloudStack, where `:<endpoint>` is the access URL (required) - -For any of these, of course, Brooklyn needs to be configured with an `identity` and a `credential`: - -{% highlight yaml %} -location: - jclouds:aws-ec2: - identity: ABCDEFGHIJKLMNOPQRST - credential: s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l -{% endhighlight %} - -The above YAML can be embedded directly in blueprints, either at the root or on individual services. -If you prefer to keep the credentials separate, you can instead store them as a [catalog entry]({{ site.path.guide }}/ops/catalog/index.html#locations-in-catalog) or set them in `brooklyn.properties` -in the `jclouds.<provider>` namespace: - -{% highlight bash %} -brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST -brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l -{% endhighlight %} - -And in this case you can reference the location in YAML with `location: jclouds:aws-ec2`. - -The Getting Started [template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties) contains more examples -of configuring cloud endpoints, including information on credential types used in different clouds. - -#### OS Initial Login and Setup - -Once a machine is provisioned, Brooklyn will normally attempt to log in via SSH and configure the machine sensibly. - -The credentials for the initial OS log on are typically discovered from the cloud, -but in some environments this is not possible. -The keys `loginUser` and either `loginUser.password` or `loginUser.privateKeyFile` can be used to force -Brooklyn to use specific credentials for the initial login to a cloud-provisioned machine. - -(This custom login is particularly useful when using a custom image templates where the cloud-side account -management logic is not enabled. For example, a vCloud (vCD) template can have guest customization that will change -the root password. This setting tells AMP to only use the given password, rather than the initial -randomly generated password that vCD returns. Without this property, there is a race for such templates: -does Brooklyn manage to create the admin user before the guest customization changes the login and reboots, -or is the password reset first (the latter means Brooklyn can never ssh to the VM). With this property, -Brooklyn will always wait for guest customization to complete before it is able to ssh at all. In such -cases, it is also recommended to use `useJcloudsSshInit=false`.) - -Following a successful logon, Brooklyn performs the following steps to configure the machine: - -1. creates a new user with the same name as the user `brooklyn` is running as locally - (this can be overridden with `user`, below). - -1. install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine, - to make it easy for the operator to `ssh` in - (override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair will be generated; - if there is a passphrase on the key, this must be supplied) - -1. give `sudo` access to the newly created user (override with `grantUserSudo: false`) - -1. disable direct `root` login to the machine - -These steps can be skipped or customized as described below. - -#### jclouds Config Keys - -The following is a subset of the most commonly used configuration keys used to customize -cloud provisioning. -For more keys and more detail on the keys below, see -{% include java_link.html class_name="JcloudsLocationConfig" package_path="org/apache/brooklyn/location/jclouds" project_subpath="locations/jclouds" %}. - -###### VM Creation - -- Most providers require exactly one of either `region` (e.g. `us-east-1`) or `endpoint` (the URL, usually for private cloud deployments) - -- Hardware requirements can be specified, including - `minRam`, `minCores`, and `os64Bit`; or as a specific `hardwareId` - -- VM image constraints can be set using `osFamily` (e.g. `Ubuntu`, `CentOS`, `Debian`, `RHEL`) - and `osVersionRegex`, or specific VM images can be specified using `imageId` or `imageNameRegex` - -- Specific VM images can be specified using `imageId` or `imageNameRegex` - -- Specific Security Groups can be specified using `securityGroups`, as a list of strings (the existing security group names), - or `inboundPorts` can be set, as a list of numeric ports (selected clouds only) - -- A specific existing key pair known at the cloud to use can be specified with `keyPair` - (selected clouds only) - -- A specific VM name (often the hostname) base to be used can be specified by setting `groupId`. - By default, this name is constructed based on the entity which is creating it, - including the ID of the app and of the entity. - (As many cloud portals let you filter views, this can help find a specific entity or all machines for a given application.) - For more sophisticated control over host naming, you can supply a custom - {% include java_link.html class_name="CloudMachineNamer" package_path="org/apache/brooklyn/core/location/cloud/names" project_subpath="core" %}, - for example - `cloudMachineNamer: CustomMachineNamer`. - {% include java_link.html class_name="CustomMachineNamer" package_path="org/apache/brooklyn/core/location/cloud/names" project_subpath="core" %} - will use the entity's name or following a template you supply. - On many clouds, a random suffix will be appended to help guarantee uniqueness; - this can be removed by setting `vmNameSaltLength: 0` (selected clouds only). - <!-- TODO jclouds softlayer includes a 3-char hex suffix --> - -- A DNS domain name where this host should be placed can be specified with `domainName` - (in selected clouds only) - -- User metadata can be attached using the syntax `userMetadata: { key: value, key2: "value 2" }` - (or `userMetadata=key=value,key2="value 2"` in a properties file) - -- By default, several pieces of user metadata are set to correlate VMs with Brooklyn entities, - prefixed with `brooklyn-`. - This user metadata can be omitted by setting `includeBrooklynUserMetadata: false`. - -- You can specify the number of attempts Brooklyn should make to create - machines with `machineCreateAttempts` (jclouds only). This is useful as an efficient low-level fix - for those occasions when cloud providers give machines that are dead on arrival. - You can of course also resolve it at a higher level with a policy such as - {% include java_link.html class_name="ServiceRestarter" package_path="org/apache/brooklyn/policy/ha" project_subpath="policy" %}. - -- If you want to investigate failures, set `destroyOnFailure: false` - to keep failed VM's around. (You'll have to manually clean them up.) - The default is false: if a VM fails to start, or is never ssh'able, then the VM will be terminated. - - ###### OS Setup - -- `user` and `password` can be used to configure the operating user created on cloud-provisioned machines - -- The `loginUser` config key (and subkeys) control the initial user to log in as, - in cases where this cannot be discovered from the cloud provider - -- Private keys can be specified using `privateKeyFile`; - these are not copied to provisioned machines, but are required if using a local public key - or a pre-defined `authorized_keys` on the server. - (For more information on SSH keys, see [here](ssh-keys.html).) - -- If there is a passphrase on the key file being used, you must supply it to Brooklyn for it to work, of course! - `privateKeyPassphrase` does the trick (as in `brooklyn.location.jclouds.privateKeyPassphrase`, or other places - where `privateKeyFile` is valid). If you don't like keys, you can just use a plain old `password`. - -- Public keys can be specified using `publicKeyFile`, - although these can usually be omitted if they follow the common pattern of being - the private key file with the suffix `.pub` appended. - (It is useful in the case of `loginUser.publicKeyFile`, where you shouldn't need, - or might not even have, the private key of the `root` user when you log in.) - -- Provide a list of URLs to public keys in `extraSshPublicKeyUrls`, - or the data of one key in `extraSshPublicKeyData`, - to have additional public keys added to the `authorized_keys` file for logging in. - (This is supported in most but not all locations.) - -- Use `dontCreateUser` to have Brooklyn run as the initial `loginUser` (usually `root`), - without creating any other user. - -- A post-provisioning `setup.script` can be specified (as a URL) to run an additional script, - before making the `Location` available to entities, - optionally also using `setup.script.vars` (set as `key1:value1,key2:value2`) - -- Use `openIptables: true` to automatically configure `iptables`, to open the TCP ports required by - the software process. One can alternatively use `stopIptables: true` to entirely stop the - iptables service. - -- Use `installDevUrandom: true` to fall back to using `/dev/urandom` rather than `/dev/random`. This setting - is useful for cloud VMs where there is not enough random entropy, which can cause `/dev/random` to be - extremely slow (causing `ssh` to be extremely slow to respond). - -- Use `useJcloudsSshInit: false` to disable the use of the native jclouds support for initial commands executed - on the VM (e.g. for creating new users, setting root passwords, etc.). Instead, Brooklyn's ssh support will - be used. Timeouts and retries are more configurable within Brooklyn itself. Therefore this option is particularly - recommended when the VM startup is unusual (for example, if guest customizations will cause reboots and/or will - change login credentials). - -- Use `brooklyn.ssh.config.noDeleteAfterExec: true` to keep scripts on the server after execution. - The contents of the scripts and the stdout/stderr of their execution are available in the Brooklyn web console, - but sometimes it can also be useful to have them on the box. - This setting prevents scripts executed on the VMs from being deleted on completion. - Note that some scripts run periodically so this can eventually fill a disk; it should only be used for dev/test. - -###### Custom template options - -jclouds supports many additional options for configuring how a virtual machine is created and deployed, many of which -are for cloud-specific features and enhancements. Brooklyn supports some of these, but if what you are looking for is -not supported directly by Brooklyn, we instead offer a mechanism to set any parameter that is supported by the jclouds -template options for your cloud. - -Part of the process for creating a virtual machine is the creation of a jclouds `TemplateOptions` object. jclouds -providers extends this with extra options for each cloud - so when using the AWS provider, the object will be of -type `AWSEC2TemplateOptions`. By [examining the source code](https://github.com/jclouds/jclouds/blob/jclouds-1.9.0/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/AWSEC2TemplateOptions.java), -you can see all of the options available to you. - -The `templateOptions` config key takes a map. The keys to the map are method names, and Brooklyn will find the method on -the `TemplateOptions` instance; it then invokes the method with arguments taken from the map value. If a method takes a -single parameter, then simply give the argument as the value of the key; if the method takes multiple parameters, the -value of the key should be an array, containing the argument for each parameter. - -For example, here is a complete blueprint that sets some AWS EC2 specific options: - - location: AWS_eu-west-1 - services: - - type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess - provisioningProperties: - templateOptions: - subnetId: subnet-041c8373 - mapNewVolumeToDeviceName: ["/dev/sda1", 100, true] - securityGroupIds: ['sg-4db68928'] - -Here you can see that we set three template options: - -- `subnetId` is an example of a single parameter method. Brooklyn will effectively try to run the statement - `templateOptions.subnetId("subnet-041c88373");` -- `mapNewVolumeToDeviceName` is an example of a multiple parameter method, so the value of the key is an array. - Brooklyn will effectively true to run the statement `templateOptions.mapNewVolumeToDeviceName("/dev/sda1", 100, true);` -- `securityGroupIds` demonstrates an ambiguity between the two types; Brooklyn will first try to parse the value as - a multiple parameter method, but there is no method that matches this parameter. In this case, Brooklyn will next try - to parse the value as a single parameter method which takes a parameter of type `List`; such a method does exist so - the operation will succeed. - -If the method call cannot be matched to the template options available - for example if you are trying to set an AWS EC2 -specific option but your location is an OpenStack cloud - then a warning is logged and the option is ignored. - - - - -See the following resources for more information: - -- [AWS VPC issues which may affect users with older AWS accounts](vpc-issues.html) -- [Amazon EC2 and Amazon Virtual Private Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html#vpc-only-instance-types) -- [Your Default VPC and Subnets](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html) -- [Amazon VPC FAQs](http://aws.amazon.com/vpc/faqs/#Default_VPCs) - \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/2_inheritance-and-named-locations.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/2_inheritance-and-named-locations.md b/guide/ops/locations/2_inheritance-and-named-locations.md deleted file mode 100644 index 3e7b50a..0000000 --- a/guide/ops/locations/2_inheritance-and-named-locations.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -section: Inheritance and Named Locations -title: Named Locations ---- - -### Inheritance and Named Locations - -Named locations can be defined for commonly used groups of properties, -with the syntax `brooklyn.location.named.your-group-name.` -followed by the relevant properties. -These can be accessed at runtime using the syntax `named:your-group-name` as the deployment location. - -Some illustrative examples using named locations and -showing the syntax and properties above are as follows: - -{% highlight bash %} -# Production pool of machines for my application (deploy to named:prod1) -brooklyn.location.named.prod1=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}") -brooklyn.location.named.prod1.user=produser1 -brooklyn.location.named.prod1.privateKeyFile=~/.ssh/produser_id_rsa -brooklyn.location.named.prod1.privateKeyPassphrase=s3cr3tCOMPANYpassphrase - -# AWS using my company's credentials and image standard, then labelling images so others know they're mine -brooklyn.location.named.company-jungle=jclouds:aws-ec2:us-west-1 -brooklyn.location.named.company-jungle.identity=BCDEFGHIJKLMNOPQRSTU -brooklyn.location.named.company-jungle.privateKeyFile=~/.ssh/public_clouds/company_aws_id_rsa -brooklyn.location.named.company-jungle.imageId=ami-12345 -brooklyn.location.named.company-jungle.minRam=2048 -brooklyn.location.named.company-jungle.userMetadata=application=my-jungle-app,owner="Bob Johnson" -brooklyn.location.named.company-jungle.machineCreateAttempts=2 - -brooklyn.location.named.AWS\ Virginia\ Large\ Centos = jclouds:aws-ec2 -brooklyn.location.named.AWS\ Virginia\ Large\ Centos.region = us-east-1 -brooklyn.location.named.AWS\ Virginia\ Large\ Centos.imageId=us-east-1/ami-7d7bfc14 -brooklyn.location.named.AWS\ Virginia\ Large\ Centos.user=root -brooklyn.location.named.AWS\ Virginia\ Large\ Centos.minRam=4096 -{% endhighlight %} - -Named locations can refer to other named locations using `named:xxx` as their value. -These will inherit the configuration and can override selected keys. -Properties set in the namespace of the provider (e.g. `b.l.jclouds.aws-ec2.KEY=VALUE`) -will be inherited by everything which extends AWS -Sub-prefix strings are also inherited up to `brooklyn.location.*`, -except that they are filtered for single-word and other -known keys -(so that we exclude provider-scoped properties when looking at sub-prefix keys). -The precedence for configuration defined at different levels is that the value -defined in the most specific context will apply. - -This is rather straightforward and powerful to use, -although it sounds rather more complicated than it is! -The examples below should make it clear. -You could use the following to install -a public key on all provisioned machines, -an additional public key in all AWS machines, -and no extra public key in `prod1`: - -<!-- tested in JcloudsLocationResolverTest --> -{% highlight bash %} -brooklyn.location.extraSshPublicKeyUrls=http://me.com/public_key -brooklyn.location.jclouds.aws-ec2.extraSshPublicKeyUrls="[ \"http://me.com/public_key\";, \"http://me.com/aws_public_key\"; ]" -brooklyn.location.named.prod1.extraSshPublicKeyUrls= -{% endhighlight %} - -And in the example below, a config key is repeatedly overridden. -Deploying `location: named:my-extended-aws` will result in an `aws-ec2` machine in `us-west-1` (by inheritance) -with `VAL6` for `KEY`: - -{% highlight bash %} -brooklyn.location.KEY=VAL1 -brooklyn.location.jclouds.KEY=VAL2 -brooklyn.location.jclouds.aws-ec2.KEY=VAL3 -brooklyn.location.jclouds.aws-...@us-west-1.key=VAL4 -brooklyn.location.named.my-aws=jclouds:aws-ec2:us-west-1 -brooklyn.location.named.my-aws.KEY=VAL5 -brooklyn.location.named.my-extended-aws=named:my-aws -brooklyn.location.named.my-extended-aws.KEY=VAL6 -{% endhighlight %} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/3.1_ssh-keys.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/3.1_ssh-keys.md b/guide/ops/locations/3.1_ssh-keys.md deleted file mode 100644 index 4adda4b..0000000 --- a/guide/ops/locations/3.1_ssh-keys.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -section: SSH Keys ---- - -### SSH Keys - -SSH keys are one of the simplest and most secure ways to access remote servers. -They consist of two parts: - -* A private key (e.g. `id_rsa`) which is known only to one party or group - -* A public key (e.g. `id_rsa.pub`) which can be given to anyone and everyone, - and which can be used to confirm that a party has a private key - (or has signed a communication with the private key) - -In this way, someone -- such as you -- can have a private key, -and can install a public key on a remote machine (in an `authorized_keys` file) -for secure automated access. -Commands such as `ssh` (and Brooklyn) can log in without -revealing the private key to the remote machine, -the remote machine can confirm it is you accessing it (if no one else has the private key), -and no one snooping on the network can decrypt of any of the traffic. - - -#### Creating an SSH Key - -If you don't have an SSH key, create one with: - -{% highlight bash %} -$ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa -{% endhighlight %} - - -#### Localhost Setup - -If you want to deploy to `localhost`, ensure that you have a public and private key, -and that your key is authorized for ssh access: - -{% highlight bash %} -# _Appends_ id_rsa.pub to authorized_keys. Other keys are unaffected. -$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys -{% endhighlight %} - -Now verify that your setup by running the command: `ssh localhost echo hello world` - -If your setup is correct, you should see `hello world` printed back at you. - -On the first connection, you may see a message similar to this: - -<pre> -The authenticity of host 'localhost (::1)' can't be established. -RSA key fingerprint is 7b:e3:8e:c6:5b:2a:05:a1:7c:8a:cf:d1:6a:83:c2:ad. -Are you sure you want to continue connecting (yes/no)? -</pre> - -Simply answer 'yes' and then repeat the command again. - -If this isn't the case, see below. - - - - -#### Potential Problems - -* **MacOS user?** In addition to the above, enable "Remote Login" in "System Preferences > Sharing". - -* **Got a passphrase?** Set `brooklyn.location.localhost.privateKeyPassphrase` - as described [here](index.html#os-setup). - If you're not sure, or you don't know what a passphrase is, you can test this by executing `ssh-keygen -y`. - If it does *not* ask for a passphrase, then your key has no passphrase. - If your key does have a passphrase, you can remove it by running `ssh-keygen -p`. - -* Check that you have an `~/.ssh/id_rsa` file (or `id_dsa`) and a corresponding public key with a `.pub` extension; - if not, create one as described above - -* `~/.ssh/` or files in that directory may have permissions they shouldn't: - they should be visible only to the user (apart from public keys), - both on the source machine and the target machine. - You can verify this with `ls -l ~/.ssh/`: lines should start with `-rw-------` or `-r--------` (or `-rwx------` for directories). - If it does not, execute `chmod go-rwx ~/.ssh ~/.ssh/*`. - -* Sometimes machines are configured with different sets of support SSL/TLS versions and ciphers; - if command-line `ssh` and `scp` work, but Brooklyn/java does not, check the versions enabled in Java and on both servers. - -* Missing entropy: creating and using ssh keys requires randomness available on the servers, - usually in `/dev/random`; see [here]({{ site.path.website }}/documentation/increase-entropy.html) for more information http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/3_localhost.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/3_localhost.md b/guide/ops/locations/3_localhost.md deleted file mode 100644 index 13e1435..0000000 --- a/guide/ops/locations/3_localhost.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -section: Localhost ---- - -### Localhost - -If passwordless ssh login to `localhost` and passwordless `sudo` is enabled on your -machine, you should be able to deploy blueprints with no special configuration, -just by specifying `location: localhost` in YAML. - -If you use a passpharse or prefer a different key, these can be configured as follows: - -{% highlight bash %} -brooklyn.location.localhost.privateKeyFile=~/.ssh/brooklyn_key -brooklyn.location.localhost.privateKeyPassphrase=s3cr3tPASSPHRASE -{% endhighlight %} - -If you encounter issues or for more information, see [SSH Keys Localhost Setup](ssh-keys.html#localhost-setup). - -If you are normally prompted for a password when executing `sudo` commands, passwordless `sudo` must also be enabled. To enable passwordless `sudo` for your account, a line must be added to the system `/etc/sudoers` file. To edit the file, use the `visudo` command: -{% highlight bash %} -sudo visudo -{% endhighlight %} -Add this line at the bottom of the file, replacing `username` with your own user: -{% highlight bash %} -username ALL=(ALL) NOPASSWD: ALL -{% endhighlight %} -If executing the following command does not ask for your password, then `sudo` should be setup correctly: -{% highlight bash %} -sudo ls -{% endhighlight %} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/4_byon.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/4_byon.md b/guide/ops/locations/4_byon.md deleted file mode 100644 index 42a52ca..0000000 --- a/guide/ops/locations/4_byon.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -section: BYON ---- - -### BYON - -"Bring-your-own-nodes" mode is useful in production, where machines have been provisioned by someone else, -and during testing, to cut down provisioning time. - -Your nodes must meet the following prerequisites: - -- A suitable OS must have been installed on all nodes -- The node must be running sshd (or similar) -- the brooklyn user must be able to ssh to each node as root or as a user with passwordless sudo permission. (For more information on SSH keys, see [here](ssh-keys.html).) - -To deploy to machines with known IP's in a blueprint, use the following syntax: - -{% highlight yaml %} -location: - byon: - user: brooklyn - privateKeyFile: ~/.ssh/brooklyn.pem - hosts: - - 192.168.0.18 - - 192.168.0.19 -{% endhighlight %} - -Some of the login properties as described above for jclouds are supported, -but not `loginUser` (as no users are created), and not any of the -VM creation parameters such as `minRam` and `imageId`. -(These clearly do not apply in the same way, and they are *not* -by default treated as constraints, although an entity can confirm these -where needed.) -As before, if the brooklyn user and its default key are authorized for the hosts, -those fields can be omitted. - -Named locations can also be configured in your `brooklyn.properties`, -using the format `byon:(key=value,key2=value2)`. -For convenience, for hosts wildcard globs are supported. - -{% highlight bash %} -brooklyn.location.named.On-Prem\ Iron\ Example=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}") -brooklyn.location.named.On-Prem\ Iron\ Example.user=produser1 -brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyFile=~/.ssh/produser_id_rsa -brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyPassphrase=s3cr3tpassphrase -{% endhighlight %} - -For more complex host configuration, one can define custom config values per machine. In the example -below, there will be two machines. The first will be a machine reachable on -`ssh -i ~/.ssh/brooklyn.pem -p 8022 myuser@50.51.52.53`. The second is a windows machine, reachable -over WinRM. Each machine has also has a private address (e.g. for within a private network). - -{% highlight yaml %} -location: - byon: - hosts: - - ssh: 50.51.52.53:8022 - privateAddresses: [10.0.0.1] - privateKeyFile: ~/.ssh/brooklyn.pem - user: myuser - - winrm: 50.51.52.54:8985 - privateAddresses: [10.0.0.2] - password: mypassword - user: myuser - osFamily: windows -{% endhighlight %} - -The BYON location also supports a machine chooser, using the config key `byon.machineChooser`. -This allows one to plugin logic to choose from the set of available machines in the pool. For -example, additional config could be supplied for each machine. This could be used (during the call -to `location.obtain()`) to find the config that matches the requirements of the entity being -provisioned. See `FixedListMachineProvisioningLocation.MACHINE_CHOOSER`. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/6_more-locations.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/6_more-locations.md b/guide/ops/locations/6_more-locations.md deleted file mode 100644 index 39099d9..0000000 --- a/guide/ops/locations/6_more-locations.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -section: Specialized Locations ---- - -### Specialized Locations - -Some additional location types are supported for specialized situations: - -#### Single Host - -The spec `host`, taking a string argument (the address) or a map (`host`, `user`, `password`, etc.), -provides a convenient syntax when specifying a single host. -For example: - -{% highlight yaml %} -services: -- type: org.apache.brooklyn.entity.webapp.jboss.JBoss7Server - location: - host: 192.168.0.1 -{% endhighlight %} - -Or, in `brooklyn.properties`, set `brooklyn.location.named.host1=host:(192.168.0.1)`. - - -#### The Multi Location - -The spec `multi` allows multiple locations, specified as `targets`, -to be combined and treated as one location. -When the first target is full, the next is tried, and so on: - -{% highlight yaml %} -location: - multi: - targets: - - byon:(hosts=192.168.0.1) - - jclouds:aws-ec2: - identity: acct1 - - jclouds:aws-ec2: - identity: acct2 -{% endhighlight %} - -The example above provisions the first node to `192.168.0.1`, -then it provisions into `acct1` at Amazon if possible, -and then to `acct2`. - - - -#### The Server Pool - -The {% include java_link.html class_name="ServerPool" package_path="org/apache/brooklyn/entity/machine/pool" project_subpath="software/base" %} -entity type allows defining an entity which becomes available as a location. - http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/byon.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/byon.md b/guide/ops/locations/byon.md new file mode 100644 index 0000000..9088754 --- /dev/null +++ b/guide/ops/locations/byon.md @@ -0,0 +1,74 @@ +--- +section: BYON +section_position: 4 +section_type: inline +--- + +### BYON + +"Bring-your-own-nodes" mode is useful in production, where machines have been provisioned by someone else, +and during testing, to cut down provisioning time. + +Your nodes must meet the following prerequisites: + +- A suitable OS must have been installed on all nodes +- The node must be running sshd (or similar) +- the brooklyn user must be able to ssh to each node as root or as a user with passwordless sudo permission. (For more information on SSH keys, see [here](ssh-keys.html).) + +To deploy to machines with known IP's in a blueprint, use the following syntax: + +{% highlight yaml %} +location: + byon: + user: brooklyn + privateKeyFile: ~/.ssh/brooklyn.pem + hosts: + - 192.168.0.18 + - 192.168.0.19 +{% endhighlight %} + +Some of the login properties as described above for jclouds are supported, +but not `loginUser` (as no users are created), and not any of the +VM creation parameters such as `minRam` and `imageId`. +(These clearly do not apply in the same way, and they are *not* +by default treated as constraints, although an entity can confirm these +where needed.) +As before, if the brooklyn user and its default key are authorized for the hosts, +those fields can be omitted. + +Named locations can also be configured in your `brooklyn.properties`, +using the format `byon:(key=value,key2=value2)`. +For convenience, for hosts wildcard globs are supported. + +{% highlight bash %} +brooklyn.location.named.On-Prem\ Iron\ Example=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}") +brooklyn.location.named.On-Prem\ Iron\ Example.user=produser1 +brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyFile=~/.ssh/produser_id_rsa +brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyPassphrase=s3cr3tpassphrase +{% endhighlight %} + +For more complex host configuration, one can define custom config values per machine. In the example +below, there will be two machines. The first will be a machine reachable on +`ssh -i ~/.ssh/brooklyn.pem -p 8022 myuser@50.51.52.53`. The second is a windows machine, reachable +over WinRM. Each machine has also has a private address (e.g. for within a private network). + +{% highlight yaml %} +location: + byon: + hosts: + - ssh: 50.51.52.53:8022 + privateAddresses: [10.0.0.1] + privateKeyFile: ~/.ssh/brooklyn.pem + user: myuser + - winrm: 50.51.52.54:8985 + privateAddresses: [10.0.0.2] + password: mypassword + user: myuser + osFamily: windows +{% endhighlight %} + +The BYON location also supports a machine chooser, using the config key `byon.machineChooser`. +This allows one to plugin logic to choose from the set of available machines in the pool. For +example, additional config could be supplied for each machine. This could be used (during the call +to `location.obtain()`) to find the config that matches the requirements of the entity being +provisioned. See `FixedListMachineProvisioningLocation.MACHINE_CHOOSER`. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/cloud-specific-setup.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/cloud-specific-setup.md b/guide/ops/locations/cloud-specific-setup.md new file mode 100644 index 0000000..56a701a --- /dev/null +++ b/guide/ops/locations/cloud-specific-setup.md @@ -0,0 +1,88 @@ +--- +section: Cloud Specific Setup +section_type: inline +section_position: 1.1 +--- + +### Cloud Specific Setup + +To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity" and +"credential" in Brooklyn terminology. + +For private clouds (and for some clouds being targeted using a standard API), the "endpoint" +must also be specified, which is the cloud's URL. + +The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring +different clouds. + + +#### AWS + +##### Credentials + +AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential +respectively. + +These keys are the way for any programmatic mechanism to access the AWS API. + +To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws) +and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html). + +An example of the expected format is shown below: + + brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST + brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm + + +##### Tidying up after jclouds + +Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see +https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group +being deleted until there are no VMs using it. However, there is eventual consistency for +recording which VMs still reference those security groups. After deleting the VM, it can sometimes +take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but +does not block for longer. + +There is utility written by Cloudsoft for deleting these unused resources: +http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds. + + +#### Google Compute Engine + +##### Credentials + +Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a private key +as the credential. + +To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google). + +An example of the expected format is shown below (note the credential is one long line, +with `\n` to represent the new line characters): + + brooklyn.location.jclouds.google-compute-engine.identity=123456789...@developer.gserviceaccount.com + brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY----- + + +##### Quotas + +GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas). + +It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en). + + +##### Networks + +GCE accounts often have a limit to the number of networks that can be created. One work around +is to manually create a network with the required open ports, and to refer to that named network +in Brooklyn's location configuration. + +To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1). + +For example, for dev/demo purposes an "everything" network could be created that opens all ports. + +| Name | everything | +| Description | opens all tcp ports | +| Source IP Ranges | 0.0.0.0/0 | +| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 | + + http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/clouds.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/clouds.md b/guide/ops/locations/clouds.md new file mode 100644 index 0000000..90f9eb4 --- /dev/null +++ b/guide/ops/locations/clouds.md @@ -0,0 +1,241 @@ +--- +section: Clouds +section_type: inline +section_position: 1.0 +--- + +### Clouds + +For most cloud provisioning tasks, Brooklyn uses +<a href="http://jclouds.org";>Apache jclouds</a>. +The identifiers for some of the most commonly used jclouds-supported clouds are +(or [see the full list](http://jclouds.apache.org/reference/providers/)): + +* `jclouds:aws-ec2:<region>`: Amazon EC2, where `:<region>` might be `us-east-1` or `eu-west-1` (or omitted) +* `jclouds:softlayer:<region>`: IBM Softlayer, where `:<region>` might be `dal05` or `ams01` (or omitted) +* `jclouds:google-compute-engine`: Google Compute Engine +* `jclouds:openstack-nova:<endpoint>`: OpenStack, where `:<endpoint>` is the access URL (required) +* `jclouds:cloudstack:<endpoint>`: Apache CloudStack, where `:<endpoint>` is the access URL (required) + +For any of these, of course, Brooklyn needs to be configured with an `identity` and a `credential`: + +{% highlight yaml %} +location: + jclouds:aws-ec2: + identity: ABCDEFGHIJKLMNOPQRST + credential: s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l +{% endhighlight %} + +The above YAML can be embedded directly in blueprints, either at the root or on individual services. +If you prefer to keep the credentials separate, you can instead store them as a [catalog entry]({{ site.path.guide }}/ops/catalog/index.html#locations-in-catalog) or set them in `brooklyn.properties` +in the `jclouds.<provider>` namespace: + +{% highlight bash %} +brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST +brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l +{% endhighlight %} + +And in this case you can reference the location in YAML with `location: jclouds:aws-ec2`. + +The Getting Started [template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties) contains more examples +of configuring cloud endpoints, including information on credential types used in different clouds. + +#### OS Initial Login and Setup + +Once a machine is provisioned, Brooklyn will normally attempt to log in via SSH and configure the machine sensibly. + +The credentials for the initial OS log on are typically discovered from the cloud, +but in some environments this is not possible. +The keys `loginUser` and either `loginUser.password` or `loginUser.privateKeyFile` can be used to force +Brooklyn to use specific credentials for the initial login to a cloud-provisioned machine. + +(This custom login is particularly useful when using a custom image templates where the cloud-side account +management logic is not enabled. For example, a vCloud (vCD) template can have guest customization that will change +the root password. This setting tells AMP to only use the given password, rather than the initial +randomly generated password that vCD returns. Without this property, there is a race for such templates: +does Brooklyn manage to create the admin user before the guest customization changes the login and reboots, +or is the password reset first (the latter means Brooklyn can never ssh to the VM). With this property, +Brooklyn will always wait for guest customization to complete before it is able to ssh at all. In such +cases, it is also recommended to use `useJcloudsSshInit=false`.) + +Following a successful logon, Brooklyn performs the following steps to configure the machine: + +1. creates a new user with the same name as the user `brooklyn` is running as locally + (this can be overridden with `user`, below). + +1. install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine, + to make it easy for the operator to `ssh` in + (override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair will be generated; + if there is a passphrase on the key, this must be supplied) + +1. give `sudo` access to the newly created user (override with `grantUserSudo: false`) + +1. disable direct `root` login to the machine + +These steps can be skipped or customized as described below. + +#### jclouds Config Keys + +The following is a subset of the most commonly used configuration keys used to customize +cloud provisioning. +For more keys and more detail on the keys below, see +{% include java_link.html class_name="JcloudsLocationConfig" package_path="org/apache/brooklyn/location/jclouds" project_subpath="locations/jclouds" %}. + +###### VM Creation + +- Most providers require exactly one of either `region` (e.g. `us-east-1`) or `endpoint` (the URL, usually for private cloud deployments) + +- Hardware requirements can be specified, including + `minRam`, `minCores`, and `os64Bit`; or as a specific `hardwareId` + +- VM image constraints can be set using `osFamily` (e.g. `Ubuntu`, `CentOS`, `Debian`, `RHEL`) + and `osVersionRegex`, or specific VM images can be specified using `imageId` or `imageNameRegex` + +- Specific VM images can be specified using `imageId` or `imageNameRegex` + +- Specific Security Groups can be specified using `securityGroups`, as a list of strings (the existing security group names), + or `inboundPorts` can be set, as a list of numeric ports (selected clouds only) + +- A specific existing key pair known at the cloud to use can be specified with `keyPair` + (selected clouds only) + +- A specific VM name (often the hostname) base to be used can be specified by setting `groupId`. + By default, this name is constructed based on the entity which is creating it, + including the ID of the app and of the entity. + (As many cloud portals let you filter views, this can help find a specific entity or all machines for a given application.) + For more sophisticated control over host naming, you can supply a custom + {% include java_link.html class_name="CloudMachineNamer" package_path="org/apache/brooklyn/core/location/cloud/names" project_subpath="core" %}, + for example + `cloudMachineNamer: CustomMachineNamer`. + {% include java_link.html class_name="CustomMachineNamer" package_path="org/apache/brooklyn/core/location/cloud/names" project_subpath="core" %} + will use the entity's name or following a template you supply. + On many clouds, a random suffix will be appended to help guarantee uniqueness; + this can be removed by setting `vmNameSaltLength: 0` (selected clouds only). + <!-- TODO jclouds softlayer includes a 3-char hex suffix --> + +- A DNS domain name where this host should be placed can be specified with `domainName` + (in selected clouds only) + +- User metadata can be attached using the syntax `userMetadata: { key: value, key2: "value 2" }` + (or `userMetadata=key=value,key2="value 2"` in a properties file) + +- By default, several pieces of user metadata are set to correlate VMs with Brooklyn entities, + prefixed with `brooklyn-`. + This user metadata can be omitted by setting `includeBrooklynUserMetadata: false`. + +- You can specify the number of attempts Brooklyn should make to create + machines with `machineCreateAttempts` (jclouds only). This is useful as an efficient low-level fix + for those occasions when cloud providers give machines that are dead on arrival. + You can of course also resolve it at a higher level with a policy such as + {% include java_link.html class_name="ServiceRestarter" package_path="org/apache/brooklyn/policy/ha" project_subpath="policy" %}. + +- If you want to investigate failures, set `destroyOnFailure: false` + to keep failed VM's around. (You'll have to manually clean them up.) + The default is false: if a VM fails to start, or is never ssh'able, then the VM will be terminated. + + ###### OS Setup + +- `user` and `password` can be used to configure the operating user created on cloud-provisioned machines + +- The `loginUser` config key (and subkeys) control the initial user to log in as, + in cases where this cannot be discovered from the cloud provider + +- Private keys can be specified using `privateKeyFile`; + these are not copied to provisioned machines, but are required if using a local public key + or a pre-defined `authorized_keys` on the server. + (For more information on SSH keys, see [here](ssh-keys.html).) + +- If there is a passphrase on the key file being used, you must supply it to Brooklyn for it to work, of course! + `privateKeyPassphrase` does the trick (as in `brooklyn.location.jclouds.privateKeyPassphrase`, or other places + where `privateKeyFile` is valid). If you don't like keys, you can just use a plain old `password`. + +- Public keys can be specified using `publicKeyFile`, + although these can usually be omitted if they follow the common pattern of being + the private key file with the suffix `.pub` appended. + (It is useful in the case of `loginUser.publicKeyFile`, where you shouldn't need, + or might not even have, the private key of the `root` user when you log in.) + +- Provide a list of URLs to public keys in `extraSshPublicKeyUrls`, + or the data of one key in `extraSshPublicKeyData`, + to have additional public keys added to the `authorized_keys` file for logging in. + (This is supported in most but not all locations.) + +- Use `dontCreateUser` to have Brooklyn run as the initial `loginUser` (usually `root`), + without creating any other user. + +- A post-provisioning `setup.script` can be specified (as a URL) to run an additional script, + before making the `Location` available to entities, + optionally also using `setup.script.vars` (set as `key1:value1,key2:value2`) + +- Use `openIptables: true` to automatically configure `iptables`, to open the TCP ports required by + the software process. One can alternatively use `stopIptables: true` to entirely stop the + iptables service. + +- Use `installDevUrandom: true` to fall back to using `/dev/urandom` rather than `/dev/random`. This setting + is useful for cloud VMs where there is not enough random entropy, which can cause `/dev/random` to be + extremely slow (causing `ssh` to be extremely slow to respond). + +- Use `useJcloudsSshInit: false` to disable the use of the native jclouds support for initial commands executed + on the VM (e.g. for creating new users, setting root passwords, etc.). Instead, Brooklyn's ssh support will + be used. Timeouts and retries are more configurable within Brooklyn itself. Therefore this option is particularly + recommended when the VM startup is unusual (for example, if guest customizations will cause reboots and/or will + change login credentials). + +- Use `brooklyn.ssh.config.noDeleteAfterExec: true` to keep scripts on the server after execution. + The contents of the scripts and the stdout/stderr of their execution are available in the Brooklyn web console, + but sometimes it can also be useful to have them on the box. + This setting prevents scripts executed on the VMs from being deleted on completion. + Note that some scripts run periodically so this can eventually fill a disk; it should only be used for dev/test. + +###### Custom template options + +jclouds supports many additional options for configuring how a virtual machine is created and deployed, many of which +are for cloud-specific features and enhancements. Brooklyn supports some of these, but if what you are looking for is +not supported directly by Brooklyn, we instead offer a mechanism to set any parameter that is supported by the jclouds +template options for your cloud. + +Part of the process for creating a virtual machine is the creation of a jclouds `TemplateOptions` object. jclouds +providers extends this with extra options for each cloud - so when using the AWS provider, the object will be of +type `AWSEC2TemplateOptions`. By [examining the source code](https://github.com/jclouds/jclouds/blob/jclouds-1.9.0/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/AWSEC2TemplateOptions.java), +you can see all of the options available to you. + +The `templateOptions` config key takes a map. The keys to the map are method names, and Brooklyn will find the method on +the `TemplateOptions` instance; it then invokes the method with arguments taken from the map value. If a method takes a +single parameter, then simply give the argument as the value of the key; if the method takes multiple parameters, the +value of the key should be an array, containing the argument for each parameter. + +For example, here is a complete blueprint that sets some AWS EC2 specific options: + + location: AWS_eu-west-1 + services: + - type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess + provisioningProperties: + templateOptions: + subnetId: subnet-041c8373 + mapNewVolumeToDeviceName: ["/dev/sda1", 100, true] + securityGroupIds: ['sg-4db68928'] + +Here you can see that we set three template options: + +- `subnetId` is an example of a single parameter method. Brooklyn will effectively try to run the statement + `templateOptions.subnetId("subnet-041c88373");` +- `mapNewVolumeToDeviceName` is an example of a multiple parameter method, so the value of the key is an array. + Brooklyn will effectively true to run the statement `templateOptions.mapNewVolumeToDeviceName("/dev/sda1", 100, true);` +- `securityGroupIds` demonstrates an ambiguity between the two types; Brooklyn will first try to parse the value as + a multiple parameter method, but there is no method that matches this parameter. In this case, Brooklyn will next try + to parse the value as a single parameter method which takes a parameter of type `List`; such a method does exist so + the operation will succeed. + +If the method call cannot be matched to the template options available - for example if you are trying to set an AWS EC2 +specific option but your location is an OpenStack cloud - then a warning is logged and the option is ignored. + + + + +See the following resources for more information: + +- [AWS VPC issues which may affect users with older AWS accounts](vpc-issues.html) +- [Amazon EC2 and Amazon Virtual Private Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html#vpc-only-instance-types) +- [Your Default VPC and Subnets](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html) +- [Amazon VPC FAQs](http://aws.amazon.com/vpc/faqs/#Default_VPCs) + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/index.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/index.md b/guide/ops/locations/index.md index 0698996..dc0c1dc 100644 --- a/guide/ops/locations/index.md +++ b/guide/ops/locations/index.md @@ -1,7 +1,7 @@ --- title: Locations layout: website-normal -show_inline_children: true +check_directory_for_children: true --- Locations are the environments to which Brooklyn deploys applications, including: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/inheritance-and-named-locations.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/inheritance-and-named-locations.md b/guide/ops/locations/inheritance-and-named-locations.md new file mode 100644 index 0000000..9854fea --- /dev/null +++ b/guide/ops/locations/inheritance-and-named-locations.md @@ -0,0 +1,80 @@ +--- +section: Inheritance and Named Locations +title: Named Locations +section_type: inline +section_position: 2 +--- + +### Inheritance and Named Locations + +Named locations can be defined for commonly used groups of properties, +with the syntax `brooklyn.location.named.your-group-name.` +followed by the relevant properties. +These can be accessed at runtime using the syntax `named:your-group-name` as the deployment location. + +Some illustrative examples using named locations and +showing the syntax and properties above are as follows: + +{% highlight bash %} +# Production pool of machines for my application (deploy to named:prod1) +brooklyn.location.named.prod1=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}") +brooklyn.location.named.prod1.user=produser1 +brooklyn.location.named.prod1.privateKeyFile=~/.ssh/produser_id_rsa +brooklyn.location.named.prod1.privateKeyPassphrase=s3cr3tCOMPANYpassphrase + +# AWS using my company's credentials and image standard, then labelling images so others know they're mine +brooklyn.location.named.company-jungle=jclouds:aws-ec2:us-west-1 +brooklyn.location.named.company-jungle.identity=BCDEFGHIJKLMNOPQRSTU +brooklyn.location.named.company-jungle.privateKeyFile=~/.ssh/public_clouds/company_aws_id_rsa +brooklyn.location.named.company-jungle.imageId=ami-12345 +brooklyn.location.named.company-jungle.minRam=2048 +brooklyn.location.named.company-jungle.userMetadata=application=my-jungle-app,owner="Bob Johnson" +brooklyn.location.named.company-jungle.machineCreateAttempts=2 + +brooklyn.location.named.AWS\ Virginia\ Large\ Centos = jclouds:aws-ec2 +brooklyn.location.named.AWS\ Virginia\ Large\ Centos.region = us-east-1 +brooklyn.location.named.AWS\ Virginia\ Large\ Centos.imageId=us-east-1/ami-7d7bfc14 +brooklyn.location.named.AWS\ Virginia\ Large\ Centos.user=root +brooklyn.location.named.AWS\ Virginia\ Large\ Centos.minRam=4096 +{% endhighlight %} + +Named locations can refer to other named locations using `named:xxx` as their value. +These will inherit the configuration and can override selected keys. +Properties set in the namespace of the provider (e.g. `b.l.jclouds.aws-ec2.KEY=VALUE`) +will be inherited by everything which extends AWS +Sub-prefix strings are also inherited up to `brooklyn.location.*`, +except that they are filtered for single-word and other +known keys +(so that we exclude provider-scoped properties when looking at sub-prefix keys). +The precedence for configuration defined at different levels is that the value +defined in the most specific context will apply. + +This is rather straightforward and powerful to use, +although it sounds rather more complicated than it is! +The examples below should make it clear. +You could use the following to install +a public key on all provisioned machines, +an additional public key in all AWS machines, +and no extra public key in `prod1`: + +<!-- tested in JcloudsLocationResolverTest --> +{% highlight bash %} +brooklyn.location.extraSshPublicKeyUrls=http://me.com/public_key +brooklyn.location.jclouds.aws-ec2.extraSshPublicKeyUrls="[ \"http://me.com/public_key\";, \"http://me.com/aws_public_key\"; ]" +brooklyn.location.named.prod1.extraSshPublicKeyUrls= +{% endhighlight %} + +And in the example below, a config key is repeatedly overridden. +Deploying `location: named:my-extended-aws` will result in an `aws-ec2` machine in `us-west-1` (by inheritance) +with `VAL6` for `KEY`: + +{% highlight bash %} +brooklyn.location.KEY=VAL1 +brooklyn.location.jclouds.KEY=VAL2 +brooklyn.location.jclouds.aws-ec2.KEY=VAL3 +brooklyn.location.jclouds.aws-...@us-west-1.key=VAL4 +brooklyn.location.named.my-aws=jclouds:aws-ec2:us-west-1 +brooklyn.location.named.my-aws.KEY=VAL5 +brooklyn.location.named.my-extended-aws=named:my-aws +brooklyn.location.named.my-extended-aws.KEY=VAL6 +{% endhighlight %} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/localhost.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/localhost.md b/guide/ops/locations/localhost.md new file mode 100644 index 0000000..86ef5b5 --- /dev/null +++ b/guide/ops/locations/localhost.md @@ -0,0 +1,33 @@ +--- +section: Localhost +section_position: 3 +section_type: inline +--- + +### Localhost + +If passwordless ssh login to `localhost` and passwordless `sudo` is enabled on your +machine, you should be able to deploy blueprints with no special configuration, +just by specifying `location: localhost` in YAML. + +If you use a passpharse or prefer a different key, these can be configured as follows: + +{% highlight bash %} +brooklyn.location.localhost.privateKeyFile=~/.ssh/brooklyn_key +brooklyn.location.localhost.privateKeyPassphrase=s3cr3tPASSPHRASE +{% endhighlight %} + +If you encounter issues or for more information, see [SSH Keys Localhost Setup](ssh-keys.html#localhost-setup). + +If you are normally prompted for a password when executing `sudo` commands, passwordless `sudo` must also be enabled. To enable passwordless `sudo` for your account, a line must be added to the system `/etc/sudoers` file. To edit the file, use the `visudo` command: +{% highlight bash %} +sudo visudo +{% endhighlight %} +Add this line at the bottom of the file, replacing `username` with your own user: +{% highlight bash %} +username ALL=(ALL) NOPASSWD: ALL +{% endhighlight %} +If executing the following command does not ask for your password, then `sudo` should be setup correctly: +{% highlight bash %} +sudo ls +{% endhighlight %} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/more-locations.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/more-locations.md b/guide/ops/locations/more-locations.md index 0a00bf5..2ed2c83 100644 --- a/guide/ops/locations/more-locations.md +++ b/guide/ops/locations/more-locations.md @@ -1,15 +1,14 @@ --- -title: More Locations -layout: website-normal -children: -- { section: Single Host } -- { section: The Multi Location } -- { section: The Server Pool } +section: Specialized Locations +section_position: 5 +section_type: inline --- +### Specialized Locations + Some additional location types are supported for specialized situations: -### Single Host +#### Single Host The spec `host`, taking a string argument (the address) or a map (`host`, `user`, `password`, etc.), provides a convenient syntax when specifying a single host. @@ -25,7 +24,7 @@ services: Or, in `brooklyn.properties`, set `brooklyn.location.named.host1=host:(192.168.0.1)`. -### The Multi Location +#### The Multi Location The spec `multi` allows multiple locations, specified as `targets`, to be combined and treated as one location. @@ -48,7 +47,7 @@ and then to `acct2`. -### The Server Pool +#### The Server Pool The {% include java_link.html class_name="ServerPool" package_path="org/apache/brooklyn/entity/machine/pool" project_subpath="software/base" %} entity type allows defining an entity which becomes available as a location. http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/cfa914de/guide/ops/locations/ssh-keys.md ---------------------------------------------------------------------- diff --git a/guide/ops/locations/ssh-keys.md b/guide/ops/locations/ssh-keys.md index f805253..e49d031 100644 --- a/guide/ops/locations/ssh-keys.md +++ b/guide/ops/locations/ssh-keys.md @@ -1,8 +1,11 @@ --- -title: SSH Keys -layout: website-normal +section: SSH Keys +section_position: 3.1 +section_type: inline --- +### SSH Keys + SSH keys are one of the simplest and most secure ways to access remote servers. They consist of two parts: @@ -21,7 +24,7 @@ the remote machine can confirm it is you accessing it (if no one else has the pr and no one snooping on the network can decrypt of any of the traffic. -### Creating an SSH Key +#### Creating an SSH Key If you don't have an SSH key, create one with: @@ -30,7 +33,7 @@ $ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa {% endhighlight %} -### Localhost Setup +#### Localhost Setup If you want to deploy to `localhost`, ensure that you have a public and private key, and that your key is authorized for ssh access: @@ -59,7 +62,7 @@ If this isn't the case, see below. -### Potential Problems +#### Potential Problems * **MacOS user?** In addition to the above, enable "Remote Login" in "System Preferences > Sharing".
