Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-08-03 Thread Server Team CI bot 
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:1265
https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/
Executed test runs:
FAILURE: 
https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=amd64/5/console
FAILURE: 
https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=ppc64el/5/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/rebuild

-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-08-03 Thread Server Team CI bot 
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:1264
https://jenkins.ubuntu.com/server/job/cloud-init-ci/4/
Executed test runs:
FAILURE: 
https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=amd64/4/console
FAILURE: 
https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=ppc64el/4/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/4/rebuild

-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-08-01 Thread Ben Howard 
Updated the provider and changed util. I also push it to [1], but I can't make 
sense of the Launchpad/Git MP process.

[1] https://code.launchpad.net/~utlemming/+git/cloud-init/+ref/digitalocean
-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-07-28 Thread Scott Moser 
Generally good, and I'd like to have this datasource enabled by default also.
Its great to see cloud's putting instance-id in dmi data.

please see the comments inline.

Diff comments:

> === modified file 'cloudinit/sources/DataSourceDigitalOcean.py'
> --- cloudinit/sources/DataSourceDigitalOcean.py   2016-05-12 17:56:26 
> +
> +++ cloudinit/sources/DataSourceDigitalOcean.py   2016-07-26 14:34:57 
> +
> @@ -14,22 +15,27 @@
>  #You should have received a copy of the GNU General Public License
>  #along with this program.  If not, see .
>  
> -from cloudinit import ec2_utils
> +# DigitalOcean Droplet API:
> +# https://developers.digitalocean.com/documentation/metadata/
> +
> +import json
> +import os
> +
>  from cloudinit import log as logging
>  from cloudinit import sources
> +from cloudinit import url_helper
>  from cloudinit import util
>  
> -import functools
> -
> -
>  LOG = logging.getLogger(__name__)
>  
>  BUILTIN_DS_CONFIG = {
> -'metadata_url': 'http://169.254.169.254/metadata/v1/',
> -'mirrors_url': 'http://mirrors.digitalocean.com/'
> +'metadata_url': 'http://169.254.169.254/metadata/v1.json',
>  }
> -MD_RETRIES = 0
> -MD_TIMEOUT = 1
> +
> +# Wait for a up to a minute, retrying the meta-data server
> +# every 2 seconds.
> +MD_RETRIES = 30
> +MD_TIMEOUT = 2

there is 'timeout' (how long to allow the url request to take before giving up)
and there is sec_between.  you've used MD_TIMEOUT for both.  might as well 
separate these to 2 variables and update your call to readurl.

>  
>  
>  class DataSourceDigitalOcean(sources.DataSource):
> @@ -40,43 +46,67 @@
>  util.get_cfg_by_path(sys_cfg, ["datasource", "DigitalOcean"], 
> {}),
>  BUILTIN_DS_CONFIG])
>  self.metadata_address = self.ds_cfg['metadata_url']
> -
> -if self.ds_cfg.get('retries'):
> -self.retries = self.ds_cfg['retries']
> -else:
> -self.retries = MD_RETRIES
> -
> -if self.ds_cfg.get('timeout'):
> -self.timeout = self.ds_cfg['timeout']
> -else:
> -self.timeout = MD_TIMEOUT
> -
> -def get_data(self):
> -caller = functools.partial(util.read_file_or_url,
> -   timeout=self.timeout, 
> retries=self.retries)
> -
> -def mcaller(url):
> -return caller(url).contents
> -
> -md = ec2_utils.MetadataMaterializer(mcaller(self.metadata_address),
> -base_url=self.metadata_address,
> -caller=mcaller)
> -
> -self.metadata = md.materialize()
> -
> -if self.metadata.get('id'):
> -return True
> -else:
> +self.retries = self.ds_cfg.get('retries', MD_RETRIES)
> +self.timeout = self.ds_cfg.get('timeout', MD_TIMEOUT)
> +
> +def _get_sysinfo(self):
> +# DigitalOcean embeds vendor ID and instance/droplet_id in the
> +# SMBIOS information
> +
> +# DigitalOcean only supports i386/x86_64 hardware
> +uname_arch = os.uname()[4]
> +if uname_arch not in ('i386', 'i686', 'x86_64'):
> +return (None, None)
> +LOG.debug("checking if instance is a DigitalOcean droplet")

I'd prefer to just accept anything that says they're DigitalOcean (per the 
bios), independent of arch.
https://bugs.launchpad.net/qemu/+bug/1243287 is a pain, which is probably 
better fixed in generic util.read_dmi_data (avoiding running on arm*) than it 
is right now.  

so either
a.) keep the check for arch
b.) fix util.read_dmi-data to not try running dmidecode on arm (see the comment 
in cloudinit/sources/DataSourceSmartOS.py for that bug and just put a similar 
work around in util).

> +
> +is_do = False
> +droplet_id = None
> +
> +# Detect if we are on DigitalOcean and return the Droplet's ID
> +vendor_name = util.read_dmi_data("system-manufacturer")
> +if vendor_name == "DigitalOcean":

i'm generally a fan of just getting out of a function early. then you can 
indent less.

if vendor_name != DigitalOcean:
return (Nonw, Nonw)

> +is_do = True
> +LOG.info("running on DigitalOcean")
> +
> +droplet_id = util.read_dmi_data("system-serial-number")
> +
> +if droplet_id:
> +LOG.debug(("system identified via SMBIOS as DigitalOcean "
> +   "Droplet {}").format(droplet_id))
> +else:
> +LOG.critical(("system identified via SMBIOS as a 
> DigitalOcean "
> +  "Droplet, but not provide an ID. "
> +  "Please file a support ticket at: "
> +  
> "https://cloud.digitalocean.com/support/tickets;
> +  "/new"))
> +
> +return (is_do, droplet_id)
> +
> +def get_data(self, apply_filter=False):
> +

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-07-26 Thread Server Team CI bot 
Review: Approve continuous-integration

PASSED: Continuous integration, rev:1262
https://server-team-jenkins.canonical.com/job/cloud-init-ci/3/
Executed test runs:
SUCCESS: 
https://server-team-jenkins.canonical.com/job/cloud-init-ci/nodes=amd64/3
SUCCESS: 
https://server-team-jenkins.canonical.com/job/cloud-init-ci/nodes=ppc64el/3

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/cloud-init-ci/3/rebuild

-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-07-26 Thread Ryan Harper 
The errors:


flake8 runtests: commands[0] |
/var/lib/jenkins/slaves/jenkins-slave-4/workspace/cloud-init-ci/nodes/amd64/cloud-init-ci-2/.tox/flake8/bin/python
-m flake8 cloudinit/ tests/ tools/
cloudinit/sources/DataSourceDigitalOcean.py:23:1: H306  imports not in
alphabetical order (cloudinit.util, cloudinit.sources)
tests/unittests/test_handler/test_handler_mcollective.py:52:26: E225
missing whitespace around operator
tests/unittests/test_handler/test_handler_mcollective.py:89:80: E501
line too long (80 > 79 characters)
tests/unittests/test_handler/test_handler_mcollective.py:105:13: E128
continuation line under-indented for visual indent
tests/unittests/test_handler/test_handler_mcollective.py:106:13: E128
continuation line under-indented for visual indent


On Mon, Jul 25, 2016 at 6:54 PM, Ben Howard  wrote:

> I'm unable to see the logs in the failed CI run. "make check" tests pass
> locally. I'm happy to make the changes if someone can post the logs.
> --
> https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
> Your team cloud init development team is requested to review the proposed
> merge of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.
>
> ___
> Mailing list: https://launchpad.net/~cloud-init-dev
> Post to : cloud-init-dev@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~cloud-init-dev
> More help   : https://help.launchpad.net/ListHelp
>

-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-07-25 Thread Server Team CI bot 
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:1260
https://server-team-jenkins.canonical.com/job/cloud-init-ci/2/
Executed test runs:
FAILURE: 
https://server-team-jenkins.canonical.com/job/cloud-init-ci/nodes=amd64/2/console
FAILURE: 
https://server-team-jenkins.canonical.com/job/cloud-init-ci/nodes=ppc64el/2/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/cloud-init-ci/2/rebuild

-- 
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp

[Cloud-init-dev] [Merge] lp:~utlemming/cloud-init/digitalocean into lp:cloud-init

2016-07-25 Thread Ben Howard 
Ben Howard has proposed merging lp:~utlemming/cloud-init/digitalocean into 
lp:cloud-init.

Requested reviews:
  cloud init development team (cloud-init-dev)

For more details, see:
https://code.launchpad.net/~utlemming/cloud-init/digitalocean/+merge/301123

This request comprises two commits to the DigitalOcean provider. 

Commit 1259: Update to the JSON meta-data endpoint
- Per the API documentation, the JSON endpoint is the preferred method of 
getting the metadata.
- Use the SMBIOS information to determine if the droplet ID has changed. In 
certain circumstances a droplet that is either migrated or snapshotted may not 
have metadata via the metadata URL. The droplets identity is authoritatively 
embedded in the SMBIOS. 
- Only hit the Metadata URL if the droplet identifies as being on DigitalOcean

Commit 1260: Update the tests cases.

-- 
Your team cloud init development team is requested to review the proposed merge 
of lp:~utlemming/cloud-init/digitalocean into lp:cloud-init.
=== modified file 'cloudinit/sources/DataSourceDigitalOcean.py'
--- cloudinit/sources/DataSourceDigitalOcean.py	2016-05-12 17:56:26 +
+++ cloudinit/sources/DataSourceDigitalOcean.py	2016-07-25 23:38:22 +
@@ -1,6 +1,7 @@
 # vi: ts=4 expandtab
 #
 #Author: Neal Shrader 
+#Author: Ben Howard  
 #
 #This program is free software: you can redistribute it and/or modify
 #it under the terms of the GNU General Public License version 3, as
@@ -14,22 +15,27 @@
 #You should have received a copy of the GNU General Public License
 #along with this program.  If not, see .
 
-from cloudinit import ec2_utils
+# DigitalOcean Droplet API:
+# https://developers.digitalocean.com/documentation/metadata/
+
 from cloudinit import log as logging
+from cloudinit import util
 from cloudinit import sources
-from cloudinit import util
-
-import functools
-
+from cloudinit import url_helper
+
+import json
+import os
 
 LOG = logging.getLogger(__name__)
 
 BUILTIN_DS_CONFIG = {
-'metadata_url': 'http://169.254.169.254/metadata/v1/',
-'mirrors_url': 'http://mirrors.digitalocean.com/'
+'metadata_url': 'http://169.254.169.254/metadata/v1.json',
 }
-MD_RETRIES = 0
-MD_TIMEOUT = 1
+
+# Wait for a up to a minute, retrying the meta-data server
+# every 2 seconds.
+MD_RETRIES = 30
+MD_TIMEOUT = 2
 
 
 class DataSourceDigitalOcean(sources.DataSource):
@@ -40,43 +46,67 @@
 util.get_cfg_by_path(sys_cfg, ["datasource", "DigitalOcean"], {}),
 BUILTIN_DS_CONFIG])
 self.metadata_address = self.ds_cfg['metadata_url']
-
-if self.ds_cfg.get('retries'):
-self.retries = self.ds_cfg['retries']
-else:
-self.retries = MD_RETRIES
-
-if self.ds_cfg.get('timeout'):
-self.timeout = self.ds_cfg['timeout']
-else:
-self.timeout = MD_TIMEOUT
-
-def get_data(self):
-caller = functools.partial(util.read_file_or_url,
-   timeout=self.timeout, retries=self.retries)
-
-def mcaller(url):
-return caller(url).contents
-
-md = ec2_utils.MetadataMaterializer(mcaller(self.metadata_address),
-base_url=self.metadata_address,
-caller=mcaller)
-
-self.metadata = md.materialize()
-
-if self.metadata.get('id'):
-return True
-else:
+self.retries = self.ds_cfg.get('retries', MD_RETRIES)
+self.timeout = self.ds_cfg.get('timeout', MD_TIMEOUT)
+
+def _get_sysinfo(self):
+# DigitalOcean embeds vendor ID and instance/droplet_id in the
+# SMBIOS information
+
+# DigitalOcean only supports i386/x86_64 hardware
+uname_arch = os.uname()[4]
+if uname_arch not in ('i386', 'i686', 'x86_64'):
+return (None, None)
+LOG.debug("checking if instance is a DigitalOcean droplet")
+
+is_do = False
+droplet_id = None
+
+# Detect if we are on DigitalOcean and return the Droplet's ID
+vendor_name = util.read_dmi_data("system-manufacturer")
+if vendor_name == "DigitalOcean":
+is_do = True
+LOG.info("running on DigitalOcean")
+
+droplet_id = util.read_dmi_data("system-serial-number")
+
+if droplet_id:
+LOG.debug(("system identified via SMBIOS as DigitalOcean "
+   "Droplet {}").format(droplet_id))
+else:
+LOG.critical(("system identified via SMBIOS as a DigitalOcean "
+  "Droplet, but not provide an ID. "
+  "Please file a support ticket at: "
+  "https://cloud.digitalocean.com/support/tickets;
+  "/new"))
+
+return (is_do, droplet_id)
+
+def get_data(self,