[Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.smartos-fabric into lp:cloud-init
The proposal to merge lp:~smoser/cloud-init/trunk.smartos-fabric into lp:cloud-init has been updated. Commit Message changed to: SmartOS: more improvements for network configuration This improves smart os network configuration - fix the SocketClient which was previously completely broken. - adds support for configuring dns servers and dns search (based off the sdc:dns_domain). - support 'sdc:gateways' information from the datasource for configuring default routes. - add converted network information to output when module is run as a main TODO: add support for sdc:routes as described at http://eng.joyent.com/mdata/datadict.html For more details, see: https://code.launchpad.net/~smoser/cloud-init/trunk.smartos-fabric/+merge/300115 -- Your team cloud init development team is requested to review the proposed merge of lp:~smoser/cloud-init/trunk.smartos-fabric into lp:cloud-init. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.smartos-fabric into lp:cloud-init
Scott Moser has proposed merging lp:~smoser/cloud-init/trunk.smartos-fabric
into lp:cloud-init.
Requested reviews:
cloud init development team (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~smoser/cloud-init/trunk.smartos-fabric/+merge/300115
--
Your team cloud init development team is requested to review the proposed merge
of lp:~smoser/cloud-init/trunk.smartos-fabric into lp:cloud-init.
=== modified file 'cloudinit/sources/DataSourceSmartOS.py'
--- cloudinit/sources/DataSourceSmartOS.py 2016-07-13 22:18:46 +
+++ cloudinit/sources/DataSourceSmartOS.py 2016-07-14 19:15:20 +
@@ -60,11 +60,14 @@
'availability_zone': ('sdc:datacenter_name', True),
'vendor-data': ('sdc:vendor-data', False),
'operator-script': ('sdc:operator-script', False),
+'hostname': ('sdc:hostname', True),
+'dns_domain': ('sdc:dns_domain', True),
}
SMARTOS_ATTRIB_JSON = {
# Cloud-init Key : (SmartOS Key known JSON)
'network-data': 'sdc:nics',
+'dns_servers': 'sdc:resolvers'
}
SMARTOS_ENV_LX_BRAND = "lx-brand"
@@ -311,7 +314,10 @@
if self._network_config is None:
if self.network_data is not None:
self._network_config = (
-convert_smartos_network_data(self.network_data))
+convert_smartos_network_data(
+network_data=self.network_data,
+dns_servers=self.metadata['dns_servers'],
+dns_domain=self.metadata['dns_domain']))
return self._network_config
@@ -445,7 +451,8 @@
class JoyentMetadataSocketClient(JoyentMetadataClient):
-def __init__(self, socketpath):
+def __init__(self, socketpath, smartos_type=SMARTOS_ENV_LX_BRAND):
+super(JoyentMetadataSocketClient, self).__init__(smartos_type)
self.socketpath = socketpath
def open_transport(self):
@@ -461,7 +468,7 @@
class JoyentMetadataSerialClient(JoyentMetadataClient):
-def __init__(self, device, timeout=10, smartos_type=None):
+def __init__(self, device, timeout=10, smartos_type=SMARTOS_ENV_KVM):
super(JoyentMetadataSerialClient, self).__init__(smartos_type)
self.device = device
self.timeout = timeout
@@ -583,7 +590,8 @@
device=serial_device, timeout=serial_timeout,
smartos_type=smartos_type)
elif smartos_type == SMARTOS_ENV_LX_BRAND:
-return JoyentMetadataSocketClient(socketpath=metadata_sockfile)
+return JoyentMetadataSocketClient(socketpath=metadata_sockfile,
+ smartos_type=smartos_type)
raise ValueError("Unknown value for smartos_type: %s" % smartos_type)
@@ -671,8 +679,9 @@
return None
-# Covert SMARTOS 'sdc:nics' data to network_config yaml
-def convert_smartos_network_data(network_data=None):
+# Convert SMARTOS 'sdc:nics' data to network_config yaml
+def convert_smartos_network_data(network_data=None,
+ dns_servers=None, dns_domain=None):
"""Return a dictionary of network_config by parsing provided
SMARTOS sdc:nics configuration data
@@ -706,9 +715,7 @@
'broadcast',
'dns_nameservers',
'dns_search',
-'gateway',
'metric',
-'netmask',
'pointopoint',
'routes',
'scope',
@@ -716,6 +723,29 @@
],
}
+if dns_servers:
+if not isinstance(dns_servers, (list, tuple)):
+dns_servers = [dns_servers]
+else:
+dns_servers = []
+
+if dns_domain:
+if not isinstance(dns_domain, (list, tuple)):
+dns_domain = [dns_domain]
+else:
+dns_domain = []
+
+def is_valid_ipv4(addr):
+return '.' in addr
+
+def is_valid_ipv6(addr):
+return ':' in addr
+
+pgws = {
+'ipv4': {'match': is_valid_ipv4, 'gw': None},
+'ipv6': {'match': is_valid_ipv6, 'gw': None},
+}
+
config = []
for nic in network_data:
cfg = dict((k, v) for k, v in nic.items()
@@ -727,18 +757,40 @@
cfg.update({'mac_address': nic['mac']})
subnets = []
-for ip, gw in zip(nic['ips'], nic['gateways']):
-subnet = dict((k, v) for k, v in nic.items()
- if k in valid_keys['subnet'])
-subnet.update({
-'type': 'static',
-'address': ip,
-'gateway': gw,
-})
+for ip in nic.get('ips', []):
+if ip == "dhcp":
+subnet = {'type': 'dhcp4'}
+else:
+subnet = dict((k, v) for k, v in nic.items()
+ if k in valid_keys['subnet'])
+subnet.update({
+'type': 'static',
+'address': ip,
+})
+
+proto = 'ipv4' if is_valid_ipv4(ip) else 'ipv6'
+
[Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init
The proposal to merge lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init has been updated. Status: Needs review => Merged For more details, see: https://code.launchpad.net/~smoser/cloud-init/trunk.net-improve-lo-dns/+merge/298035 -- Your team cloud init development team is requested to review the proposed merge of lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init
i added a test of the _write_network fallback in the fix for 1602373 (revno 1255 on trunk). and pulled that in here. going to merge this in now. -- https://code.launchpad.net/~smoser/cloud-init/trunk.net-improve-lo-dns/+merge/298035 Your team cloud init development team is requested to review the proposed merge of lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.lp1602373 into lp:cloud-init
The proposal to merge lp:~smoser/cloud-init/trunk.lp1602373 into lp:cloud-init has been updated. Status: Needs review => Merged For more details, see: https://code.launchpad.net/~smoser/cloud-init/trunk.lp1602373/+merge/300021 -- Your team cloud init development team is requested to review the proposed merge of lp:~smoser/cloud-init/trunk.lp1602373 into lp:cloud-init. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] lp:~sgolovatiuk/cloud-init/fix_mcollective into lp:cloud-init
The proposal to merge lp:~sgolovatiuk/cloud-init/fix_mcollective into lp:cloud-init has been updated. Status: Needs review => Merged For more details, see: https://code.launchpad.net/~sgolovatiuk/cloud-init/fix_mcollective/+merge/298759 -- Your team cloud init development team is requested to review the proposed merge of lp:~sgolovatiuk/cloud-init/fix_mcollective into lp:cloud-init. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] lp:~bbaude/cloud-init/azure_dhcp into lp:cloud-init
update the commit message to address the different behavior (we now write json
for 1).
is it possible to put cloud-init-dhclient-hook outside of /usr/bin ?
we could put it in /usr/lib/cloud-init and hten just name it 'dhclient-hook' .
i say this because its not really intende to be used as an executable.
other suggestions in line.
this is looking good though, thanks Brent and Lars.
Diff comments:
> === added file 'cloudinit/cmd/dhclient_hook.py'
> --- cloudinit/cmd/dhclient_hook.py1970-01-01 00:00:00 +
> +++ cloudinit/cmd/dhclient_hook.py2016-07-13 22:57:56 +
> @@ -0,0 +1,51 @@
> +import os
> +import json
> +from cloudinit import stages
> +from cloudinit import log as logging
> +
> +
> +class LogDhclient():
> +
> +LOG = logging.getLogger(__name__)
> +
> +def __init__(self):
> +self.hooks_dir = self._get_hooks_dir()
> +
> +@staticmethod
> +def _get_hooks_dir():
> +i = stages.Init()
> +return os.path.join(i.paths.get_runpath(), 'dhclient.hooks')
> +
> +def check_hooks_dir(self):
> +if not os.path.exists(self.hooks_dir):
> +os.makedirs(self.hooks_dir)
> +else:
> +hook_files = [os.path.join(self.hooks_dir, x)
> + for x in os.listdir(self.hooks_dir)]
> +for hook_file in hook_files:
> +os.remove(hook_file)
> +
> +@staticmethod
> +def get_vals(info):
> +new_info = {}
> +for k, v in info.iteritems():
> +if k.startswith("DHCP4_") or k.startswith("new_"):
> +key = (k.replace('DHCP4_', '').replace('new_', '')).lower()
> +new_info[key] = v
> +return new_info
> +
> +def record(self):
> +envs = os.environ
> +ifc_name = envs.get("interface", envs.get("DEVICE_IFACE", None))
> +if ifc_name is None:
> +return
> +ifc_file_name = os.path.join(self.hooks_dir, ifc_name + '.json')
> +with open(ifc_file_name, 'w') as outfile:
> +json.dump(self.get_vals(envs), outfile, indent=4)
return json.dumps(data, indent=1, sort_keys=True,
separators=(',', ': ')).encode('utf-8')
that is nice and pretty output. ie, please use sort_keys and separators also.
> +self.LOG.debug("Wrote dhclient options in %s", ifc_file_name)
> +
> +
> +def main():
> +record = LogDhclient()
> +record.check_hooks_dir()
> +record.record()
>
> === modified file 'cloudinit/sources/helpers/azure.py'
> --- cloudinit/sources/helpers/azure.py2016-05-12 20:52:56 +
> +++ cloudinit/sources/helpers/azure.py2016-07-13 22:57:56 +
> @@ -213,15 +228,74 @@
> return socket.inet_ntoa(packed_bytes)
>
> @staticmethod
> -def find_endpoint():
> +def _get_value_from_leases_file(lease_file):
> +leases = []
> +content = util.load_file(lease_file)
> +LOG.debug("content is {}".format(content))
> +for line in content.splitlines():
> +if 'unknown-245' in line:
can you give an example here of what that line looks like ? line.strip(''.
that is hard to parse in your head.
> +leases.append(line.strip(' ').split(' ',
> 2)[-1].strip(';\n"'))
> +# Return the "most recent" one in the list
> +if len(leases) < 1:
> +return None
> +else:
> +return leases[-1]
> +
> +@staticmethod
> +def _load_dhclient_json():
> +dhcp_options = {}
> +hooks_dir = WALinuxAgentShim._get_hooks_dir()
> +if not os.path.exists(hooks_dir):
> +LOG.debug("%s not found.", hooks_dir)
> +return None
> +hook_files = [os.path.join(hooks_dir, x)
> + for x in os.listdir(hooks_dir)]
> +for hook_file in hook_files:
> +try:
> +_file_dict = json.load(open(hook_file))
> +dhcp_options[os.path.basename(
> +hook_file).replace('.json', '')] = _file_dict
> +except ValueError:
> +LOG.info("%s is not valid JSON data", hook_file)
Probably best to raise some exception here so caller can tell the difference
between "there just wasnt data in the file" and "the file did not exist"
> +return dhcp_options
> +
> +@staticmethod
> +def _get_value_from_dhcpoptions(dhcp_options):
> +if dhcp_options is None:
> +return None
> +# the MS endpoint server is given to us as DHPC option 245
> +_value = None
> +for interface in dhcp_options:
> +_value = dhcp_options[interface].get('unknown_245', None)
> +if _value is not None:
> +break
> +return _value
> +
> +@staticmethod
> +def find_endpoint(lease_file):
> LOG.debug('Finding Azure endpoint...')
> -content = util.load_file('/var/lib/dhcp/dhclient.eth0.leases')
> value = None
> -for
Re: [Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init
Looks good. Could we add unittests for distro using apply_network, including
one without a _write_network so we can test that fallback logic?
Diff comments:
>
> === modified file 'tests/unittests/test_net.py'
> --- tests/unittests/test_net.py 2016-06-15 23:11:24 +
> +++ tests/unittests/test_net.py 2016-07-14 02:45:33 +
> @@ -323,6 +634,53 @@
> self.assertEqual(found, self.simple_cfg)
>
>
> +class TestEniRoundTrip(TestCase):
Nice!
> +def setUp(self):
> +super(TestCase, self).setUp()
> +self.tmp_dir = tempfile.mkdtemp()
> +self.addCleanup(shutil.rmtree, self.tmp_dir)
> +
> +def _render_and_read(self, network_config=None, state=None,
> eni_path=None,
> + links_prefix=None, netrules_path=None):
> +if network_config:
> +ns = network_state.parse_net_config_data(network_config)
> +elif state:
> +ns = state
> +else:
> +raise ValueError("Expected data or state, got neither")
> +
> +if eni_path is None:
> +eni_path = 'etc/network/interfaces'
> +
> +renderer = eni.Renderer(
> +config={'eni_path': eni_path, 'links_path_prefix': links_prefix,
> +'netrules_path': netrules_path})
> +
> +renderer.render_network_state(self.tmp_dir, ns)
> +return dir2dict(self.tmp_dir)
> +
> +def testsimple_convert_and_render(self):
> +network_config = eni.convert_eni_data(EXAMPLE_ENI)
> +files = self._render_and_read(network_config=network_config)
> +self.assertEqual(
> +RENDERED_ENI.splitlines(),
> +files['/etc/network/interfaces'].splitlines())
> +
> +def testsimple_render_all(self):
> +entry = NETWORK_CONFIGS['all']
> +files =
> self._render_and_read(network_config=yaml.load(entry['yaml']))
> +self.assertEqual(
> +entry['expected_eni'].splitlines(),
> +files['/etc/network/interfaces'].splitlines())
> +
> +def testsimple_render_small(self):
> +entry = NETWORK_CONFIGS['small']
> +files =
> self._render_and_read(network_config=yaml.load(entry['yaml']))
> +self.assertEqual(
> +entry['expected_eni'].splitlines(),
> +files['/etc/network/interfaces'].splitlines())
> +
> +
> def _gzip_data(data):
> with io.BytesIO() as iobuf:
> gzfp = gzip.GzipFile(mode="wb", fileobj=iobuf)
--
https://code.launchpad.net/~smoser/cloud-init/trunk.net-improve-lo-dns/+merge/298035
Your team cloud init development team is requested to review the proposed merge
of lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init.
___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init
> - use arrays for each 'section' rather than content += . This allows better
> separation of the sections and also will perform better as long strings
> with += are slow.
+1 for better separation. I don't think there is significant difference:
For an 100-line eni (that's rather high for typical usage)
% python3 t.py
Concat: 7.116918564017396
Array : 7.6817819369607605
% python2 t.py
Generating data...
Concat: 8.51635122299
Array : 7.93058490753
For a 10-line eni
% python3 t.py
Generating data...
Concat: 1.0464658139972016
Array : 1.2140009650029242
% python2 t.py
Generating data...
Concat: 0.956037998199
Array : 1.08836984634
% cat t.py
import timeit
concat = """
c=''
for x in range(0, 100):
c += 'iface eth0 inet static'
"""
append = """
c = []
for x in range(0, 100):
c.append('iface eth0 inet static')
"""
print('Generating data...')
print("Concat: %s" % timeit.timeit(concat))
print("Array : %s" % timeit.timeit(append))
--
https://code.launchpad.net/~smoser/cloud-init/trunk.net-improve-lo-dns/+merge/298035
Your team cloud init development team is requested to review the proposed merge
of lp:~smoser/cloud-init/trunk.net-improve-lo-dns into lp:cloud-init.
___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] lp:~swem/cloud-init/cloud-init.target.typo into lp:cloud-init
Chen-Han Hsiao (Stanley) has proposed merging lp:~swem/cloud-init/cloud-init.target.typo into lp:cloud-init. Requested reviews: cloud init development team (cloud-init-dev) For more details, see: https://code.launchpad.net/~swem/cloud-init/cloud-init.target.typo/+merge/300048 Fix typo in cloud-init.target. -- Your team cloud init development team is requested to review the proposed merge of lp:~swem/cloud-init/cloud-init.target.typo into lp:cloud-init. === modified file 'systemd/cloud-init.target' --- systemd/cloud-init.target 2016-03-03 22:49:10 + +++ systemd/cloud-init.target 2016-07-14 09:32:35 + @@ -1,6 +1,6 @@ # cloud-init target is enabled by cloud-init-generator # To disable it you can either: -# a.) boot with kernel cmdline of 'cloudinit=disabled' +# a.) boot with kernel cmdline of 'cloud-init=disabled' # b.) touch a file /etc/cloud/cloud-init.disabled [Unit] Description=Cloud-init target ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
