[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-09-22 Thread Scott Moser 
This bug is believed to be fixed in cloud-init in 17.1. If this is still
a problem for you, please make a comment and set the state back to New

Thank you.

** Changed in: cloud-init
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1088611

Title:
  using random hostnames to detect dns proxies allows for false
  positives

Status in cloud-init:
  Fix Released
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Fix Released
Status in cloud-init source package in Zesty:
  Fix Released

Bug description:
  === Begin SRU Template ===
  [Impact]
  Prior to this fix, cloud-init attempts to detect dns redirection by doing
  dns queries for a random hostname and two invalid hostnames.  Then, if
  the result returned for the input value was the same as the response for
  the invalid query cloud-init would assume that result was also invalid.

  The change was to replace the random string with
__cloud_init_expected_not_found__
  This is a valid hostname and resolution will use the 'search' path in
  resolv.conf where the other invalid domain names would not.

  [Test Case]
  The test case for this consists of excercising the the 'is_resolvable_url'
  method in cloudinit.util and watching dns queries.  To do this, see the
  following steps:
  a.) start an lxc container
 $ release=xenial
 $ name=$release-1088611
 $ lxc launch ubuntu-daily:$release $name
  b.) start a dnsmasq server
 $ ./run-dnsmasq lxdbr0
 ... 
 === listening on 10.75.205.2/24 ===

 # run-dnsmasq is attached and at
 #  
https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bugs/lp-1088611/run-dnsmasq

  c.) point /etc/resolv.conf at your server ip
 $ lxc exec $name -- sh -c 'exec >/etc/resolv.conf;
 echo nameserver 10.75.205.2; echo search foo;'

  d.) perform query via is_resolvable_url watch dnsmasq output, expect
  to see the random query.
 $ lxc exec $name -- python3 -c 'import sys;
  from cloudinit.util import is_resolvable_url; 
  print(is_resolvable_url(sys.argv[1]))' http://ubuntu.com

  e.) upgrade to -proposed version
  f.) perform query via is_resolvable_url, expect to *not* see random query.

  [Regression Potential]
  Immediate regression seems unlikely.  Effectively the change in cloud-init
  code path was simply to change a dns lookup attempt from rand() to a defined
  string.

  We chose a random string initially to make it difficult for a dns server to
  circumvent cloud-init's attempt to identify dns redirection.  The regression
  path really then seems to involve a dns redirection service specifically
  provding a response for '__cloud_init_expected_not_found__' that differs
  from does-not-exist.example.com.  Cloud-init could then be tricked into
  believing that a apt mirror was valid where it previously would have
  identified the dns redirection.  The failure would be seen as errors
  in package installation or 'apt-get update'.

  [Other Info]
  Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=42a7b34a12

  Original upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=1bb67be5bd

  === End SRU Template ===

  The fix that's been applied for bug #974509 checks for the presence of
  a redirector by looking of three hostnames, and treating as invalid
  any results pointing to a matching address:

   - does-not-exist.example.com.
   - example.invalid.
   - a random, unqualified 32-character alphanumeric hostname.

  The last of these carries a small but non-zero risk of colliding with
  a real hostname, and there's a small but non-zero risk that this host
  points to the same address as something we care about.  If possible,
  it would be better to not include this random-host lookup in the
  algorithm, as somewhere, some day, chances are there will eventually
  be a collision, causing an incomprehensible and unreproducible failure
  for a user.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1088611/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-09-12 Thread Launchpad Bug Tracker 
This bug was fixed in the package cloud-init -
0.7.9-233-ge586fe35-0ubuntu1~16.04.1

---
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.1) xenial-proposed; 
urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
+ debian/patches/cpick-5fb49bac-azure-identify-platform...
+ debian/patches/cpick-003c6678-net-remove-systemd-link...
+ debian/patches/cpick-1cd4323b-azure-remove-accidental...
+ debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
+ debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
- cloudinit.net: add initialize_network_device function and tests
  [Chad Smith]
- makefile: fix ci-deps-ubuntu target [Chad Smith]
- tests: adjust locale integration test to parse default locale.
- tests: remove 'yakkety' from releases as it is EOL.
- centos: do not package systemd-fsck drop-in.
- systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
- tests: Add initial tests for EC2 and improve a docstring.
- locale: Do not re-run locale-gen if provided locale is system default.
- archlinux: fix set hostname usage of write_file. [Joshua Powers]
- sysconfig: support subnet type of 'manual'.
- tools/run-centos: make running with no argument show help.
- Drop rand_str() usage in DNS redirection detection
  [Bob Aman] (LP: #1088611)
- sysconfig: use MACADDR on bonds/bridges to configure mac_address
  [Ryan Harper]
- net: eni route rendering missed ipv6 default route config
  [Ryan Harper] (LP: #1701097)
- sysconfig: enable mtu set per subnet, including ipv6 mtu
  [Ryan Harper]
- sysconfig: handle manual type subnets [Ryan Harper]
- sysconfig: fix ipv6 gateway routes [Ryan Harper]
- sysconfig: fix rendering of bond, bridge and vlan types.
  [Ryan Harper]
- Templatize systemd unit files for cross distro deltas. [Ryan Harper]
- sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
- net: fix renaming of nics to support mac addresses written in upper
  case. (LP: #1705147)
- tests: fixes for issues uncovered when moving to python 3.6.
- sysconfig: include GATEWAY value if set in subnet
  [Ryan Harper]
- Scaleway: add datasource with user and vendor data for Scaleway.
  [Julien Castets]
- Support comments in content read by load_shell_content.
- cloudinitlocal fail to run during boot [Hongjiang Zhang]
- doc: fix disk setup example table_type options [Sandor Zeestraten]
- tools: Fix exception handling. [Joonas Kylmälä]
- tests: fix usage of mock in GCE test.
- test_gce: Fix invalid mock of platform_reports_gce to return False
  [Chad Smith]
- test: fix incorrect keyid for apt repository.  [Joshua Powers]
- tests: Update version of pylxd [Joshua Powers]
- write_files: Remove log from helper function signatures.
  [Andrew Jorgensen]
- doc: document the cmdline options to NoCloud [Brian Candler]
- read_dmi_data: always return None when inside a container. (LP: #1701325)
- requirements.txt: remove trailing white space.
- Tests: Simplify the check on ssh-import-id [Joshua Powers]
- tests: update ntp tests after sntp added [Joshua Powers]
- FreeBSD: Make freebsd a variant, fix unittests and
  tools/build-on-freebsd.
- FreeBSD: fix test failure
- FreeBSD: replace ifdown/ifup with "ifconfig down" and "ifconfig up".
  [Hongjiang Zhang]
- FreeBSD: fix cdrom mounting failure if /mnt/cdrom/secure did not exist.
  [Hongjiang Zhang]
- main: Don't use templater to format the welcome message
  [Andrew Jorgensen]
- docs: Automatically generate module docs form schema if present.
  [Chad Smith]
- debian: fix path comment in /etc/hosts template. [Jens Sandmann]
- suse: add hostname and fully qualified domain to template.
  [Jens Sandmann]
- write_file(s): Print permissions as octal, not decimal [Andrew Jorgensen]
- ci deps: Add --test-distro to read-dependencies to install all deps
  [Chad Smith]
- tools/run-centos: cleanups and move to using read-dependencies
- pkg build ci: Add make ci-deps- target to install pkgs
  [Chad Smith]
- selinux: Allow restorecon to be non-fatal. [Ryan Harper]
- net: Allow netinfo subprocesses to return 0 or 1 due to selinux.
  [Ryan Harper]
- net: Allow for NetworkManager configuration [Ryan McCabe]
- Use distro release version to determine if we use systemd in redhat spec
  [Ryan Harper]
- net: normalize data in network_state object
- Integration Testing: tox env, pyxld 2.2.3, and revamp framework
  [Wesley Wiedenmeier]
- Chef: Update omnibus url to chef.io,

[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-09-12 Thread Launchpad Bug Tracker 
This bug was fixed in the package cloud-init -
0.7.9-233-ge586fe35-0ubuntu1~17.04.1

---
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~17.04.1) zesty; urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
+ debian/patches/cpick-5fb49bac-azure-identify-platform...
+ debian/patches/cpick-003c6678-net-remove-systemd-link...
+ debian/patches/cpick-1cd4323b-azure-remove-accidental...
+ debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
+ debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
- cloudinit.net: add initialize_network_device function and tests
  [Chad Smith]
- makefile: fix ci-deps-ubuntu target [Chad Smith]
- tests: adjust locale integration test to parse default locale.
- tests: remove 'yakkety' from releases as it is EOL.
- centos: do not package systemd-fsck drop-in.
- systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
- tests: Add initial tests for EC2 and improve a docstring.
- locale: Do not re-run locale-gen if provided locale is system default.
- archlinux: fix set hostname usage of write_file. [Joshua Powers]
- sysconfig: support subnet type of 'manual'.
- tools/run-centos: make running with no argument show help.
- Drop rand_str() usage in DNS redirection detection
  [Bob Aman] (LP: #1088611)
- sysconfig: use MACADDR on bonds/bridges to configure mac_address
  [Ryan Harper]
- net: eni route rendering missed ipv6 default route config
  [Ryan Harper] (LP: #1701097)
- sysconfig: enable mtu set per subnet, including ipv6 mtu
  [Ryan Harper]
- sysconfig: handle manual type subnets [Ryan Harper]
- sysconfig: fix ipv6 gateway routes [Ryan Harper]
- sysconfig: fix rendering of bond, bridge and vlan types.
  [Ryan Harper]
- Templatize systemd unit files for cross distro deltas. [Ryan Harper]
- sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
- net: fix renaming of nics to support mac addresses written in upper
  case. (LP: #1705147)
- tests: fixes for issues uncovered when moving to python 3.6.
- sysconfig: include GATEWAY value if set in subnet
  [Ryan Harper]
- Scaleway: add datasource with user and vendor data for Scaleway.
  [Julien Castets]
- Support comments in content read by load_shell_content.
- cloudinitlocal fail to run during boot [Hongjiang Zhang]
- doc: fix disk setup example table_type options [Sandor Zeestraten]
- tools: Fix exception handling. [Joonas Kylmälä]
- tests: fix usage of mock in GCE test.
- test_gce: Fix invalid mock of platform_reports_gce to return False
  [Chad Smith]
- test: fix incorrect keyid for apt repository.  [Joshua Powers]
- tests: Update version of pylxd [Joshua Powers]
- write_files: Remove log from helper function signatures.
  [Andrew Jorgensen]
- doc: document the cmdline options to NoCloud [Brian Candler]
- read_dmi_data: always return None when inside a container. (LP: #1701325)
- requirements.txt: remove trailing white space.
- Tests: Simplify the check on ssh-import-id [Joshua Powers]
- tests: update ntp tests after sntp added [Joshua Powers]
- FreeBSD: Make freebsd a variant, fix unittests and
  tools/build-on-freebsd.
- FreeBSD: fix test failure
- FreeBSD: replace ifdown/ifup with "ifconfig down" and "ifconfig up".
  [Hongjiang Zhang]
- FreeBSD: fix cdrom mounting failure if /mnt/cdrom/secure did not exist.
  [Hongjiang Zhang]
- main: Don't use templater to format the welcome message
  [Andrew Jorgensen]
- docs: Automatically generate module docs form schema if present.
  [Chad Smith]
- debian: fix path comment in /etc/hosts template. [Jens Sandmann]
- suse: add hostname and fully qualified domain to template.
  [Jens Sandmann]
- write_file(s): Print permissions as octal, not decimal [Andrew Jorgensen]
- ci deps: Add --test-distro to read-dependencies to install all deps
  [Chad Smith]
- tools/run-centos: cleanups and move to using read-dependencies
- pkg build ci: Add make ci-deps- target to install pkgs
  [Chad Smith]
- selinux: Allow restorecon to be non-fatal. [Ryan Harper]
- net: Allow netinfo subprocesses to return 0 or 1 due to selinux.
  [Ryan Harper]
- net: Allow for NetworkManager configuration [Ryan McCabe]
- Use distro release version to determine if we use systemd in redhat spec
  [Ryan Harper]
- net: normalize data in network_state object
- Integration Testing: tox env, pyxld 2.2.3, and revamp framework
  [Wesley Wiedenmeier]
- Chef: Update omnibus url to chef.io, minor doc

[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-07-31 Thread Scott Moser 
** Also affects: cloud-init (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Also affects: cloud-init (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Changed in: cloud-init (Ubuntu Xenial)
   Status: New => Confirmed

** Changed in: cloud-init (Ubuntu Zesty)
   Status: New => Confirmed

** Changed in: cloud-init (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: cloud-init (Ubuntu Zesty)
   Importance: Undecided => Low

** Changed in: cloud-init (Ubuntu Xenial)
   Importance: Medium => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1088611

Title:
  using random hostnames to detect dns proxies allows for false
  positives

Status in cloud-init:
  Fix Committed
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Confirmed
Status in cloud-init source package in Zesty:
  Confirmed

Bug description:
  The fix that's been applied for bug #974509 checks for the presence of
  a redirector by looking of three hostnames, and treating as invalid
  any results pointing to a matching address:

   - does-not-exist.example.com.
   - example.invalid.
   - a random, unqualified 32-character alphanumeric hostname.

  The last of these carries a small but non-zero risk of colliding with
  a real hostname, and there's a small but non-zero risk that this host
  points to the same address as something we care about.  If possible,
  it would be better to not include this random-host lookup in the
  algorithm, as somewhere, some day, chances are there will eventually
  be a collision, causing an incomprehensible and unreproducible failure
  for a user.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1088611/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-07-31 Thread Launchpad Bug Tracker 
This bug was fixed in the package cloud-init -
0.7.9-231-g80bf98b9-0ubuntu1

---
cloud-init (0.7.9-231-g80bf98b9-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
- tests: remove 'yakkety' from releases as it is EOL.
- systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
- tests: Add initial tests for EC2 and improve a docstring.
- locale: Do not re-run locale-gen if provided locale is system default.
- archlinux: fix set hostname usage of write_file.
  [Joshua Powers] (LP: #1705306)
- sysconfig: support subnet type of 'manual'.
- Drop rand_str() usage in DNS redirection detection
  [Bob Aman] (LP: #1088611)

 -- Scott Moser   Mon, 31 Jul 2017 09:47:34 -0400

** Changed in: cloud-init (Ubuntu)
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1088611

Title:
  using random hostnames to detect dns proxies allows for false
  positives

Status in cloud-init:
  Fix Committed
Status in cloud-init package in Ubuntu:
  Fix Released

Bug description:
  The fix that's been applied for bug #974509 checks for the presence of
  a redirector by looking of three hostnames, and treating as invalid
  any results pointing to a matching address:

   - does-not-exist.example.com.
   - example.invalid.
   - a random, unqualified 32-character alphanumeric hostname.

  The last of these carries a small but non-zero risk of colliding with
  a real hostname, and there's a small but non-zero risk that this host
  points to the same address as something we care about.  If possible,
  it would be better to not include this random-host lookup in the
  algorithm, as somewhere, some day, chances are there will eventually
  be a collision, causing an incomprehensible and unreproducible failure
  for a user.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1088611/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1088611] Re: using random hostnames to detect dns proxies allows for false positives

2017-07-11 Thread Scott Moser 
** Also affects: cloud-init
   Importance: Undecided
   Status: New

** Changed in: cloud-init
   Status: New => Confirmed

** Changed in: cloud-init
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1088611

Title:
  using random hostnames to detect dns proxies allows for false
  positives

Status in cloud-init:
  Confirmed
Status in cloud-init package in Ubuntu:
  Confirmed

Bug description:
  The fix that's been applied for bug #974509 checks for the presence of
  a redirector by looking of three hostnames, and treating as invalid
  any results pointing to a matching address:

   - does-not-exist.example.com.
   - example.invalid.
   - a random, unqualified 32-character alphanumeric hostname.

  The last of these carries a small but non-zero risk of colliding with
  a real hostname, and there's a small but non-zero risk that this host
  points to the same address as something we care about.  If possible,
  it would be better to not include this random-host lookup in the
  algorithm, as somewhere, some day, chances are there will eventually
  be a collision, causing an incomprehensible and unreproducible failure
  for a user.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1088611/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp