Re: [openstack-dev] [charms] Re-licensing OpenStack charms under Apache 2.0

[Cory Benfield]

> On 8 Jun 2016, at 11:20, James Page  wrote:
> 
> The majority of contributors are from Canonical (from whom I have permission 
> to make this switch) with a further 18 contributors from outside of Canonical 
> who I will be directly contacting for approval in gerrit as reviews are 
> raised for each repository.

Hey James,

I’m happy for you to relicense my contributions as Apache 2.0.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Maintaining httplib2 python library

[Cory Benfield]

> On 18 Mar 2016, at 17:05, Doug Wiegley <doug...@parksidesoftware.com> wrote:
>> On Mar 18, 2016, at 8:31 AM, Cory Benfield <c...@lukasa.co.uk> wrote:
>> 
>> Getting requests to talk over a Unix domain socket is not particularly 
>> tricky, and there are third-party libraries that hook into requests 
>> appropriately to make that happen. For example, the requests-unixsocket 
>> module exists that can do the appropriate things.
> 
> That’s the module that I was eyeing, but we’re just trading one dependency 
> for another. Is there something about httplib2 maintenance in particular that 
> makes us want that gone?
> 
> doug

The original message in this thread was about the fact that httplib2 is 
currently unmaintained and looking for new maintainers. I believe that was the 
impetus for the discussion.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Maintaining httplib2 python library

[Cory Benfield]

> On 18 Mar 2016, at 13:57, Brian Haley <brian.ha...@hpe.com> wrote:
> 
> On 03/17/2016 06:04 PM, Doug Wiegley wrote:
>>>> Here is the non comprehensive list of usages based on what trees I
>>>> happen to have checked out (which is quite a few, but not all of
>>>> OpenStack for sure).
>>>> 
>>>> I think before deciding to take over ownership of an upstream lib (which
>>>> is a large commitment over space and time), we should figure out the
>>>> migration cost. All the uses in Tempest come from usage in Glance IIRC
>>>> (and dealing with chunked encoding).
>>>> 
>>>> Neutron seems to use it for a couple of proxies, but that seems like
>>>> requests/urllib3 might be sufficient.
>>> 
>>> The Neutron team should talk to Cory Benfield (CC'd) and myself more about 
>>> this if they run into problems. requests and urllib3 are a little limited 
>>> with respect to proxies due to limitations in httplib itself.
>>> 
>>> Both of us might be able to dedicate time during the day to fix this if 
>>> Neutron/OpenStack have specific requirements that requests is not currently 
>>> capable of supporting.
>> 
>> Looks like neutron is using it to do HTTP requests via unix domain sockets. 
>> Unless I’m missing something, requests doesn’t support that directly. There 
>> are a couple of other libs that do, or we could monkey patch the socket. Or 
>> modify the agents to use localhost.
> 
> We have to use Unix domain sockets in the metadata proxy because it's running 
> in a namespace, so can't use localhost to talk to the agent.  But we could 
> use some other library of course.
> 

Getting requests to talk over a Unix domain socket is not particularly tricky, 
and there are third-party libraries that hook into requests appropriately to 
make that happen. For example, the requests-unixsocket module exists that can 
do the appropriate things.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documentation containing external resource links & privacy breaches

[Cory Benfield]

> On 7 Dec 2015, at 16:10, Thomas Goirand  wrote:
> 
> Cory,
> 
> Thanks a lot for this patch.
> 
> If I understand well, in Debian, I'd have to remove the:
> 
> [options]
> analytics_tracking_code = UA-17511903-1
> 
> from the default theme.conf to make the GA code go away. Right? Would
> there be a way so that I could set an environment variable instead of
> patching the file, which is always a pain?

You don’t need to remove that, you set it in `html_theme_options` in the 
conf.py for the relevant docs. That’s a fairly simple patch, but if the docs 
theme people are open to having an environment variable we could set up the 
default conf.py appropriately.

NB: You’d want html_theme_options to be set to:

html_theme_options = {
‘analytics_tracking_code’: ‘'
}

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documentation containing external resource links & privacy breaches

[Cory Benfield]

> On 7 Dec 2015, at 18:37, Thomas Goirand  wrote:
> 
> 
> The reason why I'd like an env var, is that I could simply modify
> openstack-pkg-tools (which is included in every debian/rules) to set
> that variable once and for all. For example, just adding in
> openstack-pkt-tools's pkgos.make:
> 
> export OPENSTACKDOCSTHEME_SELF_CONTAINED=yes
> 
> would do the trick for all my packages.
> 
> Your thoughts?

I am open to having such an environment variable that would control *all* 
documentation or theme related external properties, not just Google Analytics, 
but I’m also not a core on the openstackdocstheme project. ;)

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documentation containing external resource links & privacy breaches

[Cory Benfield]

> On 4 Dec 2015, at 10:06, Thomas Goirand  wrote:
> 
> That's actually a very good idea, I didn't think it was possible. Could
> you explain a bit more how this kind of patch would look like?

Sure. Rather than explain and take the credit, I’ll point you directly at the 
code in the Alabaster[0] repository that controls this and from where I 
‘borrowed’ the idea.

Here[1] is the code in the Alabaster ‘layout.html’ file that optionally 
includes the GA tracking script. Per the README[2], you can then set 
html_theme_options = {‘analytics_id’: }, which will cause the GA 
tracking script to be included with the appropriate ID.

I’m not exactly sure where the OpenStack theme lives, but assuming it’s the one 
in oslosphinx/theme/openstack, then you’d add the appropriate conditional 
template block here[3], probably with a default value in this file[4].

If this seems like something that’s generally desired, I’m open to making the 
actual patch to oslosphinx (or wherever the actual theme is mastered) myself.

Cory

[0]: https://github.com/bitprophet/alabaster
[1]: 
https://github.com/bitprophet/alabaster/blob/49f1bfec244609ab1a47592087342725163a2e88/alabaster/layout.html#L36-L52
[2]: 
https://github.com/bitprophet/alabaster/blob/49f1bfec244609ab1a47592087342725163a2e88/README.rst#theme-options
[3]: 
https://github.com/openstack/oslosphinx/blob/f8d71ac5ad023a32cbbe04e109eb2793433ee558/oslosphinx/theme/openstack/layout.html#L97-L108
[4]: 
https://github.com/openstack/oslosphinx/blob/f8d71ac5ad023a32cbbe04e109eb2793433ee558/oslosphinx/theme/openstack/theme.conf


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documentation containing external resource links & privacy breaches

[Cory Benfield]

> On 4 Dec 2015, at 13:20, Anne Gentle  wrote:
> 
> Great, thanks! This would work well for both oslosphinx and 
> openstackdocstheme:
> 
> http://git.openstack.org/cgit/openstack/openstackdocstheme/ 
> 
> 
> Anne

Ok, so to begin with I’ve filed a change for openstackdocstheme here: 
https://review.openstack.org/#/c/253592/ 


This change initially leaves GA on by default, but I’d like feedback on that 
decision because I’m open to reversing that default.

One further question: does anyone know if oslosphinx pulls in 
openstackdocstheme periodically, or if they’re entirely separate and I need to 
file a separate change for that repository?

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documentation containing external resource links & privacy breaches

[Cory Benfield]

> On 3 Dec 2015, at 13:37, Thomas Goirand  wrote:
> I've filed a bug against our docs theme, but it was marked as wontfix,
> and the patch which I started, was reviewed negatively. I've been told
> that we use Google Analytics for the openstack.org site, which I don't
> think is the right answer. I do believe we should think twice here.
> There are many alternative options to google analytics, such as web log
> analysis (webalizer, and such), and others involving local javascript of
> the same type as google analytics but without the privacy breach.
> There's ways to serve website-wide footers too (mod_footer for Apache
> for example). So I do believe there's better approaches to "we want
> statistic for openstack.org" than just Google Analytics.
> 
> So, could we have a general policy that we stop having such external
> resources in our documentations? What's the broader view of the
> community on this issue?

For Google Analytics, it’s entirely possible to make including the GA data 
conditional based on an environment variable. This would actually be a net-win, 
because it would ensure that the GA data accurately reflected the usage only on 
openstack.org. For example, the Alabaster sphinx theme does this based on 
configuration from conf.py, which could easily be set based on the 
presence/absence of an environment variable.

This would solve Debian’s problem with GA at the very least, while allowing 
openstack.org to continue to use the tool they choose to use to analyse their 
traffic.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [All] Use of self signed certs in endpoints

[Cory Benfield]

> On 16 Nov 2015, at 11:54, Sean Dague  wrote:
> That sounds pretty reasonable to me. I definitely support the idea that
> we should be using system CA by default, even if that means overriding
> requests in our tools.

Setting REQUESTS_CA_BUNDLE is absolutely the way to go about this. In requests 
2.9.0 we will also support the case that REQUESTS_CA_BUNDLE points to a 
directory of certificates, not a single certificate file, so this should cover 
all Linux distributions methods of distributing OpenSSL-compatible certificates.

If OpenStack wants to support using Windows and OS X built-in certificate 
stores, that's harder. This is because both systems do not use PEM-file based 
certificate distribution, which means OpenSSL can’t read them.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][nova] Handling lots of GET query string parameters?

[Cory Benfield]

> On 4 Nov 2015, at 13:13, Salvatore Orlando  wrote:
> 
> Regarding Jay's proposal, this would be tantamount to defining an API action 
> for retrieving instances, something currently being discussed here [1].
> The only comment I have is that I am not entirely surely whether using the 
> POST verb for operations which do no alter at all the server representation 
> of any object is in accordance with RFC 7231.

It’s totally fine, so long as you define things appropriately. Jay’s suggestion 
does exactly that, and is entirely in line with RFC 7231.

The analogy here is to things like complex search forms. Many search engines 
allow you to construct very complex search queries (consider something like 
Amazon or eBay, where you can filter on all kinds of interesting criteria). 
These forms are often submitted to POST endpoints rather than GET.

This is totally fine. In fact, the first example from RFC 7231 Section 4.3.3 
(POST) applies here: “POST is used for the following functions (among others): 
Providing a block of data […] to a data-handling process”. In this case, the 
data-handling function is the search function on the server.

The *only* downside of Jay’s approach is that the response cannot really be 
cached. It’s not clear to me whether anyone actually deploys a cache in this 
kind of role though, so it may not hurt too much.

Cory




signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

> On 14 Oct 2015, at 23:23, Thomas Goirand  wrote:
> I do understand that you don't like being called this way, though this
> is still the reality. Vendorizing still inflicting some major pain to a
> lot of your users:
> - This thread one of the demonstration of it.
> - You having to contact downstream distros is another.
> - The unbundling work inflicted to downstream package maintainers is a
> 3rd another.
> 
> So like it or not, it is a fact that it is difficult to work with
> requests because of the way it is released upstream.

As I said earlier, I’m not getting drawn into a debate about vendorizing in 
this forum. The last one of these was sufficiently toxic that I’m simply 
unwilling to have the discussion here. If you really want to talk about this 
again, I’m happy to take it out of this mailing list to somewhere where fewer 
people are going to make the discussion personal.

Note however that point 2 is not accurate. The main reason we have 
relationships with our downstream repackagers is for security release purposes. 
Per our security policy, we have exchanged GPG keys with them, and will make 
sure we contact them ahead of time so we can perform a synchronised release of 
security updates. In this instance we chose to use our relationship with our 
repackagers to get this change made, but it is not the main reason we 
communicate with them. (Also, they’re nice people!)

>> has had a policy in place for six months
>> that ensures that you can have the same result with pip and
>> system packages. For six months we have only updated to versions
>> of urllib3 that are actually released, and therefore that are
>> definitely available from pip (and potentially available from
>> the distribution).
>> 
>> The reason this has not been working is because the distributions,
>> when they unbundle us, have not been populating their setup.py to
>> reflect the dependency: only their own metadata. We’ve been in
>> contact with them, and this change is being made in the
>> distributions we have relationships with.
> 
> Though you could have avoid all of this pain if you were not bundling.
> Isn't all of this make you re-think your vendorizing policy? Or still
> not? I'm asking because I still didn't read your answer about the
> important question: since you aren't using specially crafted versions of
> urllib3 anymore, and now only using official releases, what's the reason
> that keeps you vendorizing? Not trying to convince you here, just trying
> to understand.

Again, I’m not being drawn into this discussion here.

Let me make one point, though. There are three people involved in a 
decision-making role on the requests project, and this is an important issue to 
every member of the team. This policy has been part of the requests project for 
a very long time, and we aren’t going to change it in a short space of time: 
I’m certainly not going to unilaterally do so. All I can promise you is that we 
continue to talk about this internally, and if we *unanimously* feel 
comfortable changing our policy we will do so. Until then, I’m happy to do my 
best to accommodate as many people as possible (which in this case I believe we 
have done).

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

> On 13 Oct 2015, at 07:42, Thomas Goirand  wrote:
> In this particular case (ie: a difficult upstream which makes it
> impossible to have the same result with pip and system packages)

I don’t know how carefully you’ve followed this email trail, but the “difficult 
upstream” has had a policy in place for six months that ensures that you can 
have the same result with pip and system packages. For six months we have only 
updated to versions of urllib3 that are actually released, and therefore that 
are definitely available from pip (and potentially available from the 
distribution).

The reason this has not been working is because the distributions, when they 
unbundle us, have not been populating their setup.py to reflect the dependency: 
only their own metadata. We’ve been in contact with them, and this change is 
being made in the distributions we have relationships with.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

> On 12 Oct 2015, at 01:16, Robert Collins  wrote:
> To sum up the thread, it sounds to me like a viable way-forward is:
> 
> - get distros to fixup their requests Python dependencies (and
> hopefully they can update that in stable releases).
> - fix the existing known bugs in pip where such accurate dependencies
> are violated by some operations.

Agreed.

And we’re taking this pretty seriously at requests: we got in touch with our 
downstream unbundlers at Debian and Fedora asking them to make sure they 
populate setup.py correctly. Fedora has created updates for F21, F22, and F23:

- https://bodhi.fedoraproject.org/updates/FEDORA-2015-20de3774f4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-1f580ccfa4
- https://bodhi.fedoraproject.org/updates/FEDORA-2015-d7c710a812

We’ve also heard positive noises from our Debian packager, though I don’t yet 
have a link to any change there.

If Ubuntu has a separate downstream packager from Debian, we don’t have a 
relationship with them, so it’s harder for us to affect change there.

This should mean that we’re only missing the pip fix.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

Robert Collins  writes:
> The problem that occurs is the result of a few interacting things:
>  - requests has very very specific versions of urllib3 it works with.
> So specific they aren't always released yet.

This should no longer be true. Our downstream redistributors pointed out to us
that this  was making their lives harder than they needed to be, so it's now
our policy to only  update to actual release versions of urllib3.
 
> The second is trivially insufficient - anytime requests vendored
> urllib3 is not precisely identical to a released urllib3, it becomes
> impossible to satisfy that via dependency version pinning - the only
> way to satisfy it is with the urllib3 in the distro that has whatever
> change was needed included.

Per my note above, if we restrict ourselves to relatively recent versions of
requests  (2.7.3+ IIRC) we should be fine. Of course, that doesn't mean we can
actually do that...

> The fourth approach meets the stone wall of 'but security' and 'no
> redundancy permitted' - I don't have the energy to try and get through
> the near-religious mindset I've encountered there before, though hey -
> if Fedora and Debian and Ubuntu folk are all interested in figuring
> out a sustainable way forward, that would be great: please don't feel
> cut out, I'm just not expecting anything.

It should be assumed that approach number four is a non-starter. This list has
had that  conversation before, which was a stunningly unpleasant experience for
me and not one I  want to repeat. Additionally, getting *all* of
Fedora/Debian/Ubuntu on board with not unbundling requests is about as likely
as hell freezing over.

Cory


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

> On 9 Oct 2015, at 14:40, William M Edmonds <edmon...@us.ibm.com> wrote:
> 
> Cory Benfield <cory@...> writes:
> > > The problem that occurs is the result of a few interacting things:
> > >  - requests has very very specific versions of urllib3 it works with.
> > > So specific they aren't always released yet.
> >
> > This should no longer be true. Our downstream redistributors pointedout to 
> > us
> > that this  was making their lives harder than they needed to be, so it's now
> > our policy to only  update to actual release versions of urllib3.
> 
> That's great... except that I'm confused as to why requests would continue to 
> repackage urllib3 if that's the case. Why not just prereq the version of 
> urllib3 that it needs? I thought the one and only answer to that question had 
> been so that requests could package non-standard versions.
> 

That is not and was never the only reason for vendoring urllib3. However, and I 
cannot stress this enough, the decision to vendor urllib3 is *not going to be 
changed on this thread*. If and when it changes, it will be by consensus 
decision from the requests maintenance team, which we do not have at this time.

Further, as I pointed out to Donald Stufft on IRC, if requests unbundled 
urllib3 *today* that would not fix the problem. The reason is that we’d specify 
our urllib3 dependency as: urllib3>=1.12,<1.13. This dependency note would 
still cause exactly the problem observed in this thread.

As you correctly identify in your subsequent email, William, the core problem 
is mixing of packages from distributions and PyPI. This happens with any tool 
with external dependencies: if you subsequently install a different version of 
a dependency using a packaging tool that is not aware of some of the dependency 
tree, it is entirely plausible that an incompatible version will be installed. 
It’s not hard to trigger this kind of thing on Ubuntu. IMO, what OpenStack 
needs is a decision about where it’s getting its packages from, and then to 
refuse to mix the two.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Requests + urllib3 + distro packages

[Cory Benfield]

> On 9 Oct 2015, at 15:18, Jeremy Stanley <fu...@yuggoth.org> wrote:
> 
> On 2015-10-09 14:58:36 +0100 (+0100), Cory Benfield wrote:
> [...]
>> IMO, what OpenStack needs is a decision about where it’s getting
>> its packages from, and then to refuse to mix the two.
> 
> I have yet to find a Python-based operating system installable in
> whole via pip. There will always be _at_least_some_ packages you
> install from your operating system's package management. What you
> seem to be missing is that Linux distros are now shipping base
> images which include their python-requests and python-urllib3
> packages already pre-installed as dependencies of Python-based tools
> they deem important to their users.
> 

Yeah, this has been an ongoing problem.

For my part, Donald Stufft has informed me that if the distribution-provided 
requests package has the appropriate install_requires field in its setup.py, 
pip will respect that dependency. Given that requests has recently switched to 
not providing mid-cycle urllib3 versions, it should be entirely possible for 
downstream redistributors in Debian/Fedora to put that metadata into their 
packages when they unbundle requests. I’m chasing up with our downstream 
redistributors right now to ask them to start doing that.

This should resolve the problem for systems where requests 2.7.0 or higher are 
being used. In other systems, this problem still exists and cannot be fixed by 
requests directly.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Generic question about synchronizing neutron agent on compute node with DB

[Cory Benfield]

On Sun, Mar 15, 2015 at 22:37:59, Sławek Kapłoński wrote:
 Maybe good idea for the beginning could be to implement some periodic
 task
 called from agent to check db config and compare it with real state on
 host?
 What do You think? Or maybe I'm competly wrong with such idea and it
 should be
 done in different way?

This is almost exactly what we do in our Calico ML2 driver. Each of our agents 
will periodically request its complete state from a neutron-server node and 
will ensure that its local state matches that expected state. This interval is 
configurable, to allow administrators to make a trade-off between DB/network 
load and convergence time.

With reliable transport this is in principle almost never needed (messages only 
really get lost on agent crash, and the agent will resynchronize when it starts 
back up anyway), but it provides assurances that the fabric is capable of 
bringing itself into consistency without administrator intervention.

Having similar function in other neutron agents would be valuable for the exact 
same reasons, but do bear in mind the potentially increased load this kind of 
resynchronization can place on databases and servers.

Cory
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Cory Benfield]

On Wed, Feb 04, 2015 at 08:59:54, Kevin Benton wrote:
 I proposed an alternative to adjusting the lease time early on the in
 the thread. By specifying the renewal time (DHCP option 58), we can
 have
 the benefits of a long lease time (resiliency to long DHCP server
 outages) while having a frequent renewal interval to check for IP
 changes. I favored this approach because it only required a patch to
 dnsmasq to allow that option to be set and patch to our agent to set
 that option, both of which are pretty straight-forward.

It's hard to see a downside to this proposal. Even if one of the other ideas 
goes forward as well, a short DHCP renewal interval feels like a very good idea 
to me.

Cory
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group

[Cory Benfield]

On Thu, Dec 11, 2014 at 23:05:01, Mathieu Gagné wrote:
 When no security group is provided, Nova will default to the default
 security group. However due to the fact 2 security groups had the same
 name, nova-compute got confused, put the instance in ERROR state and
 logged this traceback [1]:
 
NoUniqueMatch: Multiple security groups found matching 'default'.
 Use
 an ID to be more specific.
 

We've hit this in our automated testing in the past as well. Similarly, we have 
no idea how we managed to achieve this, but it's clearly something that the 
APIs allow you to do. That feels unwise.

 - the instance request should be blocked before it ends up on a compute
 node with nova-compute. It shouldn't be the job of nova-compute to
 find
 out issues about duplicated names. It should be the job of nova-api.
 Don't waste your time scheduling and spawning an instance that will
 never spawn with success.
 
 - From an end user perspective, this means nova boot returns no error
 and it's only later that the user is informed of the confusion with
 security group names.
 
 - Why does it have to crash with a traceback? IMO, traceback means we
 didn't think about this use case, here is more information on how to
 find the source. As an operator, I don't care about the traceback if
 it's a known limitation of Nova/Neutron. Don't pollute my logs with
 normal exceptions. (Log rationalization anyone?)

+1 to all of this.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

On Tue, Oct 28, 2014 at 21:50:09, Carl Baldwin wrote:
 Many API users won't care about the L2 details.  This could be a
 compelling alternative for them.  However, some do.  The L2 details
 seem to matter an awful lot to many NFV use cases.  It might be that
 this alternative is just not compelling for those.  Not to say it
 isn't compelling overall though.

Agreed. This is a point worth emphasising: routed networking is not a panacea 
for everyone's networking woes. We've got a lot of NFV people and products at 
my employer, and while we're engaged in work to come up with L3 approaches to 
solve their use-cases, we'd like to draw a balance between adding complexity to 
the network layer to support legacy L2-based requirements and providing better 
native L3 solutions that NFV applications can use instead.  One of the key 
challenges with NFV is that it shouldn't just be a blind porting of existing 
codebases - you need to make sure you're producing something which takes 
advantage of the new environment.

Cory

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

On Tue, Oct 28, 2014 at 20:01:40, Kevin Benton wrote:
 I think the simplest use case is just that a provider doesn't want to
 deal with extending L2 domains all over their datacenter.

This is the core motivation. As mentioned in Fred Baker's internet draft[0], 
extending layer 2 domains can be extremely challenging, and we've found it to 
be problematic to debug at setup time, let alone to operate.

If your tenants have no need of layer 2 networks (i.e. their workload is 
all-IP), the datacenter network becomes much simpler in a routed model. 
Generally speaking, simpler is better!

Cory

[0]: https://tools.ietf.org/html/draft-baker-openstack-ipv6-model-00
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

 Some of us are looking at a different model. I’d be interested in your 
 thoughts.

Fred,

Thanks for the link to the drafts. They look extremely similar to the 
approach we've been pursuing for Project Calico, and it's good to see 
that we're not the only people thinking in this direction.

It looks like the main differences between our approach and yours are 
that we've tried to come up with a model that works both for IPv4 and 
IPv6 (although we agree that moving the data center fabric to IPv6 has a 
lot of advantages - e.g. we are planning on using 464XLAT as the 
mechanism to handle IPv4 overlap).  Given this, we've focused our 
policy/security model on ACLs rather than flow labels.  An interesting 
derivative effect of that choice is that any policy or security model 
can be enforced (such as intra-tenant controls, extra-cloud controls, 
etc).

As a side note, we have been interested in using flow labels as 
namespace identifiers and for SFC.  Recently, we have moved away from 
that thinking given the guidance that the flow label should be not be 
modified in flight.  If you believe that such modifications will be 
acceptable, we would love to discuss that with you, and see where we can 
collaborate.

As it is, I believe our proposed changes to Nova and Neutron should be 
generic enough to provide a basis for implementing your approach as well 
as supporting our Project Calico ML2 driver. If they aren't, we should 
work together to make whatever changes we have to make to achieve that 
generality.

It might also be worth checking out our agent code[0]. It's in the 
middle of a rewrite at the minute so the code is unfinished, but it 
handles a lot of what you'd be doing with your proposed drafts. 
Hopefully it'd be a useful jumping off point.

Cory

[0]: https://github.com/Metaswitch/calico/tree/master/calico/felix

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] vm can not transport large file under neutron ml2 + linux bridge + vxlan

[Cory Benfield]

On Tue, Oct 28, 2014 at 08:32:11, Li Tianqing wrote:
 One more question.
 Vxlan use udp, how can vxlan promise reliability.
 

It can't, but that doesn't matter.

VXLAN emulates a single layer 2 broadcast domain: conceptually, a series of 
machines all plugged into the same Ethernet switch. This kind of network 
*isn't* reliable: you can lose Ethernet frames. There's no reason to require 
reliability from VXLAN, and it would increase VXLAN's overhead to add it in. If 
you need reliability, the encapsulated transport will provide it for you 
exactly as it does on a non-encapsulated network.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

On Tue, Oct 28, 2014 at 07:44:48, A, Keshava wrote:
 Hi,
 
 Current Open-stack was built as flat network.
 
 With the introduction of the L3 lookup (by inserting the routing table
 in forwarding path) and separate 'VIF Route Type' interface:
 
 At what point of time in the packet processing  decision will be made
 to lookup FIB  during ? For each packet there will additional  FIB
 lookup ?
 
 How about the  impact on  'inter compute traffic', processed by  DVR  ?
 Here thinking  OpenStack cloud as hierarchical network instead of Flat
 network ?

Keshava,

It's difficult for me to answer in general terms: the proposed specs are 
general enough to allow multiple approaches to building purely-routed networks 
in OpenStack, and they may all have slightly different answers to some of these 
questions. I can, however, speak about how Project Calico intends to apply them.

For Project Calico, the FIB lookup is performed for every packet emitted by a 
VM and destined for a VM. Each compute host routes all the traffic to/from its 
guests. The DVR approach isn't necessary in this kind of network because it 
essentially already implements one: all packets are always routed, and no 
network node is ever required in the network.

The routed network approach doesn't add any hierarchical nature to an OpenStack 
cloud. The difference between the routed approach and the standard OVS approach 
is that packet processing happens entirely at layer 3. Put another way, in 
Project Calico-based networks a Neutron subnet no longer maps to a layer 2 
broadcast domain.

I hope that clarifies: please shout if you'd like more detail.

Cory

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

On Fri, Oct 24, 2014 at 20:51:36, Kevin Benton wrote:
 Hi,
 
 Thanks for posting this. I am interested in this use case as well.
 
 I didn't find a link to a review for the ML2 driver. Do you have any
 more details for that available?

Sure. The ML2 driver itself isn't submitted for review yet because we're still 
working on it, but you can find the current code here[1]. If you think there's 
a risk of confusion with ML2 we'd love to hear about it.

Cory

[1]: 
https://github.com/Metaswitch/calico-neutron/blob/master/neutron/plugins/ml2/drivers/mech_calico.py

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

On Sun, Oct 26, 2014 at 19:05:43, Rohit Agarwalla (roagarwa) wrote:
 Hi
 
 I'm interested as well in this model. Curious to understand the routing
 filters and their implementation that will enable isolation between
 tenant networks.
 Also, having a BoF session on Virtual Networking using L3 may be
 useful to get all interested folks together at the Summit.

A BoF sounds great. I've also proposed a lightning talk for the summit.

Cory

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][nova] New specs on routed networking

[Cory Benfield]

All,

Project Calico [1] is an open source approach to virtual networking based on L3 
routing as opposed to L2 bridging.  In order to accommodate this approach 
within OpenStack, we've just submitted 3 blueprints that cover

-  minor changes to nova to add a new VIF type [2]
-  some changes to neutron to add DHCP support for routed interfaces [3]
-  an ML2 mechanism driver that adds support for Project Calico [4].

We feel that allowing for routed network interfaces is of general use within 
OpenStack, which was our motivation for submitting [2] and [3].  We also 
recognise that there is an open question over the future of 3rd party ML2 
drivers in OpenStack, but until that is finally resolved in Paris, we felt 
submitting our driver spec [4] was appropriate (not least to provide more 
context on the changes proposed in [2] and [3]).

We're extremely keen to hear any and all feedback on these proposals from the 
community.  We'll be around at the Paris summit in a couple of weeks and would 
love to discuss with anyone else who is interested in this direction. 

Regards,

Cory Benfield (on behalf of the entire Project Calico team)

[1] http://www.projectcalico.org 
[2] https://blueprints.launchpad.net/nova/+spec/vif-type-routed
[3] https://blueprints.launchpad.net/neutron/+spec/dhcp-for-routed-ifs
[4] https://blueprints.launchpad.net/neutron/+spec/calico-mechanism-driver

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev